Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning
Authors: Chengxing Jia
Chenxiao Gao
Hao Yin
Fuxiang Zhang
Xiong-Hui Chen
Tian Xu
Lei Yuan
Zongzhang Zhang
Zhi-Hua Zhou
Yang Yu
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Model-based Reinforcement Learning
Offline Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We introduce the idea of rehearsal into reinforcement learning in scenarios where both online and offline experiences are expensive and limited.

Abstract: Human beings can make adaptive decisions in a preparatory manner, i.e., by making preparations in advance, which offers significant advantages in scenarios where both online and offline experiences are expensive and limited. Meanwhile, current reinforcement learning methods commonly rely on numerous environment interactions but hardly obtain generalizable policies. In this paper, we introduce the idea of \textit{rehearsal} into policy optimization, where the agent plans for all possible outcomes in mind and acts adaptively according to actual responses from the environment. To effectively rehearse, we propose ReDM, an algorithm that generates a diverse and eligible set of dynamics models and then rehearse the policy via adaptive training on the generated model set. Rehearsal enables the policy to make decision plans for various hypothetical dynamics and to naturally generalize to previously unseen environments. Our experimental results demonstrate that ReDM is capable of learning a valid policy solely through rehearsal, even with \emph{zero} interaction data. We further extend ReDM to scenarios where limited or mismatched interaction data is available, and our experimental results reveal that ReDM produces high-performing policies compared to other offline RL baselines.

Published as a conference paper at ICLR 2024
POLICY REHEARSING: TRAINING GENERALIZABLE POLICIES FOR REINFORCEMENT LEARNING
Chengxing Jia 2, Chen-Xiao ; Yin', Fuxiang Zhang' Zi Chen
Tian Xu Lei Yuan Zongzhang Zhang'. Zhi-Hua Zhou', Yang 2ff National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China Polixir Technologies
{j ficcx gaocx, yinh, zhangfx, chenxh, xut, yuanl] } lamdda.... n nju Stuuede. eddu.cn (zzzhang, zhouzh, yuy) ) @n @nju. cn
ABSTRACT
Human beings can make adaptive decisions in a preparatory manner, i.c., by making preparations in advance, which offers significant advantages in scenarios where both online and offline experiences are expensive and limited. Mcanwhile, current reinforcement learning methods commonly rely on numerous environment interac- tions but hardly obtain generalizable policies. In this paper, we introduce the idea of rehearsal into policy optimization, where the agent plans for all possible outcomes in mind and acts adaptively according to actual responses from the environment. To effectively rehearse, we propose ReDM, an algorithm that generates a diverse and eligible set of dynamics models and then rehearse the policy via adaptive training on the generated model set. Rehearsal enables the policy to make decision plans for various hypothetical dynamics and to naturally generalize to previously unseen environments. Our experimental results demonstrate that ReDM is capable of learning valid policy solely through rehearsal, even with zero interaction data. We further extend ReDM to scenarios where limited or mismatched interaction data is available, and our experimental results reveal that ReDM produces high- nigh-performing policies compared to other offline RL baselines.
INTRODUCTION
In Reinforcement Learning (RL), policies are typically optimized through intensive interactions either with the target environment (Sutton & Barto 2018) or with simulators as a proxy environment (Tobin) et al. 2017 Rao et al., 2020). However, it is challenging for real-world applications since online real-world interactions are costly and sometimes dangerous, and building high-fidelity simulators requires huge amount of labor work and expertise. Offline RL (Levine et al. (2020) optimizes the policy with pre-collected datasets rather than online interactions (Fujimoto al. 2019 Kumar et
2019, 2020 Kostrikov et al. 2022) and has made remarkable progress recently (Prudencio et al. 2022). However, obtaining a comprehensive dataset, which essential for developing a reliable offline policy, can be costly in real-world applications (Kiran et al. [2021). This leads the question: Can the agent effectively make decisions without online interaction or abundant offline data?
To adapt to unfamiliar or intricate situations, humans employ cognitive mechanism known as rehearsal (Cowan & Vergauwe, 2015) Ignacio et al. 2016 Oberauer, 2019) that involves mentally practicing actions and envisioning potential outcomes. When all conceivable outcomes have been rehearsed in minds, human could make adaptive decisions based on real-world situations (Zhou 2022). Prior works have involved the concept of rehearsal in machine learning, such as continual learning (Yoon et al. (022). causality (Churamani et al. (023) and neuro-symbolic (Marconato et al., 2023). In this work, consider whether policy, by incorporating rehearsal similar to humans before taking actions, can effectively make decisions under limited real-world data.
"These authors contributed equally. Work was done during an internship at Polixir Technologies.
I Yang Yu is the corresponding author.

---

Published as a conference paper at ICLR 2024
For the mentioned consideration, we incorporate rehearsal into the reinforcement learning process to better emulate human-like learning, which we refer to as policy rehearsing. and propose Policy Rehearsing via Dynamics Model Generation (ReDM) to develop policy that can achieve satisfactory performance in environments without interaction or rich offline data. The framework of policy rehearsing iterates between two procedures. The first is to generate various tasks. Similar human rehearsal, these tasks may not directly reflect the real environment but each model represents a unique and plausible hypothesis about it. Since reward functions are often designed according to the task goals. we mainly focus on the hypothesis on dynamics model space. The second trains a meta meta-policy that can adapt to generated dynamics models. Through the ongoing generation of new dynamics models and simultaneous meta-policy learning, the policy could make rehearsals for more situations and its ability to adapt to the potential target environment gradually strengthens, despite not interacting with the target environment. However, implementing the mechanism of policy rehearsing is challenging, as the whole hypothesis space of dynamics model is large. Therefore, the core of policy rehearsing lies in how to reduce the hypothesis space of candidate dynamics models. To achieve this goal, we propose that dynamics model generation should adhere to the principles of diversity and eligibility. More specifically, we enforce diversity among the candidates by iteratively generating new candidates to minimize the worst-case performance of the current policy. While eligibility, which measures the validity of dynamics, is optimized by constraining the model to produce viable paths towards high returns. By incorporating diversity and eligibility, we effectively shrink the hypothesis space for generating candidate dynamics models and subsequently employ the meta-policy for learning.
As our model generation process is not based on directly mimicking the target, does not necessarily require interactive data. Instead, it relies on some task knowledge that can be readily obtained in practice, such as the reward function and the terminal function. Our experiments on low-dimension control tasks demonstrate that even with NO interaction data, ReDM is capable of producing valid policy that significantly outperforms the random policy. In order to extend its applicability to more complex tasks, where hypothesis space is far too large, the framework of ReDM is also compatible with offline datasets or demonstrations by using the data as a regularizer for generating models. However, it is important note that the interaction data is only used to narrow down the hypothesis space, and thus even if the provided data is limited in quantity and coverage or slightly inconsistent in dynamics, the adaptive policy produced by ReDM remains competitive. This sets ReDM apart from traditional offline RL methods where the unbiasedness of data is of vital importance for policy optimization. To validate this, we test ReDM with limited or misspecified data from D4RL, widely used benchmark for offline RL, and it turns out that ReDM outperforms other baseline methods including saate-ofthh--e model-free and model-based based offline RL algorithms. The results support that policy rehearsing is a effective approach to achieve highly generalizable policies.
2
BACKGROUND AND RELATED WORK
2.1 REINFOR LEARNING AND OFFLINE REINFORCEMENT LEARNING
The objective of RL is to learn a policy that maximizes the expected cumulative discounted reward in a Markov Decision Process (MDP) (Sutton & Barto 2018). An MDP M can be defined by six-arity tuple AS, I', r; 7; do) where S and A represent the state and action spaces respectively, T(s'Is, a) and r(s, a) represent the transition function and reward function of the dynamics, y E [0. 1] denotes the discount factor, and de denotes the distribution over initial states. In this paper, we let
Rmaax = max,,a r(s, a) be the highest reward and assume Riax 2
For a given policy π, its value function (s) = Er, EE 7'r(st, as) = s) represents the expected cumulative discounted reward for the trajectories collected by policy æ starting from so under MDP M. We define d7, to be the discounted occupancy measure of states for policy x and MDP M, satisfying dār (s) = -1 2 o yd where the occupancy of time step f satisfying (s) = Exr.r di (s')
a') and (s) = ds(s). Based on the definition of state occupancy measure (Ho & Ermon 2016), we further define the state-action distribution as d a) (s) The performance of policy x, which is the cumulative discounted reward under M, can be decomposed as the inner product of the state-action distribution and reward function: Mas (T) (s,,))az,, [r(s, a.). The goal of policy optimization, or RL, is to maximize its performance Mas (TT) in the given MDP M. The goal of offline RI., however, is optimize the policy with static
2

---

Published as a conference paper at ICLR 2024
offline dataset D and with additional online interactions. This line of research often seeks the optimal policy within the support constraint of the offline dataset, rather than searching for the global optimal policy. The effectiveness of this approach depends highly whether the offline dataset provides sufficient coverage of the state-action space interest, which in turn limits its practical application since substantially large amount of dataset still be hard to collect.
2.2 MODEL-BASED REINFORCEMENT LEARNING
To reduce the demand for interaction data, model-based RL algorithms extract dynamics models from interaction data to generate synthetic trajectories. However, the learned dynamics can be inaccurate at places where no interaction available, and such error can accumulate over long- term planning (Asadi et al. 2019). thus negatively affecting subsequent policy optimization (Janner ct al. 2019). In offline setting, various methods are proposed to mitigate this issue, such as reward penalty Yu et al.) 2020; Sun et al. 2023). modification to MDP (Kidambi et al., 2020 Yu et al. 202I), or adversarial traming (Rigter et al. 2022). all of which align with the spirit of conservatism light of uncertainty. Alternatively, MAPLE (Chen al. 2021) proposes to learn dynamics- policy on a bunch model ensembles, which can identify and adapt to the real environment during test time. However, the model ensemble in MAPLE generated via supervised training the offline dataset, while in ReDM we are considering an extreme case where no or limited interaction data is available. Such scenario issues a demand for the efficiency of the model learning process as we demonstrate in the next section.
2.3 ENVIRONMENT GENERATION
Environment generation is commonly used technique that targets generating multiple MDPs which share similar components or structures to the target environments, to improve the generalization ability and robustness when deploying policy. Several methods that have effectively improved zero-shot generalization can be classified as instances of environment generation. Among them,
domain randomization (DR) (Peng et al.] 2018 Tobin et al.) 2017 Zhou et al.] 2023] Kadokawa cct al., 2023) perturbs the parameters of a simulator designed by domain experts. Another example
procedural content generation (PCG) (Cobbe et al. 2020 Küttler et al. 2020 Dutra al. 2022
Agarwal & Shridevi 2023), which assumes an overall generation structure of the environment and creates range of MDPs by varying some pre-defined features according to random seeds.
In this paper, we are considering a more general scenario where neither such parameter ranges nor a high-fidelity and adjustable simulator available. Recent research (Khirodkar et al., 2018 Rigter ct al. 2022; Rigaki & Gareia 2023) introduces adversarial training in environment generation, but this may result -ver-pessimistic and unsolvable dynamies (Dennis al. [020 Kirk al. 2023 Mediratta et al. (023).. While in ReDM, we also motivated to design our candidate tasks to effectively facilitate the optimization. However, both DR and PCG require the pre-defined parameters range of the environment or designed simulator to ensure that the generated environment is solvable.
3 METHOD
In this section, introduce the ingredients of ReDM, which for the first time incorporates the idea of rehearsal into the RL process. In Section 3.1 we present the overall framework of policy rehearsing. In Section 3.2] we outline two principles model generation, which we identify keys to efficient candidate dynamics generation. In Section 3.3] we propose an iterative algorithm as practical implementation. Lastly in Section [3.4] we explore ways incorporate offline data to extend ReDM towards application in such scenarios.
3.1 FRAMEWORK OF POLICY REHEARSIIN
In RL, the ultimate goal is to obtain policy that performs well on target MDP M (S,,,T", r, However, the target MDP typically unavailable during the training pro- cess, and optimizing the policy in one inaccurate environment may lead to potential risk of mismatch. In policy rehearsing, we shift the focus to meta-optimizing the policy over a set candidate dynamics models. Specifically, assume the dynamics differ only in their transition function, and let the
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Framework of Policy Rehearsing
1: Input:
Initial context-based policy na, context extractor dhy, and knowledge IC about the envi- ronment (can be in any form).
2:
Obtain the candidate model set .MC via knowledge AC
3:
Learn policy xy and context extractor dy over M via Algorithm in Appendix
space M [(S, A. Tú, r, 7, do)); contains all possible dynamics. The target MDP, M is the actual MDP of interest and lies somewhere in the space M. Although M not available directly for policy optimization, we can generate candidate models from M to serve as proxies for optimization.
Unlike previous works that rely on a generative structure or simulator, we consider generating dynamics models only based on simple underlying knowledge about the target environment, such as the reward function, terminal function, or rough estimation of transition. Such knowledge helps to narrow down the whole hypothesis space M to reasonable candidate set M For example, reward functions aid in distinguishing good trajectories from bad ones, terminal functions early- stop unrealistic trajectories, while the range of the state space prevents transitioning unreachable states, thus effectively eliminating unrealistic trajectories. In our setting, we consider generating the transition models with such knowledge of reward and termination.
An adaptive policy π- is later trained on the set M to optimize the objective л" arg max; MM Thx (TT) In order adapt various dynamics models, we adopt context-based policy to serve as the adaptive policy x", which consists of two components: a context extractor ø and context-depee depeextde policy æ. and meta-training style pipeline to optimize the policy under different dynamics in the training stage. Then we obtain the formulation for adaptive policy optimization:
L Σ 2 - (. (r(,,)rrrrrr,, az) + H H((1ss)],
(1)
where 2, = ((c)) is context embedding of history c {si, 0t 82, ... 0t-1, 8t With highly expressive adaptive policy formulation, we can achieve near-optimal control in all dynamics M E M simultancously. Thus, long as the target M could be roughly included in the set MS the optimal control can be achieved in the target environment. We summarize the framework of policy rehearsing in Algorithm|] [
Random exhaustion of all models seems a way to construct M However, naively constructing M can inefficient as the space for qualified dynamics searching is still too large. almost impossible to obtain all the models. Therefore, we turn to a more effective way by limited selection of models from the whole set, where the selected models, called candidate models, aim to significantly enhance the generalization of adaptive policy on the potential target environments. We discuss how to select representative candidate models in Section 3.2
3.2 PRINCIPLES OF MODEL GENERATION
To efficiently generate candidate dynamics, we first analyze how improve the performance of the learned adaptive policy. To begin with, we first characterize the gap between the candidate model set M and the target MDP M
Definition 3.1. (Optimal Policy Gap). Let all candidate dynamics in M be identical to the target MDP M except for the transition function. The optimal policy gap e, is defined as the worst-case performance gap between the optimal performance in each M E M and M
max. MEM:
[Ma (T TaM (TT)) S Ce.
where म Jis . is the optimal policy in M and TMM is the optimal policy in M.
This discrepancy highlights the candidate model set's capability to derive proficient policy in the learned model itself. can be optimized if the ad-hoc optimal policy Tis in each candidate is performant measured by the reward function. Instead of maintaining separate optimal policies for each candidate dynamics, ReDM employs a single adaptive policy x" to achieve near-optimal control in all dynamics. Consequently, it is necessary to consider the performance cost accompanying the adaptation. We here make a simplified and mild assumption about the cost during the adaptation
process like (Chen al. 2021
4

---

Published as a conference paper at ICLR 2024
Assumption 3.2. For any candidate MDP model M from M the adaptive policy x" arg max; mM (T) satisfies
TM((=*) 2 nM
where x is the optimal policy in M and ea is the adaptive cost which takes positive value.
Then we come to the following theorem, which describes the performance bound of policy rehearsing: Theorem 3.3. Given an MDP model set {M,} and its optimal adaptive policy TO = arg maxx" EME(M 7M, TM,(TT) (T²),, , if the target MDP M satisfies min; Drv (dh d S 6ms where cm > 0, then we have:
aM- (x²) 2 7M- (1-) co. -2 Eaz
where x² is the optimal policy in " and Drv(das,, E ldbr, (s, a) -dar- (8, a)l denotes the total variance divergence.
The proof can be found Appendix A.3 This theorem highlights the factors which bound the performance loss when deploying the adaptive policy, namely ce and cm Here, E. is determined by the performance of the ad-hoe optimal policy on each dynamics. To control ce, we need to ensure that the candidate model is capable of deriving a performant policy, i.e., make sure that each candidate is eligible for policy optimization. Secondly, the term c. measures the minimum distance between the candidate model set M and the target MDP M A practical strategy control this term is to diversify the candidate set within our reach so that the target will fall into the vicinity of the candidates with higher probabilities. Finally, the term c, describes the cost of the meta-policy in identifying the dynamics characteristics. It is not solely dependent on the model generation process. To sum up the intuitions from Theorem 3.3] we propose two principles that should be taken into consideration when generating candidate dynamics: diversity and eligibility.
3.3 IMPLEMENTATIONS OF MODEL GENERATION
In this section we elaborate on how ReDM practically implements the aforementioned principles of model generation, including diversity and eligibility, where:
(a) Diversity. Quantifying the divergence hetween models can be challenging since environment dynamics is complex and high-dimensional in nature and we don't assume the generative structures available probing data in ReDM. Inspired by recent advances in applying adversarial training (Good
fellow 2016). ReDM leverages the performance gap of current policy to measure the divergence between models. Specifically, suppose at k-th iteration we have a candidate model set Mi and an adaptive policy o which has been optimized over MG such that Tद arg max,' E E. 7MM (x').
EME If we manage to generate a new candidate dynamics M' where π performs poorly, then M' should be distinct from previous candidate models MG in both trajectory and single step level. The relationship between model divergence and policy performance can be characterized with Lemma
Lemma 3.4. Given a set of MDP models M = ( policy x = arg max Ei= TM,, (x'), and an MDP model Me+1 satisfying min,e ,b} 1JM, (T) 1]Mx+ (T) 2 5, then we have the occupancy discrepancy between Me+1 and M that satisfies min;g _} DTV (dy diseen)
2Rmax and
single-step discrepancy satisfies min,e E (Tv Te+1 T T; is the transition of the MDP model M,.
(11s, a))l 2
5(1-7) where 2Rmaxx
The derivation is provided in Appendix A.2 This motivates us to iterate between adaptive policy optimization and candidate dynamics generation, and for the k-th generation, our goal to find new dynamics model M,+1 that satisfies: M arg minM na (m.). Note that when gī is fixed, this objective can be deemed as a standard RL objective where M is the agent and gद serves as the environment. The objective can further be rewritten as
M = arg min E
M (s.a)<d" M (s')),
(2)
where (s()) = E. i ~e Ir(s', a') ]')] The derivation is given in Appendix A.I In principle, any RL algorithm can be employed to optimize this objective, with the dynamics model being treated as a distinct agent that needs be learned.
5

---

Published as a conference paper at ICLR 2024
(b) Eligibility. If dynamics model is eligible, it should be able to produce viable paths toward high returns with given reward function. To enhance eligibility, we encourage the generated candidate dynamics to transit to states from which a planning- -bannd algorithm can obtain high returns, by adding state-dependent intrinsic reward defined as r=(') = max), (s)),)"), where (Ti(5') are random trajectories starting from s' and (s))(s')) = E,,,, rrr, denotes the discounted cumulative reward of the trajectory. The reward is computed on-the-fly and utilized optimize the generated dynamics models by an RL method. To sum up, our final objective for model generation is an RL objective involving both r" and r":
Mi+1 arg max EE M (s.a)=dd)) M ( (s'),
where À balances diversity and eligibility. In our implementation, we employ Proximal Policy Optimization (PPO) (Schulman et al. (2017) to optimize this objective, and the pseudo-code of model generation in ReDM is presented in Algorithm 2 (see Appendix B).
3.4 MODEL GENERATION WITH OFFLINE DATA
As tasks become increasingly complex, the hypothesis space of dynamics models catastrophically explode such that the generated candidate models cannot effectively approximate the target environ- ment. In such circumstances, generating dynamics with the guidance of reward function solely can no longer suffice policy rehearsing. One relaxed assumption is the availability of pre-collected offline interaction data from the target or a misspecified but similar environment. Such data can help narrow down the hypothesis space of dynamics model since it provides knowledge about the ground truth transition of the target environment on state-action pairs covered by the offline dataset. light of this, we extend the framework of ReDM by augmenting it with regularization term to incorporate the offline dataset, leading to the following objective:
M arg max
(a.a)=d M . (s') 1 + (r'(
('( Dog T(' as,
(4)
where D3 is the given offline dataset and the coefficient balancing RL and regularization. The regularization term resembles previous offline model-based RL algorithms, which extract an accurate model from the offline dataset. However in ReDM, we don't 't impose strict demands on the dataset, and the dataset can be small in amount or slightly mismatched in dynamics. Another remark is that we used a pre-trained policy as the planner to calculate the eligibility reward r The extended algorithm is termed ReDM-o, and the pseudo-code is provided in Appendix B
EXPERIMENTS
In this section, we aim investigate the following questions: (1) Can policy rehearsing and ReDM facilitate policy optimization when no interaction data is available? (See Section 4.1). (2) As an environment generation method, can ReDM efficiently approximate the target environment? Are the building blocks of ReDM really necessary? (See Section4.22 (3) Does ReDM-o, the ReDM extension
with limited or mismatched data, aid policy optimization complex tasks? (See Section 4.3)
4.1 REDM WITH No INTERACTION DATA
To answer question (1), we conduct experiments on three representative Gym (Brockman al. 2016) environments with continuous action spaces, namely InvertedPendulum, MountainCar (Continuous) and Acrobot. To validate ReDM, we instantiate each environment with five tasks by varying specific parameters of its simulator. For InvertedPendulum, the five tasks are created by setting the gravity coefficient to 0.5, 0.8, 1.0, 1.2 and 1.5 times the standard value of -9.81. For MountainCar, the five tasks differ in the angle of the mountain which is controlled by parameter named coef. We set ooe = 2.0, 2.5, 3.0, 3.5 and 4.0, respectively, with the original value being 3.0. For Acorbot, we change the frequency time of the 4th-order Runge-Kutt method to 0.15, 0.2, 0.25, 0.3 and 0.4, respectively. In the model generation process, we only utilize the true reward function, the terminal function, and we sample the initial states for rollouts from a Gaussian distribution. (0, I). We test ReDM with 5 independent runs for each task and record the means and standard deviations. Figure illustrates the relative performance of ReDM against random policy, and it can be observed that

---

Published as a conference paper at ICLR 2024
InvertedPendullum
Acrobat
200
125
100
Relative Performance
Relalive Performance
250
26
100
:0
Reintive
26
100
20
arglen2. angles2.5 anglerd angien3.5 onglens.0
6:0.1
8:0.16 16
Paramelers
Parameters
Paramelers
Figure 1: Relative performance between ReDM and the random policy on different environments with different hlper-paramenn
InvertedPendulum
MountainCarContinuous
Acrobot
ReOM dimm
Min Model Emror
0.00100
Random
ReDM
0.
RIOM Random
11
0.3
Min Modal
0
0.00050
Min Model
0.2
0.00025
0.05
0.1
0.00000
0
10
20
30
10
20
30
10
20
30
Number of Models
Number of Models
Number Models
Figure 2: Model loss of ReDM on different environments. All results are averaged across 5 seeds.
ReDM significantly outperforms the random policy when interaction data is available, indicating the effectiveness of policy rehearsing and ReDM. We also provide the learning curves in Appendix To illustrate the adaptation ability of the final learned policy, fine-tune it with few online trajectories and compare its performance to learning from scratch with the same policy architecture and the same amount of data The learning curves are plotted in Figure 3 Within few trajectories, our learned adaptive policy could make quicker adaptation than directly learning.
4.2 CLOSER LOOK AT THE MODEL GENERATION IN REDM
In this section we will check each building block of ReDM verify where improvements stem from. We start with introducing minimal model error A. which is defined the minimal ror achieved by a set models M, i.e. AM =
InvertedP InvertedPendulum endulum
Episodic Return
Turing
Drectt Leaming
min [s, a) - s/1 where s' is the ground- truth transition of the target environment and M(s a)
100
the predicted next state by MDP model M. To track minimal model error, at the end of each iter- ation, we collect data by the learned policy in the target environment and report the average minimal model error AM over the collected data. As shown Figure? 2 with more candidate models generated and included into the set M the average error A. gradually decreases, indicating that the set of candidate dynamics more closely approximates the target environment. This decrease in minimal example,
0.0
1.0
2.0
Log Number Trajectories
Figure 3: Learning curves of ReDM tuning and direct online learning on InvertedPendu- lum. All results are averaged across 5 seeds.
model error in turn results in improved performance. Take MountainCar (Continuous) the minimal model eиrоr (orange line) reaches its lowest with approximately 20 models, which corresponds to roughly 200 gradient steps at which the performance of the adaptive policy begins improve. To provide a comparison, we generate another candidate model set Mrrandom by iteratively adding random dynamics models. However, found that the minimum model error of this (red line in the figure) is larger than that of ReDM with a non-negligible gap.
Then visualize the information of trajectories generated by learned models. For the InvertedPendu- lum task, since we have generated dynamics models and learned the policy correspondingly. we use the learned policy rollout in the real environment (1.0 times gravity) and the generated candidate models respectively. We select three models, including the most accurate one and two randomly selected models. We plot the angles and positions the states changed over time steps in Figure (() [bb] The results indicate that although the other generated models are quite distinguished from real environment, there exists one model that is similar to the target environment.
7

---

Published as a conference paper at ICLR 2024
Position
Angle
0.1
0.0
-0.1
Value
0.9
-0.2
10 15
20
Time Steps
Time Steps
(a)
(b)
Figure 4: The data rollout by timesteps from distinguished generated models and target environments in InvertedPendllum. (a) is the position information of the data. (b) is the angle information of the
data. There exists model (red line) that could generate similar data as real environment (blue line).
ReDM Model Rollout: 10 models
Random Model Rollout: 10 models
L.SNE y-Dimension
t-SNE x-Dimension
t-SNÈ x-Dimension
10
(b)
Figure 5: The t-SNE results of the data rollout from distinguished generated models. We use the final checkpoint of the adaptive policy to roll out on ten candidate models in the candidate set and collect the interaction data. The data are then reduced into woodiieessoon via t-SNE. (a) is the projected data from our generated models, and (b) is the projecte
Relalive Performance in Model
InvertedPendulum
To illustrate the diversity of the candidate model set generated by ReDM, we employ t-SNE (Van der
60
Maaten & Hinton| 2008) to visualize the distribution of the interaction data produced by each candidate model in Figure (aa Similarly we present the t-SNE of Mrundom in Figure 5(b) It is evident from the plots that the data manifolds of each candidate generated by ReDM are clearly grouped into different clusters, which means ReDM generates more distinct dynam- ics and provides more benefits for policy rehearsing compared to the baseline.
20
Model Index
Fbo
BOOK
Tla
RialRd wel
Figure 6: The sorted relative performance between the first 10 generated models and target environment in each policy, where the red column is the model of ReDM, the green is that without diversity, and the blue is ReDM without eligibility.
I Lastly, we ablate the design of diversity and eligi- bility. As a preliminary check, we run RcDM on InvertedPendulum without the diversity reward or the eligibility reward. We found that neither of them yields valid policy. To identify the root cause of the degeneration, we calculate the accumulated return of the meta-policy given by (1) ReDM, (2) ReDM without diversity, and (3) ReDM without eligibility in their generated models. Specifically, upon the generation of the 10th candidate model, we use the policy that iteration to interact with each candidate model to obtain the performances of the i-th candidate, i e [10]. Later we rollout the same policy in the ground truth environment to obtain the reference performance. The relative performance calculated as the i-th performance minus the reference performance and reported in Figure 6 We sort the indices of candidate models according to their relative performance. The result reveals that (I) without eligibility, candidate models tend to be overly pessimistic, resulting in a significantly lower accumulated return than the ground truth; (2) without diversity, the candidate models can be overly optimistic, as the policy achieves much higher performance in all candidate models; (3) by incorporating both diversity and eligibility constraints, the evaluations of the policy are diversified and there even exists one model (e.g., model 3 in Figure whose evaluation is similar that in the target environment.
8

---

Published as a conference paper at ICLR 2024
4.3 REDM-O WITH INTERACTION DATA
In this section, we aim to test ReDM-o, which incorpo- rates external offline data or demonstrations in more com- plex tasks to answer the ques- tion (3). Our evaluations in this section differ from stan- HalfCheetah-
Table Performance of ReDM-o and baselines on D4RL with limited data. All scores presented are averaged over 5 seeds and normalized by the way proposed by (Fu al. 2020j. We bold the highest mean. Hyper-parameters can be found in Appendix 3
Type
Data Size
ReDM-o
MB-best
MF-best
dard offline RL in that we test on offline data limited in both HallCheetah-
200
2.5 + 0.9
2.2
0.6
5000
19.0 + 1.5
9.3
9.5
quantity and quality, or data collected from mismatched en- Hopper-Randee
200
23.8 1 8.0
14.3
1.1
Hupper-Random
5000
31.4 0.3
13.8
2.5
vironments. Standard offline
RL methods may be compromised in these settings, the coverage, quality and un- biasedness of the data are vital for policy optimization. However, we expect ReDM- have better performance We compare ReDM-o with several offline RL methods, such as model-free methods including CQL (Kumar et al. 2020). TD3BC (Fujimoto & Gu 202T), IQL (Kostrikov al. 2022), and model-basee based methods including MOPO (Yu et al. (020) and MAPLE (Chen et al. 2021). Here MAPLE is model- method that generates an ensemble dynamics models via supervised learning and also employs an adaptive policy to meta-train on the generated dynamics. As it only involves supervised learning, can be a direct ablation of the model generation designs in ReDM-o. We report the best results among the model-free basclines (MF-best) and model-based baselines (MB-best).
ReDM-o does not rely solely on the provided data.
For the first experiment, we sample a subset of data, which is only 200 or 5000 transitions, from random datasets in D4RL and feed the samples to each al-
Average Performance on MuJoCo
80
gorithm. Results presented in Table [T] The per- formance of standard offline RI algorithms signifi- cantly deteriorates when data is scarce and of poor quality, while ReDM-o achieves higher performance compared to the baselines, including MAPLE, which validates the effectiveness of our model generation process. To comprehensively evaluate the generaliza- tion ability, we test ReDM-o with full datasets from D4RL, including domains like HalfCheetah, Hopper, Walker2D and dataset qualities like random, medium, mediuum-replan and medium-expert. We further eval- uate the optimized meta-policies in mismatched dy- namics, by multiplying the coefficient of gravity by 0.5, 1.0, and 1.5. We report the scores averaged over all tasks and all gravity factors in Figure 7 ReDM-o is still performant compared other baselines. For more detailed information, such as the performance of the ReDM-o and baselines on each task and gravity coefficient, please refer to Appendix
Normalized Score
50
40
30
20
10
ReDM- MOPO
IQL
TD3BC COL
Figure 7: Averaged performance of RcDM-o and baselines, which are evaluated on all the dataset of D4RL. For each dataset, methods are evaluated on environment, where the grav- ity changed by multiplying 0.5, 1.0, 1.5. All scores presented are averaged over seeds and normalized (Fu al. 2020). Hyper- parameters can be found in Appendix
CONCLUSION AND FUTURE WORK
In this paper, we aim optimize policies in scenarios where interactions or rich offline data are not available. Our method, Policy Rehcarsing via Dynamics Model Generation (ReDM), incorporates the idea of rehearsal into reinforcement learning and generates candidate dynamics models for policy rehearsing. To scale to practical tasks, we establish principles of diversity and eligibility for model generation and also discuss how to leverage offline demonstrations to further narrow down the hypothesis space of models. We tested ReDM with three different settings: learning with no data, with a limited amount of data, and with mismatched data, and ReDM demonstrates consistent improvement in these scenarios. This work offers fresh perspective on reevaluating reinforeement learning and dynamics model generation. While our method primarily concentrates on generating candidate models, the exploration of adaptive policy is still avenue that requires further investigation.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Jigyasa Agarwal and S. Shridevi. Procedural content generation using reinforcement learning for disaster evacuation training in virtual 3d environment. IEEE Access, 11:98607-98617, 2023.
Vladimir Araujo, Alvaro Soto, and Marie-Francine Moens. memory model for question answering from streaming data supported by rehearsal and anticipation of coreference information. In Findings of the Association for Computational Linguistics, pp. 13 13124-13138, 2023.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L. Lituman. Combating the compounding- error problem with multi-step model. arXiv preprint vrXiv:1905.133200 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprini arXiv: 00w.66666005540 2016.
Kiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning, In Advances in Neural Information Processing Systems 34, pp. 8432-8443, 2021.
Nikhil Churamani, Jiaee Cheong, Sinan Kalkan, and Hatice Gunes. Towards causal replay for knowledge rehearsal in continual learning. In AAAI Bridge Program on Continual Causality, pp. 63-70, 2023.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In 2nd Annual Conference Robot Learning, pp. 617-629, 2018.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, pp. 2048-2056, 2020.
Nelson Cowan and Evie Vergauwe. Applying how adults rehearse understand how rehearsal may develop. Frontiers Psychology, 5:1538, 2015.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems 33, pp. 13049-13061, 2020.
Paulo Vinícius Moreira Dutra, Saulo Moraes Villela, and Raul Fonseca Neto. Procedural content generation using reinforcement learning and entropy measure as feedback. In 21st Brazilian Symposium Computer Games and Digital Entertainment, pp. 1-6, 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06001 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems 34, pp. 20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference Machine Learning, pp. 2052-2062, 2019.
Dibya Ghosh, Chethan Anand Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. In Proceedings of the 40th International Conference on Machine Learning, pp. 11321-11399 2023.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1856-1865, 2018.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29. pp. 4565 4573, 2016.
Jeanette Ignacio, Diana Dolmans, Albert Scherpbier, Jan-Joost Rethans, Violeta Lopez, and Sok Ying Liaw. Development, implementation, and evaluation of mental rehearsal strategy improve clinical performance and reduce stress: A mixed methods study, Nurse Education Today, 37:27-32, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems 32, pp. 12498-12509, 2019.
Shengqin Jiang, Yaoyu Fang, Haokui Zhang. Peng Wang, Yuankai Qi, and Qingshan Liu. Teacher agent: A non-knowledge distillation method for rehearsal-based video incremental learning. arXiv preprint arXiv:2306. 003993 2023.
Yuki Kadokawa, Lingwei Zhu, Yoshihisa Tsurumine, and Takamitsu Matsubara. Cyclic policy distillation: Sample-efficient sim-to-real reinforcement learning with domain randomization. Robotics and Autonomous Systems, 165:10, 2023.
Rawal Khirodkar, Donghyun Yoo, and Kris M Kitani. Vadra: Visual adversarial domain randomization and augmentation. arXiv preprint arXiv:1812.0044 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems 33, pp. 21810-21823, 2020.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23(6):4909 4900-449999996,,,,, 2021.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 201-264 2023.
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In 10th International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems 33, pp. 1179-1191, 2020.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Advances in Neural Information Processing Systems 33. pp. 7671-7684, 2020.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems 35, 35:34478-344 344911, 2022.
Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jínwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 5757-5766, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Iearning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Runji Lin, Ye Li. Xidong Feng, Zhaowei Zhang, Xian Hong Wu Fung, Haifeng Zhang, Jun Wang, Yali Du, and Yaodong Yang. Contextual transformer for offline meta reinforcement learning. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.

---

Published as a conference paper at ICLR 2024
Daofeng Liu, Fan Lyu, Linyan Li, Zhenping Xia, and Fuyuan Hu. Centroid distance distillation for effective rehearsal in continual learning. arXiv preprint arXiv:2303.02954, 2023.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In 7th International Conference on Learning Representations, 2019.
Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso. Neuro-symbolie continual learning: Knowledge, reasoning shortcuts and concept rehearsal. In Proceedings of the 40th International Conference on Machine Learning, pp. 23915 229366, 2023.
Ishita Mediratta, Minqi Jiang. Jack Parker-Holder, Michael Dennis, Eugene Vinitsky, and Tim Rocktäschel. Stabilizing unsupervised environment design with a learned adversary. arXiv preprint arXiv:2308. 10797, 2023.
Luckeciano C. Melo. Transformers are meta-reinforcement learners. Proceedings of the 39th International Conference Machine Learning. pp. 15340-15359, 2022.
Klaus Oberauer. Is rehearsal an effective maintenance strategy for working memory? Trends in cognitive sciences, 23(9):798-80) 2019.
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M.
Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 7487-7498, 2020.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems 33, 33:18050-1888
2020.
Francesco Pelosin and Andrea Torsello. Smaller better: An analysis of instance quantity/quality trade-off in rehearsal-basee continual learning. In International Joint Conference on Neural Networks, pp. 1-8, 2022.
Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation, pp. 1-8, 2018
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems, arXiv preprint drX1v:2003.01387, 2022.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, pp. 5331-5340, 2019.
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to- real. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11154-11163, 2020.
Maria Rigaki and Sebastian Garcia. The power of MEME: adversarial malware creation with modeel-aseed reinforcement learning. arXiv preprint arXiv:2308. 16562, 2023.
Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: robust adversarial model-based offline reinforcement learning. In Advances in Neural Information Processing Systems 35, pp. 16082- 16097, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1707.06347, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine. Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In 7th International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2024
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman incon- sistency for model-based offline reinforcement learning. In Proceedings of the 40th International Conference Machine Learning, pp. 33177-33194, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement Learning; An Introduction. MIT press, 2018
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: desearch-oriented deep offline reinforcement learning library. arXiv preprint arXiv:2210.07100 2022.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30, 2017.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in
Neural Information Processing Systems 34, 34:13-23, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems 30, pp. 6000-6010 2017.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems 33, pp. 15737-15749, 2020a.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems 33, pp. 15737-15749, 2020b.
Jiachen Yang, Brenden K. Petersen, Hongyuan Zha, and Daniel M. Faissol. Single episode policy
transfer in reinforcement learning. In 8th International Conference Learning Representations, 2020.
Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, Jialian Li, Dong Yan, and Jun Zhu. Reward informed dreamer for task generalization reinforcement learning, arXiv preprint arXiv:2303. 05002, 2023.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. In 10th International Conference on Learning Representations, 2022.
Kenny John Young, Aditya Ramesh, Louis Kirsch, and Jürgen Schmidhuber. The benefits of model- based generalization in reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, pp. 40254-40276, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems 33, pp. (4129-14142) 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Advances in Neural Information Processing Systems 34, pp. 18954-28961 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 45(04)):3964441 2023.
Zhi- Hua Zhou. Rehearsal: Learning from prediction to decision. Frontiers of Computer Science, 16
(4):164352, 2022.
13

---

Published as a conference paper at ICLR 2024
A PROOF
Lemma A.1. For the objective M arg minas (T)( where TM (T) (ss,))((s,))], and T is the transition of M, the objective is equal to
arg min (s,,a) (s))s M
where (s(') = ((o
Proof. For the objective Mas (T)) we have
NM (x) = (a. [r(s.a)]
= (1 - 7)
(s,aa)ad [r(s,a)]
1=0
bd (1
= (1 >) B (1(, + 77)
"B (B(' (')) a.0) Bay)
t=1
= (1 - 7) B - + (1 - 7) Σ
a.) (s'))
((.0) +7 7)
Σ
(s))]
- ([L))))iss)aabbbsss.) of (r"
where only the part (b) is related with the transition model as the part (a) is only with the initial distribution. Thus, minimizing (TT) equal minimizing part (b).
Lemma A.2 (Lemma 3.4). Given set of MDP models M
{M policy
arg maxx' Eiz nMs (T'), an MDP model M8+1 satisfying minie J. JM, (अ) 77 (T)) where 8 is the performance gap, we have the cumulative discrepancy between the new model and the model set satisfying
min iE(1,--
Drv(d di d 2 2R
and single step discrepancy satisfying
8(1 -))
min .}
E. Beal ()
21Rmx
where T₃ is the transition of MDP model M, and Drv(dT, dār,) = E.. ldhii, (s. -d7,, as, (s, is the total variance divergence.
Proof. First, we have
min 1JM, (T)
-MM++(()) =min
Σ (ssaa))d3, disearb
min Σ Rimax ddss, Ireezali
14

---

Published as a conference paper at ICLR 2024
then as min, Ma,, (Tt)
((T)) 2 8, we have
min
Rmax ddss,
1 dibtt1
8
min
Σ
Rmax ddM
-dMuial
2
8,a
min
Σ
d dMin21
2
2
2Rmax
8,0
8
min (ry d 2R
For the single step transition, from the theoretical results in (Xu et al. 2020b). the relationship
between the TV divergence of transitions of model M1 and M2 and their occupancy measure under model M is
Drv(d d S E..a Drv [ (1s, T2(+s,
(1-7)
where and T2 are the transitions of M1 and M2. Finally we have
Sinve..DDVT(T(( The 5(1 7)
2Rmax
The proof is completed.
Lemma A.3 (Theorem 3.3). Given a set of MDP model {M?} and its optimal adaptive policy T
arg max EME 7M, (r"), if the target MDP model M satisfies min; Drv (d My dzg.) S c.. where Em is positive value, we have
Tas (T²) 2 7M- (M=-(T") - Ce 2m Car
where x² is the optimal policy M".
Proof. For the distance condition min, Drv (dr,, dīr-) S cm, have min 7MM (T²) 775 (T²)
min Σ r(s, a))ad, dh
8,a
min
Rmax ddAs. d3
3,a
min 2 R Dry (dis,
2 Rmaax Cy.
Then by the assumption, we have
1M, (T 2 Tis. -
2 Mas- (TT*) Ce Car
where x² is the optimal policy the model M.
Then Vi that Mas, (T*) 2 7M- (TT) ce Ea and 3i that TMs, (TT) I Mas ) + 2R finally we have
Tas- T)M-(x=") TJM- TMM-(T") Eo - 2mmaxem Ea.
The proof is completed.
Then we provide simple analysis for convergence:
Lemma A.4. Given a set of MDP models Mc. {Mi)+ C M, policy mi arg max. Σ (Tr),, and an MDP model M E M satisfying min .k) 7as, (Tx) Maieas (TT) = dy and MR+1 = arg rigminMeN Ma (Tx), if we assume that nma (T&+1) TM (Tx) = 0, vk, VM M &Mi41 and mince (1."., .k) MAs, (TR) min.e(1,... 1++1)
.k+ 13 Mas, (Tk+1) 0, which
implies an increasing adaptive cost by k, then for any 0, there must exists a certain K. for any k > K we have &g <
15

---

Published as a conference paper at ICLR 2024
Algorithm 2 Candidate Model Generation
1: Input: context-basee policy parameterized by 0 and context extractor Φy parameterized by
2: Initialize a new candidate model M, with its transition function T₁ parameterized by μs.
3: Initialize an empty data buffer Dodell =
4: for = 1 to Emodel do
5: Sample initial state So-
6: for = 0 to H do
7: Sample action dg ~ (-(1 (e(()) where & is the context containing history transitions.
8:
Sample next state 81411 Tp (-1st, az) and calculate the reward r?. Sample N trajectories from 804-1 with random policy to compute eligible reward (8+41)
9:
by Eq. 2]
10:
Mix reward -rg with ry by Eq.
Add data to model buffer Moddl-
12:
end for
Update y with model buffer by PPO.
14: Reset the data buffer D model 0.
15: end for
16: Return candidate model M.
Proof. First, we have
min TM, (TK) - 1M (Tk)) (e k)
2
min
TM, (Tx)
T]Mu+a (TT))
iE(,, .))
min
TM. (Tx) T7Ma+a (TT++1
(e( .)
2 min
nar, (TK(TTK))
([11--x1]) Tat, (TK+1)
min
re(1. .b)
+
min
7M, (T&+1) 71 (T&+1)
iE (E x+1)
min TM. (Tk+1) MM (T++1)
re(1. k+1}
Thus, we have 6% 2 -x-1. And when we have all the models in our candidate set, 8₃ should be zero. òg will converge zero by the increasing of k.
The theorem demonstrates that as the number of generated models increases, becomes increasingly difficult to find models that perform poorly within the hypothesis space. This implies that our policy gradually adapts to the entire hypothesis space.
B ALGORITTM
Algorithms 2 are the pseudo-codes of candidate model generation, policy optimization, candidate model generation with offline data, ReDM and ReDM-o, respectively.
Algorithm deseribes the process generating one candidate model. First, we initialize a parameter- ized dynamics model T₁ as the candidate model M. In lines 2-10, we rollout the current meta-policy Ty on the candidate model M. each timestep, the meta-policy selects actions based on the state 8 and the context feature
(st), and the model predicts the next state $141 and the immediate reward rf. In line we use a random shooting planner to compute the eligibility reward rf. After the data collection, employ Proximal Policy Optimization (PPO) optimize the candidate model (line 11). This process will repeat for Emodel epochs, after which the new candidate will be returned.
Algorithm ssmmmarizss the optimization of the meta-policy. The algorithm takes up-to-date meta policy Te. the associated context encoder Φv. and the current candidate model set M as input. During each update epoch, randomly select a candidate model from the set and rollout the
16

---

Published as a conference paper at ICLR 2024
Algorithm 3 Policy Optimization
1: Input: Context-based policy and context extractor dv. candidate model set M
2: Initialize the data buffer D Dpolicy
3: for i == to Epolicy do
4: Randomly sample a model M from M as current model. Denote T as its dynamics function, and sample initial state 50-
5: to 0 to H do
6: Sample action de To(-18,, (o(-)ss, where e is the context containing history transitions. Sample next state S2+11 ~ [-(ss,,, St, ar) and calculate the reward fz. Add reward penalty to rs based on the uncertainty of models.
7:
8:
9: Add (st, ae, St+1, re) to dataset D policy.
end for
11: Update 9 and vb with D policy using SAC.
end for
13:
Return context-based policy and context extractor Φv
Algorithm ReDM
1: Initialize the context-hasse -hased policy To and the context encoder Φx-
2: Initialize the candidate model set M 0.
3: for k k= to Feotal do
4: Generate new candidate dynamics model M via Algorithm
5: Let M + MM U }.
6: Update policy my and context extractor de via Algorithm
7: end for
meta-policy to collect data (line 3-8), with which we update the meta-policy and the encoder using SAC (line 9). The optimization continues for Epolicy epochs.
The pseudo-code for ReDM outlined in Algorithm ReDM iteratively performs policy optimiza- tion and candidate model generation. In each iteration, it will call Algorithm to generate new candidate dynamics model, followed by meta-policy optimization by Algorithm
For ReDM-o, it differs slightly from ReDM only in the model generation process due to incorporating the offline data. We list the pseudo-code for candidate model generation in Algorithm [ Major fit the offline data after initialization, and further updated via Eq. Qi the loop; (2) we used pre-trained policy ny in replace of the random planner for efficiency. The policy ma is trained via SAC with the offline data, plus additional BC regularize similar to TD3+BC (Fujimoto & Gu 2021); (3) the initial states are sampled from offline data. follow existing practices (Yu et al. 2020) to train the meta-policy with a mixture of offline data and synthetic rollouts.
differences are (1) the model is pre-trate mre-trained
Apart from these, the offline dataset also involved in policy optimization,
C DETAILS OF EXPERIMENTS
In this section, we introduce the baselines and provide details for Section 4
how we obtained the results in
C.I BASELINE METHODS
IQL (Kostrikov et al. 2022). Implicit Q-Learning aims to mitigate errors caused by out-of-distribution data by focusing on estimating the Q-value and improving the policy solely within the in-distribution data. The algorithm achieves this through two main steps: value iteration using expectile regression and policy learning using weighted regression. During value iteration, IQL employs expectile regression to approximate the in sampe optimal state value, by assigning higher weights to samples with positive advantages. The policy optimization behavior clones offline data with weight of exponent of the advantage. The whole process eliminates the risk of incorporating out-of-datasse actions, thus providing more safe and robust improvement.
17

---

Published as a conference paper at ICLR 2024
Algorithm 5 Candidate Model Generation with Offline Data
1: Input: Context- based policy parameterized by 8 and context extractor Φy parameterized by
w. Offline data buffer Dofftine and a pre-trained policy T6 for planning.
2: Initialize a new candidate model M, with its transition function Tμ parameterized by p. Fit Tμ
with the offline data for several epochs.
3: Initialize empty data buffer model 0.
4: for = 1 to Emodel do
5: Sample initial state So from Doffline-
6: for O to H do
7: Sample action de ~ (-(s (c())) where er is the context containing history transitions.
8:
Sample next state 81+1 ~ [-1ss ax) and calculate the reward rg- Sample N trajectories from 8(41 with T6 to compute eligible reward rf (st+1) by Eq.
9:
10:
Mix reward -ry with ry by 3
11:
Add data to model buffer Doodel
12:
end for
Update y with D modell and Dottine by Equation
14: Reset the data buffer D Model = 0.
15:
end for
16:
Return candidate model M.
TD3BC (Fujimoto & Gu 2021). Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning combines the TD3 (Twin Delayed Deep Deterministic Policy Gradient) and behavior cloning techniques. TD3BC incorporates delayed updates for the target Q-networks, which further stabilizes the learning process. For policy improvement, TD3BC additionally incorporates regularization term which forces the policy to imitate actions in offline data.
CQL Kumar al. 2020). Conservative Q-Learning addresses the problem of overestimation in Q-learning methods and provides conservative estimate of the Q-values to ensure cautious policy updates. It achieves this by adding a regularization term to the standard Q-learning objective. This term penalizes OOD actions with high Q-values. By doing CQL encourages the agent to be more conservative and avoids overly optimistic estimates.
MОРO Yu et al. 2020). Model-Based Offline Policy Optimization is algorithm that utilize the dynamics model to learn a more generalizable policy than model-free methods. The algorithm consists of two main steps: model learning and policy optimization. In the model learning phase, MOPO trains a dynamics model using the offline dataset, which is used to generate synthetic rollouts. Later, MOPO combines the offline data and the synthetic rollouts for policy optimization. To account for model bias or model error, MOPO penalizes the reward of the synthetic rollouts based on the uncertainty of the models.
MAPLE (Chen et al. 2021). Model-Based Adaptive Policy Learning is a reinforcement learning algorithm that combines dynamics models and context-based policies learn adaptive policies in complex tasks. In MAPLE, the model learning process is the same as that in other model- -bsed methods like MOPO, but introduces an RNN policy to fit different dynamics of ensemble models. With the RNN policy, MAPLE can adaptively handle the OOD area in the target environment.
Code Sources. For model- free offliie algorithms including TD3BC, CQL and IQL, we use imple- mentations from CORL (Tarasov et al. (022). and keep the hyperparameter identical to the original papers. For MOPO and MAPLE, we used the implementation from the OfflineRI. codebase All parameters are kept the same as their original papers.
C.2 REDM
The hyperparameters we used are listed Table 2 (for ReDM) and Table (for ReDM-o). The hyper-parameters of concern include:
X, which balances between the diversity reward and eligibility reward in Eq.
https:://gfthhbb.comm////////
18

---

Published as a conference paper at ICLR 2024
Table 2: Eyper-parametery used in experiments without interaction data.
Model Generation
Policy Optimization
Others
Environment
A
NN
Penalty
11
Emodel
Epolicy
Bioial
InvertedPendllum
0.2
50
200
20
25
MountainCarContin
0.2
50
20
200
20
25
AcRobot
0.2
50
20
200
20
25
Table 3: Hyperparameters used in ReDM-o experiments with D4RL datasets.
Model Generation
Policy Optimization
Others
Environment
Type
a²
N
Penalty
Emodet
Epolicy
Ectaal
HallCheetah
random
0.1
0.01
50
0.25
40
HalfChoetah
medium
0,1
0.01
0.25
40
HalfCheetah
medium-repla
0.1
0.01
50
0.25
20
50
40
HalfCheetah
medium--eppert
0.1
0.01
40
Hopper
random
0.1
0.01
50
0.25
20
Hopper
medium
0.1
0.01
50
0.25
20
50
20
Hopper
mediuni-replay
0.1
0.01
0.25
20
Hopper
medium-cxpert
0.1
0.01
0.25
20
Walker2d
random
0.1
0.01
50
0,25
10
50
20
Walker2d
medium
0.1
20
Walker2d
medium-replay
0.1
0.01
50
0.25
20
Walker2d
mediun--xxpeit
0.1
0.01
50
0,25
20
50
20
Pen
0.1
0.01
0.1
Pen
human
0.1
0.01
Hammer
cloned
0.1
0.01
50
0.1
10
50
10
Hammer
N, which is the number of trajectories that the planner collects compute the eligibility reward.
Penalty, which is the coefficient of reward penalty calculated the same as that in MOPO Yuu et al. 2020) and MAPLE (Chen et al. 202T).
H, which is the policy rollout horizon.
Fmodel, Epplicy; Frotal, which are the training epochs for model optimization, policy opti- mization, and outer loop respectively.
For ReDM-o experiments, there is an additional we actually multiply the coefficient on the loss terms defined by diversity and eligibility constraints, rather than the regularization term. Thus, Table For those parameters that are not involved in the tables, we kept them identical to MAPLE for fair comparison.
in Eq. 4 In the implementation.
a to denote the coefficient, and its value is listed in
C.3 ENVIRONMENTS
InvertedPendullm. It is based on the CartPole enrironment. In this environment, there is cart that can move linearly along track. Attached one end of the cart is pole, which has another end that is free to move. The objective is to balance the pole on top of the cart by applying forces to the cart. The available actions this enviromment involve pushing the cart either to the left or right.
ConttionCar The Mountain Car is deterministic MDP that involves a car positioned stochastically at the bottom of sinusoidal valley. this version of the problem, the car can take continuous actions by applying accelerations in either direction. The objective of the MDP is to skillfully accelerate the car to reach the goal state situated on top of the right hill.
Acrobot. The Acrobot is dynamic system comprised of two links connected linearly to form chain, with one end of the chain fixed. The joint between the two links actuated, allowing for the application of torques. The objective of the Acrobot task is to strategically apply torques to the actuated joint in order to swing the free end the chain above a specified height, starting from an initial state where the chain hangs downwards.
19

---

Published as a conference paper at ICLR 2024
Table 4: Performance of ReDM-o and other baselines on D4RL benchmark with limited data. All scores presented are averaged over seeds and normalized by the way proposed by (Fu et al. 2020).
We bold the highest mean. Hyper-panee peper-paraeteers for each task can be found in Table 3
Env
Type
Size
ReDM-o
MAPLE
MOPO
CQI.
IQL.
TD3BC
HalfChoctah
Random
200
2.5 + 0.9
1.0
2.2
-1.5
0.6
-1.5
HallCheetah
Random
5000
19.0 1.5
9.3
2.4
1.7
2.2
9.5
Hopper
Random
200
23.8 8.0
14.3
0.6
0.8
1.1
0.9
Hopper
Random
5000
31.4 1.4 0.3
13.8
2.5
1.6
halfcheetah-random gravity=0..5
halfcheetah--andom gravity=1.0
halfcheetah-random gravity=1.5
Epinodic Retum
Egisodic Retum
8
Epieodic Retum
%
20
10
00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0 Epachs
1.5
2.0
0.0
0.5
1,0
1.5
2.9
Epochs
Epochs
Figure 8: Learning curves of ReDM-o on D4RL wawttcchheta--aanndooo dataset with different hyper-
parameters. All results averaged acroos seeds.
Hopper. The Hopper environment in MuloCo features a 2D one-legged robot, resembling grasshop- per or a pogo stick. The goal is to control the robot's actions to make it hop and maintain balance while moving forward. The agent must learn to apply appropriate forces to the leg joints generate hopping motions and effectively navigate the environment.
Walker2d. The Walker2d environment simulates a 2D bipedal robot with two legs. The objective is to control the robot's 's movements to make it walk and maintain stability. The agent needs to learn how to generate coordinated leg movements and adjust joint angles ensure smooth and balanced walking gaits.
The HalfCheetah environment represents 2D model of a cheetah-like quadruped robot. The task is to control the robot's actions to achieve fast and efficient running. The agent must learn to generate coordinated leg movements and apply appropriate forces to the leg joints to maximize speed and maintain stability during locomotion.
HalfCheetah.
D ADDITIONAL RESULTS OF EXPERIMENTS WITH LIMITED DATA
The detailed results of all the methods are shown in Table
E ADDITIONAL RESULTS OF EXPERIMENTS WITH MISMATCHED DATA
We chose 12 tasks of D4RL, including 3 domains (HallfCheetah, Hopper and Walker2D) and 4 dataset qualities (random, medium. medium-replay and medium-expers To create a mismatch setting, the test environment is modified by scaling its gravity by 0.5, 1.0 and 1.5, and 1.0 means the unbiased case. All results are listed in Table 5 ReDM-o outperforms other methods on the majority of the tasks
halfcheetal-meedum gravity=0.5
halfcheetah-medium gravity=1.0
hallcheetaa-mmelum gravity=1.5
Episodic Retum
Epieodic Relum
Epleodic Retum
90
20
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Epochs
Epochs
Epochs
Figure 9: Learning curves of ReDM-o on D4RL halfcheetah mediin dataset with different hyper- parameters. All results averaged acroos seeds.
20

---

Published as a conference paper at ICLR 2024
Table Performance of ReDM-o and other baselines on D4RL benchmark. All scores presented
averaged over 5 seeds and normalized by the way proposed by (Fu et al. 2020 mean. Hyper-parameters for each task can be found in Table
We bold the highest
Enviromment
Type
Purameter
ReDM-o
MAPLE
MОPO
IQL
TD3BC
CQL
HaliCheetah
random
Gravity-0.5
33.9 +
32.2
32.2
11.0
11.0
12.0
HaltCheetah
random
Gravity-10
39.7 2.4
33.1
37.0 0
12.0
10.9
15.8
HaliCheetah
random
Gravity-1.5
29.7
8.2
HallCheetah
medium
Gravity-0,5
5.79=2.7
58.1
60.1
39.5
43.7
41.3
HaliCheetah
medium
Gravity- 1.0
70.1
48.1
HaltCheetah
medium
Gravity-1.5
56.3 1.8
53.2
64.2
40.0
38.4
37.8
HaltCheetah
medium-reppay
Gravity-0.5
56.2-6
65.4
51.9
27.8
33.1
31.1
HallCheetak
medium- replay
Gravity-
61.8-17
65.6
47.0
43.9
44.7
45.2
HaltCheetah
medium-rcpplay
Gravity-1.5
5.3
16.6
35.2
35.2
34.9
33.6
HaliCheetah
medium- expert
Gravity-0.5
6.67
39.1
47.6
HallCheetah
medium expert
Gravity 1.0
96.1 96.1
95.7
70.3
91.5
89.0
90.1
HaltCheetah
medium- tredium-expert expert
Gravity-
1.6
52.7
32.2
36.3
Walker2d
random
Gravity-0.5
2..6
17.7
5.8
6.2
1.0
3.7
Walker2d
random
Gravity-1.0
21.8+0.2
21.8
3.1
4.3
1.0
3.6
Walker2d
random
Gravity 1.5
2..7
17.6
3.9
0.8
3.6
Walker2d
medium
Gravity-0.5
37.2
18.5
53.8
12.6
19.6
33.9
Walker2d
medium
Gravity-
81.7
Walker2d
medium
Gravity 1,5
$2.4
41.1
15.5
11.7
18.3
13.9
medium--rpping
Gravity-0.5
56.1=288
67.4
52.8
49.3
Walker2d
medium-replay
Gravity-100
70.1=3.8
75.2
84.1
81.7
76.3
81.0
Walker2d
mediamn-rople humm--pppayy
Gravity-1.5
74.3
42.4
34.6
32.0
14.7
18.0
Walker2d
medium expert
Gravity 0.5
74.1 6.7
87.5
51.2
64.4
56.3
46.3
Walker2d
medium-expett expert
Gravity-1.0
101.242.3
109.1
43.6
112.5
110.3
107.1
Walker2d
medium expert
Gravity 1.5
82.61
Hopper
random
Gravity-0.5
31.5
29.8
11.6
8.1
9.3
6.4
Hopper
random
Gravity-
31.6=02
31.8
10.0
7.5
8.2
Hopper
random
Gravity-1.5
31.5=02
31.6
9.5
7.5
7.T
6,9
Hupper
mediam
Gravity-O3 0.5
38.8
36.7
11.0
13.4
13.0
26.6
Hopper
medium
Gravity- 1.0
104.7
41.4
54.3
63.2
58.6
57.6
Hopper
medium
Gravity-1.5
61.4138.2
38.9
33.8
25.0
15.6
18.1
Hopper
mediam replay
Gravity-O.
21.3
Hopper
medium-replaag
Gravity- 1.0
102.5
58.4
100.9
90.6
58.4
55.0
Hopper
medium-reppaa
Gravity-1.5
31.9
23.5
22.3
21.5
14.6
Hupper
medium- expert
Gravity-0.5
48.4
59.6
12.8
21.6
17.1
46.3
Hopper
medium expert
Gravity 1.0
111.5
64.2
34.0
87.9
103.0
109.7
Hopper
medium-eree eepeit expert
Gravity-1.5
41.4++.4.7
56.3
10.7
14.5
23.7
31.1
Average
Mismatched
50.4
45.8
30.7
24.4
24.0
29.3
Average
Matched
74.2
60.3
52.4
60.4
57.6
58.3
(22/36), especially on mismatched tasks (17/24), which validates the generalization capability of ReDM-o. Additionally, we provide demonstration of the average performance in different scenarios. specifically the matched scenario, which averages performances over all tasks with Gravity- 1.00, and the mismatched scenario, which averages performances over all tasks with Gravity-O.5 and Gr. These results highlight that our method performs well on both matched and mismatched tasks.
We also provide the learning curves of ReDM-o to illustrate the stability of our method. We take halfcheetah-random in Figure and setffchhetah-mmdiine in Figure dataset stable improvement our method is evident from the curves obtained under three different values of gravity. We also find that in nulfcheetah-randenn since choose 50 the interval of generating new candidate dynamics models, an apparent improvement in the performance can be witnessed nearly every 50 epochs.
example. The
We conducted additional experiments on the standard D4RL Adroit tasks, which feature higher dimensions compared previous tasks. For instance, the state dimension in the hammer tasks is 46, while in the pen tasks it is 45. The results, as presented in Table 6 demonstrate the effectiveness of our method in handling more complex tasks.
F ADDITIONAL RESULTS OF EXPERIMENTS WITHOUT INTERACTION DATA
We additionally provide the expert policy performance on InvertedPendullm, MountainCar and Acrobot, to demonstrate the extent of improvement ReDM achieved over random policies. We also
21

---

Published as a conference paper at ICLR 2024
Table Performance of ReDM-o and other baselines on D4RL Adroit benchmark with full dataset. All scores presented are averaged over 5 seeds and normalized by the way proposed by (Fu et al. 2020 We bold the highest mean. Hyper-parameters for each task can be found in Table
Environment
Type
ReDM-o
MAPLE
MOPO
CQL.
TD3RC
IQI.
Pcn
Cloned
57.3 + 16.5
45.7
54.6
27.2
-2.1
54.8
Pen
Human
35.7 4: 17.2
27.5
10.7
35.2
1.0
65.4
Hammer
Cloned
1.5 1.5 0.5
0.9
0.5
1.4
-0.1
1.1
Hammer
Human
0.3
0.3
0.6
1.3
InvertedPendullum
MountainCarConinuuus
Acrobot
1002
Reletvee Performance
Reletivve Performence
Relative Performances
420
000
320
600
200
900-05
90--1
9mw-12
grov-1.5
orgo-2
01ge-25 ung
ange-35 angis4.0
0-0.1
41-023
Parameters
Figure 10: Relative performance between expert policy and the random policy on different environ- ments with different ryper-parameters.
organize the results in the relative performance of the expert policy versus the random policy, and the results are illustrated in Figure [0] worthwhile note that, although the expert policy can do better, ReDM managed to achieve about 30% and 70% of the expert performance in MountainCar and Acrobot respectively.
We also plot the learning curve of the meta-policy w.r.t. the number of generated candidate dynamics models, as shown in Figure[T]] As the number of gradient steps increases, candidate dynamics are generated and included in the candidate set, resulting in gradual performance improvement of ReDM over the random policy in most of the tasks.
Finally, we provide the t-SNE results of the model rollouts with a total of 3 and 5 models respectively. We collect the model rollouts with the same procedure as described in the main text. Results in Figures] 12 13 validate that models generated by ReDM are more diversified.
G DETAILED RELATED WORK
Model-Bate Based Reinforcement Learning. In complex tasks, traditional RL methods often require millions of interactions with the environment fully optimize the policy (Haarnoja al. 2018). In order to mitigate the issue of optimization, model-based reinforcement learning provides a promising approach by extracting dynamics model from the interaction data to serve as the proxy environment (Janner et al. 2019 Luo et al. 2019) or provide extra generalization ability (Young et al. 2023; Ying et al.]| Lee
2020). The most frequently used formulation of dynamics model learning is to maximize the one-step log-likelinng likelihoohd of the observed state transition (Janner et al.
InverladPendulum
MountainCarronbnuous
Acrobot
200
grauid
Eplsodiic Return
-200
19:61
grav-2 26-000000000011
unde-25
d--.15
gray- lt
a
-300
100
pww-11.772
voge-15
45-225
pww-16.715
rnddon
400
.500
200
400
800
600
200
400
600
0
200
400
600
Gradient Stope
Gradient Stops
Gradiient Steps
Figure 11: Learning curves of ReDM on different environments with different results averaged across 5 seeds.
All
22

---

Published as a conference paper at ICLR 2024
ReDM Model Rollout: 3 models
Random Model Rollout: : 3 models
t-SNE y-Dimension
I-SNE y-Dimension
64
(-SNE X- Dimensionn
I-SNE Dimension
(b)
Figure 12: The t-SNE results of the data rollout from distinguished 3 generated models. (a) is the data from our generated models with no interaction data, and (b) the data from random parameterized models.
ReDM Model Rollout models
Random Model Rollout: : 5 models
t-SNE J-Dimension
I-SNE y-Dimension
t-SNE x- XDimennsion
L-SNE X-Dimension
Figure 13: The t-SNE results of the data rollout from distinguished 5 generated models. (a) is the data from our generated models with no interaction data, and (b) the data from random parameterized models.
23

---

Published as a conference paper at ICLR 2024
2019 Luo et al.) 2019: Clavera et al. 2018). However, such MLE-estimated dynamics models known to suffer from compound error, which means the error in predicted states grows quadratically with the rollout horizon H. To control the compound error, MBPO (Janner et al.) 2019) introduces branch rollout, which rollouts the policy for a short planning horizon starting from states chosen from the off-policy buffer uniformly. An alternative for model learning is imitate the environments with GAIL (Ho& & Ermon 2016), whose error is proved to grow linearly with the horizon H (Xu et al. 2020ap.
Learning dynamics models can also benefit offline reinforcement learning in that they can augment the offline dataset with synthetic rollouts. Nevertheless, the inaccuracy of the learned models bears a more serious effect on policy optimization in the offline setting, as no more interaction data is available to provide the corrective feedback. To remedy this, a wide range of literature focuses on how to prevent the policy from accessing the uncertain area by modifying the learned MDP. For example, MOPO Yu et al., 2020) and MORel. (Kidambi et al.) 2020) learn an ensemble of models and utilize their disagreement to modify the model. RAMBO Rigter et al. 2022), on the other hand, formulates the model learning as zero-sum game with the policy and thus adversarially encourages the model to transition to low-value states. However, the above-mentioned methods all employ the principle of pessimism in the face of uncertainty, often leading to an wver-ooneervative policy. A recent approach, MAPLE (Chen et al. 2021), learns large set of models through supervised learning on the onfline dataset and then learns a dynamics-aware policy that is capable of adapting to the real dynamics by identifying the most consistent dynamics model from the training ensemble. Although MAPLE is similar to our proposed method, the ensemble set of dynamics lacks diversity and cligibility, which hinders the overall performance of the final policy.
Adaptive Policy Learning. In order to adapt to different tasks or dynamics models, it is common to augment RL policies with a context encoder so that the policy can identify the environment during execution and make corresponding adjustments to its decision. The context encoder can be implemented with various architectures.
common approach is to use RNN-based networks to encode the up-to-date history and take RNN's final state as the context, as demonstrated by RI.² and MAPLE (Chen et al. 202II. PEARL (Rakelly et al. 2019) extracts the context information via the multiplication of Gaussian distributions, each of which encodes the information carried by a single transition tuple obtained from the environment. On the other hand, recent studies have shown that the attention mechanism has an unparalleled ability in terms of feature extraction, as witnessed in natural language processing (Vaswani et al. 2017). Thus, some recent studies (Yang et al.] 2020)] Parisotto et al., 2020; Melo 2022 Lin al. 2022) adopted Transformer-like structures to encode features of transition tuples or trajectories. In this paper, we employ standard architectures of context encoders (RNN networks or Transformer encoders) to ensure a fair comparison against other baseline methods as our main focus is on how to efficiently generate dynamics model.
Unsupervised Reinforcement Learning. Through the lens of unsupervised reinforcement learning, ReDM can be categorized as learning an effect policy without knowledge about the task dynamics. Previous works in unsupervised RL typically assume that the agent can interact with the online environment without the reward signal, or that it can access offline datasets without reward anno- tations. One line of research in unsupervised RL focuses on extracting meaningful skills that can later be utilized or composed to establish an effective policy when a task reward is given in the future. To realize this, they often seek to enhance the diversity among the policy population or the extracted skills (Eysenbach et al. 2018 Laskin et al. 2022, Parker- Holder Polder-Holder et al.. 2020 Sharma jet al. 2019). Another strategy involves pre-training the agent to ancapsulate environmental dynamics in representations, thereby enabling rapid adaptation to tasks upon the availability of rewards, as investigated by (Ghosh et al., 2023 Touati & Ollivier (022)) In this paper, we examine a unique scenario where the reward function is known, but the model of dynamics remains undiscovered. Our method, ReDM, aligns with the broader unsupervised RL literature in its emphasis on diversifying the candidate dynamics set.
Rehearsal Learning. The concept of rehearsal is widely used in many areas of machine learning. In the context of causality (Zhou 2022), the concept of rehearsal involves taking proactive measures to control actionable factors that can influence the occurrence or non-occurrence of desired or undesired events. By manipulating these factors, the goal is to increase the probability of the desired event and decrease the probability of the undesired event. In the context of continual learning (Liu et al. 2023 Yoon et al.) 2022. Pelosin & Torsello| (002),, rehearsal is a technique used to mitigate catastrophic forgetting, which refers to the phenomenon where a model's performance previously learned
24

---

Published as a conference paper at ICLR 2024
Model generation iterations
learned
①
2
x Diversity
Eligibilit
(8) Policy rehearsing
the target environment
Figure 14: An illustrative example of policy rehearsing when a robot try to find the treasure. During (a) policy rehearsing, the agent is trained through cycles of generating diverse and eligible candidate dynamics models. When (b) acting in the target environment, the agent subsequently adapts to the target environment by matching it to similar candidate models it have learned.
tasks degrades significantly when learning new tasks. The goal of rehearsal is to retain knowledge of previous tasks while learning new ones, allowing the model to maintain its performance across multiple tasks over time. Rehearsal involves storing and periodically revisiting subset of previously encountered data samples (Jiang et al.) 2023) or knowledge (Churamani et al. (023) during the learning process. When training data to create a combined training set. By including past data, the model is exposed to a mixture of old and new information, which helps in preserving the knowledge acquired from previous tasks. There are also some other areas that involve rehearsal, such as nature language process (Araujo et al. 2023 and incremental learning (Jiang al. 2023).
tasks, these stored samples mixed with the current task's s
H AN ILLUSTRATION OF POLICY REHEARSING
In Figure 14 we demonstrate the idea of policy rehearsing and how helps decision-making in the target enviromment. Supposing task where a robot try to find treasures in the environment, the robot may envision possible routes and outcomes of accomplishing the task. This process reflects the dynamics model generation of ReDM. During the iterations, dynamics similar to previous outcomes or unable to complete the task will be masked out according to designed diversified and eligible metrics. A meta-policy is trained to adapt those generated dynamics models and make decisions. When deployed to the target environment, the policy can identify it and build connections with previously generated dynamics. As result, the agent can successfully adapt to the target environment with the assistance of policy rehearsing.
25


Reviews:
[{"review_id": "MvDiSJ90Uc", "reviewer": "Reviewer_YzNF", "summary": "Human beings can make adaptive decisions in a preparatory manner, i.e., by making preparations in advance, which offers significant advantages in scenarios where both online and offline experiences are expensive and limited. Meanwhile, current reinforcement learning methods commonly rely on numerous environment interactions but hardly obtain generalizable policies. In this paper, the authors introduce the idea of *rehearsal* into policy optimization, where the agent plans for all possible outcomes in mind and acts adaptively according to actual responses from the environment. To effectively rehearse, they propose ReDM, an algorithm that generates a diverse and eligible set of dynamics models and then rehearse the policy via adaptive training on the generated model set. Rehearsal enables the policy to make decision plans for various hypothetical dynamics and to natually generalize to previously unseen environments. Their experimental results demonstrate that ReDM is capable of learning a valid policy solely through rehearsal, even with zero interaction data. Besides, they further extend ReDM to scenarios where limited or mismatched interaction data is available. The provided empirical results reveal that ReDM produces high-performing policies compared with other offline RL baselines.", "questions": "1. I am curious if totally no interaction data, how can the generated dynamics model approximates the real dynamics in the target environment. It seems there lacks enough grounding points to support this potential. Does there exist the probability that the generated dynamics models are far from the dynamics in the target environment? I hope to see more analysis on this during the rebuttal.\n2. The D4RL benchmark in your experiments is all Mujoco tasks with low input dimensions. Could you please consider incorporating some more high-dimensional task, in which the hypothesis space is too large to narrow down?\n3. In the paper, you claim that the interaction data is only used to narrow down the hypothesis space. But could you please consider how to utilize these interaction data in a more direct way to better facilitate the policy learning as the complement to the purely dynamics model learning, like finetuning the learned meta policy? Besides, I cannot agree the statement that the biasedness in the interaction data will somehow hinder the policy optimization in traditional offline RL methods. If such pre-collected trajectories are expert ones or near-optimal ones, such *biasedness* can actually help avoid some low-value and dangerous states.\n4. Considering your method encourages the diversity in the model learning part, some learned dynamics models may be unreasonable though the meta policy can still achieve high returns via planning in such models, like violating the physics laws or economics laws. And I can hardly expect the *eligibility* part in your method can help alleviate this 'short-path' issue. More explanations and discussions are encouaged during the rebuttal phase.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The problem of policy rehearsing in offline reinforcement learning is interesting and challenging as an academic topic.\n2. The description to the problem modeling and the methods is clear and generally easy-understanding.\n3. The proposed method is well motivated by comprehensive preliminary theoretical analysis.\n4. The experiment analysis is in-depth and insightful, which helps the readers bettere understand the effectiveness and underlying mechanism of the propose methods.", "weaknesses": "1. The environments used in the experiments are still limited. I encourage to supplement more environments to demonstrate the applicability of your proposed method is possible. Otherwise, we may argue if the solution can only be effective on some specific kinds of tasks.\n2. Considering the proposed method needs to train the new dynamics models and meta-policy simultaneously, the complexity of this method and the training stability/convegence are encouraged to be clarified and analyzed.\n3. The assumed accessibility to the task reward function and initial state distribution is often unrealistic in the real applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Rcc8xrwzz", "reviewer": "Reviewer_anZu", "summary": "The paper proposes a method for offline model-based reinforcement learning. The idea is to generate a set of candidate dynamics models and learn an adaptive policy that optimizes the original reward on this candidate set. If the true dynamics are in the distribution of the candidate set, the adaptive policy should perform well on the true task. The central problem lies in generating a candidate set of dynamics models. The authors propose optimizing over dynamics models with RL using a reward that incentivizes (1) diversity among the set and (2) the tendency for random trajectories to achieve high reward. The method alternates between optimizing for a new dynamics model to add to the set and optimizing for a new adaptive policy given the current set. When interaction data from the true task is available, it is used to regularize the optimization over dynamics models. Experiments show the method can work with no interaction data on low-dimensional continuous control tasks (inverted pendulum, mountain car, acrobot). On two D4RL tasks (hopper, half-cheetah) with a small amount of random interaction data, the method outperforms prior offline model-free and model-based RL methods.", "questions": "- The explanation of the optimal policy gap is confusing. Specifically this sentence: \"This discrepancy highlights the candidate model set’s capability to derive a proficient policy in the model itself.\" Does \"model\" here mean the true dynamics?\n- \"we conjecture that a diversified dynamics model set will correspond to a smaller ϵa since recognizing the dynamics is much easier\" It's not clear to me why a more diverse candidate model would lower the adaptation cost. Could you explain this?\n- In Figure 6, what is the shown performance relative to? Is this the performance of the policy at each iteration in the model at that iteration minus the performance of the policy at that iteration in the ground truth model?\n- Figure 7: Are these results averaged over Hopper and HalfCheetah and averaged over each gravity level?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method is similar to MAPLE but replaces the dynamics model generation process with a more directed procedure (RL on a custom reward vs learning an ensemble of models). The reward used in the dynamics model generation process is well motivated by formal analysis of error bounds. The different components of the method are analyzed/ablated.", "weaknesses": "My main concern is the limited applicability of this method beyond low-dimensional benchmark tasks due to some significant assumptions. The method assumes access to a query-able reward/termination function and the initial state distribution. Though more importantly, the method assumes that the dynamics can be easily parameterized and optimized over with RL. Additionally, the method assumes that random plans through the candidate dynamics models will achieve some non-zero reward (to optimize the dynamics models for the eligibility reward). These assumptions makes the method difficult to apply (if not impossible) in sparse-reward or high-dimensional (e.g image-based) environments. In principle these issues could be solved by providing the method with enough interaction data to learn a good dynamics model initialization. However, then prior offline model-based or model-free methods might also work well. Additionally, this still wouldn't make the method applicable to sparse reward problems. \n\nAnother concern is the limited scope of the experiments relative to prior work. The evaluations on InvertedPendulum, MountainCar, and Acrobot are good for analyzing the method, however for the comparison to prior work, experiments are only shown for HalfCheetah and Hopper. It would be good to additionally include at least Walker2d. Additionally, the experiments with interaction data only test random interaction data and relatively small amounts of data (200 and 5000 transitions). While it is understandable that this is the setting where the proposed method would excel, it would be good to also show comparison to prior work with interaction data of varying optimality and amounts (including the full D4RL datsets). \n\nThere is no discussion of MAPLE in the related work section. MAPLE is very related (just a different model generation process) so the similarities and differences should be addressed here. It would also be good to include a brief mention of meta-learning in the related work as the proposed method uses similar concepts when optimizing for the adaptive policy.\n\nSmaller comments:\n- Algorithm 1 does not say a lot about the method. It could be replaced by algorithms 5/6 from the appendix. \n- Figure 1 should use a more descriptive x-axis label like \"Tasks\".\n- Figure 3 needs a more descriptive caption that explains what \"model loss\" means here.\n- The locations of Figure 2 and 3 should be switched.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkc3EaL3i4", "reviewer": "Reviewer_GFGi", "summary": "This paper presents a pretty interesting idea called rehearsal, which is able to **initialize or warm up a generalizable policy with zero interaction data or limited mismatched offline data**. Concretely, the proposed method, *ReDM*, takes as input a reward function and a termination function and generates a set of transition functions or models. Imaginary trajectories can thus be generated by rolling out these transition models and used to warm up the policy. As some of the models may produce data close to the target environment dynamics, the policy warmed up with these data can have a good initialization when deployed to the target environment, which is helpful for subsequent fine-tuning. Additionally, the method can be modified for offline-RL settings, allowing it to learn a robust and generalizable policy even with a small amount of offline data mismatched with target environment dynamics. \n\nThe method is motivated theoretically and contains lots of analysis like performance bound, laying foundations for future study in this new direction. Besides, the experiments on the standard gym and D4RL environment empirically prove the effectiveness of the method for both online and offline policy learning.", "questions": "1. How about replacing the random model for calculating the eligible reward with a human-crafted planner? It is supposed to be helpful for improving the performance as well. I guess this can be a good direction for exploration and to make this method more practical. A simple rule-based planner is also as easily accessible as a reward function in most practical settings like robotics. \n2. In the zero interaction data setting, the method indeed works well in three simple gym environments. I wonder if the method still works well in the more complex Mujoco environment without any pre-collected interaction data. I am curious about its performance on high-dimensional control tasks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The idea is novel unlike traditional model-based RL, this new idea suggests learning a bunch of transition models from reward function and termination functions, exempting the need for interaction data. \n2. In terms of soundness, it proves empirically and theoretically that the transition models learned in this way can help warm up the policy and improve its performance when deployed in environments with diverse transition dynamics.", "weaknesses": "1. The paper writing is not attractive. In my perspective, the main paper contains too much tedious content regarding the theoretical analysis and lacks an explanation for the rehearsal framework. My suggestion would be to move some theoretical content to the appendix and include at least one figure to explain the procedures of this new rehearsal framework and what it can achieve or why we need it. People don't care about the theoretical stuff until they are attracted by the idea and want to dive into it. Thus I suggest making some figures to explain the idea or the method.\n2. No standard deviation is included for experiments in Table 1. Also, there is no error bar in Figure 7. \n3. What is the $D_{TV}$ should be explained in the main paper. It is strongly related to your main theorem but without definition.\n4. What is relative performance? Is it calculated through minus the baseline performance?\n5. The axis *Number of models* in Figure 3 should be [0, 10, 20, 30, 40], right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ws0eaKlryg", "reviewer": "Reviewer_AWzJ", "summary": "This work considers training an agent without online interaction or abundant offline data but only with the reward function of the target environment. Borrowing the idea of rehearsal from the cognitive mechanism, this work proposes policy rehearsal. In detail, this work hopes to train an array of models to imitate the target model. Theoretical analyses indicate that the target environment performance gap between the policy trained in these imitated models and the optimal policy can be bounded by three terms, which are further summarized as diversity and eligibility. Based on these two criteria, this work proposes two corresponding reward functions for training imitated models and then uses these models to train the policy. Also, the proposed ReDM can easily combined with offline datasets. Extensive results show the effectiveness of ReDM.", "questions": "- In my opinion, the considered setting is that the agent can only get the reward function of the target task but has no knowledge about the dynamic of the target task. Is it right? Given the offline data, it is understandable that the agent can learn the dynamic to some degree. But without an offline dataset, it seems that there is no idea for the agent to learn the dynamic of the target task. \n\n- Based on the previous question, I'm confused about the setting of Experiment 4.1 \" ReDM With no Interaction Data\". As there are no data about the environment and the agent can not interact with the environment, how does the agent to learn about the environment?\n\n- As Unsupervised RL considers training an agent in the environment without reward, in my opinion, the setting in this work is like training an agent and models in the environment with reward but without dynamic. As the dynamic of the target environment will vary a lot, whether finetuning the agent (as well as the model) in the target environment with few steps will be more reasonable?\n\n- About $r_e$ for Eligibility. The proposed method is to randomly sample N trajectories and estimate the biggest return. Is this inefficient as the state space and action space are continuous in experiments? Also, what is the choice of N in experiments?\n\n- I'm curious about the performance of ReDM in the D4RL setting (Sec. 4.3) but without any Interaction Data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The ideas about the setting are novel and important, minimizing interaction with the environment as much as possible is an important problem in the RL community. Also, introducing rehearsal into RL is novel and enlightening.\n\n- The writing of Sec 3.2 is clear and solid, I have roughly read all the proofs, which are written quite clearly.\n\n- The proposed ReDM utilizes two novel terms for learning an imitated model, which is interesting and helpful.\n\nCurrently, my evaluation of this paper is really Boardline. If authors can address my concerns in Weaknesses and Questions, or point out what I have misunderstood, I'd like to update my scores accordingly. Also, I will keep active in the following discussion stage.", "weaknesses": "- The connection between diversity and controlling $\\epsilon_e, \\epsilon_a$ is unclear. For example, if all environments are the same, i.e., there is no diversity, it is obvious that $\\epsilon_a=0$ is minimal. There also needs more explanation about why $\\epsilon_e$ can be controlled via diversity.\n\n- Based on the previous points, one of my major concerns is why the proposed methods can help optimize the gap calculated in Thm 3.3. The authors have summarized the three errors in Thm 3.3 as diversity and eligibility, which indeed provides insights for analyzing this problem. But I think a more direct connection, like whether the objective in Sec 3.3 can be proven to directly control the three errors in Thm 3.3, will make the analyses more solid.\n\n- In experiments, providing the results directly trained in the target environments as the reference will better show the results.\n\n- Lack of some related works, like utilizing model-based methods for improving generalization [1-3], and finding diverse skills for unsupervised RL [4-6] as this work hopes to find diverse models.\n\n[1] Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning\n\n[2] Task Aware Dreamer for Task Generalization in Reinforcement Learning\n\n[3] The Benefits of Model-Based Generalization in Reinforcement Learning\n\n[4] Diversity is All You Need: Learning Skills without a Reward Function\n\n[5] Effective diversity in population based reinforcement learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)