Model Imitation for Model-Based Reinforcement Learning
Authors: Yueh-Hua Wu
Ting-Han Fan
Peter J. Ramadge
Hao Su
Venue:  (2020)
Keywords: Model-Based Reinforcement Learning

Abstract: Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizon, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return.

Under review as a conference paper at ICLR 2020
MODEL IMITATION FOR MODEL-BASED REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Model-based reinforcement learning (MBRL) aims to learn dynamic model to reduce the number of interactions with real-world environments. However, due estimation error, rollouts in the learned model, especially those of long horizons, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRI.. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step tran- sition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the syn- thesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real tran- sition and the learned one. Our experiments also show that the proposed model imitation method outperforms the sfate-of-the-art in terms of sample complexity and average return.
INTRODUCTION
Reinforcement learning (RL) has become of great interest because plenty of real-world problems can be modeled as a sequential decision-making problem. Model-free reinforcement learning (MFRL) is favored by its capability of learning complex tasks when interactions with environments are cheap. However, in the majority of real-world problems, such as autonomous driving, interactions are ex- tremely costly, thus MFRI. becomes infeasible. One critique about MFRL is that it does not fully exploit past queries over the environment, and this motivates us to consider the model-based rein- forcement learning (MBRL). In addition to learning an agent policy, MBRL also uses the queries to learn the dynamics of the environment that our agent is interacting with. If the learned dynamic is accurate enough, the agent can acquire the desired skill by simply interacting with the simulated environment, so that the number of samples to collect in the real world can be greatly reduced. As result, MBRL has become of the possible solutions to reduce the number of samples required to learn an optimal policy.
Most previous works of MBRL adopt supervised learning with ba-based errors (Luo et al. 2019 Kurutach et al. 2018 Clavera et al., 2018) or maximum likelihood (Janner et [ to obtain an environment model that synthesizes real transitions. These non-trivial developments imply that optimizing a policy on synthesized environment is challenging task. Because the estimation error of the model accumulates as the trajectory grows, it is hard to train policy on a long synthesized trajectory. On the other hand, training on short trajectories makes the policy short-sighted. This issue is known as the planning horizon dilemma (Wang et al.) 2019). As a result, despite having strong intuition at first sight, MBRI has to be designed meticulously.
Intuitively, we would like to learn a transition model in a way that can reproduce the trajectories that have been generated in the real world. Since the attained trajectories are sampled according to a certain policy, directly employing supervised learning may not necessarily lead to the mentioned result especially when the policy is stochastic. The resemblance in trajectories matters because estimate policy gradient by generating rollouts; however, the one-step model learning adopted by many MBRL methods do not guarantee this. Some previous works propose multi-step training (Luo jet al.,, 2019; Asadi et al. 2019) Talvitie| 2017); however, experiments show that model learning fails to benefit much from the multi-step loss. We attribute this outcome the essence of super-

---

Under review as a conference paper at ICLR 2020
Rollout world
Rollout model
Initial state
distribution
Supervised learning
Distrbuution
matchhing
Figure I: Distribution matching enables the learned transition to generate similar rollouts to the real ones even when the policy stochastic or the initial states are close. On the other hand, training with supervised learning does not ensure rollout similarity and the resulting policy gradient may be inaccurate. This figure considers a fixed policy sampling in the real world and a transition model.
vised learning, which elementally preserves only one-step transition and the similarity between real trajectories and the synthesized ones cannot be guaranteed.
In this work, we propose to learn the transition model via distribution matching. Specifically, WGAN (Arjovsky et al. 2017) to match the distributions
sitte-actionneet--stee triple (s, o, s') in real/learned models that the agent policy can generate similar trajectories when interacting with either the true transition or the learned transition. Figure illlstraaes the difference between meth- ods based supervised learning and distribution matching. Different from the ensemble methods proposed in previous works, our method is capable of generalizing to unseen transitions with only one dynamic model because merely incorporating multiple models does not alter the essence that one-step (or few-step) supervised learning fails to imitate the distribution of multi-step rollouts.
Concretely, we gather some transitions in the real world according to a policy. To learn the real transition T, we then sample fake transitions from our synthesized model T with the same policy. The synthesized model serves as generator the WGAN framework and there a critic that discriminates the two transition data. We update the generator and the critic alternatively until the synthesized data cannot be distinguished from the real one, which we will show later that it gives T' T" theoretically.
Our contributions are summarized below:
We propose an MBRL method called model imitation (MI), which enforces the learned transition model to generate similar rollouts to the real one so that policy gradient is accu- rate;
We theoretically show that the transition can be learned by MI the sense that 7 - T" by consistency and the difference in cumulative rewards can be bounded by the training error
WGAN;
To stabilize model learning, we deduce guarantee for our sampling technique and investi- gate training across WGANs;
We experimentally show that MI is more sample efficient than state-of-the-art MBRL and MFRL methods and outperforms them on four standard tasks.
RELATED WORK
In this section, we introduce our motivation inspired by learning from demonstration (Lff) SShhall 1997 and give a brief survey of MBRL methods.
2.1 LEARNING FROM DEMONSTRATION
A straightforward approach to LID is to leverage behavior cloning (BC), which reduces LfD to a supervised learning problem. Even though learning a policy via BC time-efficient, it cannot imitate policy without sufficient demonstration because the error may accumulate without the guidance of expert (Ross et al.] 2011). Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) is another state-of-the-art L.fD method that learns optimal policy by utilizing
2

---

Under review as a conference paper at ICLR 2020
generative adversarial training match oceupaney measure (Syed et al. 2008 GAIL learns an optimal policy by matching the distribution of the trajectories generated from an agent policy with the distribution of the given demonstration. Ho & Ermon (2016) shows that the two distributions match and only if the agent has learned the optimal policy. One of the advantages of GAIL. is that it only requires a small amount of demonstration data obtain an optimal policy but it requires a considerable number of interactions with environments for the generative adversarial training to converge.
Our intuition is that transition learning (TL) is similar to learning from demonstration (LfD) by exchanging the roles of transition and policy. LfD. trajectories sampled from a fixed transition are given, and the goal is to learn a policy. On the other hand, in TL, trajectories sampled from a fixed policy are given, and we would like imitate the underlying transition. That being said, from LfD to TL, we interchange the roles of the policy and the transition. It is therefore tempting to study the counterpart of GAIL. TL; i.e., learning the transition by distribution matching. Fortunately, by doing so, the pros of GAIL remain while the cons are insubstantial in MBRL because sampling with the learned model is considered to be much cheaper than sampling in the real one. That GAIL learns better policy than what BC does suggests that distribution matching possesses the potential to learn a better transition than supervised learning.
2.2 MODEL-BASER REINFORCEMEN LEARNING
For deterministic transition, ba-based error is usually utilized to learn the transition model. Naga bandi et al. (2018), an approach that uses supervised learning with seean-squared error ob- jective, is shown to perform well under fine-tuning. To alleviate model bias, some previous works adopt ensembles (Kurutach et al. 2018 Buckman al., 2018). where multiple transition models with different initialization are trained at the same time. In a slightly more complicated manner, Clavera et al. (018)) utilizes meta-learning to gather information from multiple models. Lastly, on the theoretical side, SLBO (Luo et al. 2019) is the first algorithm that develops from solid theoretical properties for model- based deep RL via a joint model-policy optimization framework.
For the stochastic transition, maximum likelihood estimator moment matching are natural ways to learn a synthesized transition, which is usually modeled by the Gaussian distribution. Following this idea, Gaussian process (Kupesik al. 2013; Deisenroth et al., 2015) and Gaussian process with model predictive control (Kamthe & Deisenroth 2017) are introduced as an uncertainty-awer version of MBRL. Similar to the deterministic case, to mitigate model bias and foster stability, an ensemble method for probabilistic networks (Chua et al. (018)) is also studied. An important distinction between training a deterministic or stochastic transition is that although the stochastic transition can model the noise hidden within the real world, the stochastic model may also induce instability if the true transition deterministie. This is a potential reason why an ensemble of models is adopted to reduce variance.
BACKGROUND
3.1 REINFORCHMENT LEARNING
We consider the standard Markov Decision Process (MDP) (Sutton & Barto 1998). MDP is repre- sented by tuple (S, A,T, r, 7), where S is the state space, A is the action space, T T(s [st, at) is the transition density of state 8t-11 at time step ++ given action a, made under state 8t, r(s, a) is the reward function, and (0, 1) is the discount factor.
A stochastic policy (aass) is density of action given state s. Let the initial state distribution be a. The performance of the triple (a, π,, T') is evaluated in the expectation of the cumulative reward in the ddiscouuted infinite horizon setting:
H Σ 1=
R(a, ग. T) = E Σ ris an) T, =
E
r( a() a. N, a)) T,
Equivalently, R(a, n, T) is the expected cumulative rewards in length- H trajectory {sr, a, ) generated by (a, n, T) with H Geometric(1 - 7)) When a and T are fixed, R(-) becomes
3

---

Under review as a conference paper at ICLR 2020
function that only depends on n, and reinforcement learning algorithms (Sutton & Barto 1998 aim to find a policy π to maximize R(T).
3.2 OCCUPANCY MEASURE
Given initial state distribution a(s), policy <{als) and transition s,) the normalized occu- 0; pancy measure Pr (s, a) generated by (a, x, T) is defined as
H-1
PT PT (s, (s.))= s,0 s, =,a== vT, =
(1-7)
P(s: Σ 1=
= = 5, TT), (2)
t=0
where P(-) is the probability measure and will be replaced by density function if S or A is con- tinuous. Intuitively, PT"
a. (8.a) .) is distribution of (s, a) in a length- H trajectory {8,, Gt )
H H-1 with
[[=)
H Geometric(1 - -) following the laws of (ax; n, T). From Syed et al. (2008), the relation be- a.
tween PT a.z and (a, π,, I") is characterized by the Bellman flow constraint. Specifically, defined in Eq. ils the unique solution to:
PT as
7))(() +7 / a (''',a')T(ssss' d')ds dd)dsdd') x(s,u) a) 2 0.
((,a) = (all)) (1-7)((s))
(3)
In addition, Theorem 2 of Syed et al. (2008) gives that x (als)) and Pr a. (s, a) have an one-to-one correspondence with (s)) and [s8'|s, a) fixed; i.e., (als)) - pls.a) occupancy measure is
J p(s.a)da is the only policy whose
With the occupancy measure, the cumulative reward Eq. can be represented
R(a, x, T) = [=aa))) (r(s, (1 - 7)-
(4)
The goal of maximizing the cumulative reward can then be achieved by adjusting p.."". and this
motivates us adopt distribution matching approaches like WGAN (Arjovsky al. (007)) to learn transition model.
4 THEORETICAL ANALYSIS FOR WGAN
In this section, we present consistency result and error bounds for WGAN (Arjovsky et al. 2017). All proofs of the following theorems and lemmas can found Appendix A
In the setting of MBRL, the training objective for WGAN is
min max E (()a)))7 ( 8 a s')] E (s,0)>ppp) (a
(5)
T If S1
By Santorovich-Rubinstei exactly W, (p(s, a, s')(lp' a, s')) where a,s.a, s') oi "ss"ss,, as,a)T(ss'ss a) is the discounted distri- bution of (s, a, Thus, by minimizing over the choice of T", we are essentially finding p' that minimizes Wi (p(s, a, s')llp' a, a,)'), which gives the consistency result.
duality (Villani 2008). the optimal value of the inner maximization is
Proposition 1 (Consistency for WGAN), Let T and T' be the true and synthesized transitions respectively. If WGAN is trained to its optimal point, we have
T(s'ss, a) =T'(s')s, V(s,a) Supp(er
where Supp(p 0.5 is the support of PT
n.
The support constraint inevitable because the training data is sampled from pr and guaranteeing anything beyond it can be difficult. Still, we will empirically show that the support constraint is not an issue in our experiments because the performance boosts up in the beginning, indicating that Supp(p a. may be large enough initially.
Now that training with WGAN gives a consistent estimate of the true transition, it is sensible to train synthesized transition upon it. However, the consistency result is too restrictive it only discusses the optimal case. The next step is to analyze the non-optimal situation and observe how the cumulative reward deviates w.r.t. the training error.
4

---

Under review as a conference paper at ICLR 2020
Theorem 1 (Error Bound for WGAN). Let PT (s,"), a), PT" (s. a) be the normalized occupancy measures generated by the true transitiin T and the synthesized one TV. If the reward function is L, Lipsschcittz and the training error of WGAN is we have R(T,, T) R( T cL,
Theorem Lindicates that if WGAN is trained properly, i... having small the cumulative reward on the synthesized trajectory will be close to that on the true trajectory. As MBRL aims train a policy on the synthesized trajectory. the accuracy of the cumulative reward over the synthesized trajectory thus the bottleneck. Theorem T also implies that WGAN's error is linear to the (expected) length of the trajectory (1 ? This is a sharp contrast to the error bounds in most RL literature, as the dependency on the trajectory length is usually quadratic (Syed & Schapire, 2010 Ross et al. al.| [201 or of an even higher order. Since WGAN gives us a better estimation of the cumulative reward in the learned model, the policy update becomes more accurate.
5 MODEL IMITATION FOR MODEL-BASED REINFORCEMENT LEARNING
In this section, we present a practical MBRL method called model imitation (MI) that incorporates the transition learning mentioned in Section
5.1 SAMPLING TECHNIQUE FOR TRANSITION LEARNING
Due to the long-term digression, it is hard to train the WGAN directly from a long synthesized trajectory. To tackle this issue, we use the synthesized transition T" sample N short trajectories with initial states sampled from the true trajectory.
To analyze this sampling technique, let 7 be the discount factor of the short trajectories so d that the expected length is [[L] (1 3)-1. To simplify the notation, let pru. Pr. Pr. Pr be the normalized occupancy measures of synthesized short trajectories, empirical true short trajectories, true short trajectories and the true long trajectories. Notice both p7. and 8 are generated from the same initial distribution pr. The Wassersstiin distance can be bounded by
Wi (P < Wi (1 (1 + WT (I + (1(Plller).
(e ((f.)) is upper bounded by the training error of WGAN on short trajectories, which can be small empirically because the short ones are easier to imitate. W1 (p li = EE
O(((1 - /B)N))///3) by Canas Rosasco 2012) and Lemma where d is the dimension of (s. Wi (p S diam(S x A)(1 - 7)8/(1 3) by Lemma 2 and W1 < Drv-diam(S x A) (Gibs & Su. 2002), where diam(-) (-) the diameter. The second term encourages to be large while the third term does the opposite. Besides, 3 need not be large if N is large enough; in practice we may sample N short trajectories to reduce the error from (pr lPT)) to W1 (Pfrller) Finally, since PT. is the occupancy measure we train on, from the proof of Theorem
we deduce that
[R(T,T) T) - R(T,T') < Wi (er) (1- 7).
Thus, WGAN may perform better under this sampling technique.
5.2 EMPIRICAL TRANSITION LEARNING
To learn the real transition based on the occupancy measure matching mentioned in Section 4 we employ a transition learning scheme by aligning the distribution of (s. o, between the real and the learned environments. Inspired by how GAII. (Ho & Ermon 2016) learns to align (s, a) via solving an MDP with rewards extracted from a discriminator, we formulate an MDP with rewards from discriminator over (s, a, s'). Specifically, the WGAN critic f(s, a, s') in Eq. is used as the (psuedo) rewards r(s, a, s') of our MDP. Interestingly, there is a duality between GAIL and our transition learning: for GAIL, the transition is fixed and the objective is to train policy to maximize the cumulative pseudo rewards, while for our transition learning. the policy is fixed and the objective is to train a synthesized transition to maximize the cumulative pseudo rewards.
In practice, since the policy is updated alternatively with the synthesized model, we are required to train number of WGANs along with the change of the policy. Although the generators across
5

---

Under review as a conference paper at ICLR 2020
WGANs correspond the same transition and can be similar, observe that WGAN may get stuck at a local optimum when we switch from one WGAN training to another. The reason is that unlike GAN that mimics the Jensen-Shannon divergence and hence its inner maximization is upper bounded by 12g( WGAN mimics the Wasserstein distance and the inner maximization is unbounded from above. Intuitively, such unboundedness makes the WGAN critic so strong that the WGAN generator (the synthesized transition) cannot find way out and get stuck at a local optimum. Thereby, we have modify the WGAN objective to alleviate such situation. To ensure the boundedness, for a fixed 8 > 0, we introduce cut-offs at the WGAN objective so that the inner maximization is upper bounded by 26:
min max E (8.0)~p7 [min(6, f(,, 5') 1 (s.a)m-piu" [min (ò, -f(s, a 11 1/ (T(T(1)),))
(6)
s S' (1s,a)
As 8 co, Eq. 6 recovers the WGAN objective, Eq. Therefore, this is truncated version of WGAN. To comprehend Eq. furtherr notice that it is equivalent to
min max E (s.a) [min(0, f(,, a, -8] (a.))-pp)" ('(ee())
[min (0i (0,-f(s,a,s)) 6)]
T' I1 (TTaaa))
<> min min (sa))=p7 [max(0, 5 f(s a, 8 s())] + (s.a) an s (-1saa
mnxx00,6+[(0 s))]
T [1 d'T(-11s,a)
which is a hinge loss version of the generative adversarial objective. Such WGAN is introduced in Lim & Ye (017), where the consistency result provided and further experiments are evaluated in Zhang et al. (2018). According to Lim & Ye (017), the inner minimization can be interpreted as the soft-margin SVM. Consequently, provides geometric intuition of maximizing margin, which potentially enhances robustness. Finally, because the objective of transition learning is to maximize
the cumulative pseudo rewards on the MDP, T does not directly optimize Eq. 7 Note that the truncation only takes part in the inner minimization:
min (ssaa=pp: (1n 8 - S(8, a, 8')1 (x.a)~pp) [max((00 ( a))),,)); 3'
(8)
Illl <'-T(118.))
which gives us a WGAN critic f(s, a, s'). As mentioned, f will be the pseudo reward function. Later, we will introduce transition learning version of PPO (Schulman al. 2017) optimize the cumulative pseudo reward.
Algorithm Model Imitation for Model-Based Reinforcement Learning
1: Parameterize policy म,, T. and WGAN critic f with 6, o, and w respectively. Initialize an empty
environment dataset ID Dany
2: for - == 0, 2....... do
3: Take actions in real environment according to Te; Denv <- Dany UD;
4: Pre-train T' and f. by optimizing Eq. 8 and II with D, and Dan
5: for N epochs do
6: for Alamsttion epochs do
7: optimize Eq. and 11 over & and w with D;
8: end for
9: for Ppllicyy epochs do
update अy by TRPO on the data generated by Te
11: end for
end for
13: end for
After modifying the WGAN objective, to include both the stochastic and (approximately) deter- ministic scenarios, the synthesized transition is modeled by a Gaussian distribution (' (s, a) = Ta(s'ls, a) ((us(s,)) E, a)). Although the underlying transitions of tasks like MuJoCo Todorov et al. 2012) are deterministic, modeling by a Gaussian does not harm the transition learn-
ing empirically.
Recall that the synthesized transition is trained on an MDP whose reward function is the critic of the truncated WGAN. To achieve this goal with proper stability, employ PPO (Schulman al.,

---

Under review as a conference paper at ICLR 2020
Hopper
HalfCheetah
average return
ave rare return
100
-200
-300
23
thousand steps
thousand steps Reacher
Ant
average return
average
thousand steps
thousand steps
MI (Ours)
PETS
METRPO
STEVE
SLBO
PPO
TRPO
Learning curves of MI versus two model-free and four model-based baselines. The solid lines indicate the mean of five trials and the shaded regions suggest standard deviation.
Figure 2:
2017), which is efficient approximation of TRPO (Schulman et al. 2015). Note that although the PPO is originally designed for policy optimization, can be adapted to transition learning with fixed sampling policy and the PPO objective (Eq. Schulman al. (2017))
(oro(o) = E min(r:())A,, clip(r:(0), -, +))
(9)
where
To (5644411))
re(
A advantage func. derived from the pseudo reward [8t, or, 81+1).
Tom (8t+1 8t, at)
(10)
To enhance stability of the transition learning, in addition to PPO, we also optimize maximum likelihood, which can be regarded regularization. We empirically observe that jointly optimizing both maximum likelihood and the PPO objective attains better transition model for policy gradient.
The overall loss of the transition learning becomes
Cransition -PpO + aLmiee
(11)
where Lile the loss MLE, which policy-agnostic and can be estimated with all collected real transitions. For more implementation details, please see Appendix B.1
We consider a training procedure similar to SLBO (Luo et al. 2019). where they consider the fact that the value function is dependent on the varying transition model. As result, unlike most of the MBRL methods that have only one pair of model-policy update for each real environment sampling, SLBO proposes take multiple update pairs for each real environment sampling. Our proposed model imitation (MI) method is summarized in Algorithm
EXPERIMENTS
In the experiment section, we would like to answer the following questions. (1) Does the proposed model imitation outperforms the state-of-the-art in terms of sample complexity and average return? (2) Does the proposed model imitation benefit from distribution matching and superior to its model-free and model-basee counterparts, TRPO and SLBO?
To fairly compare algorithms and enhance reproducibility, we adopt open-sourced environments re- leased along with model-based benchmark paper (Wang al. 2019), which is based on physical simulation engine, MuJoCo (Todorov et al. 2012). Specifically, we evaluate the proposed algorithm MI on four continuous control tasks including Hopper, HalfCheetah, Ant, and Reacher. For hyper- parameters mentioned in Algorithm I and coefficients such as entropy regularization X, please refer to Appendix B.2
7

---

Under review as a conference paper at ICLR 2020
Proportion of bench-marked RL methods that are inferior to MI in terms of 5% t-test. z/y indicates that among y approaches, MI is significantly better than 2' approaches. The detailed performance can be found in Table of Wang et al. (2019). It should be noted that the reported results in Wang et al. (2019) are the final performance after 200k time-steps whereas ours are no more than 100k time-steps.
Table 1:
Hopper
HalfCheetah
Ant
Reacher
MBRL
8/10
10/10
8/10
8/10
MFRL
3/4
2/4
4/4
3/4
We compare to two model-free algorithms, TRPO (Schulman et al. 2015) and PPO (Schulman eet al. 2017). to assess the benefit of utilizing the proposed model imitation since our MI (Algo- 1 uses TRPO for policy gradient to update the agent policy. We also compare MI to four modci-based methods. SLBO (Luo et al., 2019) gives theoretical guarantees of monotonic improve- ment for model-based deep RL and proposes update a joint model-policy objective. PETS (Chua et al.] 2018) propose to employ uncertainty-awart dynamic models with sampling- based uncertainty to capture both aleatoric and epistemic uncertainty. METRPO (Kurutach et al., 2018) shows that insufficient data may cause instability and propose to use an ensemble of models to regularize the learning process. STEVE (Buckman et al., 2018) dynamically interpolates among model rollouts of various horizon lengths and favors those whose estimates have lower error.
rithm
Figure 2 shows the learning curves for all methods. In Hopper, HalfCheetah, and Ant, MI converges fairly fast and learns policy significantly better than competitors'. In Ant, even though MI does not improve the performance too much from the initial one, the fact that it maintains the average return at around 1,000 indicates that MI can capture better transition than other methods do with only 5,000 transition data. Even though we do not employ an ensemble of models, the curves show that our learning does not suffer from high variance. In fact, the performance shown in Figure 2 indicates that the variance of MI is lower than that of methods incorporating ensembles such as METRPO and PETS.
The questions raised at the beginning of this section can now be answered. The learned model enables TRPO to explore the world without directly access real transitions and therefore TRPO equipped with MI needs much fewer interactions with the real world to learn a good policy. Even though MI is based on the training framework proposed in SLBO, the additional distribution match- ing component allows the synthesized model to generate similar rollouts to that of the real environ- ments, which empirically gives superior performance because we rely on long rollouts to estimate policy gradient.
To better understand the performance presented in Figure 2 we further compare MI with beneh- marked RL algorithms recorded in Wang al. (2019) including state-of-the-art MFRL methods such as TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al. 2018). It should be noted that the reported results of Wang et al. (2019) are the final performance after 200k time-steps but only use up to 100k time-steps to train MI. Findictts that MI significantly outperforms most of the MBRL and MFRL methods with 50% fewer samples, which verifies that MI is more sample-efficient by incorporating distribution matching.
CONCLUSION
We have pointed out that the state-of-the-art methods concentrate on learning synthesized models in a supervised fashion, which does not guarantee that the policy is able to reproduce similar trajectory in the learned model and therefore the model may not be accurate enough to estimate long rollouts. We have proposed to incorporate WGAN to achieve occupancy measure matching between the real transition and the synthesized model and theoretically shown that matching indicates the closeness in cumulative rewards between the synthesized model and the real environment.
To enable stable training across WGANs, we have suggested using truncated version of WGAN to prevent training from getting stuck at local optimums. The empirical property of WGAN application such as imitation learning indicates its potential to learn the transition with fewer samples than supervised learning. We have confirmed it experimentally by further showing that MI converges much faster and obtains better policy than ofte-of-theeart model- and model-free algorithms.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Lon Bottou. Wasserstein gan, 2017.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L Littman. Combating the compounding-error problem with multi-step model. arXiv preprini arXiv:/905. 2019.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8224-8234. Curran Associates, Inc., 2018.
Guillermo D. Canas and Lorenzo A. Rosasco. Learning probability measures with respect to optimal transport metries. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 2, NIPS' 12, pp. 2492-25000 USA, 2012. Curran Associates Inc.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine, Deep reinforcement learning handful of trials using probabilistic dynamics models. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems 31, pp. 4754 4765. Curran Associates, Inc., 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Picter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference on Robot Learning, pp. 617-629, 2018
M. P. Deisenroth, D. Fox, and C. E. Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Puttern Analysis and Machine Intelligence, 37(2): 408-423, Feb 2015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.0077 2018.
Alison L. Gibbs and Francis Edward Su. On choosing and bounding probability metrics. Interna- tional Statistical Review, 10934:419-435, 2002.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01299. 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. NeurIPS, pp. 4565-
4573, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. arXiv preprint arXiv:1906.088 08253, 2019.
Sanket Kamthe and Mare Peter Deisenroth. Data-cfficiemt reinforcement learning with probabilistic model predictive control. In AISTATS, 2017.
Andras Gabor Kupesik, Marc Peter Deisenroth, Jan Peters, and Gerhard Neumann. Data-efficient generalization of robot skills with contextual policy search. In Proceedings of the Twenty-Seventh AAAI Conference Artificial Intelligence, AAAI'I 13, pp. 1401-1407. AAAI Press, 2013.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representationss 2018. n Eorum? wi=SmmiinsSessese......
URL httpthttts///peeeeeehe htw.://wwwwweeeeeeeeeeeer.eaee.a
Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv: 028944 2017.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. Interna- fional Conference on Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2020
A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Confer- on Robotics and Automation (ICRA), pp. 7559-7566, May 2018. doi: 10.1109///...001.. 8463189.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
Stefan Schaal. Learning from demonstration. In Advances in neural information processing systems, pp. 1040-1046, 1997.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Richard S Sutton and Andrew G Barto. Introduction to Reinforcement Learning, volume 135, MIT press, 1998.
Umar Syed and Robert Schapire. A reduction from apprenticeship learning to classification. In J. D. Lafferty, C. K. I. Williams, J. Shawe Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2253-2261. Curran Associates, Inc., 2010.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. In ICML, pp. 1032-1039, 2008.
Erik Talvitie. Self-correcting models for model-based reinforcement learning. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033, 2012.
C Villani. Optimal transport Old and new, volume 338, pp. xxii+973, 01 2008.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. 2019.
David Wood. The computation of polylogarithms. Technical Report 15-92*, University of Kent, Computing Laboratory, University of Kent, Canterbury, UK, June 1992.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318, 2018.
10

---

Under review as a conference paper at ICLR 2020
A PROOFS
A.1 PROOF FOR WGAN
Proposition 1 (Consistency for WGAN), Let a(s). r(als). (s'sss, a) be initial state distribution, a. policy and synthesized transition. Let T be the true transition, p(s, p(s,a,s') a, =PT (s. T (s be the discounted distribution of the triple (s, a, under the true transition. If the WGAN is trained
to its optimal point, we have
(8)1 = (d'(s,,)) V(s a) E Supp(ep 2
Proof. Because the loss function of WGAN the I-Wasserstein distance, we know p(s, a, a',,, a, 8') at its optimal points. Plug in to the Bellman flow constraint Eq. (3.
PTi (s,a) = <<als) (1 ()((s)
+~ (1-
d') d'sds'da
= (1-)as)
())(s) +7 p'(st, s)ds' dd)
- a(al)) ma(()) +7 / pls',d, s)ds dd! = PT (s, a)
That is,
WGAN is opt. + p(s (,,,,,5) = = a's,, 8) Bellman Finally, recall s') & Pf (s.a)T(s'lss. and as, e PT a.
p7 (),a)= = P7 (s, a).
(s' we arrive
at
WGAN is opt. T()lss.)) (8))ss,)), a(s,a) E Supp(p?"))
a. (s, a) be normalized occupancy measures generated by the true transition T and the synthesized T". Suppose the reward is L, -ipisccchizz. the training enror of WGAN is then (R(TTT) R(x,T')] S
a. Theorem 1 (Two-sided Errors for WGAN), Let pT (s,a), PT"
Proof. Observe that the occupancy measure 2. (s, is a marginal distribution of p(s, a, a. (s, a..a"((ssll (s,a),,, a). Because the distance between the marginal is upper bounded by that of the PT joint, we have
Wi (s, P (s, a)) S (i a, s s')lll' (s. e,
where W, is the 1- W-Wasserstein distance. Then, the cumulative reward is bounded by
R(TT, T) = 1- / r(s, a. a)ddda = R(T, T') +
1 1 / r(s:a) a. (8, 0.5 (s, a)) dsda
R(TT, T') + 1-11 / r(s,a) (pT PT (s, dsda
L,
L
Lr
sup / (15,a) a- (s,
< T') + -->
p
0.5 7 a) dsda
Tar
= RR(T, T') + 1 1 1-
sup B( [f(s, (f a)) B(aa)~ o (f(
L, W1 (1 11 < R(T, T")
L
= R(T,TT)++ +
where the first inequality holds because r(s, a) //Lp. is I-Lipschitz and the last equality follows from Setitooovicc-Rubinstte duality Villani (2008). Since W, distance is symmetric, the same conclu- sion holds we interchange T and 7", so we arrive at
RR(T, T) R( T'I cL,
11

---

Under review as a conference paper at ICLR 2020
A.2 LEMMAS FOR SAMPLING TECHNIQUES
Lemma 1. Let L ~ Geometric(1 -). 1fd 1, then E[L = (((10 3)
Proof.
E 1/d (1-1)81 = 1 3
3
1- 3
B 1/4
1)
3 li (i))
1
where is the polylogarithm function. From Wood (1992).
the limiting behavior of is
Lit/a(e [11 1 1/d)// as l 0+
where I' is the gamma function. Since e - - μ when μ - 0+, we know when 3 - 17. ( + oe--r(111/// 1/d)(1- 3) Finally, since r(1 - 1/d) s 1, we conclude that E[L-1/d)
O((1 19)
Lemma 2. Let p8(8,a) be the normalized occupancy measure generated by the triple (a, n, T) with discount factor "i- Let (s, a) be the normalized occupancy measure generated by the triple (pT,n,1) T) with discount factor B. If > 3, then Drv S (1 - (0133////>>>> 3).
Proof. By definition of the occupancy measure we have
pr(s,a) = E (1-1))''((s,0
(s) = (-
1= 1=0
where fe(8, a) is the density of (s, a) at time if generated by the triple (a, ,, T). The TV distance bounded by
Drv(prlle
(1
7)h
Σ -1-1)) 11-1) (1-))38)) 2 1=0
Σ (1
11--2111- 1 Ea->
1=0
j=n
1
1
3(1 - -7)+ (2) i+1 (1- (1-
(1 7)
7
1=0
11-3)B M-1
-(1 7)m + +11 (1- (1 -))
Σ
7- (n - M
7- 3
(1 7)3
3)7 a strictly decreasing function in i. Since y > B, its sign flips from + to at some index; say M. Finally, the sum of the absolute value are the same from 5M-1 and from Die because the total probability conservative, and the difference
where (+) comes from that 3[ 7)
( (
side is the same as that the other.
B EXPERIMENTS
B.1 IMPLEMENTATION DETAILS
We normalize states according to the statistics derived from the first batch of states from the real world. To ensure stability, maintain the same mean di and standard deviation dg throughout the training process.
Instead of directly predicting the next state, estimate the state difference 81+1 S, (urrutach et al. 2018 Luo et al. 2019). Since we incorporate state normalization, the transition network is trained output (8++1 8t Mn)/00-
12

---

Under review as a conference paper at ICLR 2020
To enhance state exploration, sample real transitions according to policy 3 (H) o), where (((s) is the mean our Gaussian parameterized policy тe and a is fixed standard deviation. In addition, since model the transition Gaussian distribution, found that matching PTr with PT a.8 is empirically more stable and more sample-efficiet than matching pT with o
For policy update, it is shown that using the mean Hs of the Gaussian-paramettrizze transition can accelerate policy optimization and better balance exploration and exploitation. In order to enforce the Lipschitz constraint to the WGAN critic f, employ gradient penalty (Gulrajani et al. 2017) with weight 10.
B.2 HYPERPARAMETERS
HalfCheetah
Hopper
Reacher
Ant
N
10
C
10
Teransition
100
Rpolicy
20
60
100
30
horizon for model update
20
10
30
entropy regularization
0.001
0.005
Table List of spper-parameters adopted our experiments.
13


Reviews:
[{"review_id": "HJlTfB8Ttr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the model-based reinforcement learning (MBRL)\nproblem in which one of the main parts is to learn the transition\nprobability matrix of the underlying MDP (called Transition Learning -\nTL). The motivation is that if the transition model can be learnt from\nsome real-world trajectories, the agent can improve by interacting with\nsimulated environment built from the learnt transition model; hence,\nthe overall number of real-world interactions is reduced.\n\n\nGiven a policy $\\pi(a|s)$, and a transition $T(s’|s,a)$, the occupancy\nmeasure $\\rho(s,a)$, defined in Eq (2) in the paper, can be\ninterpreted as the discounted distribution of state-action pairs in\nthe rollouts. The main idea is to use Wasserstein-GAN (WGAN) to match\nthe distribution of $p(s,a,s’) = \\rho(s,a) T(s’|s,a)$. In particular,\ngiven a policy, the idea is to get some real-world trajectories and\nfeed into the WGAN framework in which the generator tries to generate\nsynthesized trajectories and the discriminator tries to discriminate\nbetween them. These trajectories are then used in policy optimization\nto obtain a new policy.\n\n\n\nThe authors presented two theoretical results: 1) Consistency for\nWGAN: if WGAN is trained optimally, then the synthesized transition is\nthe same as the real transition; and 2) Error bound for WGAN: the\ndifference between cumulative reward under synthesized transition and\nthat under real transition is upper bounded by a linear function of\nWGAN training error. On the experimental side, the authors used the\nmodel-based benchmark library (MBBL, Wang et al. 2019) to compare the\nproposed algorithm with several existing algorithms on four MuJoCo\ntasks: Hopper, HalfCheetah, Ant, and Reacher.\n\n\n\nOverall, I think this work is positive. The theoretical part is\ntechnically sound (there might be an error in the proof of Proposition\n1 but it is fixable). The experimental part is also sensible, although\nit would be good to include the performance comparisons for other\ntasks in MBBL. Details in the comments below.\n\n\n\nMain   comments/suggestions:\n\n- MBBL (Wang et al. 2019) has many other tasks (18 in total, the\n  authors only include 4 in this work). It would be good if the\n  authors can also include the performance of the proposed algorithm\n  with respect to those tasks.\n  \n\n- The claim at the end of Proposition 1’s proof (in side the proof), in my assessment, is not\n  established. It is clear from the proof that $p(s,a,s’) = p’(s,a,s’)$ is\n  a sufficient condition for $\\rho_{T}(s,a) = \\rho_{T’}(s,a)$, but I’m\n  not convinced that it is also a necessary condition. Note that\n  $\\rho_{T}(s,a)$ and $\\rho_{T’}(s,a)$ are two unique solutions of two\n  *different* Bellman equations. It would be good if the authors can\n  provide the detailed proof for the necessary condition if it is\n  true. Nevertheless, the statement of Proposition 1 only claims the\n  sufficient condition so this is fixable.\n  \n\n\n\n\nOther minor comments:\n\n- End of Section 1, page 2: the notations $T$, $T’$, $R(T)$, $R(T’)$ are used but not introduced until Section 3.\n\n- Typo, page 2, last paragraph: “IfD method” ==> “LfD” method?\n\n- Algorithm 1, page 6: what are the $\\phi$ and $w$ parameters? It looks to me that they are taken from WGAN paper but with no explanation.\n\n- Two references “Syed et al. (2008a)” and “Syed et al. (2008b)” are actually the same.\n\n- Reference “Langlois et al. (2019)” should be corrected as “Wang et al. (2019)” -- see https://arxiv.org/abs/1907.02057\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gHX_waKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In model-based reinforcement learning methods, in order to alleviate the compounding error induced in rollout-based planning, the paper proposes a distribution matching method, which should be better than a regular supervised learning approach (i.e. minimizing mean square error). Experiments on continuous control domains are presented to show the algorithm’s advantages. The author compares the proposed method with several well-known baselines. The proposed approach is interesting, however, there are some issues.\n\n1. Missing citations. The author should include discussions regarding model-correction methods. For example, Self-Correcting Models for Model-Based Reinforcement Learning, Combating the Compounding-Error Problem with a Multi-step Model. Also, there is a highly relevant work “Learning Latent Dynamics for Planning from Pixels”, in section 4, methods regarding multi-step prediction issue are introduced. \n\n2. The experimental results do not reflect the key issue the author attempts to resolve. MBRL algorithms are very diverse, there are different ways of learning a model and of using a model. Even though the proposed algorithm seems to be better, it is unclear whether it is due to better model accuracy. I think the author should show model accuracy in a rollout of samples comparing against traditional strategies. \n\n===================\nI read the author's response and take a look at other reviews. I reconsidered the contribution of the paper and found that my previous concerns may not be that important. I am fine with accepting the paper. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgHrnDiYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review for \"Model Imitation for Model-Based Reinforcement Learning\".\n\nThe paper proposes a type of model-based RL that relies on matching the  distribution of (s,a,s') tuples rather  than using supervised learning to learn an autoregressive model using supervised learning.\n\nI vote to reject the paper for three reasons.\n\n1. The motivation for matching distributions as opposed to learning the model the traditional autoregressive way is lacking. In particular, consider the table lookup case with discrete states and a single actions. Learning the model in this case corresponds to learning a stochastic matrix / Markov Chain. Call this chain P. Define a diagonal matrix whose diagonal contains the ergodic distribution of the chain \\Xi. Your framework corresponds to learning a matrix \\Xi P, while standard autoregressive models would just learn P. Knowing one gives information about the other - you can go from \\Xi P to by normalizing the rows and go from P to \\Xi P by computing the stationary distributions. On the other hand, you seem to claim in Figure 1 and in the introduction that your framework is qualitatively different from standard autoregressive models, but the above analysis suggests you are simply approximating a slightly different object, without much of an argument about why this is preferable.\n\n2. The theory section seems a bit underwhelming. In particular:\n- Proposition 1 says that we will learn a perfect model given infinite data. That is true, but I am not sure how it helps motivate the paper.\n- The presentation of Theorem 1 makes it unclear. In particular, in equation 1, you define R (the return) to depend on the transition model and the policy, but in Theorem 1, you seem to suggest that there is no dependence on the policy. \n\n3. In the experimental section, the Ant plot shows no learning for your method (MI). MI performs well when initialized and does not seem to learn anything (the curve is flat). Please justify why this happens.\n\nI will re-evaluate the paper if the above doubts are cleared up during the revision phase.\n\nMinor point:\nPlease have the paper proof-read. If you can't find help, please run it through an automated grammar checker. The current version has severe writing problems, which make the writing unclear. Examples:\n\"we analogize transition learning\"\n\"For deterministic transition, it (what?) is usually optimised with l2-based error\"\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses challenges in offline model learning, i.e., in the setting where some trajectories are given and can be used for learning a model, which in turn serves to train an RL agent or plan action sequences in simulation. A key issue in this setting is that of compounding errors: as the simulated trajectory deviates from observed data, errors build up, leading to suboptimal performance in the target domain. The paper proposes a distribution matching approach that considers trajectory sequence information and provides theoretical guarantees as well as some promising empirical results.

Several issues were raised by reviewers, including missing references, clarity issues, questions about limitations of the theoretical analysis, and limitations of the empirical validation. Many of the issues raised by reviewers were addressed by the authors during the rebuttal phase.

At the same time, several issues remain. First, the authors committed to adding results for additional tasks (initially deemed too easy or too hard to show differences). Even if the tasks show little separation between methods, these would be important data points to include as they support additional comparisons with prior and future work. The AC has to assess the paper without taking promised additional results into account. Second, questions about the results for Ant are not sufficiently addressed. The plot shows no learning. The author response mentions initialization but this is not deemed a sufficient explanation. Given the remaining questions, my assessment is that the quality and contribution of the submission are not yet ready for publication at the current stage.
Decision: Reject