Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In model-based reinforcement learning methods, in order to alleviate the compounding error induced in rollout-based planning, the paper proposes a distribution matching method, which should be better than a regular supervised learning approach (i.e. minimizing mean square error). Experiments on continuous control domains are presented to show the algorithm’s advantages. The author compares the proposed method with several well-known baselines. The proposed approach is interesting, however, there are some issues.

1. Missing citations. The author should include discussions regarding model-correction methods. For example, Self-Correcting Models for Model-Based Reinforcement Learning, Combating the Compounding-Error Problem with a Multi-step Model. Also, there is a highly relevant work “Learning Latent Dynamics for Planning from Pixels”, in section 4, methods regarding multi-step prediction issue are introduced. 

2. The experimental results do not reflect the key issue the author attempts to resolve. MBRL algorithms are very diverse, there are different ways of learning a model and of using a model. Even though the proposed algorithm seems to be better, it is unclear whether it is due to better model accuracy. I think the author should show model accuracy in a rollout of samples comparing against traditional strategies. 

===================
I read the author's response and take a look at other reviews. I reconsidered the contribution of the paper and found that my previous concerns may not be that important. I am fine with accepting the paper. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper considers the model-based reinforcement learning (MBRL)
problem in which one of the main parts is to learn the transition
probability matrix of the underlying MDP (called Transition Learning -
TL). The motivation is that if the transition model can be learnt from
some real-world trajectories, the agent can improve by interacting with
simulated environment built from the learnt transition model; hence,
the overall number of real-world interactions is reduced.


Given a policy $\pi(a|s)$, and a transition $T(s’|s,a)$, the occupancy
measure $\rho(s,a)$, defined in Eq (2) in the paper, can be
interpreted as the discounted distribution of state-action pairs in
the rollouts. The main idea is to use Wasserstein-GAN (WGAN) to match
the distribution of $p(s,a,s’) = \rho(s,a) T(s’|s,a)$. In particular,
given a policy, the idea is to get some real-world trajectories and
feed into the WGAN framework in which the generator tries to generate
synthesized trajectories and the discriminator tries to discriminate
between them. These trajectories are then used in policy optimization
to obtain a new policy.



The authors presented two theoretical results: 1) Consistency for
WGAN: if WGAN is trained optimally, then the synthesized transition is
the same as the real transition; and 2) Error bound for WGAN: the
difference between cumulative reward under synthesized transition and
that under real transition is upper bounded by a linear function of
WGAN training error. On the experimental side, the authors used the
model-based benchmark library (MBBL, Wang et al. 2019) to compare the
proposed algorithm with several existing algorithms on four MuJoCo
tasks: Hopper, HalfCheetah, Ant, and Reacher.



Overall, I think this work is positive. The theoretical part is
technically sound (there might be an error in the proof of Proposition
1 but it is fixable). The experimental part is also sensible, although
it would be good to include the performance comparisons for other
tasks in MBBL. Details in the comments below.



Main   comments/suggestions:

- MBBL (Wang et al. 2019) has many other tasks (18 in total, the
  authors only include 4 in this work). It would be good if the
  authors can also include the performance of the proposed algorithm
  with respect to those tasks.
  

- The claim at the end of Proposition 1’s proof (in side the proof), in my assessment, is not
  established. It is clear from the proof that $p(s,a,s’) = p’(s,a,s’)$ is
  a sufficient condition for $\rho_{T}(s,a) = \rho_{T’}(s,a)$, but I’m
  not convinced that it is also a necessary condition. Note that
  $\rho_{T}(s,a)$ and $\rho_{T’}(s,a)$ are two unique solutions of two
  *different* Bellman equations. It would be good if the authors can
  provide the detailed proof for the necessary condition if it is
  true. Nevertheless, the statement of Proposition 1 only claims the
  sufficient condition so this is fixable.
  




Other minor comments:

- End of Section 1, page 2: the notations $T$, $T’$, $R(T)$, $R(T’)$ are used but not introduced until Section 3.

- Typo, page 2, last paragraph: “IfD method” ==> “LfD” method?

- Algorithm 1, page 6: what are the $\phi$ and $w$ parameters? It looks to me that they are taken from WGAN paper but with no explanation.

- Two references “Syed et al. (2008a)” and “Syed et al. (2008b)” are actually the same.

- Reference “Langlois et al. (2019)” should be corrected as “Wang et al. (2019)” -- see https://arxiv.org/abs/1907.02057



review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Review for "Model Imitation for Model-Based Reinforcement Learning".

The paper proposes a type of model-based RL that relies on matching the  distribution of (s,a,s') tuples rather  than using supervised learning to learn an autoregressive model using supervised learning.

I vote to reject the paper for three reasons.

1. The motivation for matching distributions as opposed to learning the model the traditional autoregressive way is lacking. In particular, consider the table lookup case with discrete states and a single actions. Learning the model in this case corresponds to learning a stochastic matrix / Markov Chain. Call this chain P. Define a diagonal matrix whose diagonal contains the ergodic distribution of the chain \Xi. Your framework corresponds to learning a matrix \Xi P, while standard autoregressive models would just learn P. Knowing one gives information about the other - you can go from \Xi P to by normalizing the rows and go from P to \Xi P by computing the stationary distributions. On the other hand, you seem to claim in Figure 1 and in the introduction that your framework is qualitatively different from standard autoregressive models, but the above analysis suggests you are simply approximating a slightly different object, without much of an argument about why this is preferable.

2. The theory section seems a bit underwhelming. In particular:
- Proposition 1 says that we will learn a perfect model given infinite data. That is true, but I am not sure how it helps motivate the paper.
- The presentation of Theorem 1 makes it unclear. In particular, in equation 1, you define R (the return) to depend on the transition model and the policy, but in Theorem 1, you seem to suggest that there is no dependence on the policy. 

3. In the experimental section, the Ant plot shows no learning for your method (MI). MI performs well when initialized and does not seem to learn anything (the curve is flat). Please justify why this happens.

I will re-evaluate the paper if the above doubts are cleared up during the revision phase.

Minor point:
Please have the paper proof-read. If you can't find help, please run it through an automated grammar checker. The current version has severe writing problems, which make the writing unclear. Examples:
"we analogize transition learning"
"For deterministic transition, it (what?) is usually optimised with l2-based error"

review_title: Official Blind Review #1

Meta-review: This paper addresses challenges in offline model learning, i.e., in the setting where some trajectories are given and can be used for learning a model, which in turn serves to train an RL agent or plan action sequences in simulation. A key issue in this setting is that of compounding errors: as the simulated trajectory deviates from observed data, errors build up, leading to suboptimal performance in the target domain. The paper proposes a distribution matching approach that considers trajectory sequence information and provides theoretical guarantees as well as some promising empirical results.

Several issues were raised by reviewers, including missing references, clarity issues, questions about limitations of the theoretical analysis, and limitations of the empirical validation. Many of the issues raised by reviewers were addressed by the authors during the rebuttal phase.

At the same time, several issues remain. First, the authors committed to adding results for additional tasks (initially deemed too easy or too hard to show differences). Even if the tasks show little separation between methods, these would be important data points to include as they support additional comparisons with prior and future work. The AC has to assess the paper without taking promised additional results into account. Second, questions about the results for Ant are not sufficiently addressed. The plot shows no learning. The author response mentions initialization but this is not deemed a sufficient explanation. Given the remaining questions, my assessment is that the quality and contribution of the submission are not yet ready for publication at the current stage.
Decision: Reject