Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The reviewed paper is concerned with a fundamental challenge of model-based reinforcement learning, namely that the objective used to train the dynamics model may not align with the final objective of return maximization. To this end, the authors propose a globally valid lower bound on the expected return from which both a dynamics model and policy optimization objective can be derived. Intuitively, the dynamics model is trained to produce transitions which are hardly distinguishable from real transitions, resembling a GAN objective. Additionally, policy and dynamics model training are entangled, with measures of each appearing in the training objectives of the other. The authors show the functioning of their method convincingly on a set of toy-, but also complex high-dimensional manipulation environments.

strengths_and_weaknesses: Strengths:
  * The paper is extremely clearly written, it was very enjoyable to read!
  * The method is well-grounded in prior work and soundly derived theoretically
  * The experiments are well-chosen to demonstrate the functioning of the method and presented very clearly
  * In my opinion, this work is very relevant to the NeurIPS community, providing solid theoretical groundwork for model-based reinforcement learning approaches

Weaknesses:
I have some **minor** comments on the presentation of the experiments:
  * In Fig. 4b there seems to be a mismatch between the caption and the legend. Does "RL on real" stand for SAC?
  * In Fig. 6a it is not clear to me what Q-values are shown exactly

questions: Would it be possible to combat the exploration problem by only modifying the reward with the classifier
term for experience which comes from the model, and use the original reward for transitions
observed on the real system? I.e., train $Q$ and $\pi$ on both type of transitions, but only
modify the reward on the simulated ones? Intuitively, this would make sense to me, as it would give lower
reward to potentially mismatching transitions which deviate from the real system.

limitations: One advantage of MBRL, in addition to often being more sample-efficient than purely model-free methods,
is the possibility to quickly transfer to different tasks (by generating transitions from the model and
training the policy with the changed reward, ideally requiring no additional interactions on the real system).
It would be beneficial if the authors could comment on if their proposed method also gives advantages 
in these situations where the model is to be re-used for different tasks.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Model-Based Reinforcement Learning (MBRL) is usually treated as a supervised learning problem where we first fit the data from previously observed data by reducing MSE and then do planning using this learned model. This paper highlights that this way of learning the model for planning leads to the objective mismatch problem: The model training objective is disconnected from the policy optimization process. While highlighting the problems caused by this objective mismatch, the authors propose a workaround by a unified objective for both model and policy learning. More specifically by viewing model learning as a latent variable problem, the author leverage theory from VI and propose a novel lower bound (similar to ELBO) which they maximize during the learning process. Then the authors propose a practical GAN-style algorithm where the aim is to learn a model which fools the discriminator by generating realistic-looking fake trajectories. The policy network is trained to maximize the log of the reward. Since all these optimizations are done with just a single objective, the model learning and policy learning phases are coherent resulting in learning high-return policies.   The authors support the theory presented in the main text from a series of experiments on simple tasks demonstrating the proposed algorithm's efficacy. 
strengths_and_weaknesses: Strenghts:
1. The contributions are clear and the authors did a good job of reiterating their contributions every now and then. This helped in getting an overall big picture
2. The writing is very clear. 
3. The authors discussed the limitations of their approach clearly which is commendable. 

Weaknesses:
1. Most of the experiments are toy-ish/limited and I think performing more experiments on a wide variety of tasks and benchmarks would be helpful in understanding the efficacy of MnM algorithm. 
2. MnM approx algorithm is not completely clear. 
3. The theory could be stated more formally (More on this in the next section)
4. No ablation studies 
5. Experiments are performed on limited random seeds (3 seeds) which makes it hard to comment on the validity of the reported learning curves.


questions: 1. Line 163 states “...... overestimate the policy’s return, violating lemma 3.1”. However, I skimmed through the entire paper searching for a formal statement for lemma 3.1 with little success. Could the authors state where is lemma 3.1 in the paper? 
2. Regarding difficulties of including the classifier term while doing policy optimization for continuous control tasks, I see that this is just stated without any experimental evidence. This makes it unclear for readers to understand if this is indeed causing an issue and if yes, in what way it is hurting the exploration. 
3. The authors say that they performed experiments on the OpenAI benchmark while citing the OpenAI gym paper. As far as I know, there are several environments in the OpenAI gym, but the authors seem to perform experiments only on 3 environments which makes it hard to comment on the scope of the algorithm. Could you please clarify a) the motivation behind picking these specific environments and b) the validity of your approach on the entire mujoco suit? 
4.Since all of the continuous control experiments are based on MnM approx algorithm, it would be helpful to have pseudocode highlighting the differences between MnM and MnM approx. 


limitations: The limitations are clearly mentioned in the paper and I am happy that the authors are transparent about the validity of their method. While having a single objective to train both policy and model is helpful, the authors do mention it seems to scale poorly with the problem complexity and one must employ some tricks as the authors did in MnM approx method to handle these inefficiencies. This begs the following question: Is single objective training a right way to go about building model based algorithms? If yes, why do you think so?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The proposed method provides a strategy based on a standard variational inference framework to perform model-based reinforcement learning by addressing the objective-mismatch issue. The algorithm “Mismatched-no-More” (MnM) jointly optimizes the model and policy with the help of an objective that serves as a “global” lower bound for the expected return. Practically, the approach uses a new augmented reward function containing the error between learned and real transition probabilities simplified with the help of a GAN-like classifier to differentiate between the two. The efficacy of the method is validated on 2 tabular MDPs, 3 locomotion tasks, and 4 manipulation tasks.
strengths_and_weaknesses: The paper is well written with several strong contributions:

- The paper clearly distinguishes itself from very similar prior works (like VMBPO) in terms of its theoretical contribution to maximizing the lower bound on expected return. To the best of the reviewer’s knowledge, the approach of using a GAN-like objective for model learning is novel.

- The additional analysis of model performance, model exploitation, and training performance of the MnM model with the MSE objective adds additional support to the claims made in the paper.

However, the difference between the proposed theoretical formulation and the experimental implementation makes it difficult to realize the true significance of using MnM. The approach is proposed with the intention of optimizing the same objective for policy and model learning but conducts most of the experiments only with model learning.

- Further, the comparison results are not clear on what advantage we get from MnM. In continuous control experiments, the authors show a clear analysis of the superiority of MnM-approx over other methods only on `DClawScrewRandom-v0` out of the seven environments. 
- On all the OpenAI Gym environments, the improvement in performance over MBPO is not considerable, given that an additional classifier was trained to replace the maximum likelihood objective. 
- Finally, the reviewer could not correspond the VMBPO scores presented in the paper with their published one. Looking at the original values, there seems to be hardly any difference between the performance of VMBPO and MnM which is expected given that they are solving the same objective eventually. (referring: https://www.ijcai.org/proceedings/2021/0316.pdf)
questions: - Given that “MnM-approx” is need to ensure appropriate exploration, the reviewer is curious to see the results of using MnM-approx in the tabular settings as well. Is training the policy and model with the same objective really needed at all in practice?
- I appreciate the Manipulation experiments but since the most similar algorithm to MnM is VMBPO, the authors should at least conduct experiments in all the environments shown by the original VMBPO paper. Also, the authors should properly justify the discrepancy between the VMBPO scores shown in the submitted paper and the original one.
- Is it possible to include VMBPO in Fig. 5 since the authors have included it for the tabular settings? That would complete it.
- By looking at the HalfCheetah-v2 model prediction results, the reviewer lacks clarity in realizing the advantage of using such a model. While taking an action in the initial state topples the robot in the real world, it helps the robot to move forward in the learned model. Can the authors bring some clarity given that the final evaluation of the policy is on the real environment?
- It is confusing to see the performance analysis on `DClawScrewFixed-v0`, MSE analysis on `DClawScrewRandom-v0` task and Q-value analysis on `metaworld-drawer-open-v2`. Since, the reviewer can still see a noticeable difference in the performance between MnM-approx and MBPO on `DClawScrewFixed-v0`, something more convincing will be to conduct all the analysis on the same. Moreover, adding the performance of the additional environments as the authors already have it will be the most appreciated.

limitations: The reviewer acknowledges all the limitations of the approach as mentioned by the authors and also agrees with the same. However, due to a very selective analysis of the comparison performance, the experimental segment is still not convincing enough.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to solve objective mismatch in model-based RL which claims that the objective of the model learning is different from the objective of policy learning. The authors propose Mismatch no More (MnM) which optimizes the lower bound of the expected return to jointly training the dynamics model and the policy. To optimize this lower bound practically, they introduce a modified reward function which is estimated by a trained GAN-like classifier. Experimental results show that MnM can achieve performance competitive with prior model-based methods, and better performance on certain hard exploration tasks.
strengths_and_weaknesses: Strengths:

1. To my knowledge, the idea is novel, and the idea of training a classifier to distinguish the real transitions from the fake transitions is very interesting.
2. The paper is well-written and easy to follow.

Weaknesses:

1. My main concern is about the experiments. This paper aims to solve the objective mismatch in MBRL and it is very related to Value-Awareness Model-based RL. However, they don't compare with the SOTA Value-Awareness Model-based RL methods such as VaGarm [1]  in both OpenAI Gym and ROBEL evaluations. In ROBEL environment, they even don't compare with VMBPO.

2. Besides, an important part of dyna-style model-based RL is that the learned dynamics model can perform multi-step rollout for policy learning to improve sample efficiency. I want to know whether the learned dynamics model of MnM can perform multi-step rollout, because this model is completely dependent on the value function, and has not learned the knowledge related to transition. How to control model compounding error in MnM if multi-step rollout is possible?

Typos:

In Eq.8, does $\tilde{Q}_\psi$ mean ${Q}_\psi$? I didn't find the definition of $\tilde{Q}_\psi$.

[1] Voelcker, Claas A., et al. "Value Gradient weighted Model-Based Reinforcement Learning." International Conference on Learning Representations. 2022.
questions: See Weaknesses above.
limitations: See Weaknesses above.
review_title: null

Meta-review: metareview: The paper addresses an important problem in model-based RL: the objective mis-match problem, in which the objective being optimized is different from the actual objective needed. The paper proposes a lower bound function on the expected return. The training resembles a GAN approach. The paper also presents numerical study to justify the method.

The reviewers unanimously agree that the paper could be accepted to NeurIPS. The reviewers appreciate the good writing of the paper and also recognize the technical novelty. However a major concern remains about the experiments. For instance, the reviewer believes there is a gap between the theory presented in the paper and the empirical experiments. It would be good to investigate deeper why the classifier term hurts exploration.
confidence: Certain
Decision: Accept