Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Authors: Benjamin Eysenbach
Alexander Khazatsky
Sergey Levine
Ruslan Salakhutdinov
Venue: NeurIPS 2022 Accept (2022)
Keywords: theory
reinforcement learning
model-based RL
GAN
joint optimization
unified objective

Abstract: Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning. However, models that achieve better training performance (e.g., lower MSE) are not necessarily better for control: an RL agent may seek out the small fraction of states where an accurate model makes mistakes, or it might act in ways that do not expose the errors of an inaccurate model. As noted in prior work, there is an objective mismatch: models are useful if they yield good policies, but they are trained to maximize their accuracy, rather than the performance of the policies that result from them.  In this work, we propose a single objective for jointly training the model and the policy, such that updates to either component increase a lower bound on expected return. To the best of our knowledge, this is the first lower bound for model-based RL that holds globally and can be efficiently estimated in continuous settings; it is the only lower bound that mends the objective mismatch problem. A version of this bound becomes tight under certain assumptions. Optimizing this bound resembles a GAN: a classifier distinguishes between real and fake transitions, the model is updated to produce transitions that look realistic, and the policy is updated to avoid states where the model predictions are unrealistic. Numerical simulations demonstrate that optimizing this bound yields reward maximizing policies and yields dynamics that (perhaps surprisingly) can aid in exploration. We also show that a deep RL algorithm loosely based on our lower bound can achieve performance competitive with prior model-based methods, and better performance on certain hard exploration tasks.

Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
Benjamin Eysenbach" 12 Alexander Khazatsky*
Sergey Levine Ruslan Salakhutdinoy
Carnegie Mellon University, beysenba@ca.. cmu. edu,
Google Brain, UCC Berkeley
khazatsky@cs. www.ssaa...eeooooommm
Abstract
Many model-based reinforcement learning (RL) methods follow similar template: fit model to previously observed data, and then use data from that model for RL or planning. However, models that achieve better training performance (e.g.. lower MSE) are not necessarily better for control: an RL agent may seek out the small fraction of states where an accurate model makes mistakes, that do not expose the errors of an inaccurate model. As noted in prior work, there is an objective mismatch: models are useful if they yield good policies, but they are trained to maximize their accuracy, rather than the performance of the policies that result from them. In this work, we propose a single objective for jointly training the model and the policy, such that updates to either component increase lower bound on expected return. To the best of our knowledge. this the first lower bound for model-based RL that holds globally and can be efficiently estimated in continuous settings; is the only lower bound that mends the objective mismatch problem. A version of this bound becomes tight under certain assumptions. Optimizing this bound resembles a GAN: a classifier distinguishes between real and fake transitions, the model is updated to produce transitions that look realistic, and the policy is updated to avoid states where the model predictions are unrealistic. Numerical simulations demonstrate that optimizing this bound yields reward maximizing policies and yields dynamics that (perhaps surprisingly) can aid exploration. We also show that deep RI. algorithm loosely based lower bound can achieve performance competitive with prior model-based methods, and better performance on certain hard exploration tasks.
might act in ways
Introduction
Much of the appeal of model-based RL is that model learning simple and scalable supervised learning problem. However, even after learning a very accurate model, it is hard to say whether that model will actually be useful for model-based RL [17, 29]. For example, model might make small mistakes in critical states that cause policy to take suboptimal actions. Alternatively, model with large errors may yield policy that attains high return if the model errors occur in states that the policy never visits.
The underlying problem is that dynamics models trained differently from how they are used. Typical model-based methods train a model using data sampled from the real dynamics (e.g., using maximum likelihood), but apply these models by using data sampled from the learned dynamics [1 23, 24, 48). Prior work has identified this objective mismatch issue [17, 29, 30]: the model is trained using one objective, but the policy is trained using different objective. Designing an objective for model training that is guaranteed to improve the expected reward remains an open problem.
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

So. how should we train a dynamics model so that it produces high-return policies when used for model-based RL?
The key idea in this paper is to view model-hased RI. as a latent-variable problem: the latent variable is the trajectory and the cumulative reward is interpreted as the probability that the trajectory solves the task. Inferring the latent variable corresponds to learning a dynamics model. Latent variable models are typically learned via an evidence lower bound, and we show how a similar evidence lower bound provides yields a new objective for model-based RL. In the same way that the evidence lower bound is a joint optimization problem over two variables, our objective will jointly optimize the model and the policy using the same objective: to produce realistic and high-return trajectories. Our objective differs from standard model-based RL objectives, where it is more common to pit the model against the policy [5, 33, 37J. A consequence of maximizing the lower bound is that the dynamics model does not learn the true dynamics, but rather learns optimistic dynamics that facilitate exploration.
The main contribution of this work is an objective for model-based RL. To the best of our knowledge. this is the first lower bound for model-based RL that holds globally (unlike Luo et al. [30]) and can be efficiently estimated in continuous settings (unlike Kearns and Singh [26]). It is the first lower bound that jointly optimizes the model and policy using the same objective. We also present a more complex version of this bound that becomes tight under some assumptions. Through numerical simulations in simple tasks, we demonstrate that optimizing the bound yields reward maximizing policies and yields an optimistic dynamics model that can aid exploration. We also demonstrate that our bound gracefully accounts for function approximation error in the model - the policy is penalized for taking transitions that the model cannot represent. Finally, we show that we can use parts of our theoretically- motivated -ootiiaally objective to design maatically-aaplicabble deep RL method that, despite deviating from the theory, can match the performance of prior model-based methods.
2 Related Work
Most model- based RL methods use maximum likelihood to fit the dynamics model and then use RL to maximize the expected return under samples from that model [10, 11, 23, 24, 48]. As noted in prior work, this maximum likelihood objective is not aligned with the RL objective: models that achieve higher likelihood do not necessarily produce better policies [17, 29, 30, 45, 52]. This issue is referred to as the objective mismatch problem: the model and policy (or planner) are optimized using different objectives. This problem arises in almost all model-based RL approaches. including those that train the model to predict the value function [34, 41] or that perform planning [10, 41].
Some prior work addresses this problem by decreasing the rewards states where the model is inaccurate [27, 30, 42, 49, 51]. a strategy that our method will also employ through discriminator. While some of these methods also use discriminators in this manner, ours the first optimize a lower bound on returns. Other work modifies the model objective to include multi-step rollouts [3, 4, 17, 25, 45, 47J. Our method will modify the model objective in a different way, so that the model objective is exactly the same as the policy objective. Some prior work directly optimizes the model to produce good policies [2, 13, 32, 35, 43], as theoretically analyzed in Grimm et al. [20]. While our aim is the same as these prior methods, our approach will not require differentiating through unrolled model updates or optimization procedures.
Our work builds on prior lower bounds for model-based RL. Kearns and Singh [26] pro- vide lower bound that holds globally. but that is only com- putable in tabular settings. Luo lower bound that can be efficiently es- timated, but which only holds for nearby policies and models. As shown in Table 1, our bound combines the strengths of these prior works, providing lower bound that holds globally and can be efficiently estimated in MDPs with continuous states and actions. Unlike prior work, our bound also mends the objective mismatch problem.
Table 1: Lower bounds model-based RL.
Luo et al. [30]
Kearns and Singh [26]
MnM
MnM (Eq. 10)
(Eq. 2)
holds globally efficient to compute J
X
et al. [30] provide
unified objective
X
X
tight at optimality
X
X
Discrepancy mensure is no0-22ro in stochastic envirooments.

---

Our theoretical derivation builds on prior work that casts model-based RL as two-player game between a model-player and a policy-player [5, 33, 36, 38]. Whereas prior work pits model and policy against one another, our formulation will result in a cooperative game: the model and policy cooperate to optimize the same objective (a lower bound on the expected return). Our approach, though structurally resembling a GAN, is different from prior work that replaces a maximum likelihood model with a GAN model [6, 8, 28].
The most similar prior work is VMBPO [9), which also jointly optimizes the model and the policy using the same objective. However, while our objective is lower bound on expected return, VMBPO maximizes different, risk- seeking objective, which is an upper bound on expected return (see Appendix B. This different objective can be expressed as the expected return plus the variance of the return, so VMBPO has the undesirable property of preferring policies that receive slight lower return if the variance of the return much larger (see Appendix B. Indeed, while most of the components of our method (e.g., classifiers, GAN-like models) have been used in prior work, our paper the first to provide a precise recipe for combining these components into an objective that provable lower bound.
3 A Unified Objective for Model-Based RL
Notation. We focus on the Markov decision process with states 8t, actions at, initial state distri- bution (s()s) positive reward function r(st, a+) > 0, and dynamics p(s++1 (s)) Our aim is to
learn a control policy ae(at 8e) with parameters that maximizes the expected discounted return:
Ex'r(sr.ai)
max ESso
(I)
We use transitions (St, de; Tt, St+1) collected from the (real) environment to train the dynamics model (s St, at), and use transitions sampled from this learned model to train the policy. To simplify notation, we will define a trajectory T _ (sp: ag, $1, a1 ...) as a sequence of states and actions visited in an episode. We define R(T) & Eno (rss as) as the discounted return of a trajectory. We define two distributions over trajectories: p" (T) and (r) for when policy my interacts with dynamics p(3141
St, 0z) and qa ( st, at), respectively:
(r) (Po())) II t=
p(s441 sc, ao)oo st), (²()) Po((s)) II go(s11 st, ae)oo((a st). ==
Desiderata. Our aim is to design an objective L(0) with properties. First, this objective should be a lower bound on the expected return in the true environment: if the policy does well the learned model, are guaranteed that the policy will also do well in the true environment, The expected return under the learned model, which most prior model-based RL methods use to train the policy, is not a lower bound on the expected return [26, 30J. While prior work has made strides in developing lower bounds for model-base based RL, even the best lower bounds do not hold for all models [30] or are limited tabular settings [26].
Second, this objective should be the same for the policy and the dynamics model, such that updates to the model would improve the policy, and vice versa. This desiderata is important because prior work has found that training the model to be more accurate (increase likelihood) can decrease the policy's 's expected return under that model [17, 29, 30, 45, 52]. One prior method (VMBPO [9]) does train the model and policy with the same objective, but this objective corresponds to a risk-seeking, upper-bound on the expected returns.
An objective for model-basee RL. We now introduce an objective that achieves these aims. The key idea to deriving this result is to take a probabilistic perspective on decision making: we view the trajectory as an unobserved random variable, and the reward function as the probability that a trajectory solves task. Then, the problem of inferring this random variable is equivalent to learning the dynamics model. Using an evidence lower bound, we obtain an objective for jointly training the model and policy. We provide the full derivation in Appendix B.3.
Our resulting objective is the policy's reward when interacting with the learned model, but using a new reward function. The new reward function combines the task reward with term that measures
3

---

the difference between the learned model and the real environment. We define our objective
((0) A (T))
Σ 7 ae s41))
(2)
where the modified reward function is defined
F(st, az; S141) (1 7) log (st,, a.) + log ((( | (8,
(((441 st, - (1 7)) log(1 -)).
(3)
Intuitively, the new reward function F penalizes the policy for taking transitions that are unlikely under the true dynamics model, similar to prior work [15, 46, 49]. Later, we will show that we can estimate this augmented reward without knowing the true environment dynamics by using classifier.
We will optimize this lower bound with respect both the policy Ty (a: s;) and the dynamics model go (8 8t, at). For the policy, we maximize the modified reward using samples from the learned model; the only difference from prior work is the modification to the reward function. Training the dynamics model using this objective is very different from standard maximum likelihood training. The model is optimized to sample trajectories that are similar to real dynamics (like a GAN) and that have high reward (unlike GAN). This objective differs from VMBPO [9] by taking the log(-) of the original reward functions; our experiments demonstrate that excluding this component invalidates our lower bound and results in learning suboptimal policies (Fig. 3a).
Our objective achieves the two desiderata. First, our objective a lower bound on the expected return. To state this result formally, we will take the logarithm of the expected return. Of course, maximizing the log(-) of the expected return is equivalent to maximizing the expected return. Theorem 3.1. The following bound holds for any dynamics ( 8t, 8t,02) and policy x xa st):
log E
Σ t=
('r(st, an) 2 C()).
The proof is presented in Appendix B.3. To the best of our knowledge, this the first global (unlike Luo et al. [30]) and computtaeee (unlike Kearns and Singh [26]) lower bound for model-based RL. The model and the policy are trained using the same objective: updating the model not only increases the objective for the model, but also increases the objective for the policy.
Sec. 4 will introduce an algorithm maximize this lower bound. While this lower bound may not be tight, experiments in Sec. 5 demonstrate that optimizing this lower bound yields policies that achieve high reward across wide range of tasks. In Appendix A, we propose variant of this bound that does become tight at convergence. Because this tight bound is more complex, we focus on the simple bound in this paper.
The optimal dynamics are optimistic. We now return to analyzing the simpler lower bound (C(0) in Eq. 2). In stochastic environments, the dynamics model that optimizes this lower bound is not equal to the true environment dynamics. Rather, it is biased towards sampling trajectories with high return. Ignoring parametrization constraints, the dynamies model that optimizes our lower R(r) bound is (T) = J R(")) (proof in Appendix B.4.). While it may seem surprising that the cojective-optimiziin dynamics would differ from the true dynamics, this result is analogous to VAE, where the BoBt-optigiziig encoder differs from the prior. The optimism in the dynamics model may accelerate policy optimization, hypothesis we will test in Sec. 5.1.
Would the optimistic dynamics overestimate the policy's return, violating Theorem 3.1? While using the optimistic dynamics with the original reward function will overestimate the true return, using the optimistic dynamics with the augmented reward function yields a valid lower bound. We demonstrate this effect in Fig. 3b.
4 Practical Optimization of the Lower Bound
The previous section presented a single (global) lower bound (C from Eq. 2) for jointly optimizing the policy and the dynamics model. In this section, we develop practical algorithm for optimizing

---

this lower bound. The main challenge in optimizing this bound is that the augmented reward function depends on the transition probabilities of the real environment, p(st+1 St, az), which are unknown. We address this challenge by learning classifier (Sec. 4.1). Of course, Theorem 3.1 only holds if the classifier Bayes-optimal. We then describe the precise update rules for the policy, dynamics model, and classifier (Sec. 4.2).
4.1 Estimating the Augmented Reward Function
To estimate the augmented reward function. which depends on the transition probabilities of the real environment, we learn a classifier that distinguishes real transitions from fake transi- tions. This approach is similar to GANs [19] and similar to prior work in RL [15, 49]. We use Co(st, at, St-1) E [0, 1] to denote the classifier, which we train to distinguish real versus model transitions using the standard cross entropy loss:
max Lo real real model, o) - loog (s( at s0 + log (1 - (st of model $141
Note that the real transition (sTaa, aft s(+1 and model transition (s!aal af model) have the same initial state and initial action. Once trained, use the classifier's predictions estimate the augmented reward function:
Oe: De: Cp(Se, ae (1,1),,
F(st, ais 8641) RJ log (oggrss., a.) log
(5)
The approximation above reflects function approximation error in learning the classifier. Now that estimate the augmented reward function, we can apply any RL method to maximize the augmented reward under transitions sampled from the dynamics model. The following section describes a particular instantiation using off- off-policy RL algorithm.
4.2 Updating the Model and Policy
Classifee
We now present our complete method, which trains three components: classifier, a policy, and a dynamics model. Our method alternates between (I) updating the policy (by perform- ing RL using model experience with augmented rewards) and (2) updating the dynamics model and classifier (using a GAN-like objective). In describing the loss functions below, superscripts (-)cca) and (-)medal to denote tran- sitions that have been sampled from the true environment dynamics or the learned dynamics function. To reduce clutter, we omit the super- scripts when unambiguous.
pollcy
dynamics model
Learnid
policy
dynamics model
Learned
Learne
dynamics model
Figure 1: Mismatched No More is a model-based RL algorithm that learns policy, dynamics model, and classifier. The classifier distinguishes real transitions from model transitions. The policy and dynamics are jointly optimized to sample transitions that yield high returns and look realistic, as estimated by the classifier.
the
Updating the policy. The policy is optimized maximize the augmented reward on transitions sampled from the learned dynamics model. While this optimization can be done using any RL algorithm, including on on-policy methods, we will focus on an off off-policy actor-critic method.
We define the Q function as sum of augmented rewards under the learned dynamics model:
Q(81, a1) & E
<(aelat).
Σ od "(sss av)
(6)
We approximate the Q function using a neural network Q (st, ar) with parameters o. We train the Q function using the TD loss on transitions sampled from the learned dynamics model:
Lo (ss; ae Th, model +b) = (Qu a1) - 2
(7)
where L-Jag is the stop-gradient operator and ye = F(st, ax smodee ) alsgntt)) [Q a.+1)] is the TD target. The augmented reward F is estimated using the learned classifier (Eq. 5). To estimate
5

---

Algorithm Mismatched no More (MnM) an algorithm model-based RI.. The method alternates between training the policy experience from the learned dynamics model with augmented rewards and updating the model+classifier using GAN-like loss. While we use an off-policy RL. algorithm on LA, any other RL algorithm can be substituted.
1: while not converged do
2: Sample experience from the learned model.
3: Modity rewards using the classifier (Eq. 5).
4: Update the policy and Q function using the model experience and modified rewards (Eq.s 8 and 7).
5: Update model and classifier using GA losses (Eq.s 4 and 9).
6: (Infrequently) Sample experience from the real model.
7:
return policy (at st).
the corresponding value function, we use a I -sammll approximation: (8t))) = Qu(se, az) where
ag (a st)). The policy is trained to maximize the Q function:
max Lr (#6; (#t; 0) _ (xg(( (st ac)]
(8)
In implementation, we regularize the policy by adding an additional entropy regularizer. Following prior work [18), we maintain two Q functions and two target Q functions, using the minimum of the two to compute the TD target. See Appendix D for details.
Updating the dynamics model. To optimize the dynamics model, we rewrite the lower bound in terms of a single transition (derivation in Appendix B.6):
Calstal grcal gmodel Co ar smpple)
L (s) of 0) E co .ati))
feal real
Vo(o model) + log
This loss an approximation of our original lower bound (Eq. 2) because we estimate the difference in dynamics using classifier. This approximation is standard in prior work on GANs [19] and adversarial inference [12, 14].
Intuitively, the procedure for optimizing the dynamics model and the classifier resembles a GAN [19]: the classifier is optimized distinguish real transitions from model transitions, and the model is updated to fool the classifier (and increase rewards). However, our method is not equivalent to simply replacing maximum likelihood model with GAN model. Indeed, such approach would not optimize a lower bound on expected return. Rather, our model objective includes an additional value term and our policy objective includes additional classifier term. These changes enable the model and policy to optimize the same objective, which is a lower bound on expected return.
Algorithm summary. We summarize the method in Alg. and provide illustration in Fig. 1. We call the method MISMATCHED NO MORE (MnM) because the policy and model optimize the same objective, thereby resolving the objective mismatch problem noted in prior work. While the model and policy are optimized using the same objective, that objective can stop being a lower bound the learned classifier not Bayes-optimal.
Implementing MnM on top of standard model-based RL algorithm is straightforward. First, create additional classifier network. Second, instead of using the maximum likelihood objective train the model, use the GAN-like objective in Eq. 9 to update both the model and the classifier. Third, add the classifier's logits to the predicted rewards (Eq.. 5). Following prior work [24], we learn neural network to predict the true environment rewards.
4.3
A Note about Exploration
The classifier term in the augmented reward (Eq. 3) is double edged sword. Our theoretical derivation suggests that this term should appear, and our didactic experiments demonstrate that removing this term results in suboptimal behavior. Experiments also show that this term can effectively combat errors in the learned model. Prior work in offline RL has found similar model-error reward terms critical for achieving good performance in the offline setting [27, 42, 49, 51].
However, when scaling MnM to continuous control tasks in the online setting, we found that including this term hurts performance (Fig. 10). This makes sense: this classifier term is exactly the opposite

---

2.2
Mart
tours)
VMBPO
Deturn
modified rewards tound)
0.75
orgina rewarts aigmented "ward
2.3
5.0
0
200
435
500
500
1903
1000
1420
iperations
(a) Stochastic Gridworld
(b) Inaccurate Models
Figure 2: Gridworld experiments. (Left) We apply MrM to a navigation task with transition noise that moves the agent to neighboring states with equal probability. MrM solves this task more quickly than Q-learning and VMBPO. The dynamics learned by MnM are different from the real dynamics, changing the transition noise (blue arrows) point towards the goal. (Right) We simulate function approximation by learning model that makes same predictions for groups
3 states, resulting a model that inaccurate around obstacles. The modified reward compensates for these errors by penalizing the policy for navigating near obstacles.
of exploration objectives based on model error (e.g., [44]), and penalizes the policy for performing exploration. Our continuous control experiments in Sec. 5.2 will deviate from our theoretical derivation: they will use the model objective suggested by our theory, but the same policy objective as prior. That is, the policy maximizes r(5t; 0+) instead of F(s,, at) (Eq. 3). We call this method "MmM-approx." We note that many theoretical model-based RL papers also find it necessary to implement practical algorithms that differ from the theory [30, 46, 49].
5 Numerical Simulations
The primary aim of our experiments is to verify that our objective is a valid lower bound, and that maximizing the bound yields maximizing policies. We study these questions in Sec. 5.1, where we will use tabular problems so that we can study the objective in the absence of function approximation error. Then, in Sec. 5.2, we show adapting part of our objective into scalable model- based RL algorithm yields method that, while deviating from the theory, can achieve competitive results on benchmark tasks.
5.1 Understanding the Lower Bound and the Learned Dynamics
Our didactic experiments test whether optimizing the lower bound produces optimal policies and study how the components of the lower bound. We use tabular domains in this section, as they allow us to analytically compute the optimal policy for comparison, and allow us to evaluate the lower bound in the absence of function approximation error.
Our first experiment compares the lower bound to an oracle method that applies Q-learning to a perfect dynamics model. In comparison to this baseline, MnM learns dynamics model using the GAN-like objective (Eq. 9) and maximizes a modified reward function (Eq. 5). We also compare to VMBPO, which learns a dynamics model similar to MnM but omits the log-transformation of the reward function; this transformation ordinarily encourages pessimistic behavior. For this experiment, 10 x 10 gridworld with stochastic dynamics and sparse reward, shown in Fig. 2a (Left). The results, shown in Fig. 2a, show that MnM outperforms both Q-learning with the correct model and VMBPO this task. We hypothesize that VMBPO performs poorly this task because maximizes an upper bound on performance; and confirm this in the following experiments.
We hypothesize that MnM outperforms Q-learning with the correct model because it learns "optimistic" dynamics. We test this hypothesis by visualizing the dynamics model learned by MnM (Fig. 2a (Left)). While the true environment dynamics have stochasticity that moves the agent in random direction with equal probability, the MnM dynamics model biases this stochasticity to lead the agent towards the goal (blue arrows point towards the goal). Of course, use the true environment dynamics, not the optimistic dynamics model, for evaluating the policies.
Our augmented reward function contains two crucial components, (1) the classifier term and (2) the logarithmic transformation of the reward function. We test the importance of the classifier term in correcting for inaccurate models. To do this, we limit the capacity of the MnM dynamics model so that it makes "low-resolution" predictions, forcing all states in 3 x 3 blocks have the same dynamics. We will use the gridworld shown in Fig. 2b (Left), which contains obstacles that occur

---

Hloi VMSPO ofjective
24-
Overestimate
Return
400
Underestimate
Hogi Expected Retur
Miskt objective
+50
+1 +51
200
10 heretions
Q-leaming
MnM
VMBРO
(ours)
(a) Testing for risk secking behavior
(b) Comparing objectives.
Figure 3: Analyzing the lower bound. (Left) On a 3-state MDP with stochastic transitions in one state (red arrows), MnM learns the reward- maximmiiii policy while VMBPO learns strategy with lower rewards and higher variance (as predicted by theory). (Right) We apply value iteration the gridworld from Fig. 2a analytically compule various objectives. As predicted by our theory, the MrM objective a lower bound on the expected return, whereas the VMBPO objective overestimates the expected return.
at a finer resolution than the model can detect. When the "low resolution" dynamics model makes predictions for states near the obstacle, it will average together some states with obstacles and some states without obstacles. Thus, the model will (incorrectly) predict that the agent always has some probability of moving in each direction, However, the classifier (whose capacity we have also limited) detects that the dynamics model is inaceurate in these states, so the augmented reward is much lower at these states. Thus, MnM is able to solve this task despite the inaccurate model; an ablation of MnM that removes the classifier term attempts navigate through the wall and fails to reach the goal.
if that direction is actually blocked by obstacle.
Like MnM, VMBPO includes classifier term in the reward function but omits the logarithmic transformation, difference expect to cause VMBPO to prefer suboptimal, risk-secking policies. To test this hypothesis, we use the 3-state MDP in Fig. 3a (Right), where numbers indicate the reward at each state. While moving to the right state yields slightly higher rewards, "wind" knocks the agent out of this state with probability 50% so the reward-mant maaaddmmiiiizzngggg strategy is to move to the left state. While MnM learns the reward maxammiiiiiiinggggg. strategy, VMBPO learns a policy that goes to the right state and receives lower returns.
Finally, we verify Theorem 3.1 by comparing the MnM objective to the true expected return. We also compare to the objective from VMBPO, which looks similar to the MnM objective but omits the logarithmic transformation; our theory predicts that the VMBPO objective will therefore be an upper bound on the expected return (see Appendix B.I). We use the gridworld from Fig. 2a and use a version of MnM based value iteration to avoid approximation error. Plotting the MnM objective in Fig. 3b (Right), we observe that it always lower bound on the (log) expected return, as predicted by our theory. Also predicted by the theory, the VMBPO objective overestimates the expected return, illustrating the importance of the logarithmic transformation.
Of the oft-cited benefits of model-based RL is that the learned dynamics model can be re-used to solve new tasks. MnM presents a twist on that story, because MnM does not learn the true environment dynamics but rather learns optimistic dynamics (see Sec. 3). In Appendix C (Fig. 13), we examine MnM's effectiveness at transferring dynamics different tasks in the stochastic gridworld from Fig. 2a. We find that the (optimistic) dynamics learned by MnM do not slow learning on dissimilar tasks, but can accelerate learning of challenging, similar tasks.
5.2 Comparisons On Higher- Dimeesioamn Tasks
Our next experiments use continuous-contrr robotic tasks to study whether the new model-learning objective suggested by our lower bound can be stably applied to higher-dimennional control tasks. While the MnM-approx method tested in this section deviates from our theoretical derivation (see Sec. 4.3), these experiments nonetheless useful for providing preliminary evidence about whether the proposed model objective can be efficiently estimated and stably optimized.
We use MBPO [24] as a baseline for mosel-based RL because it achieves state-of-the results and is prototypical example of model-based RL algorithms that use maximum likelihood models. Because our method differs from the baseline (MBPO) along only one dimension now (new model update, same policy update), these experiments will directly test the utility of our proposed model update. Experimental details are in Appendix D.
8

---

RoN tours)
MnM lount HellChrettdhor
MBPO
Tappere2
VHBPO
VaGraM
PooeRandom-o do
Ranooooo
Serewmoed v0
FrrewRandom- 40
(r)
0.0
2V0r300 nitur
overage return
-10
0.5
0.3
0.5
0.2
cnvironment stops
105
environimennt steps
(a) OpenAl Gym benchmark
(b) ROBEL manipulation benchmark
Figure Comparison on two benchmarks. (Left) On the OpenAl gym benchmark [7], MnM-approx performs on par with prior state-of-the-art method (MBPO [24]). while consistently outperforming a recent
method that addresses objective mismatch (VMBPO [9]). (Right) The ROBEL manipulation benchmark [1] contains complex contact dynamics that challenging model. MBPO performs poorly these tasks, often worse than model-free fedel-free SAC [22).
MnM Dynamics:
Real Dynamics:
Figure 5: Optimistic Dynamies: (Left) On the Pusher-v2 task, the MnM dynamics model makes the puck move towards the puck move towards the gripper before being grasped. (Right) On the HalfCheetah- v2 task, the MnM dynamics model helps agent stay upright after tripping.
We first use three locomotion tasks from the OpenAl Gym benchmark [7] to compare MnM-approx MBPO and VMBPO. The VMBPO curves taken directly from that paper. As shown in Fig. 4a, MnM-approx performs roughly on par with MBPO and outperforms [9], more recent model-based method also addresses the objective mismatch problem by maximizing an upper bound on expected returns (I-sided p-values: p = 0.04, 0.00, 0.03).
We next the ROBEL manipulation benchmark [1] to compare how MnM-approx and MBPO handle tasks with more complicated dynamics. As shown in Fig. 4b, MBPO struggles to learn three of the four tasks, likely because the dynamics hard. In contrast, MnM-approx solves these tasks, likely because the GAN-like model is more accurate. MnM-approx outperforms MBPO on tasks (p 0.03) and outperforms the model-free baseline 2/4 tasks S 0.02)
We next compare to prior methods for addressing the objective mismatch problem using the DC1awScre=Fixedwo task. We compare VAML [17] and the welue-weighted maximum likeli- hood approach proposed in Lambert et al. [29]. As shown in Fig. 6, these alternative model learning objectives perform poorly on this task.
To better understand why MnM-approx
sometimes outperforms the maximum like- lihood baseline (MBPO), we visualized the Q-values throughout training. We used -btaworlddddrave--opn------
AverageRetur
4000
Mntd-aprmee
MEPO
a task where
VAML
Value-Weighted
we found noticeable difference in the per- formance between MBPO and MnM-approx. Fig. 7a shows that MnM-approx yields Q values that are more accurate and more stable than MBPO, perhaps because MBPO learns policy that exploits inaccuracies in the learned model.
0.25
0.50
1.00
1.25
Env steps
Figure 6: Alternative model learning objectives: Us- ing the DC1 auScresFixed approx and MBPO [24] to two additional model lcarning objectives suggested in the literature, VAML [17] and value-weighted maximum likelihood [29]. MnM-approx outperforms these alternative approaches.
task, we compare MnM-
To study the stability of MnM-approx, we plot the validation MSE of the MnM-approx model when training on the DClauScrewRandom- task, As shown in Fig. 7b, the MSE decreases stably, indicating that the adversarial nature of the MnM-approx model training does not create instabilities. In Appendix C (Fig. 8, 9) we visualize Q

---

Validation Model MSE
0.033
values
200
MBPO
0035
100
Mopoappprxx
0.005
0.000
0.0
3.5
24
235
0.0
0.4
0.6
1.0
Training steps
gradient updates
107
(a) Model exploitation
(b) MnM-approx trains stably.
Figure 7: Analyzing MmM-approx. (Left) The very large values of MBPO suggest model exploitation. which our method appears to avoid. (Right) Despite resembling GAN, MnM-approx dynamics model trains stably. Different colors correspond to different random seeds, and the dashed line corresponds to the minimum validation MSE of an MLE dynamics model.
value stability and MSE model error on additional environments. In Appendix C (Fig. 12), we study the effect of model horizon on Mn. in continuous control task, and find that performance degrades linearly as we increase this value.
We visualize the dynamics learned by MnM on two robotic control tasks in Fig 5. These tasks have deterministic dynamics, so our theory would predict that am idcalized version of MnM would learn a dynamics model exactly equal to the deterministic dynamics. However, our implementation relies function approximation (neural networks) to learn the dynamics, and the limited capacity of function approximators makes dtherwis-ddetrminnis
dynamics appear stochastic. On the Pusher-v2 task, the MnM dynamics cause the puck to move towards the robot arm even before the arm has come in contact with the puck. While this movement not physically realistic, it may make the exploration problem easier. On the HalfCheetah--z task, the MnM dynamics increase the probability that the agent remains upright after tripping, likely making it easier for the agent learn how run. We expect that the implicit stochasticity caused by function approximation to be especially important for real-world tasks, where the complexity of the real dynamics often dwarfs the capacity of the learned dynamics model.
Conclusion
The main contribution of this paper is an objective for model-based RL that is both efficient to compute and globally valid. Moreover, the model and policy are jointly optimized using this same objective, thereby addressing the objective mismatch problem. This joint optimization will ease and accelerate the design of future model-based RL algorithms.
The main limitation of this paper is the classifier term. Because this term is estimated, the objective used in practice can fail to be a lower bound when the classifier is not Bayes-optimal. Additionally. this classifier term hinder exploration and degrade performance online setting. We encourage future investigations into joint optimization based approaches that do explicitly address the role of exploration, perhaps including the dataset as additional optimization variable.
continuous control tasks in the
Acknowledgements. We thank Laura Smith, Marvin Zhang, and Dibya Ghosh for helpful discussions, and thank Anusha Nagabandi and Michael Janner for help with experiments. We thank Yinlam Chow for help sharing the VMBPO baselines and thank Danijar Hafner for reviewing a draft of the paper. This material is supported by the Fannie and John Hertz Foundation and the NSF GRFP (DGE1745016).
References
[1] Ahn, M., Zhu, H., Hartikainen, K.. Ponte, H., Gupta, A., Levine, S., and Kumar, V. (2020). ROBEL: Robotics benchmarks for learning with low-cost robots. In Conference on Robot Learning, pages 1300- 1313. PMLR.
[2] Amos, B., Rodriguez, I., Sacks, J., Boots, B., and Kolter, Z. (2018). Differentiable MPC for end-to-end planning and control. Advances in neural information processing systems.
10

---

[3] Asadi, K., Misra, D., Kim, S., and Littman, M. L. (2019). Combating the compounding-emor problem with multi-step model. arXiv preprint 1xXiv:1905.13300
[4] Asadi, K., Misra, D., and Littman, M. (2018). Lipschitz continuity in model- reinforcement learning. In International Conference on Machine Learning, pages 264 274-273. PMLR.
[5] Bagnell, J. A., Ng. A. Y., and Schneider, J. G. (2001). Solving uncertain Markov decision processes.
[6] Bai, X., Guan, J., and Wang, H. (2019), model-based reinforcement learning with adversarial training for online recommendation. Advances in Neural Information Processing Systems, 32:10735 10746.
[7] Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAl gym. arXiv preprint 1606.01144
[8] Chen, S., Jiang, J., Zhang, X., Wu, J., and Lu, G. (2020). Gan-based planning model in deep reinforcement learning. In International Conference on Artificial Neural Networks, pages 323-334. Springer.
[9] Chow, Y., Cui, B., Ryu, M., and Ghavamzadeh, M. (2020). Variational model- policy optimization. arXiv preprint 2tte:22000055444
[10] Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, pages 4754 4765.
[11] Deisenroth, M. and Rasmussen, C. E. (2011). PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning (ICML), pages 465-472.
[12] Donahue, J., Krähenbühl, P., and Darrell, T. (2016). Adversarial feature learning. arXiv preprint arXiv:1605.0972 09782.
[13] D'Oro, P.. Metelli, A. M., Tirinzoni, A., Papini, M., and Restelli, M. (2020). Gradient-aware model-based policy search. In Proceedings the AAAI Conference Artificial Intelligence, volume 34, pages 3801-3808.
[14] Dumoulin, V., Belghazi, Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., and Courville, A. (2016). Adversarially learned inference. arXiv preprint arXiv:1606.0000 00704.
[15] Eysenbach, B., Chaudhari, S., Asawa, S., Levine, S., and Salakhutdinov, R. (2020a). Off-dynamics reinforcement learning: Training for transfer with domain classificrs. In International Conference on Learning Representations.
[16] Eysenbach, B., Salakhutdinov, R., and Levine, S. (2020b). C-learning: Learning to achieve goals via recursive classification. In International Conference Learning Representations.
[17] Farahmand, A.m., Barreto, A., and Nikovski, D. (2017). Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statisticx, pages 1486-1494.
[18] Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing Function Approximation Error in Actor- Critic Methods. International Conference on Machine Learning (ICML).
[19] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. In Advances in Neural Information Processing Systems (NIPS).
[20] Grimm, C., Barreto, A., Singh, S., and Silver, D. (2020). The value equivalence principle for model-based reinforcement learning. arXiv preprint arXiv:2011.05: 1.03566.
[21] Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E.,, Fishman, S., Wang, K., Gonina, E., Wu, N., Kokiopoulou, E., Sbaiz, L., Smith, J.. Bartók, G., Berent, J., Harris, C., Vanhoucke, V., and Brevdo, E. (2018). TF-Agents: library reinforcement learning tensorflow. [Online; accessed 25-June-2019].
[22] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In International Conference on Machine Learning.
[23] Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations.
[24] Janner, M., Fu, J., Zhang, M., and Levine, S. (2019). When to trust your model: Model-based policy optimization. In Advances Neural Information Processing Systems, pages
11

---

[25] Joseph, J., Geramifard. A., Roberts, J. W. How. J. P., and Roy, N. (2013). Reinforcement learning with misspecified model classes. In 2013 IEEE International Conference Robotics and Automation, pages 939-946. IEEE.
[26] Kearns, M. and Singh, S. (2002). Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2-3):209-23..
[27] Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. (2020). Morel reinforcement learning. ArXiv, 0os/2005.05555/
Model-based offline
[28] Kurutach, T., Tamar, A., Yang, G., Russell, S. J., and Abbeel, P. (2018). Learning plannable representations with causal InfoGAN. In Advances in Neural Information Processing Systems, pages 8733-8744.
[29] Lambert, N., Amos, B.. Yadan, O., and Calandra, R. (2020). Objective mismatch in model-basee reinforcement learning. In Learning for Dynamics and Control, pages 761-770. PMLR.
[30] Luo. Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. (2019), Algorithmic framework for model-bas based deep reinforcement learning with theoretical guarantees. In ICLR (Poster).
[31] Mihatsch, O. and Neuncier, R. (2002). Risk-sensitive reinforcement learning. Machine learning, 49(2):667- 290.
[32] Nikishin, E., Abachi, R., Agarwal, R., and Bacon, P.-L. (2021). Control-oriented model--ased reinforce- ment learning with implicit differentiation. arXiv preprint arXiv:2106.00 03273.
[33] Nilim, A. and EI Ghaoui, L. (2003). Robustness in Markov decision problems with uncertain transition matrices. In NIPS, pages 839-889-8888 846. Citeseer.
[34] Oh, J., Singh, S., and Lee, H. (2017). Value prediction network. arXiv preprint 1rx.:
[35] Okada, M., Rigazio, I.., and Aoshima, T. (2017). Path integral networks: End-to-end differentialle optimal control. arXiv preprint 1706
[36] Rajeswaran, A., Mordatch, and Kumar, V. (2020). A game theoretic framework for model based reinforcement learning. International Conference on Machine Learning, pages 7953-7963. PMLR.
[37] Ross, S. and Bagnell, D. (2012). Agnostic system identification for model-based reinforcement learning. In ICML.
[38] Ross, S., Gordon, G. J., and Bagnell, J. A. (2011). A Reduction of Imitation Learning and Structured Prediction No-Regret Online Learning. International Conference on Artificial Intelligence and Statistics (AISTATS).
[39] Rudner, T. G., Pong, V. H., McAllister, R., Gal, Y., and Levine, (2021). Outcome-driven reinforcement learning via variational inference. arXiv preprint arXiv:2104.1011 10190.
[40] Salimans. T., Goodfellow, 1., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training GANs. Advances in neural information processing systems, 29:2234-2242.
[41] Schritwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, I.,, Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., al. (2020). Mastering atari, go, chess and shogi hy planning with learned model. Nature, 588(7889))6004-1)
[42] Sorg. J., Singh, P., and Lewis, L. (2010). Internal rewards mitigate agent boundedness. In ICML.
[43] Srinivas, A.. Jabri, A.. Abbeel, P., Levine. S., and Finn, C. (2018). Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference Machine Learning. pages 4732-4741, PMLR.
[44] Stadie, B. C., Levine, S., and Ahheel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 157.000507008//4
[45] Talvitie, F. (2014). Model regularization for stable sample rollouts. In UAI, pages 780-789.
[46] Vemula, A., Oza, Y., Bagnell, J. A., and Likhachev, M. (2020). Planning and execution using inaccurate models with provable guarantees. arXiv preprint 2003::000030439.
[47] Venkatraman, A., Capobianco, R., Pinto, L., Hebert, M., Nardi, D., and Bagnell, J. A. (2016). Improved learning of dynamics models for control. In International Symposium on Experimental Robotics, pages 703-713. Springer.
12

---

[48] Williams, G., Wagener. N., Goldfain, B., Drews, P. Rehg. J. M., Boots, B., and Theodorou, E. A. (2017). Information Theoretic MPC for Model-Based Reinforcement Learning. In International Conference Robotics and Automation (ICRA).
[49] Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. (2021). Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363
[S0] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C.,, and Levine, S. (2020a). Meta-World benchmark evaluation for multi task and meta reinforcement leaming. In Conference on Robot Learning. pages 1094 1100. PMLR.
[51] Yu, T., Thomas,
L., Ermon. Zou, J., Levine, S., Finn, C., and Ma. T. (2020b). MOPO: Model-based offline policy optimization. arXiv preprint arXiv:2005. 133.9
[52] Ziebart, D. (2010). Modeling purposeful adaptive behavior with principle of maximum causal entropy.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Our main claim is that our proposed objective is global lower bound on the expected return (proof in Appendix B).
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] While RL algorithms might be applied to applications with positive and negative societal implications, our theoretical analysis does not dictate these applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] Experimental details are included in Appendix D. Code will be released upon publications.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Plots show the mean and standard deviation across the random sceds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No] Proprietary.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "qgwuYnxQO7C", "reviewer": "Reviewer_27GT", "summary": "This paper aims to solve objective mismatch in model-based RL which claims that the objective of the model learning is different from the objective of policy learning. The authors propose Mismatch no More (MnM) which optimizes the lower bound of the expected return to jointly training the dynamics model and the policy. To optimize this lower bound practically, they introduce a modified reward function which is estimated by a trained GAN-like classifier. Experimental results show that MnM can achieve \u0010performance competitive with prior model-based methods, and better performance on certain hard exploration tasks.", "questions": "See Weaknesses above.", "limitations": "See Weaknesses above.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. To my knowledge, the idea is novel, and the idea of training a classifier to distinguish the real transitions from the fake transitions is very interesting.\n2. The paper is well-written and easy to follow.\n\nWeaknesses:\n\n1. My main concern is about the experiments. This paper aims to solve the objective mismatch in MBRL and it is very related to Value-Awareness Model-based RL. However, they don't compare with the SOTA Value-Awareness Model-based RL methods such as VaGarm [1]  in both OpenAI Gym and ROBEL evaluations. In ROBEL environment, they even don't compare with VMBPO.\n\n2. Besides, an important part of dyna-style model-based RL is that the learned dynamics model can perform multi-step rollout for policy learning to improve sample efficiency. I want to know whether the learned dynamics model of MnM can perform multi-step rollout, because this model is completely dependent on the value function, and has not learned the knowledge related to transition. How to control model compounding error in MnM if multi-step rollout is possible?\n\nTypos:\n\nIn Eq.8, does $\\tilde{Q}_\\psi$ mean ${Q}_\\psi$? I didn't find the definition of $\\tilde{Q}_\\psi$.\n\n[1] Voelcker, Claas A., et al. \"Value Gradient weighted Model-Based Reinforcement Learning.\" International Conference on Learning Representations. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PmSHYtVZra8", "reviewer": "Reviewer_Y1gL", "summary": "Model-Based Reinforcement Learning (MBRL) is usually treated as a supervised learning problem where we first fit the data from previously observed data by reducing MSE and then do planning using this learned model. This paper highlights that this way of learning the model for planning leads to the objective mismatch problem: The model training objective is disconnected from the policy optimization process. While highlighting the problems caused by this objective mismatch, the authors propose a workaround by a unified objective for both model and policy learning. More specifically by viewing model learning as a latent variable problem, the author leverage theory from VI and propose a novel lower bound (similar to ELBO) which they maximize during the learning process. Then the authors propose a practical GAN-style algorithm where the aim is to learn a model which fools the discriminator by generating realistic-looking fake trajectories. The policy network is trained to maximize the log of the reward. Since all these optimizations are done with just a single objective, the model learning and policy learning phases are coherent resulting in learning high-return policies.   The authors support the theory presented in the main text from a series of experiments on simple tasks demonstrating the proposed algorithm's efficacy. ", "questions": "1. Line 163 states “...... overestimate the policy’s return, violating lemma 3.1”. However, I skimmed through the entire paper searching for a formal statement for lemma 3.1 with little success. Could the authors state where is lemma 3.1 in the paper? \n2. Regarding difficulties of including the classifier term while doing policy optimization for continuous control tasks, I see that this is just stated without any experimental evidence. This makes it unclear for readers to understand if this is indeed causing an issue and if yes, in what way it is hurting the exploration. \n3. The authors say that they performed experiments on the OpenAI benchmark while citing the OpenAI gym paper. As far as I know, there are several environments in the OpenAI gym, but the authors seem to perform experiments only on 3 environments which makes it hard to comment on the scope of the algorithm. Could you please clarify a) the motivation behind picking these specific environments and b) the validity of your approach on the entire mujoco suit? \n4.Since all of the continuous control experiments are based on MnM approx algorithm, it would be helpful to have pseudocode highlighting the differences between MnM and MnM approx. \n\n", "limitations": "The limitations are clearly mentioned in the paper and I am happy that the authors are transparent about the validity of their method. While having a single objective to train both policy and model is helpful, the authors do mention it seems to scale poorly with the problem complexity and one must employ some tricks as the authors did in MnM approx method to handle these inefficiencies. This begs the following question: Is single objective training a right way to go about building model based algorithms? If yes, why do you think so? \n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The contributions are clear and the authors did a good job of reiterating their contributions every now and then. This helped in getting an overall big picture\n2. The writing is very clear. \n3. The authors discussed the limitations of their approach clearly which is commendable. \n\nWeaknesses:\n1. Most of the experiments are toy-ish/limited and I think performing more experiments on a wide variety of tasks and benchmarks would be helpful in understanding the efficacy of MnM algorithm. \n2. MnM approx algorithm is not completely clear. \n3. The theory could be stated more formally (More on this in the next section)\n4. No ablation studies \n5. Experiments are performed on limited random seeds (3 seeds) which makes it hard to comment on the validity of the reported learning curves.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OcIVZWTIZ39", "reviewer": "Reviewer_HuRX", "summary": "The reviewed paper is concerned with a fundamental challenge of model-based reinforcement learning, namely that the objective used to train the dynamics model may not align with the final objective of return maximization. To this end, the authors propose a globally valid lower bound on the expected return from which both a dynamics model and policy optimization objective can be derived. Intuitively, the dynamics model is trained to produce transitions which are hardly distinguishable from real transitions, resembling a GAN objective. Additionally, policy and dynamics model training are entangled, with measures of each appearing in the training objectives of the other. The authors show the functioning of their method convincingly on a set of toy-, but also complex high-dimensional manipulation environments.\n", "questions": "Would it be possible to combat the exploration problem by only modifying the reward with the classifier\nterm for experience which comes from the model, and use the original reward for transitions\nobserved on the real system? I.e., train $Q$ and $\\pi$ on both type of transitions, but only\nmodify the reward on the simulated ones? Intuitively, this would make sense to me, as it would give lower\nreward to potentially mismatching transitions which deviate from the real system.\n", "limitations": "One advantage of MBRL, in addition to often being more sample-efficient than purely model-free methods,\nis the possibility to quickly transfer to different tasks (by generating transitions from the model and\ntraining the policy with the changed reward, ideally requiring no additional interactions on the real system).\nIt would be beneficial if the authors could comment on if their proposed method also gives advantages \nin these situations where the model is to be re-used for different tasks.\n\n", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n  * The paper is extremely clearly written, it was very enjoyable to read!\n  * The method is well-grounded in prior work and soundly derived theoretically\n  * The experiments are well-chosen to demonstrate the functioning of the method and presented very clearly\n  * In my opinion, this work is very relevant to the NeurIPS community, providing solid theoretical groundwork for model-based reinforcement learning approaches\n\nWeaknesses:\nI have some **minor** comments on the presentation of the experiments:\n  * In Fig. 4b there seems to be a mismatch between the caption and the legend. Does \"RL on real\" stand for SAC?\n  * In Fig. 6a it is not clear to me what Q-values are shown exactly\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J82qdlAQns", "reviewer": "Reviewer_FWhQ", "summary": "The proposed method provides a strategy based on a standard variational inference framework to perform model-based reinforcement learning by addressing the objective-mismatch issue. The algorithm “Mismatched-no-More” (MnM) jointly optimizes the model and policy with the help of an objective that serves as a “global” lower bound for the expected return. Practically, the approach uses a new augmented reward function containing the error between learned and real transition probabilities simplified with the help of a GAN-like classifier to differentiate between the two. The efficacy of the method is validated on 2 tabular MDPs, 3 locomotion tasks, and 4 manipulation tasks.", "questions": "- Given that “MnM-approx” is need to ensure appropriate exploration, the reviewer is curious to see the results of using MnM-approx in the tabular settings as well. Is training the policy and model with the same objective really needed at all in practice?\n- I appreciate the Manipulation experiments but since the most similar algorithm to MnM is VMBPO, the authors should at least conduct experiments in all the environments shown by the original VMBPO paper. Also, the authors should properly justify the discrepancy between the VMBPO scores shown in the submitted paper and the original one.\n- Is it possible to include VMBPO in Fig. 5 since the authors have included it for the tabular settings? That would complete it.\n- By looking at the HalfCheetah-v2 model prediction results, the reviewer lacks clarity in realizing the advantage of using such a model. While taking an action in the initial state topples the robot in the real world, it helps the robot to move forward in the learned model. Can the authors bring some clarity given that the final evaluation of the policy is on the real environment?\n- It is confusing to see the performance analysis on `DClawScrewFixed-v0`, MSE analysis on `DClawScrewRandom-v0` task and Q-value analysis on `metaworld-drawer-open-v2`. Since, the reviewer can still see a noticeable difference in the performance between MnM-approx and MBPO on `DClawScrewFixed-v0`, something more convincing will be to conduct all the analysis on the same. Moreover, adding the performance of the additional environments as the authors already have it will be the most appreciated.\n", "limitations": "The reviewer acknowledges all the limitations of the approach as mentioned by the authors and also agrees with the same. However, due to a very selective analysis of the comparison performance, the experimental segment is still not convincing enough. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written with several strong contributions:\n\n- The paper clearly distinguishes itself from very similar prior works (like VMBPO) in terms of its theoretical contribution to maximizing the lower bound on expected return. To the best of the reviewer’s knowledge, the approach of using a GAN-like objective for model learning is novel.\n\n- The additional analysis of model performance, model exploitation, and training performance of the MnM model with the MSE objective adds additional support to the claims made in the paper.\n\nHowever, the difference between the proposed theoretical formulation and the experimental implementation makes it difficult to realize the true significance of using MnM. The approach is proposed with the intention of optimizing the same objective for policy and model learning but conducts most of the experiments only with model learning.\n\n- Further, the comparison results are not clear on what advantage we get from MnM. In continuous control experiments, the authors show a clear analysis of the superiority of MnM-approx over other methods only on `DClawScrewRandom-v0` out of the seven environments. \n- On all the OpenAI Gym environments, the improvement in performance over MBPO is not considerable, given that an additional classifier was trained to replace the maximum likelihood objective. \n- Finally, the reviewer could not correspond the VMBPO scores presented in the paper with their published one. Looking at the original values, there seems to be hardly any difference between the performance of VMBPO and MnM which is expected given that they are solving the same objective eventually. (referring: https://www.ijcai.org/proceedings/2021/0316.pdf)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept