A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning
Authors: Ran Wei
Nathan Lambert
Anthony D McDonald
Alfredo Garcia
Roberto Calandra
Venue: Accepted by TMLR ()

Abstract: Model-based Reinforcement Learning (MBRL) aims to make agents more sample-efficient, adaptive, and explainable by learning an explicit model of the environment. While the capabilities of MBRL agents have significantly improved in recent years, how to best learn the model is still an unresolved question. The majority of MBRL algorithms aim at training the model to make accurate predictions about the environment and subsequently using the model to determine the most rewarding actions. However, recent research has shown that model predictive accuracy is often not correlated with action quality, tracing the root cause to the objective mismatch between accurate dynamics model learning and policy optimization of rewards. A number of interrelated solution categories to the objective mismatch problem have emerged as MBRL continues to mature as a research area. In this work, we provide an in-depth survey of these solution categories and propose a taxonomy to foster future research.

Published in Transactions on Machine Learning Research (02/2024)
A Unified View on Solving Objective Mismatch in Model- Based Reinforcement Learning
Ran Wei
rw4220tamu.ck edu
VERSES Research Lab
Nathan Lambert
nathanl@kallenai.cmm
Allen Institute for AI
Anthony McDonald
admedonaldd@viscc.ee
University of Wisconsin-Matien
Alfredo Garcia Texas AUM University
prteddo.gariia@tamm.. edu
Roberto Calandra TU Dresden
roberto. calandra Steennesdennddddnnne de
Reviewed on OpenReview: https: openreview. net/ htr/m? id= tQVZguXhZb
Abstract
Model-based Reinforcement. Learning (MBRL) aims to make agents more sample-efficient, adaptive, and explainable by learning an explicit model of the environment. While the ca- pabilitios of MBRL agents have significantly improved in recent years, how to best learn the model is still an unresolved question. The majority of MBRL algorithms aim at training the model to make accurate predictions about the environment and subsequently using the model to determine the most rewarding actions. However, recent research has shown that model predictive accuracy is often not correlated with action quality, tracing the root cause to the objective mismateh between accurate dynamics model learning and policy optimiza- tion of rewards. A number of interrelated solution categories the objective mismatch problem have emerged MBRL continues to mature as research area. In this work, provide in-depth survey of these solution categories and propose taxonomy to foster future rescarch.
1 Introduction
Reinforcement, learning (RL) has demonstrated itself as promising tool for complex optimization landscapes and the creation of artificial agents by exceeding human performance in games (Mnih et al., 2015; Silver et al., 2016), discovering computer algorithms (Fawzi et al., 2022), managing power plants (Degrave et al., 2022), and mumerous other tasks. The premise of RL is that complex agent behavior and decisions are driven by desire maximize cumulative rewards in dynamic environments (Silver et al., 2021). RL methods focus on learning reward-optinal policy from sequences of state-action-reward tuples. These methods can be broadly classified as model-free RL (MFRL) and model-based RL (MBRL). MFRL methods directly learn the policy from the environment samples, whereas MBRL approaches learn explicit model of the environment and use the model in the policy-learning process. MBRL methods are advantagcous because they can make deep RL agents more sample-efficient, adaptive, and explainable. Prior work has shown that MBRL methods allow agents to plan with respect to variable goals or diverse environments (Zhang et al.,
Work done Texas A&M University

---

Published in Transactions on Machine Learning Research (02/2024)
2018; Hafner et al., 2023) and that designers can introspect an agent's decisions (van der Waa al., 2018). which can help in identifying potential causes for failures (Räuker et al., 2023).
Despite the benefits of MBRL, there is considerable divergence in existing algorithms and no consensus on the aspects the environment to model and how the model should be learned (e.g., model architecture, data arrangement, objective functions). For example, Dyna-style MBRL algorithms are trained to make accurate predictions about the environment, then find the optimal actions policy with respect to the trained model (Sutton & Barto, 2018). The intuition behind these approaches is that improving the model's accuracy in predicting environment dynamics should facilitate better action selection and improved algorithm performance. However, recent research found that improved model accuracy often does not correlate with higher achieved returns (Lambert et al., 2020). While the underperformance of policies trained on the learned models is often due to the models' inability to sufficiently capture environment dynamics and the policies exploiting errors in the models (Jafferjee al., 2020), Lambert et al. (2020) attributed the root cause of this phenomenon to the objective mismatch between model learning and policy optimization: while the policy is trained to maximize return, the model trained for a different objective and not aware of its role in the policy deeiiioon-maiing process. This objective mismatch problem represents substantial and fundamental limitation of MBRL algorithms, and resolving it will likely lead to enhanced agent capabilities.
In this review, we study existing literature and provide unifying view of different solutions to the objec- tive mismatch problem. Our main contribution is a taxonomy of four categories decision-aware MBRL approaches: Distribution Correction, Control-As-Inferene which derive modifications to the model learning, policy optimization, both processes for the purpose of aligning model and policy objectives and gaining better performance (e.g., achieving higher returns). For each approach, we discuss its intuition, implementation, and evaluations, well implications for agent behavior and applications. This review is complementary to prior broader introductions of MBRL (e.g., see Moerland et al. (2023); Luo et al. (2022)) in its in-depth analysis of solutions to the objective mismatch problem and illustrations of implications for MBRL approaches.
Value- Equivalence, and Differentiable Planning,
2 Background
To facilitate comparisons between the reviewed approaches, we adopt the common notation and premise for MBRL based on (Sulton Barto, 2018). In the subsequent sections, we introduce Markov Decision Processes, reinforcement learning, and the objective mismatch problem.
2.1 Markov Decision Process
We consider reinforcement learning in Markov Decision Processes (MDP) defined by tuple (S, M, R. pt, 7), where S is the set of states, A is the set of actions, M SxA + A(S) is the environment transition probability function (also known as the dynamics model), R S x A + R the reward function, μ S A(S) is the initial state distribution, and 7 [0,1) is the discount factor. The RL agent interacts with the environment using policy : S A(A) and generates trajectories evaluated by discounted cumulative rewards (also known the return) R(r). P(T) and R(() are defined respectively as:
(oo,T,007) distributed according to P(T) and
T
T M ((rls)) R(T) = Σ o ((st, 1=
P(T) = ((30) II
(1)
t=
Abusing notation, we also use P(T) and R(T) to refer respectively to the probability measure and discounted reward for infinite horizon sequences (S0.00; )o:o)) We further denote the marginal stato-action density of policy x in the environment M (also known as the normalized occupancy measure) ds, a) (1 (Ppr)lZ 7 Pr(s: 8,a, a, = =)).
2

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm Basic algorithm of model-based reinforcement learning
Require: Environment, dynamics model M, policy x, data buffer D, time budget T while 2 7 do
Interact with the environment and collect data D t- (s, Update model M Update policy æ end while
2.2
Reinforcement Learning
The goal of the RL agent is to find policy that maximizes the expected return ( in the environment with dynamics M, where (x) is defined
JM(3) = [R())
(2)
Importantly, the agent does not know the true environment dynamics and has to solve (2) without this knowledge. We assume the reward function is known.
(2) is often solved by estimating the state value function ((s) and state-action value function Q(s, a) of the optimal policy using the Bellman equations:
Q(s, a) R(s, a) P P(le))
(3)
V(s) max (ss.a)) aEA
Then the optimal policy can be constructed by taking actions according to:
<(ass) arg max Q(s,a). aEA
(4)
Variations to (3) and (4) exist depending on the problem at hand. For example, continuous action space, the maximum over action typically found approximately using gradient descent. (Lillicrap el al., 2015; Schulman et al., 2017a). Policy iteration algorithms estimate the value of the current policy by defining V(s)= - Ba-x(
a) Bawr( ()(s,, a)] and improve upon the current policy. We refer to the process of finding the optimal policy for an MDP, including these variations, policy optimization.
In model-froe RL, the expectation in (3) is estimated using samples from the environment. Model-based RL instead learns a model M and estimates the value function using samples from the model (often combined with environment samples). These algorithms alternate between data collection in the environment, updating the model, and improving the policy (see Algorithm 1). We will be mostly concerned with forward dynamics model of the form (s'ss, a). although other types dynamics models such as inverse dynamics models, also be used (Chelu et. al., 2020).
We use (s) and (s) to distinguish value functions associated with different policies and dynamics. When it is clear from context, we drop M and to refer to value functions of the optimal policy with respect to the learned dynamics model, since all reviewed hods are model-based. We treat all value functions as estimates since the true value functions can not be obtained directly.
2.3 Model Learning and Objective Mismatch
Many MBRL algorithms train the model M to make accurate predictions of environment transitions. This is usually done via maximum likelihood estimation (MLE):
max LE EE log (s' M
(MLE 5)
This choice justified by the well-known simulation lemma (Kearns Singh, 2002), which has been further refined in recent state-of-the-art MBRL algorithms to account for off-policy learning:
3

---

Published in Transactions on Machine Learning Research (02/2024)
Theorem 2.1. (Lemma 3 in (Xu et al., 2020)) Given an MDP with bounded reward: maX,,a [R(s, a)l Rmaz and dynamies M, a data-collecting behavior policy wo, and learned model M with I Ext, for arbitrary policy with bounded divergence en z
((s,a)ddd D D , max, ( (=1)s=mmm]]]]]m((11)) the policy evaluation error is bounded by:
(T) - JM((T] S V2yBmax 71- VeM + 2/22 VEa.
(G)
(1-
Thus, one way to reduce the policy evaluation error of the optimizing policy is to make the model accurate as possible in state-action space visited by the behavior policy while maintaining small policy divergence. There are Iwo issues with this approach. First, unlike the tabular selting, the dynamics error might not reduce to zero even with infinite data in complex, high dimensional environments due to limited model capacity or model misspecification. Second, maintaining small policy divergence in the second term requires knowledge of the behavior policy in all states and additional techniques for constrained policy optimization. The former requirement can be demanding with an evolving behavior policy as in most online RL settings, with an unknown behavior policy as in most offline RL settings, and the latter requirement can be undesirable since the goal in most RL settings is to quickly improve upon the behavior policy.
Towards better understanding of model learning and policy performance, Lambert et al. (2020) found that model predictive accuracy is often not correlated with the achieved returns. They attributed the root cause of this finding to the fact that, in the common practice, the model learning objective (i.e., maximizing likeli- hood) is different from the policy optimization objective (i.e., maximizing return) and they coined the term 'objective mismatch' to refer to this phenomenon. One of the main manifestations of objective mismatch is that the dynamics model learned by one-step prediction can be inaccurate for long-horizon rollouts due to compounding error (Lambert et al., 2022). These inaccuracies can then be exploited during policy op- timization (Jafferjee et al., 2020). Various methods have been proposed to improve the dynamics model's long-horizon prediction performance or avoid exploitation by the policy, for example by using multistep ob- jectives (Tuo et al., 2018), training the model to self-correct (Talvitie, 2017), directly predicting the marginal density (Janner et al., 2020), training a model with different temporal abstractions (Lambert al., 2021), or quantifying epistemic uncertainty in the model (Chua et al., 2018).
Nevertheless, the relationship between model learning and policy optimization has been found to be highly nuanced. There also exist studies highlighting the diminishing contribution of model accuracy to policy per- formance (Palenicek et al., 2023). From a Bayesian RL perspective, model learning from one-step transitions is theoretically optimal as it provides the sufficient statistics for both model learning and policy optimization (Duff, 2002; Ghavamzadeh et al., 2015). However, the optimal Bayesian RL policy has to account for the uncertainty in the dynamics model parameters and maintaining such uncertainty is generally intractable. Thus, the complex interaction between model learning and policy optimization and efficient methods to bridge these misaligned objectives remains substantial research gap.
2.4 Towards Decision-Aware MBRL
Overcoming the objective mismatch problem has important implications for safe and data-efficient RL. In domains where environment interaction is expensive or unsafe, off-policy or offline RT. are used to extract optimal policies from limited dataset (Levine et al., 2020). In offline MBRL, the dynamics model is typically fixed after an initial pretraining stage and other methods are required to prevent model-exploitation from the policy, such as by designing pessimistic penalties (Yu et al,, 2020b; Kidambi et al., 2020). Decision-aware MBRL has the potential to simplify or remove the design of these post hoc methods
Beyond data-efficiency and safety, decision-aware MBRL can polentially address the gaps in current auto- mated decision decision-making software systems in various domains, such as transportation and health care (MeAl- lister et al., 2022; Wang et al., 2021). These systems are traditionally developed and tested in a modular fashion. For example, in automated vehicles, the trajectory forecaster is typically developed independently of the vehicle controller. As result, modules which pass the unit. test may still fail the integration test.
4

---

Published in Transactions on Machine Learning Research (02/2024)
Thus, the core motivation for solving the objective mismatch problem is to improve MBRL agent capabilities and downstream performance by designing model learning objectives that are aware of its role in or directly contribute to policy optimization, policy objectives that are aware of model deficiencies, or unified objectives that contribute to both (Lambert et al., 2020). It is therefore desirable if guiding principles can be identified to lacilitate the design of future objectives. The goal of this survey is to identify these principles by synthesizing existing literature.
3 Survey Scope and Related Work
The focus of this survey is on existing approaches that address the objective mismatch problem in MBRL. We identified these approaches by conducting literature search of the terms 'objective mismatch,," "model- based," and 'reinforcement learning' and their permutations on Google Scholar and Web of Science. We compounded these searches by examining articles citing (Lambert et al., 2020) and their reference lists. We focused on (Lambert et al., 2020) because it was the first work to coin the term "objective mismatch' and the reference tree rooted at this paper led us to valuable older works that did not used this term. The initial search yielded 85 results which were screened for relevance. To be included in the survey, an article has to specifically address a solution to the objective mismatch problem propose decision-aware objectives for model learning and policy optimization. Articles that simply discussed the objective mismatch problem without providing a solution were excluded. After abstract and full text screening, we retained a total of 46 papers 1
Before presenting the included papers, we briefly cover related research areas that are deemed out-of-scope by our screening critcria in a non-cxhaustive manner; these include state abstraction, representation learning, control-aware dynamics learning, decision-focused learning, and meta reinforcement learning.
MDP state abstraction focuses on aggregating raw state features while preserving relevant properties of the dynamics model, policy, or value function (Li et al., 2006; Abel et al., 2020). For example, bisimulation-baser aggregation methods aim to discover models with equivalent transitions and have been applied in the deep RL setting (Zhang et al., 2020). Recent works on representation learning for control aim to factorize controllable representations from uncontrollable latent variables using, for example, inverse dynamics modeling (Lamb et al., 2022). These research directions are related to the objective mismatch problem in that they also aim to overcome the diffieulty of modeling complex and irrelevant observations for control, however, they are out-of-scope for the current survey because they do not consider the mutual adaptation of model and policy while learning. Control-aware dynamics learning focus dynamics model regularization based on smoothness or stabilizability principles in order to address compounding error in synthesizing classic (e.g.. LQR) or learned controllers (Levine et al., 2019; Singh et al., 2021; Richards et al., 2023). However, they do not directly focus on the source of the problem, i.e., misaligned model and policy objectives, and thus are out-of-scope for the current survey. Objective mismatch also related the broad topic on decision- focused learning, where a model is trained to predict the parameters of an optimization problem, which is subsequently solved (Wilder et al., 2019; Elmachtoub & Grigas, 2022). Due to the focus on sequential decision-making problems (i.e., RL), more general work on decision-focused learning is out-of-scope.
Finally, we omit meta RL (Beck et al., 2023) in the survey and the related topics of Bayesian RL (Ghavamzadch et al., 2015) and active inference, a Bayesian MBRL approach rooted in neuroscience (Da Costa et al., 2020; 2023). Meta RL based on the hidden parameter and partially observable MDP formulations (Doshi-Velez & Konidaris, 2016; Duff, 2002) has a particularly strong connection with decision- aware MBRL in that the policy is aware of the error, or rather uncertainty, in the dynamics model parameters and predictions by virtue of planning in the belief or hyper-state space. However, these approaches often fo- cus on obtaining Bayes-optimal exploration strategies using standard or approximate Bayesian learning and belief-space planning algorithms as opposed to developing novel model and policy objectives (Zintgraf et al., 2019). We also side-step the open question on unified objectives in active inference (Milligge et al., 2021; Imohiosen et al., 2020) but discuss potential connections between active inference and control-as an-ineerncc (Levine, 2018; Millidge et al., 2020) in section 4.2.
The full list of papers can be found at aattpa://gitie httppp///.ttth..cc//aawrv/a//tte paperr..
5

---

Published in Transactions on Machine Learning Research (02/2024)
Error-Agnostic
Policy-Shift POML,
AMPL Yang et 2022
et 2023
Model-Shift
DAM Haghoo et al, 2021
Error-Aware
GAMPS, D'Ora
Controi- Control-As-inferene As-inferenne
VMBPO, Chow 2020
MNM Eysenbach et 2022
Chugare
Distribution Correction
Decision Awaree MBRL
Value- Equuv--quivalecce
Differentiable Planning
Farquher al.,
OMD, Nikisin et a.. 2022
DCBM, Amos
Value- Prediction
VAML Farahmand
H-MML Farihmmnntt 2018
Voloshin 2021
al,
Robust Control
VE, Grimm al, 2020, 2023, 2022
Value-Based
LAMPS, Vemula et si. 2023
PAMI, Atachi 2020
RAMBO, Rigter et. 2022
ct
MuZero, Schvittwieer et 2020 2021
Policy-Based
TD Hameen 2022. 2023
Figure 1: Schematic of the relationships between the core surveyed decision-aware MBRL approaches. Di- roct relationships are shown in solid arrows. Indirect relationships are shown in dashod connections. The algorithms in each category are sorted by the order which they are presented the paper.
4 Taxonomy
Our synthesis of the 46 papers identified 4 broad categories of approaches to solving the objective mismatch problem. We consolidated these into the following taxonomy of decision-aware MBRL:
Distribution Correction adjusts for the mismatched training data in both model learning and policy optimization.
Control-As-Infereen provides guidance for the design of model learning and policy optimization objectives by formulating both under single probabilistic inference problem.
Value-Equivalence searches for models that are equivalent to the true environment dynamics terms of value function estimation.
Differentiable Planning embeds the based policy optimization process in differentiable program such that both the policy and the model can be optimized towards the objective.
A schematic of the relationships between these approaches is shown in Figure 1. The figure illustrates that Value- Eatee-qqqvveecee approaches are the most prevalent in the literature, however, more recent approaches have concentrated on distribution correction and control-as-inferenee. The remaining sections discuss these approaches with a focus on their model learning and policy optimization objectives. Comparisons of the design and evaluations of the core reviewed algorithms are provided in Table and Table 2, respectively.

---

Published in Transactions on Machine Learning Research (02/2024)
4.1 Distribution Correction
Distribution correction aims to correct for training errors atlributable to policy oplimization on samples not from the true environment (model-shift) or to samples from an outdated or different policy (policy-shift). Our literature search identified one approach focused on addressing model-shift and four approaches focused on policy-shift.
4.1.1 Addressing Model-Shift
Haghgoo et al. (2021) proposed to address model-shift by using the learned model as the proposal distribution for an importance sampling estimator the expected return in the true environment as defined in (7):
P(T) R(T) Q(T)
(P(r)[((T)] = BQ(r))
(7)
where P(T) is the trajectory distribution in the true environment, Q(F) = ((s0) II (st41|s as ( (a,ls)) st) is
the trajectory distribution induced by the learned model .M (known the proposal distribution in importance sampling), and w(r) is the importance weight. Since importance sampling is agnoslic to the proposal distribution (i.e., Q(T)), this method can be applied to models trained using any objectives.
Given that both P(r) and Q(T) correspond to the MDP structure, the importance weight can be decomposod over time steps as:
M ( am)
(80.: aoaa)
II m=0
II m=0
(5mm amyy 8m+1)
(8)
M (8m+1 8m, am)
Although not directly available, the per-time step importance weight w(8m: 4m 8m+1) can be obtained using density ratio estimation through binary classification, similar to Generative Adversarial Networks (Sugiyama et al., 2012; Goodfellow et al., 2020). The policy is then optimized with respect to the importance-weiggtee reward. Thus, the authors refer to this method as Discriminator Augmented MBRL (DAM).
To address the possibility that the estimator can have high variance for long trajectorics due multiplying the importance weights, the authors proposed to optimize towards model that achieves the lowest variance for the estimator, which is shown in (Goodfellow et al,, 2016) to have the following form:
("(r) x R(T)
Such a model can be found by minimizing DRr (+))(( resulting in a seturn-weighted model training objective. DAM's model learning and policy optimization objectives are thus defined as follows:
max Ep(r) M
R(T) log (8411 ax)
(DAM 10)
max EQ(r)
II m=0
(Smm am, smt1) A(s al
The authors showed that DAM outperforms the standard Dyna MBRL algorithm with a MLE model objective in custom driving environments with multi-modal environment dynamics. They further found that including the importance weights in the policy objective is crucial for its superior performance.
4.1.2 Addressing Policy-Shift
Rather than focusing on model-shift, the policy-shift approaches address objective mismatch by re-weighting the model training data such that samples less relevant or collected far away from the current policy's marginal state-action distribution are weighte,
Model training data in MBRL is typically collected by policies different from the current policy due continuous policy updating. Wang et al. (2023) found that when training the model to make accurate

---

Published in Transactions on Machine Learning Research (02/2024)
predictions all data, the model produces more error the most recent data compared to its overall performance. To this end, they propose an algorithm called Policy-adapted Dynamics Model Learning (PDML) which stores all historical policies and uses an estimate of the divergence between the data-collecting policy and the current policy to weight data samples for MLE model training.
In line with Wang et al. (2023) to address inferior model predictions on updated policies, Ma et al. (2023) proposed weighted model training scheme motivated by the following lower bound of the log oog-traasformed expected return:
log JM = -log Bay, e') R(ss, a)]
dī ds R(s,a)
log Baz, (s.a,
dr M
(s, a,
(11)
dīe as, log dr
2 Eay it (s
log R(s,a)
(5, (s, 11
2 - -Dylit (8, a, s')lldrr a8, 8.5 + Bag Mt (a.0, s) flog a(s.
where (s, a, s') (1- Pr(st = s, = a, S6-1 = s')l denotes the normalized occupancy measure for transitions (s, a, s'), and D, denotes a chosen f-divergence measure. The first. term in the lower bound suggests that the learned dynamics model should induce similar transition occupancy to the true environment under the current policy. To this end, the authors proposed the Transition Occupancy Matching (TOM) algorithm to optimize the first term while alternating with optimizing the second term using model-based policy optimization (MBPO) (Janner et al., 2019).
Although one could optimize the first term by simulating æ in M, such an approach requires repeated simulation for each dynamics model update and fails to leverage data collected by the behavior policy, Instead, the authors proposed using dual RL techniques to minimize D, with respect d where a dual value function (s),a) is introduced to penalize d from violating the Bellman flow constraint (Nachum & Dai, 2020). Upon obtaining the optimal dual value function (*((s, a), which can be estimated from the collected data, the dual RL formulation yields the following importance weight estimator for finding the optimal transition couupancy-matchine dynamics using weighted MLE:
(s, a, 5') wTOM(S,a,s') a,) (s a. 8')
f2 (1og dir(s, (s, BB 1"x,
(12)
where the log density ratio on the right hand side is estimated using discriminator and f. is the derivative of the Fenchel conjugate of the chosen f-divergence function. The resulting TOM model and policy objectives are as follows:
max E( WTOMM(S a, s') log (s'(ss))))
(TOM 13)
max E (s [log R(s,a))
Empirically, the authors showed that TOM correctly assigns higher importance weights transition samples more similar to the current policy and that TOM is more sample efficient and achieves higher asymptotic performance in MuJoCo environments (Todorov et al., 2012) than MBPO and PDML which uses heuristic policy divergence based weighting. The authors also compared TOM with VaGradM (a different class of decision-aware MBRI algorithm reviewed in section 4.3.1) 1) and found TOM performs significantly better except for one environment.
In contrast the above online RL approaches, Yang et al. (2022) addressed the offline RL setting and proposed the Alternating Model and Policy Learning (AMPL.) algorithm to optimize the following lower
8

---

Published in Transactions on Machine Learning Research (02/2024)
bound of the expected return:
JM 2 M
V2 \2 V (M,
Rmma
(14)
2 J M
where the divergence in the second term which characterizes policy-shift is defined as:
Dx(M, = (3 WAMPL(Ss a) ()m a)ma (I a))(((())
(15)
dīr(3,0) is the marginal statc-action density ratio between the current policy and the da (e,a)
and WAMPL(s,a) =
behavior policy.
Similar to TOM, the bound in (14) suggests MLE dynamics model learning weighted by (8, How- ever, for policy training, (14) suggests not only maximizing reward but also minimizing the divergence so that the ovaluation error in the second term is controlled for. Instead of using dual-RL. techniques to estimate w in TOM, the authors proposed novel fixed point estimator. They further approximated Dx M) for policy optimization using the output of discriminator: log(1 a)s, The APML model and policy objectives are thus defined follows:
max E. B/C../1-D(WAMMP(.... a) log a)],
(AMPL 16)
max B(,.a)ydy, [R(s - log(1 - a)))
A similar method was proposed in (Hishinuma & Senda, 2021) which instead estimates the importance weight using full-trajectory model rollouts.
Empirically, Yang et al. (2022) showed that APML outperforms number of SOTA model-based and model- free offline RL algorithms (e.g., MOPO; (Yu et al., 2020b), COMBO (Yu et al., 2021), CQL (Kumar et al., 2020)) in most D4RL environments (Fu et al., 2020) and that including the importance weight w in the model objective is crucial for its performance.
Also in the offline RL setting, D'Oro et al. (2020) focused specifically on policy-gradient estimation using samples from the offline dataset, but the value function is estimated using the learned model (model-value gradient; MVG). This setup can be advantageous since the model bias is limited to the value estimate but not the sample-based evaluation of expectation. They proposed the Gradient-Aware Model-based Policy Search (GAMPS) algorithm which trains the model to minimize the following bound on the difference from the true volicy-gradiens
2mman
V+JM(TT) JVG
(1 (1- V ((aaa))nng DKLL)(LLL..... ())
(17)
where Z is constant and n/. is state-action density weighted by the policy-gradient norm. This suggests model training objective where samples with higher policy-gradies norms are up-weighted.
The GAMPS model and policy objectives are defined as follows:
max ENND M
I
T (aels)) (a
V log m(alssi)]] log M (s411l1))
(st)
L=D
m=u
(GAMPS 18)
T(als:) my(a+(s.)
max ENED
Q(5s, a:) log (a.s()))):)
x
m==
Empirically, the authors first used a diagnostic gridworld environment to show that GAMPS can accurately estimate policy gradients with a constrained dynamics model class which conditions only on actions but not on states. On two MuJoCo tasks, they showed that GAMPS achieved higher asymptotic performance than ML.E. model learning and two classical model-free RL algorithms adapted to the offline setting.

---

Published in Transactions on Machine Learning Research (02/2024)
Distribution Correction Summary: All approaches in this category focused on return estimation under mismatched distributions. The single approach addressing model-shift adopted straightfor- ward importance sampling estimator, while the rest of the approaches have focused on bounding policy or policy-gradient evaluation error with respect to policy-shift as a result of deviating from the behavior policy. All approaches also adopted a weighted maximum likelihood model learning ob- jective where the weights represent relevance to policy optimization, such as higher return, closeness to the current policy, or potential for policy improvement. However, by changing weights over the course of training, these models are able to adapt changes in the current policy as opposed to being policy-agnostic in standard MLE model learning.
4.2 Control-As-
Besides the distribution correction approaches, other methods have attempted to leverage existing validated approaches to solve the objective mismatch problem. One such principled approach is to leverage the control- as-inference framework (Levine, 2018) and formulate both model learning and policy optimization as a single probabilistic inference problem.
The core concept of nuntro-as-infferend is that optimal control can be formulated probabilistic inference problem if we assume optimal behavior were to be observed in the future and compute posterior distribution over the unknown actions that led to such behavior. Most control-as infferencc methods define optimality using a binary variable O where the probability that an observed state-action pair is optimal (O = 1) is
defined as follows:
P(O, = 1|8, a.) = eepp(((sss) at))
(19)
The posterior, represented as variational distribution Q(T), is found by maximizing the lower bound of the marginal likelihood of optimal trajectorics as follows:
log p(00: = 1) = log S :(
log Ep(T)
(P(O = 1|T]]
0
(20)
log Er(r)
exp
A(st, ax)
1=0
2 [Q(7)
Σ
R(ses az)
-DuI((G))P(t))
The prior P(r) is usually defined using the structure of the MDP:
P(r) ((so) II M(s
(21)
where P(als) a prior or default policy usually set to a uniform distribution.
Most of the design decisions in control-as-inference are incorporated in defining the variational distribution Q(T). Prior control-as-ing -nifeerenc approaches, such as the ones reviewed in (Levine, 2018), define Q(T) using the environment dynamics and do not incorporate a learned model in the formulation. Thus, the majority of those algorithms are model-free.
In contrast to those approaches, Chow et al. (2020) proposed a joint model learning and policy optimization algorithm under the control-as-inference framework called Variational Model-Based Policy Optimization (VMBPO) by incorporating the learned dynamics model in the variational distribution defined as follow:
Q(T) p(s))) II ( (a)(s)) 1=0
(22)
10

---

Published in Transactions on Machine Learning Research (02/2024)
Although Q(r) has the same structure the proposal distribution in the importance sampling-bass algo- rithm DAM (see (7)); the model learning and policy optimization objectives of VMBPO are derived from a single marginal likelihood lower bound defined as:
20
P(a,ls,) + log M (86,
M(++ ( 18))
max (T()) M,s
Σ 76
(()) + log x(a st)
(VMBPO 23)
Defining Bellman-like recursive equation based on (23):
P(als) (s,, a) = R[sr, at) + log s)
+ E (1))..))) V(s') + log a)
(24)
M
The authors showed that the optimal variational dynamics and policy have the following form:
(s'ls, ss, a) x exp(V(s') + log M(''ss, a)), 7 Taass x exp(Q(s, a) + log P)a
(25)
Similar to DAM, the authors used a discriminator to estimate the dynamics density ratio.
Interestingly, the optimal dynamics are encouraged to be not only similar to the true dynamics via the log likelihood term, but also have a higher tendency to sample high value states. Such an optimistic dynamics model is similar to that of DAM, albeit for a different reason that the dynamics is conditioned on the opti- mality variable while performing variational inference. Also similar to DAM's importance weight-angmented reward, the policy optimizes not only reward but also the negative log density ratio between the learned and the ground truth dynamics. This encourages the policy visit states where the learned dynamics is accurate (lower cross entropy with the ground truth dynamics) and also states where the learned dynamics is uncertain (higher entropy).
Empirically, the authors showed that VMBPO outperforms SOTA model-based (e.g., MBPO) and model-free baselines (e.g., SAC (Haarnoja et al., 2018)) in six MuJoCo environments for a range of policy learning rate settings. However, they did not evaluate specific agent propertics discussed in the previous paragraph.
While Chow al. (2020) was the first propose a unified decision-awars objective for model learning and policy optimization, Eysenbach al. (2022) pointed out that the objective (23) is an upper-bound on the RL objective, which can be decomposed as the expected return and its variance, and thus undesirable. They instead proposed an algorithm called Mismatched--NoMMM whose distribution is conditioned on trajectories instead of state-action pairs:
(MNM) with alternative optimality variable
P(O 1/7) R(T) o (((s,.)))
(26)
Using this definition of the optimality variable, they derived a modified log marginal likelihood lower bound:
log P(O = 1) = log / P(O = 1,
llog Epr) (P(O 1|7)
EQ(r) [log R(T) + log (rr) - log Q(r)]
(27)
EQ(T))
log
R(s,, az) - log P(T) - log Q(T)
1=0
2 BQ(r)
Σ n²t log R(s,, a.) + log P(T) log Q(T))
1=0
The benefit of this modification is that it is a lower bound on the RL objective so that improving this bound guarantees improving upon agent performance.
11

---

Published in Transactions on Machine Learning Research (02/2024)
MNM also optimizes single objective for both model learning and policy optimization, except that the reward is log-transformed:
P(ac(se) M (6sH1lls,,a)) (1)) log R(s;, at) + log [T(((s)))
max EQ(r) M,
Σ 7
+ log M (8241131, at) ((suillshaa))
(MNM 28)
To evaluate MNM, the authors first showed that in a gridworld goal-reaching environment, MNM solves the task faster than VMBPO and Q-learning and it robust to constraints on the dynamics model to only make low-resolution predictions. On robotic control tasks with contact dynamics and sparse reward (e.g., door-opening) in MuJoCo, Deepmind Control suite (DMC) (Tassa al., 2018), Metaworld (Yu et al., 2020a), and ROBEL (Ahn et al., 2020) environments, MNM frequently outperformed MBPO and SAC by large margin and it consistently performed well in all tasks. Furthermore, the authors found that MNM's model-based value estimates were stable and did not explode towards large values throughout the learning process, which suggest robustness to model-exploitaatinn Visualizing the learned dynamics, the authors also found the MNM dynamics model tends to generate transitions towards high value states, which provides evidence for the optimistic dynamics to speed up learning.
Extending MNM to the visual RL setting, Ghugare et al. (2022) replaced the state dynamics model with a latent dynamics model M2''2z,a) a) and an observation encoder E(z[s) in an algorithm called Latent Aligned Model (ALM). By designing the following prior and variational distributions:
P(T) = ((50)
II t=
M (8 (t) ((a
(29)
20
Q(T) = (
II (st41 se, at) (t) a:)
(30)
where T(aaz) is latent space policy, the ALM's model (M, B} and policy jointly optimize the following objective:
P(a((z) Σn r² log R(s,, uz) + log (a.l21) 00
+ log B(2011 )
max EQG) M.E.
(ALM 31)
M (t)
Interestingly, the third term in the augmented reward, which can be estimated using a discriminator similar to VMBPO and MNM, is the information gain of the latent dynamics model, thus connecting this approach with prior work on intrinsic motivation and thformation-theortiicc approaches in RL (Eysenbach et al., 2021; Rakelly et al., 2021; Sun al., 2011) and the pragmatiic-episemmic value decomposition of the expected free energy objective function for planning in active inference (Millidge, 2020; Sajid et al., 2021; Fountas et al., 2020).
However, different from VMBPO and MNM's weighted-regresstee dynamics model learning objective, the ALM loss implies "self-predictive" model learning process (Schwarzer et al., 2020), where the latent dy- namics model is trained to predict the encoding of observations (and reward) in the collected dataset and the encoder is trained to match the predictions of the latent dynamics (similar to Bootstrap Your Own Latent (Grill et al., 2020)). While theoretical understanding of self-predictive objectives is still lacking, Sub- ramanian et al. (2022) showed that, when reward prodiction is also incorporated into the objective function, the learned latent state z is sufficient statistic (or information state) for the optimal policy with respect to the true environment and Tang et al. (2023) proposed that it implicitly performs cigen-decomposition of the true environment dynamics. Thus, the dynamics model learned by ALM is likely more task-agnostic than the dynamics models of VMBPO and MNM. This also suggests that decision-aware objectives may not necessarily need to bias model learning, aligning this method further with active inference.
Empirically, the authors found that ALM significantly outperformed SAC and SAC-SVG (Amos et al., 2021), MLE-based MBRL algorithm with latent dynamics, with 2c5 environment steps (1/5 of the usual RL. training steps) in five MuJoCo environments. Using the Q value evaluation protocol from (Chen et al.,
12

---

Published in Transactions on Machine Learning Research (02/2024)
2021) and (Fujimoto et al., 2018), they found ALM value estimates to consistently have negative bias (underestimation). Similar to MNM, they also found that including the density ratio term in the objective is crucial.
Control-as-Inference Summary: The control-as-inference calegory is closely related to the distribution-correetio category in that the variational distribution can be interpreted as the pro- posal distribution for an importance sampling estimator the expected return. However, control-as- inference focuses more directly on return optimization as opposed estimation as signified by the maximization of the likelihood of optimal trajectories.
The main advantage of the the derivation of model and policy objectives as long as factorization of the prior and variational distributions are given. However, a major downside of current mutrol-as--nferonn the factorization of these distributions are largely hand-designed, which is the key to the attractive properties in (31) but potentially a bottleneck for future objective design. Although automated design of posterior distributions leveraging graphical model factorization has been proposed for variational inference in probabilistic predictive models (Webb al., 2018), extensions the RL setting have not been considered developed.
inference framework is that it provides a clear guidance to
approaches is that
4.3 Value Equivalnee
A core case of the learned dynamics model is that the agent sample from it to augment the limited environment samples for estimating the expected return or value function. Thus, as long the dynamics can lead to accurate value estimates, it does not need to model the environment at all. In other words, the model is free to discover any dynamics that are equivalent to the true dynamics in terms of estimating value without wasting resources on irrelevant aspects of the environment. This class of approaches focuses almost entirely on model learning and performs standard model-based policy optimization. In this section, we first summarize how the literature formulates equivalent dynamics and how identifying such dynamics malue-prediction problem. We then summarize how such equivalence is related to the
can be formulated robustness properties of the learned dynamics.
4.3.1 Palui-Prediction
The value- pelleeprediition approaches propose to directly predict states with accurate values rather than accurate features. This is desirable if the dynamics model has limited capacity modeling all aspects of the environment laithfully, for example when using a uni-modal distribution to predict multi-model transitions, but the model can generate states whose values are close to the ground truth future state values. The models don't 't necessarily have to predict the true expected return value which is unknown, and major design decision in these approaches is what value function to predict.
As the first approach in this class, Farahmand et al. (2017) proposed to train the dynamics model such that it yields the same Bellman backup (i.e., (3)) as the ground truth model. They formulated this intuition the following objective called Value-Awan Model Loss (VAML):
min (' max Vx V(s') (2 M
(VAML 32)
Since the true value function is not known, robust formulation is used to optimize against the worst-case value function.
Empirically, the authors showed that VAML achieves smaller value-estimation error and higher return than MLE dynamics models in a finite MDP using a low-resolution dynamics model class.
To account for the fact that the optimizing policy и may be different from the behavior policy Th Voloshin et al. (2021) introduced density ratio correction into an objective similar to VAML. Furthermore, to account for unknown density ratio and value function, they propose to optimize against the worst-case for both in
13

---

Published in Transactions on Machine Learning Research (02/2024)
their Minimax Model Learning (MML) objective:
min max M W.V
B( (w(ssa) (V(s))-Ber- -i B MM [(+")]]
(MML. 33)
where w(s, a) = Be(sa)) is the unknown density ratio between the optimizing policy and the behavior policy 2n (s.a)
in the ground truth environment. They showed that this objective a tighter bound on the policy evaluation error than the VAML objective.
Asadi et al. (2018) showed that the VAML loss function is equivalent to minimizing the Wasserstein distance between the learned and ground truth dynamics. As a result, the learned dynamics model tends to have attractive smoothness properties (e.g. being K-Lipschitz) and are less prone to compounding error and model exploitation.
However, the robust formulation of VAML poses a challenging optimization problem. As a response, Farah- mand (2018) proposed replace the worst-case value function with the most recent estimate in an objective called iterative VAML. In (Ayoub et al., 2020), the authors paired iterative VAML with an optimistic planning algorithm to derive a. novel regret bound for the RI. agent. Related to iterative VAML, MuZero "Schrittwieser et al., 2020; 2021) and Value Prediction Network (VPN) (Oh et al., 2017) replace the most recent value es- timate with one bootstrapped from Monte Carlo Tree Search and multi-step look-ahead search, respectively (for a more nuanced discussion of VAML and MuZero, see (Voelcker et al., 2023)). These algorithms all similar architecture and
train the model to perform reward-prediction in addition to valuo-prediction. loss function was used in the Predictron model to evaluate the return of fixed deterministic policies (Silver et al., 2017a). MuZero has been shown to outperform prior state-of-the-art algorithms (e-g., AlphaZero: (Silver et al., 2017b)) in Go, Shogi, and Chess and large scale image-based online and offline RL settings (Schrittwieser et al., 2020; 2021).
More recently, Hansen et al. (2022: 2023) proposed combining value-prediction, reward-prediction, and self- prediction (i.e., the model learning objective in (31) in their TD-MPC algorithm and performing model- predictive control with a learned terminal value function. They showed that with a small number of architec- tural adaptations to handle environments with distinct observation and action spaces and reward magnitudes, TD-MPC agents trained with single set of hyperparameters substantially outperformed prior state of the art (e.g. Dreamer-v3 (Hafner et al., 2023)) on large set of 104 continuous control tasks. Most notably, the authors showed in a multi-task setting across 80 tasks that increasing the number of model parameters led to substantial increase in returns and that multi-task training improved agent performance by almost two-fold when fine-tuned on new tasks compared to training from scratch.
Extending the VAML approach, Grimm et al. (2020) considered calue-prediction with respect to a set of policies 1l and a set of arbitrary value functions V (i.e., not associated with any particular policies). They formulated the following objective based on the proposed Value-Equivalens Equivalence (VE) principle:
min ΣΣ B(s)-Daaaa(1))) |B (Vo) - - Be (V M VELVEE
-
(VE 34)
The primary difference between the VE loss and the VAML loss is that it is formulated using arbitrary policies and values, rather than just the data-collecting policies and its associated value estimates. The benefit of this is that as we increase the size of the set of policies and value functions considered, the space of value-oquivalent models shrink before eventually reducing to the single ground truth model. Furthermore, when the model has limited capacity even at predicting values, one can trade off between the policy and value set considered and the desired value-equuvallenee loss (Grimm et al., 2021; 2022).
Empirically, the authors showed that VE significantly outperformed MLE on two tabular environments (Catch and Four Rooms) and Cartpole when high rank constraints where imposed on the dynamics model, the agent performance improved with increasing size of the value function set.
Despite the simplicity and popularity of the VAMI-famill loss, Voelcker et al. (2022) suggested that it can cause undesirable optimization behavior when querying ouitt--distribution value function estimates, especially when the learned model is not regularized to be close to the true environment. Using the inductive
14

---

Published in Transactions on Machine Learning Research (02/2024)
bias that the learned model should predict states similar to the true environment for next state samples s' in the dataset, they replaced the value function estimate in the VAMI. loss with its Taylor expansion around s': (s)]ar = V(s') + (V.V (s)
s'). (s's). Assuming deterministic dynamics and applying the Cauchy Schwartz incquality: (Ei=1 x2 I n Em= x7, the VAML loss reduces to the following Value-Gradien Weighted Model
Loss (VaGradM):
min (. (((laa)) - dit))iiig(( M
(VaGradM 35)
When evaluated against VAML and MLE dynamics model training approaches with expressive dynamics (e.g,, sufficiently large neural networks) in two MuJoCo environments, the authors found VaGradM to be robust to loss explosion when the value estimates were inaccurate in the initial training steps and converged to similar solutions to the MLE dynamics. However, VaGradM perporrreeed MLF when the dynamics model was constrained to fewer neural network layers and when distracting state dimensions were added.
The value-prediction approach can also be applied policy-based RL methods, where instead of training the model to make accurate predictions of values, the model is trained make accurate predictions of policy gradients. Abachi (2020) proposed the (multi-step) Policy-Aware Model Loss (PAML) defined as follows:
K
K Σ k=
h F( an) - Er-Q(T)
min M
E E P(T)
A F(s ai)
(PAML 36)
R=
where Q(r) = ((5)) Ie 1s,
a(s, (logm(als) and Q" a) is model-free
value estimate.
In a quadar- control setting, the author found PAML to learn more slowly and achieves lower asymplotic performance than MLE but performs substantially better when irrelevant state dimensions are added. However, this result did not transfer to the MuJoCo environments where PAML and MLE only performed on par regardless of whether there were irrelevant dimensions.
Value-Prediction Summary: Value-prediction approaches are similar distribution correction and control-as-inerrnnce approaches in that it also tries to obtain more accurate value estimates, however, it focuses estimating the Bellman operator opposed to the return directly. Compared to two previous categories of approaches, value-prediction approaches have the benefit of requiring fewer moving parts, e.g., the importance weight estimator, and they do not by-default generate optimistic or pessimistic dynamics. However, by focusing on value-based RL algorithms, value- prediction approaches can be less flexible than the other classes of approaches.
4.3.2 Robust Control
While seemingly bearing no obvious connection to valueeprediction, the robust control approaches reviewed in this section are in fact derived from the same valuo-equivalence principle for the purpose of learning value-equivalent dynamics to the ground truth environment dynamics. As we discuss below, these ap- proaches highlight an inherent pessimism in value-equivalent dynamics models which neglected in the value-prediction approaches.
Modhe et al. (2020; 2021) suggested that the VAML. and value-oquivalence loss functions can be understood from the perspective of model advantage defined as:
A (s, s') = (a 77)
(37)
where positive model advantage corresponds to optimistic dynamics and negative model advantage corre- sponds to pessimistic dynamics. However, by minimizing the norm of the valuo-prediction error, VAML and the VE optimize towards zero model advantage.
15

---

Published in Transactions on Machine Learning Research (02/2024)
A more complete relationship between valuo-equivalence and model advantage is depicted in (Vemula et al., 2023) where the authors derived the following decomposition of the return gap between the optimizing policy π and the behavior policy Tb in terms of IT's value in the learned dynamics model:
(1 (1-T) - Jas((T))]
E(oa))ad (s) 22 (s,
Model-based advantage under data distribution
y (e 32 (e [V (s') E M (-1sa) [V
(s")
(38)
Model dis-advantage under learner distribution
(3,0)
(er (s.a)
E
[V (s")] E/-M(()a
[V (s')
Model advantage under data distribution
The second and third terms in (38) suggest that the return gap between the two policies is more nuanced than just valuo-prodiction error; it also involves model advantage and disadvantage under the data and learner distributions.
This relationship suggests a recipe for jointly optimizing model and policy to improve upon the behavior policy, namely, training the policy in the learned model starting from states visited by the behavior policy and simultaneoussy training the model to increase advantage under the data distribution and decrease advantage under the unknown learner distribution. Using this insight, Vemula et al. (2023) proposed an algorithm called Lazy bodel-based Policy Search (LAMPS) with the following model and policy objectives:
max E ((..)) (1 [5') (s M i (s))
M max EgwD (1)1
(LAMPS 39)
where d is obtained by simulating the learner policy in the learned dynamies. By optimizing (39), the dynamics model will be optimistic under the data distribution, similar to However, the dynamics model will be pessimistic rather than just entropic outside the data distribution, resulting in a robust formulation.
approaches.
Empirically, the authors found LAMPS to consistently achieve higher performance with fewer environment steps than MBPO across four MuJoCo enviromments.
In the context of offline RL, Rigter et al. (2022) proposed similar algorithm to LAMPS called Robust Adversarial Model-basod Offline Policy Optimization (RAMBO). The RAMBO model and policy objectives are defined as follow:
max XB ( log M ss,a) E. ((,)) s' '' M(15a) (s')]
(RAMBO 40)
max E( [R(s,a)]
where the model advantage loss (the first term in (39) model objective) is replaced with the standard MLE model objective and a hyperparameter X is used to weight against the second term optimizing model disadvantage. The policy is optimized using the learned model rather than the collected data. As a result, the learned model is only pessimistic and only in state-actions where there no data. This treatment makes intuitive sense the offline RL setting since the agent not allowed to interact with the environment to explore and correct for optimistic mistakes.
Theoretically, Uehara & Sun (2021) showed that for sufficiently accurate learned dynamics model the offline data distribution (e.g,, as measued by Drev))-DDvvl [LM ( MLE dynamics), simultaneously minimizing model advantage and optimizing policy on the learned dynamics model yields a policy that is competitive with any policy found on the offline dataset. This can be done by setting the À parameter to be sufficiently high in the RAMBO algorithm.
a) M))s a)] S e where MLE is the
Empirically, Rigter et al. (2022) showed that RAMBO achieves the best overall performance amongst other model-based and model-free baselines in four MuJoCo tasks with varying dataset qualities and it significantly
16

---

Published in Transactions on Machine Learning Research (02/2024)
outperforms other model-based baselines in the AntMaze environment which has more challenging contact dynamics. Compared with COMBO (Yu et al., 2021), SOTA model-based offline RL algorithm, RAMBO learns a smoother dynamics model, potentially making it less prone to local optima.
Robust Control Summary: Robust control approaches reveal a hidden insight behind value- prediction approaches: in the process of finding value-equivalent MDPs, the dynamics model is ac- tually becoming optimistic on some state-action pairs and pessimistic others depending its visitation. The inherent pessimism makes robust control approaches espocially suited for addressing distribution-shift, such as in offline RL.
4.4 Differentiable Planning
Instead of explicitly defining model learning objectives, differentiable planning approaches embed the policy (or trajectory) optimization process with respect to the learned model as a differentiable program and update the model with respect to a higher level objective, such as maximizing return in the true environment or the standard Bellman crror loss using environment samples (Mnih et al., 2013). These approaches typically take bi-level optimization format where optimality with respect to the learned model is defined constraint.
Farquhar et al. (2018) first recognized that for discrete actions and deterministic dynamics, the multi-step look-ahead search in VPN (Oh et al., 2017) can be interpreted as an neural network layer, which can be used to process state inputs before outputting the final value prediction. They proposed the TreeQN architecture which can be formulated as the following bi-level optimization problem:
min B (o (R(s, a) + +s' a(s,a))² M.Q
K-1
(TreeQN 41)
s.t. V(8) = arg max E. ao.k
Σ K=
R ax) + V =
M
In a box-pushing environment, the authors showed that TreeQN significantly outperformed DQN (Mnih et al., 2013) and its performance improved when increasing the look-ahead depth from to steps. the Atari environments, TreeQN outperformed DQN in all except environment and consistently achieved higher performance with fewer environment steps. However, the effect of look-ahead depth in Atari were not observable.
Nikishin et al. (2022) proposed a similar bi-level optimization approach called Optimal Model Design (OMD). However, instcad of using a multi-step look-ahcad scarch, the constraint was defined using the first-order optimality condition for Bellman error minimization with respect to the learned model. The OMD model and policy objectives are defined as follows:
min E (a (R(s,a)
M.Q
(OMD 42)
s.t. M B (. (R(s, + -(s)))((ssaa)) =0,
where the soft-value function ((s) == log E. cxp(Q(s, was used to make the constraint differentiable.
A novelty of the OMD approach is the use of the implicit function theorem to differentiate through the constraint. This method was later extended in (Bansal et al., 2023) where only metric the model loss was included in the upper objective to automatically weight task-relevant features while the standard prediction objective under the optimized metric was still used for model learning and formulated as an additional constraint. Sharma et al. (2023) used a similar implicit differentiation approach in a setting where the reward function is sampled from a distribution. Wang et al. (2021) used a sample-baser approximation of implicit gradients in the setting of predicting missing parameters in MDP from logged trajectories and then perform offline RI.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Empirically, Nikishin et al. (2022) showed that OMD outperformed MLE in Cartpole and one MuJoCo environment when the dynamics model hidden dimensions parameter norm were constrained and when distracting state dimensions were added.
In contrast to the above value- based upper objectives, Amos Yarats (2020) proposed to re-paramcterize the policy by embedding the model differentiable Cross Entropy Method (DCEM) trajectory optimizer. Each action outputted by the DCEM policy is computed by running the DCEM trajectory optimization algorithm for finite number of iterations. Then, the dynamics model parameters are updated with respect to the upper level objective of the expected return of the policy in the true environment. The DCEM model and policy objectives are defined as follows:
max (( [R(s,a)]
M
(DCEM 43)
s.l. T 8; M) = arg max EM dg
Σ
- =
The upper level objective is optimized using the Proximal Policy Optimization algorithm (Schulman et. al., 2017b).
Empirically, the authors showed that DCEM matched the performance of Dreamer (Hafner et al., 2019), popular MBRL algorithm, on two MuJoCo tasks with an order-of-magnitudee fewer enviromment samples.
Prior to the above deep learning based differentiable planners, Joseph et al. (2013) and Bansal et al. (2017) explored similar bi-level optimization formulations of model learning where the model parameters are updated using finiteddifeerence gradient approximation and Bayesian optimization, respectively. These methods were applied to simple and low-dimensional problems and they are difficult to scale to high-dimenniooall settings with large dynamics model paramcterizations.
Recently, Gehring et al. (2021) theoretically studied differentiable planners in a simplified reward prediction sctting and found that their learning dynamics can be understood as pre-conditioned gradient descent, which can significantly increase convergence speed depending on the initialization of the value estimate.
Despite limited applications in RL, differentiable planners are a popular approach for inverse RL and imita- tion lcarning (Tamar al., 2016; Okada al., 2017; Srinivas et al., 2018; Karkus et al., 2017; Amos et al., 2018). Aiming to improve the efficiency of learning differentiable planners, Bacon et al. (2019) proposed Lagrangian method which bypasses solving the lower optimization problem for every upper optimization step. Leveraging the relationship between value-equivalence and robust control (i.c., (38)), Wei et al. (2023) showed that favorable properties of differentiable inverse planners can be attributed to learning pessimistic dynamics and robust policies.
Differentiable Planning Summary: Differentiable planning represents the most direct approach to solving objective mismatch by embedding minimal set of standard MBRL components (e.g., only a model and a policy) in a differentiable program so that all components optimize the same objective. This class of approaches bypasses return estimation and focuses directly on return optimization. It also involves the least amount of human intervention in the objective design process.
5 Discussion
This paper reviewed solutions to the objective mismatch problem in MBRL and classified the approaches into a taxonomy based on their structure and intuition. The taxonomy highlights that these approaches have influences on key components of MBRL which we discuss below:
MBRL objective design and the search for value optimizatiion-equivaaeee in both the model and the policy.
Important agent properties, such as ground truth model identifiabllity, agent exploration behavior, and model optimism and pessimism.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Comparisons of architecture and implementation of the core reviewed decision-avee -aasin MBRL. algorithms. For each algorithm, we list its ob- jective mismatch solution category DDC=distrbbution correction, MS- MS-model shift PS S---ooll------ shift, CAI= CAl=control-ne catrooll---nfffeeee inferffeee,, VE- waw---aaleeeeeeeee tatte---uuilllenee
VP= VP-volue volue- predictiioo RC- RC-rrbbus robust control, DP= dddfferrnntiabll planning). dynamics model type, whether ensembling of dynamios was used, whether the dynamies model was trained to make multi-ster predictions, the policy optimization algorithm, and other agent components.
Discriminatod model Q Discriminator, IW estimator
Discriminator encoder
Adversarial V Adversarial V IW
Other com ponentsens
Discriminator
Discriminator
Discriminator
Encoder
DQN, TD search
Policy algo.
Discrete SAC
DQN, MCTS
Discrete SAC
Shooting
SAC
TD3
PG
SAC
SAC
DDPG
VI
MPPI
DQN
SAC
DDPG
SAC
SAC
DQN
CEM
No Multi-step
No
No
No
No
No.
Yes
No
No
Yos
Yes
Yes
No
No
Yes
No
No
No No
No No
No No
Ensemble
No
Yes
Yes
No
Yes
Yes
No
No
No
No
No
No
No
Yes
No
Yes
Yes
Mixture density network
Mixture density network
Linear Caussian
Latent Ganssian
Latent deterministin
Latent deterministic
Latent deterministic
Model type.
Gaussian
Gaussian
Gaussian
Tabular
Gaussian
Deterministic
Deterministic
Deterministic
Deterministic
Gaussian
Gaussian
Deterninistic
Deterministic
Category
DC-MS
DC-PS
DC-PS
DC-PS
VE-VP
VE-VP
VE-VP
VE-VI
VEEVP
VE-VP
VE-VP
VE-VP
VE-RC
VE-RC
CAI
CAI
CAT
DP
DP
DP
TD-MPC Hansen et al. (2022; 2023)
MuZero (Schrittwiesed ot al., 2020)
VAML (Farahmand et al., 2017)
VaGradM (Voeleker et al,, 2022)
DAM (Haghgoo al., 2021)
GAMPS (D'Oro et al., 2020)
VMBPO (Chow et al,, 2020)
MNM [Eysenbach al., 2022)
ALM (Ghugare et al., 2022)
MML (Voloshin et al., 2021)
LAMPS (Vemula et al., 2023)
RAMBO (Rigter al., 2022)
TreeQN (Farquhar et al., 2018)
OMD (Nikishin et al., 2022)
DCEM (Amos & Yarats, 2020)
AMPL (Yang al., 2022)
VE (Grimm et al., 2020)
TOM (Ma et al., 2023)
VPN (Oh et al., 2017)
PAML (Abachi, 2020)
A Algorithm
Table 1:
19

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Comparisons of eevauaatonn and experiments of the core reviewed decision aware MBRL algorithms. For each algorithm, we list its main eval tutiin environments, main baseline algorithms, the types of model misspecification diliaatton evaluated, the metric used for evaluating model exploitation (policy gradient or Q estimation accuracy), and whether the it's designed for the online or offline RL setting Algorithm
On offline
Online
Online
Offline
Offiine
Online
Online
Online
Online
Offline
Online
Online
Online
Online
Online
Online
Online
Offiine
Online
Online
Online
Exploitation
PG
Q
Q
Q
Q
NN size, distractor Distractor
Remove features
Misspccffiation
Multimodal
Low-res.
Low-res.
Rank
NN size
MBPO, PDML, VaCradM
MBPO, SAC-SVG
Baselines
COMBO, CQL PG
MBPO, SAC
MBPO, SAC
MILE, VAMI
Alpha
DQN, MLE
SAC, Dreamer-v2
MBPO, VAML
MLE, DDPG
MBPO
COMBO, CQL
DQN, A2C
Dreamer
MLE
MLE
MLE
MLE
MuJoCo, DMC, Metaworld, ROBEL
Four rooms, Catch, Cartpole
Maze2D, MuJoCo, Adroit
Atari, Go, Chess, Shogi
DMC, Metaworld
MuJoCo, AntMaaee
MuJoCo, Cartpole DMC
LQR, Cartpole
Environments
Driving
MuloCo
MuJoCo
MrJoCo
MuJoCo
Tabular
MuJoCo
MuloCo
MIJoCo
Atari
Atari
TD- TD-MP MPC (Hansen et al., 2022; 2023)
MuZero (Schrittwiesee et al., 2020)
VAML Farahmand et al., 2017)
VaGrudM (Voelcker et al., 2022)
DAM (Haghgoo et al., 2021)
GAMPS (D'Oro et al., 2020)
VMBPO (Chow et al., 2020)
MNM Eysenbach et al,, 2022)
ALM (Ghugare et al., 2022)
MML (Volashin et al., 2021)
LAMPS (Vemula et al., 2023)
RAMBO (Rigter al., 2022)
TreeQN [Farquhar et al., 2018)
OMD (Nikishin et al., 2022)
DCEM (Amos & Yarats, 2020)
AMPL (Yang et al., 2022)
VE (Grimm et al., 2020)
TOM (Ma et al., 2023)
VPN (Oh et al., 2017)
PAML (Abachi, 2020)
20

---

Published in Transactions on Machine Learning Research (02/2024)
Optimization approaches and how to extract maximum information from novel objectives.
Downstream applications and re-using dynamics models.
Benchmarking and improving rigor in MBRL.
5.1 Decision-Aware Objective Design
The proposed taxonomy suggests a single principle decision-aware model and policy objective design: value optimization- -ew.uuttoeeeeuuueeeeeeeeeee where both the model and the policy should be trained to optimize the expected return in the real environment. The value optimization-equivalenee principle extends the value- equivalence principle, a previously proposed principle for MBRL (Grimm et. al., 2020; 2021; 2022) and also reviewed in Section 4.3.1, in two ways. First, it suggests that in addition to decision-aware model learning, which was proposed in VE, policy learning should also be decision-aware, or rather model-aware. Second, and differentiable planning) rather
model learning can focus on value optimization (e.g., control-as-inferee than just value or Bellman operator estimation (e.g., distribution correction and value-cquivalence). Below, we discuss how the value optimization-eqquiallae identify split among the reviewed approaches similar to the policy-based based vs. value-based and model-free vs. model-based paradigms in RL and whether these approaches are designed to explicitly handle errors occurring at different stages of the learning process.
principle is manifested in the reviewed approaches. We
Error-awareness: Error-aware approaches in the distribution correction and control-as-inferencc categories achieve value optimization iedimization-equivlleneee by explicitly modeling the errors simulated trajectories using distri- bution correction and adapting the model lcarning and policy optimization objective better approximato policy performance in the real enviromment. These error-moeeling schemes fit imperfect models to relevant data akin to locally weighted learning (Atkeson et al., 1997). Methods that only address model-shift, e.g., by redesigning policy objectives, can be more permissive towards the specifie model-training method. However, authors have also identified model-training objectives with better optimization behavior from the policy evolution, policy-gradient matching, variance reduction, or control-as-inferennee perspectives.
On the other hand, error-agnostic approaches achieve value-optimization equivalence by constructing equiv- alent classes of MDP dynamies, either explicitly as in value-equivalence or implicitly as in differentiable planning, such that evaluating policies in the learned dynamics is close to that of the true dynamics. These approaches simplify the training pipeline as they do not alter the standard model-based value estimation procedure, for example by adding density-ratio estimators and auxiliary model-based reward bonus in DAM and MNM.
However, many approaches in both the crror-aware and crror-agnostic categorics still do not explicitly control for policy-shift. (i.e., limiting policy divergence in (6)) or introduce policy-shift. to new agent. components. In error-aware approaches, the value pntimizaaiinn-equvallee estimator trained past data cannot generalize to new policies. Similarly, in error-agnostic approaches, minimizing the value- predictred loss on past data may not ensure small loss new data. Instead, policy- shift is implicitly addressed by taking small optimization steps and relying on fast alternation between model and policy updates (Ma et al., 2023), optimizing against the worst-case density-ratio (Voloshin et al., 2021), against the worst-case dynamics (Rigter et al., 2022), or introducing reward bonus (Haghgoo et al., 2021: Eysenbach et al., 2022; Yang et al., 2022).
property may be hindered if the density-ratio
Value-awareness: In RL, basec and value-based approaches are distinguished by whether a value function is necessary component of the algorithms, and policy- based approaches can operate on Monte Carlo return estimators rather than explicit value function estimators. Using this analogy, can classify distri- bution correction and differentiable planning as policy-based and control-as-inferennc and valno-oquivalence as value-based. Similar to policy optimization in RL, value-based approaches for model learning are tied to the quality of the value function estimator, which can require special care either in objective design (e.g., identifying the extrapolation errors of value function estimates (Voclcker et al., 2022; Farahmand et al., 2017: Voloshin et al., 2021)) or implementation (e.g., making sure the value function estimator is sufficiently accurate before applying to model learning (Fysenbach et al., 2022))-
21

---

Published in Transactions on Machine Learning Research (02/2024)
Model-awareness: The value optimization-eeuivalenee principle also implics a hybridization of model- free and model-based RL, approaches, which is manifested in most reviewed works. The traditional view of model-based approaches is to build an as accurate as possible model of the environment. The value optimizatoon-qquiallonee principle suggests to instead stay as close as possible to model-free RL in the sense that sampling from the model or evaluating samples from the model is close to having samples drawn from the true environment. This is intuitive because model-free RL evaluates returns on true samples; this is as if we were running MBRL with privileged knowledge of the true dynamics. However, there are still several advantages of MBRL that are beyond just having better environment sample complexity. As Gehring et al. (2021) suggested, value opéinization-qquivllntt MBRL likely enjoys optimization advantages due to the relationship with accelerated gradient descent methods. Value ietimiization-quivlleee MBRL may also enjoy exploration and safety advantages depending on whether and in what part of the stale-action space the learned dynamics is optimistic or pessimistic.
5.2 Properties of Decision-Awar Agents
Decision-aware MBRL methods can mirror the enactive view of cognitive science (Di Paolo & Thompson, 2014), where the role of perception is not to build a true model of the environment but to serve as an interface between the agent and the environment to enable flexible and adaptive behavior. Under this view, the learned model is free to deviate from the environment. It is thus useful to understand how learned models deviate from the true model, how agents behave during the learning process, and implications for downstream applications.
The fullest picture is depicted by Vemula et al. (2023) who showed in an exact decomposition of the return gap that the learned model is optimistic in the in-distribution region of the state-action space and pessimistic in the out-of-distributiion region. The adversarial model learning approach by Rigler el al. (2022) adopts the pessimistic perspective, but instead learns objective (accurate) model in the in-distribution region. Agents trained with this objective will likely be more conservative in their exploration behavior and less prone to model-exploitation. In contrast, mntrrolas-inferenn approaches learn an optimistic model of the dynamics in the in-distribution region, and they do not explicitly model out-of-distribution behavior in the model learning objective. Instead, control-as-inferennd learns conservative behavior in the policy-optimization process by encouraging agents to follow well-predicted states and only deviate when additional information about the dynamics model can be gained. These agents will likely exhibit more exploratory behavior than robust control agents in more structured manner. For example, Eysenbach et al. (2022) found that their agent tends to make the goal positions appear closer than they actually are in model rollouts so that the goals are easier to reach, which is beneficial in sparse reward settings to accrue more feedback for the policy, Control-as-inferennce agents also represent an interesting point of contact with recent information-theore approaches to agent objective design in both task-driven and unsupervised RL (Hafner et al., 2020; Rhinchart et al., 2021).
A substantial question for value optimizatiion-equivaleee agents is whether the true environment model can be identified. The findings of Nikishin et al. (2022) suggest that the true environment model is not identifiable given that two different dynamics models can lead to the same optimal policy. This equivalence is related to the research on state-abstraction in MDPs where functionally similar states are aggregated (in the current case ignored) in order to avoid modeling task-irrelevant state features (Li et al., 2006). One way to improve identifiability is to add an environment or reward prediction loss as in (Schrittwieser et al., 2020; 2021), however, this introduces a trade-off between value-equivalent vs. accurate models which has to be specified by the modeler. Nikishin et al. (2022) suggested that unidentiifability is likely blessing rather than a curse because there are likely value-equivalent models that are casier to learn than the true model. However, Wei et al. (2023) showed in inverse RL setting with differentiable planners that favorable robustness properties of valuo-oquivalent models may succumb to overly high inaccuracy. They further suggested regularizing the model's slate-prediction accuracy. As noted by several authors, the set of va value-equivalent models reduces with increasing number of policies and values considered (Grimm et al., 2020; Voloshin et al., 2021). This suggests that identifiabiliyy is possible in a multi-task or meta-learning sctting and an experimental validation is provided in (Berke et al., 2022) from a cognitive science perspective. However, currently most works on decision-aware MBRL has focused on the single-task setting.
22

---

Published in Transactions on Machine Learning Research (02/2024)
5.3 Optimization Approaches
In order to gain the full benefit from novel objective formulations designed to solve objective mismatch, dif- ferent optimization techniques may be needed. We highlight two optimization approaches from the survey. The majority of reviewed works use manually designed objectives for model learning and policy optimization mostly by bounding the true return and leveraging existing optimization techniques such policy-gradient and adversarial training. However, certain properties of decision- aware agents can be lost in manual for- mulation and optimization. For example, optimistic or pessimistic behavior characteristic of decision-aware agents is lost in models learned using value-prediction In contrast to manual design of component objec- lives, differentiable planning approaches use a single objective of the true return and offload the complexity of optimization to differentiable programming. These approaches mostly take on a bilevel-optimization for- mat, where update of both the model and the policy need to take into account the optimal policy with respect to the current model. To this end, earlier differentiable planners such as (Farquhar et al., 2018; Amos & Yarats, 2020) relied on finite-horizon or truncated- horizon policy objectives where the gradients can be computed using backpropagation. More recent approaches such as (Nikishin et al., 2022; Bansal et al., 2023) have explored alternative optimization techniques such as implicit differentiation which can more precisely compute the gradient of infinite-horizon policies. However, the authors have also commented on the added complexity and approximation to these approaches. While manually designed optimization is advantageous for understanding the propertics of decision-aware agents, more efficient end-to-end optimization approaches may be desirable for scaling and broadening application domains.
5.4 Downstream Applications
An important use case for MBRL is transfer learning, where the dynamics model learned in source task can be used an environment simulator for target task to reduce or eliminate the number of environ- ment samples (Taylor & Stone, 2009; Zhu et al., 2023). For traditional environment prediction-bar based model learning approaches, transferring would simply require the model to generalize in state-action space covered by the target task. However, transfer learning potentially presents a larger challenge for value optimization- equivalent agents because of model unidentifiabillty and the fact that decision-aware models are usually biased. In this setting, it is not clear whether model bias, especially the optimistic bias, which aids opti- mization in the source task is also help the target task. However, given that model-identiffabiiit can be improved with multi-task training, decision-aware agents may be especially suited for tasks that require con- tinuous model fine-tuning or adaplation such as in some instances of meta RL (Nagabandi et al., 2018b;a) and lifelong learning (Khetarpal et al., 2022). Within a limited set of multi-task decision- aware MBRL works, Tamar et al. (2016) and Karkus et al. (2017) showed that differentiable planners trained on a varicty of imitation tasks led to promising task transfer capabilities. However, Karkus et al. (2017) observed that multi-task training did not contribute to learning more accurate models and the models were still "wrong yet useful", On the other hand, Hansen et al. (2023) achieved an almost two-fold improvement after fine-tuning the TD-MPC model pretrained on significant 80 RL tasks but did not examine the accuracy of the learned model. Thus, the relationship between model accuracy, task transfer, and decision-aware MBRL is still an open question.
Another important utility of MBRL is to enhance the transparency and explainability of RL agents through the separation of model and policy, where designers can introspect the learned model to understand agent be- havior and potentially correct agent failures. An important aspect of transparency is that the agent designer can easily comprehend and identify sub-optimalities in the model and make precise editing decisions (Räuker et al., 2023). The biases and unidentifiability in decision-aware agents introduce significant challenges for model comprehension since the true environment is no longer the ground truth or the optimization target and it may not be appropriate to correct the model towards the ground truth only on some identified states but not others since it may alter the learned value-equivalence.
These application considerations suggest a nced to better understand the properties of decision-aware MBRI agents, such as value-equivalent MDPs, in order to reap their benefits without losing that of traditional MBRL.
23

---

Published in Transactions on Machine Learning Research (02/2024)
5.5 Evaluations and Benchmarks
While resolving objective mismatch focuses on designing novel agent objectives or training procedures, the qualities of the objectives should be measured based on agent behavior. Since the ultimate goal of decision- aware agents is to achieve high returns, the final performance and learning speed (i.e., the number of enviromment steps to reach performance threshold) are the primary evaluation metrics which have been used by most reviewed decision-aware MBRL. works. However, more fine-grained evaluation of decision- aware MBRL should probe its advantages over traditional MBRL and model-frce RL, most importantly, robustness to model misspecification and model-exploitation in both online and offline RI, which are the top two motivators for most reviewed methods backed by theory (6). Many reviewed works such as (D*Oro al., 2020; Eysenbach et al., 2022; Farahmand et al., 2017; Voelcker al., 2022; Abachi, 2020; Nikishin et al., 2022; Grimm et al., 2020) compared agent performance under different levels of model misspecificatinn by varying the capacity of the dynamics models, removing state features, or adding distracting state features (see Table 2). Evaluation protocols of model-exploitation have also been developed in recent years by measuring value- estimation bias (Chen et al., 2021; Fujimoto et al., 2018). We recommend future work on decision-aware MBRL to include these experiments.
Beyond model misspecification and exploitation, some authors also probed more specific agent properties (e.g., exploration behavior). For example, Eysenbach et al. (2022) and Rigter et al. (2022) assessed the optimism and smoothness, respectively, of their learned dynamics models through visualizations in selected environments. Since these properties do not necessarily apply to all decision-aware MBRL approaches, differ- ent works performed different evaluations in different enviromments, which makes comparison of approaches challenging if not incomplete. Thus, as our understanding of the properties and utilities of decision-awarr agents continues to develop and mature, we may benefit from documenting the agent properties that cach environment is designed test and developing behavior suites (Osband al., 2019) for decision-aware agents.
The consistency between theory and implementation in decision-aware MBRL also warrants additional at- tention. Several works have remarked on the theory-imllementation gap (Eysenbach et al., 2022; Modhe et al., 2021; Lovatto et al., 2020), which are mostly due to additional moving components and optimization challenges (e.g., requirements on value estimation accuracy as a result of including value estimates in model training objectives). Thus, evaluations of decision-aware MBRL should focus on not only the final perfor- mance but also implementation easiness, training stability, debugging tools (Lambert, 2021), sharing trained models (Pineda et al., 2021), and other optimization failure modes.
6 Conclusion
While resolving objective mismatch promises to boost the capability MBRL agents, how to design aligned objectives for different agent components remains an open question. To this end, we found that all current efforts to address the objective mismatch problem can be understood along the lines of 4 major categories: distribution correction, control-as anntrol-a-iiffernnce inference, malue-equivalencee, and differentiable planning. These efforts point to single principle for designing decision-aware objectives: value apiimization-qquiulleeee. Under this principle, both the dynamics model and the policy should be trained to optimize the expected return, effectively achieving a hybridization of model-free and model-based RL. We recommend future work to continue to enhance our understanding of decision-aware agents, identify their practical utilities, and design appropriate evaluation suites to fully harvest their benefits.
Broader Impact
This paper synthesizes theoretical and empirical results for solving objective mismatch Lowards building more capable MBRL agents. RL and automated decision-making system can have important ramifications, especially when directly interfacing with humans. A major subset of these ramifications stems from misspoc- ified reward and training environments. While we have focused on correctly specified reward and training environments, we remark that identifying and debugging these misspecifications in decision-aware MBRI.
24

---

Published in Transactions on Machine Learning Research (02/2024)
agents is likely harder than traditional RL agents. We believe developing better theoretical understand- ing and empirical testing suites are a first step towards the transparency required to mitigate harms from awaae MBRL agents.
Acknowledgments
This work was partly supported by the German Research Foundation (DFG, Deutsche Forschungsgemein- schaft) as part of Germany's Excellence Strategy EXC 2050/1 Project ID 390696704 Cluster of Fx- cellence "Centre for Tactile Internet with Human-in-the-Loopp (CeTI) of Technische Universität Dresden, and by Bundesministerium für Bildung und Forschung (BMBF) and German Academic Exchange Service (DAAD) in project 57616814 (SECAT, School of Embedded and Composite AT).
References
Romina Abachi. Policy-aware model learning for policy gradient methods. University of Toronto (Canada), 2020.
David Abel, Nate Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and Michael Littman. Value preserving state-action abstractions. In International Conference on Artificial Intelligence and Statistics, pp. 1639-1650. PMLR, 2020.
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conference on robol learning, pp. 1300 1313. PMLR, 2020.
Brandon Amos and Denis Yarats. The differentiable cross-entropy method. In International Conference on Machine Learning, pp. 291-302. PMLR, 2020.
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. Differentiable mpe for end-to- end planning and control. Advances in neural information processing systems, 31, 2018.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamicss and Control, pp. 20. PMLR, 2021.
Kavosh Asadi, Evan Cater, Dipendra Misra, and Michael L Littman. Equivalence between wasserstein and valuo-aware loss for model-based reinforcement learning. arXiv preprint arXiv:1806.0 01265, 2018.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, pp. 11-73, 1997.
Alex Ayoub, Zeyu Jia, Csaba. Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, pp. 463 474. PMLR, 2020.
Pierre-Luc Bacon, Florian Schäfer, Clement Gehring, Animashree Anandkumar, and Emma Brunskill. A lagrangian method for inverse problems in reinforcement learning. In Optimization in RL workshop al NewrIPS, volume 2019, 2019.
Dishank Bansal, Ricky TQ Chen, Mustafa Mukadam, and Brandon Amos. Taskmet: Task-driven metric learning for model learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Somil Bansal, Roberto Calandra, Ted Xiao, Sergey Levine, and Claire Tomlin. Goal-driven dynamics learning via bayesian optimization. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pp. 5168 5173. IEFE, 2017.
25

---

Published in Transactions on Machine Learning Research (02/2024)
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon White- son. A survey of metaa-reinforoement learning. arXiv preprint or NNu:2301008022 2023.
Marlene D Berke, Robert Walter-Terrill. Julian Jara-Ettinger, and Brian J Scholl. Flexible goals require that inflexible perceptual systems produce veridical representations: Implications for realism as revealed by evolutionary simulations. Cognitive Science, 46(10):e1319,, 2022.
Veronica Chelu, Doina. Precup, and Hado P van Hasselt. Forethought and hindsight in credit assignment. Advances in Newral Information Processing Systems, 33:2270-2283 1770002281,220 2020.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Representations, 2021.
Yinlam Chow, Brandon Cui, MoonKyung Ryu, and Mohammad Ghavamzadeh. Variational model-based policy optimization. arXiv preprint arXiv:2006.00 05448, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Lancelot Da Costa, Thomas Parr, Noor Sajid, Sebastijan Veselic, Victorita Neacsu, and Karl Friston. Active inference on discrete stato-spaces: A synthesis. Journal of Mathematical Psychology, 99:102447, 2020.
Lancelot Da Costa, Noor Sajid, Thomas Parr, Karl Friston, and Ryan Smith. Reward maximization through discrete active inference. Neural Computation, 35(5):807-852, 2023.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414 419, 2022.
Ezequicl Di Paolo and Evan Thompson. The enactive approach. The Routledge handbook of embodied cognition, pp. 68-78, 2014.
Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli. Gradient- aware model-based policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp. 3801-3808, 2020.
Finale Doshi-Velez and George Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In IJCAI: proceedings of the conference, volume 2016, pp. 1432. NIH Public Access, 2016.
Michael O'Gordon Duff. Optimal Learning: Computational procedures for Bayes- Markov decision processes. University of Massachusetts Amherst, 2002.
Adam N Elmachtoub and Paul Grigas. Smart "predict, then optimize". Management Science, 68(1):9 26, 2022.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems, 34:27813 27825, 2021.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Russ R Salakhutdinov. Mismatched no more: Joint polecy optimization for model-based rl. Advances in Neural Information Processing Systems, 35:23230-2333 232443, 2022.
Amir-massoud Farahmand. Iterative value-aware model learning. Advances in Neural Information Processing Systems, 31, 2018.
Amir-masson Farahmand, Andre Barreto, and Daniel Nikovski. Valuo-aware loss function for model-basee reinforcement learning. In Artificial Intelligence and Statistics, pp. 1486-194, PMLR, 2017.
26

---

Published in Transactions on Machine Learning Research (02/2024)
Gregory Farquhar, Tim Rocktäschel, Maximilian Igl, and Shimon Whiteson. Treequ and atreec: Differen- tiable tree-structured models for deep reinforcement learning. In International Conference on Learning Representations, 2018.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, al. Dis- covering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
Zafeirios Fountas, Noor Sajid, Pedro Mediano, and Karl Friston. Deep active inference agents using monte- carlo methods. Advonces in neural information processing systems, 33:11662-11675, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. DArl: Datasets for deep data- driven reinforcement learning. arXiv preprint Stl::0040002199, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018
Clement Gehring, Kenji Kawaguchi, Jiaoyang Huang, and Leslie Kaelbling. Understanding end-to-ent model- based reinforcement learning methods as implicit parameterization, Advances in Neural Information Pro- cessing Systems, 34:703 714, 2021.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, al. Bayesian reinforcement learning: A survey. Foundalions and Trends0 in Machine Learning, 8(5-6):359 483, 2015.
Raj Ghugare, Homanga Bharadhwaj,, Benjamin Eysenbach, Sergey Levine, and Russ Salakhutdinov. Simpli- fying model-based rl: Learning representations, latent-space models, and policies with one objective. Tn The Eleventh International Conference on Learning Representations, 2022.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning, pp. 590- MIT press, 2016.
lan Goodfellow, Jean Abaget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139 139-144, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-211 212244, 2020.
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for masel-based reinforcement learning. Advances in Neurol Information Processing Systems, 2020.
Christopher Grimm, André Barreto, Greg Farquhar, David Silver, and Satinder Singh. Proper value equiv- alence. Advances Neurul Information Processing Systems, 34:7773 7786, 2021.
Christopher Grimm, Andre Barreto, and Satinder Singh. Approxinate value equivalence. Advances in Newral Information Processing Systems, 01333099-33444 2022.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint Xfo::1882. 05900, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Danijar Hafner, Pedro A Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception divergence minimization. Xix preprint 00:::200900199: 2020.
27

---

Published in Transactions on Machine Learning Research (02/2024)
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. orXiv preprinl arXi::2301.04 104, 2023.
Behizad Haghgoo, Allan Zhou, Archit Sharma, and Chelsea Finn. Discriminator augmented model-based reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint Xrx:::2000. 16828, 2023.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387 8406. PMLR, 2022.
Toru Hishinuma and Kei Senda. Weighted model estimation for offline model-based reinforcement learning. Advances in neural informalion processing systems, 34:17789 17800, 2021.
Abraham Imohiosen, Joe Watson, and Jan Peters. Active inference control as inference? unifying view. In Active Inference: First International Workshop, IWAI 2020, Co-located with ECML/PKDD 2020, Ghent, Belgium, September 14, 2020, Proceedings 1, pp. 19.19. Springer, 2020.
Taher Jafferjee, Ehsan Imani, Erin Talvitie, Martha. White, and Micheal Bowling. Hallucinating value: A pitfall of dyna-style planning with imperfect environment models. arXiv preprint ar Xiv:200066 04363, 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model:
Model-based
policy optimization. Advances in newral information processing systems, 32, 2019.
Michael Janner, Igor Mordatch, and Sergey Levine. gamma-models: Generative temporal difference learning for unfinite-horizon prediction. Advances in Neural Information Processing Systems, 33:1724-177 1733,, 2020.
Joshua Joseph, Alborz Geramifard, John W Roberts, Jonathan How, and Nicholas Roy. Reinforcement learning with misspecified model classes. In 2013 IEFF International Conference Robotics and Au- tomation, pp. 939 946. IEEE, 2013.
Peter Karkus, David Hsu, and Wee Sun Lee. Qudp-net: Deep learning for planning under partial observ- ability. Advances in neural informalion processing systems, 30, 2017.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49:209-230 2002.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810 21823, 2020.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 1191, 2020.
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford. Guaranteed discovery of controllable latent states with multi-step inverse models. arXio preprint ar orXin:2207.0822 2022.
Nathan Lambert. Debugging model-based reinforcement learning systems. 2021.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model- based reinforcement learning. In Learning for Dynamics and Control (L4DC), pp. 761-770, 2020. URL https://arxiv.ort 200. 04523.
Nathan Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learning accurate long-term dynamics for model-based reinforcement learning. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 2880-20 2887. IEFE, 2021.
28

---

Published in Transactions on Machine Learning Research (02/2024)
Nathan Lambert, Kristofer Pister, and Roberto Calandra. Investigating compounding prediction errors in learned dynamics models. arXic preprint arXiv:2203.0000 09637, 2022.
Nir Levine, Yinlam Chow, Rui Shu, Ang Li, Mohammad Ghavamzadeh, and Hung Bui. Prediction, con- sistency, curvature: Representation learning for locally-lincar control. arXiv preprint arXiv: 15:9:0000001100 000000 2019.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprinl Xix: 00000009,000,, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offine reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXi::2005.01655 2020.
Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. AIBM, 2006.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wwXiv:1509002971 2015.
Angelo Gregório Lovatto, Thiago Pereira Bueno, Denis Mauá, and Leliane Nunes de Barros. Decision-awaad aware model learning for actor-critic methods: When theory does not meet practice. In "I Can't Believe It's Not Netrlr! 2020 workshop, 2020.
Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-basod reinforcement learning. arXiv preprint Xix:2220 09328, 2022.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic frame- work for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2018.
Yecheng Jason Ma, Kausik Sivakumar, Jason Yan, Osbert Bastani, and Dinesh Jayaraman. Learning policy- aware models for mosel-based reinforcement learning via transition occupancy matching. In Learning for Dynamics and Control Conference, pp. 259 271. PMLR, 2023.
Rowan McAllister, Blake Wulfe, Jean Mereat, Logan Ellis, Sergey Levine, and Adrien Gaidon. Control- aware prediction objectives for autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA), pp. 01 08. IEEE, 2022.
Beren Millidge. Deep active inference as variational policy gradients. Journal of Mathematical Psychology. 96:102348, 2020.
Beren Millidge, Alexander Tschantz, Anil K Seth, and Christopher L Buckley. On the relationship be- tween active inference and control as inference. In Active Inference: First International Workshop, TWAT I 2020, Co-located with ECML/PKDD 2020, Ghent, Belgium, September 14, 2020, Proceedings 1, pp. 11. Springer, 2020.
Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Whence the expected free energy? Neural Compulation, 437244447-4 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXie preprint arXin:1312.5000 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andrcas K Fidjcland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(754)):529 533, 2015.
Nirbhay Modhe, Harish K Kamath, Dhruv Batra, and Ashwin Kalyan. Bridging worlds in reinforcement learning with model-advantage. In 4th Lifelong Machine Learning Workshop al. ICML 2020, 2020.
29

---

Published in Transactions on Machine Learning Research (02/2024)
Nirbhay Modhe, Harish Kamath, Dhruv Batra, and Ashwin Kalyan. Model-advantage and value-aware models for model-based reinforcement learning: Bridging the gap in theory and practice. orXiv preprinl arXiv:2106.14008 2021.
Thomas M. Mocrland, Joost Brockens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trendoo in Machine Learning. 16(1):1 118, 2023.
Ofrr Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXi::201.008866, 2020.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning, In International Conference on Learning Represenlations, 2018a.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine Deep online learning via meta-learning: Continual adaptation for model- based rl. In International Conference on Learning Representations, 2018b.
Evgenii Nikishin, Romina Abachi, Rishabh Agarwal, and Pierre-Lue Bacon. Control-oriented model-based reinforcement learning with implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7886-7894, 2022.
Junhyuk Oh, Satinder Singh, and Honglak Tee. Value prediction network. Advances in neural information processing systems, 30, 2017.
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-cnd differentiable optimal control. arxie preprint Xiv: 706.09597, 2017.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2019.
Daniel Palenicek, Michael Lutter, Joao Carvalho, and Jan Peters. Diminishing return of value expansion methods in model-based reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Luis Pineda, Brandon Amos, Amy Zhang, Nathan Lambert, and Roberto Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. arXiv preprint arXiv:2104.10159, 2021.
Kale Rakelly, Abhishek Gupta, Carlos Florensa, and Sergey Levine. Which mutual-information represen- tation learning objectives are sufficient for control? Advances in Newral Information Processing Systems, 34:26345-26357, 2021.
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan HadfieldMMenlll. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trushworthy Machine Learning (SaTMT.), pp. 464-483. IEFE, 2023.
Nicholas Rhinehart, Jenny Wang, Glen Berseth, John Co-Reyes, Danijar Hafner, Chelsea Finn, and Sergey Levine. Information is power: intrinsic control via information capture. Advances in Neural Information Processing Systems, 34:10745-10758. 2021.
Spencer M Richards, Jean-Jacques Slotine, Navid Azizan, and Marco Pavone. Learning control-oriented dynamical structure from data. arXiv preprint arXiv:2302.02525 2023.
Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-basod offline reinforce- ment learning. In Advances in Neural Information Processing Systems, 2022.
Noor Sajid, Philip J Ball, Thomas Parr, and Karl J Friston. Active inference: demystified and compared. Neural computation, (113))6774713 2021.
30

---

Published in Transactions on Machine Learning Research (02/2024)
Julian Schrittwieser, Ioannis Antonoglon, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barckatain, Joannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34:27580-211 275991 2021.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence wet policy gradients and soft q-learning. arXiv preprint arXii::104..000 704.06440, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- rrXxi:1707 1700:777.000 707.06337, 2017b.
tion algorithms. arXiv preprint arv;v
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data- efficient reinforcement learning with self-predictive representations. In International Conference on Learn- ing Representations, 2020.
Abhishek Sharma, Sonali Parbhoo, Omer Gottesman, and Finale Doshi-Velez. Robust decision-focused learning for reward transfer. arXiv preprint 2xXx::2304.03365, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nalure, 529(7788):484 489, 2016.
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191 3199. PMLR, 2017a.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXie preprint arXi::1712.0188 2017b.
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Arlificial Intelligence, 299:103535, 2021.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques E Slotine, and Marco Pavone. Learn- ing stabilizable nonlinear dynamics with contraction-baset regularization. The International Journal of Robotics Research, 40(10-11):1123-111 -1500, 2021.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp. 4732 4741. PMLR, 2018
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning partially observed systems. The Journal of Ma- chine Learning Research, 23(1):483-56) 565, 2022.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Densily rulio estimation in machine learning. Cambridge University Press, 2012.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In Artificial General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3-6, 2011. Proceedings 4, pp. 41-51. Springer, 2011.
Richard S Sutton and Andrew G Barlo. Reinforcemenl learning: An introduction. MIT press, 2018.
Erik Talvitie. Self-correcting models for model-based reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
31

---

Published in Transactions on Machine Learning Research (02/2024)
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. Advances in neural information processing systems, 29, 2016.
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chandak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, et al. Understanding self-predictive learning for reinforcement learning, In International Conference Machine Learning, pp. 33632-33656 PMLR, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv:181100000 .00690, 2018.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey, Journal of Machine Learning Research, 10(7); 2009.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial cov- orage. arXiv preprint arXiv:2107.06228 2021.
Jasper der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. Contrastive expla- nations for reinforcement lcarning in terms of expected consequences. arXiv preprint 2018.
0876,
Anirudh Vemula, Yuda Song, Aarti Singh, Drew Bagnell, and Sanjiban Choudhury. The virtues of laziness in model-based rl: A unified objective and algorithms. In International Conference on Machine Learning, pp. 34978 35005. PMLR, 2023.
Claas Voelcker, Arash Ahmadian, Romina Abachi, Igor Gilitschenski, and Amir-massoud Farahmand. X- ac: Learning latent decision-aware models for reinforcement learning in continuous state-spaces. ar preprinl X3v: 17366, 2023.
Claas A Voelcker, Victor Liao, Animesh Garg, and Amir-massoud Farahmand. Value gradient weighted model-based reinforcement learning. In International Conference Learning Representations, 2022.
Cameron Voloshin, Nan Jiang, and Yisong Yue. Minimax model learning. In International Conference on Artificial Intelligence and Statistics, pp. 1612 1620. PMLR, 2021.
Kai Wang, Sanket Shah, Haipeng Chen, Andrew Perrault, Finale Doshi-Velez, and Milind Tambe. Learn- ing mdps from features: Predict-then-optimiz for sequential decision making by reinforcement learning. Advances in Newral Information Processing Systems, 34:8795 8806, 2021.
Xiyao Wang, Wichayaporn Wongkamjan, Ruonan Jia, and Furong Huang. Live in the moment: Learning dynamics model adapted to evolving policy. In International Conference on Machine Learning, pp. 36470 36493. PMLR, 2023.
Stefan Webb, Adam Golinski, Rob Zinkov, Tom Rainforth, Yee Whye Tch, Frank Wood, et al. Faithful inversion of generative models for effective amortized inference. Advances in Neural Information Processing Systems, 31, 2018.
Ran Wei, Siliang Zeng, Chenliang Li, Alfredo Garcia, Anthony McDonald, and Mingyi Hong. A bayesian approach to robust inverse reinforcement lcarning. In Conference on Robot Learning. PMLR, 2023.
Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1665, 2019.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33:15737-15749, 2020.
32

---

Published in Transactions on Machine Learning Research (02/2024)
Shentao Yang, Shujian Zhang, Yihao Feng, and Mingyuan Zhou. A unified framework for alternating offline model training and policy learning. Advances in Neural Informalion Processing Systems, 35:17216-177 111222 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Mcta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robol learning, pp. 1094 1100. PMLR, 2020a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 38:14129-14142, 2020b.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954 28967, 2021.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning, arvi preprint arXiv:1804.1000 2018.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2020.
Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Trunsactions Pottern Analysis and Machine Intelligence, 2023.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian 1gl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learnig. In International Conference Learning Represendations, 2019.
33


Reviews:
[{"review_id": "7nIyrbLMHf", "reviewer": "Reviewer_uYW8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The classification of previous approaches into the four categories is very useful to gain an understanding of the field. \n- The paper is well-written, and flows nicely between sections. \n- Each approach discussed is explained in a sufficient level of depth.\n\n# Weaknesses\n- The notation $P$ is overloaded as both the transition operator of the MDP and the symbol for an underlying probability (e.g. $P(\\tau)$, $P(a_t|s_t)$, $P(\\mathcal{O}_t)=1$, among others). A bit more care with the notation would make it easier for people to understand. \n- The term \"homomorphic\" appears in Section 4 with no prior definition: \"Value-Equivalence searches for models that are homomorphic to the true environment and value function.\" Homomorphisms are a well-established concept in algebra (and more recently RL) but this seems to use neither definition. I would suggest the authors to either define this term as used in their work or use another term to avoid confusion. \n- Almost all losses/objectives are not written properly mathematically. For example, equation (5) states $\\max_{\\hat{P}} L_{MLE}(\\hat P) = \\mathbb{E}_{s,a,s'}[\\log(\\hat{P}(s'|s,a) )] $ -- I know that the authors are trying to convey that we want to maximize the RHS, but as it is currently written it is not mathematically correct, and is likely to cause confusion (it at least caused confusion for me in the later sections).\n- \"Then the optimal policy....\" This language suggests that the optimal policy is unique.\n- Very minor: \"f-divergence\" => \"$f$-divergence\".\n- \"We refer to the process of finding the optimal policy for an MDP, including these variations, as policy optimization\" - the authors can use any terms they please, however policy optimization is commonly reserved for policy-based methods in RL and using this terminology is likely to confuse readers who don't read the background closely.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is a survey to approaches related to addressing the objective-mismatch problem in model-based RL. They group the previous approaches into four broad categories, and provide an overview of each category and the high-level approaches used for each. They finally provide a lengthy discussion, and suggest a unified principle of value optimization-equivalence: the joint training of the model and the policy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the weaknesses highlighted above are addressed (I think they are mostly minor and will help readers understand the paper).", "broader_impact_concerns": "The paper's broader impact statement seems sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aPJMnnX3KX", "reviewer": "Reviewer_e1y7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does a good job contextualizing and summarizing various approaches for objective mismatch. This work puts forth a reasonable categorization of these methods and should provide a good entry point for researcher wanting to learn more about objective mismatch in RL.\n\nThe writing is clear and the flow is good. The notation of the summarized work has been standardized and made consistent across the whole survey.\n\nI overall enjoyed reading this paper and only have superficial comments and corrections which I discuss below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a summary of recent work related to objective mismatch in model-based reinforcement learning (MBRL). The collection of work is broken down into 4 categories, distribution correction, control-as-inference, value-equivalence, and differentiable planning. Contributions of each paper are summarized and compared to each other. The paper concludes with a significant discussion about various components of these methods for objective mismatch in MBRL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Necessary corrections\n\nEq. (33) to (37) are missing \"min\" on the rhs of the equation.\n\np.16, \"Empirically, the authors showed that OMD [...]\", this is ambiguous. Which authors? I presume this refers to Nikishin et al. but other work was cited since they were referenced last.\n\n# Recommendations\n\nOverall, I recommend making the distinction between optimal state/state-action value functions, state/state-action value functions and approximate state/state-action value functions explicit with separate notation, e.g., $Q^*, Q^\\pi, \\hat{Q}^*,  \\hat{Q}^\\pi$.\n\nThe paper might benefit from a summarization of the motivations of the compared methods, possibly in a similar style as tables 1 and 2 or, alternatively, as a few paragraphs in the discussion section. For instance, some papers seem to have a focus on the offline RL setting and some more on model-misspecification. Summarizing this information would help guide readers to the papers most relevant to the motivation behind their current research. Though it's also possible that there isn't enough differences between the stated (or inferred) motivations of each paper to be worthwhile, but it would be a nice addition if there is.\n\nEq (13), In what sense is $\\tilde{Q}$ a dual? Maybe add a sentence or two to clarify.\n\nSec 4.2, in what way is $\\mathcal{O}$ representing \"optimality\"? Is there any justification or explanation that could be summarized and included in this survey?\n\nbottom p. 11, \"While theoretical understanding of self-predictive objectives is still lacking [...]\", Subramanian, et al. [1] provide some theoretical discussion about self-predictive objectives and provide conditions under these latent variables (which they call information states) satisfy a form of value-equivalence.\n\nControl-as-inference summary, It's not clear to me what exactly is meant by hand-designed.\n\nSec 4.3.2 felt generally too sparse on details and discussion. For instance, in the robust control summary, It's wasn't clear to me where the discussion about pessimism came from.\n\np. 16, the name OMD was taken from Bacon, et al. [2]. In addition, it would be good to discuss this work as they take a different approach to implicit differentiation and instead consider using the Lagrangian to solve the Bi-level optimization problem.\n\np. 20, \"we can classify distribution correction and differentiable planning as policy-based [...]\", I'm not sure I would classify differentiable planning as being only policy-based or value-based. Some differentiable planning methods (e.g., OMD) learn a state-action value function.\n\np. 21, \"decision-aware MBRL has focused on the single-task setting\" & \"transfer learning presents a larger challenge for value optimization-equivalent agents\", I would argue the focus of value iteration networks (Tamar et al., 2016) and QMDP-Net (Karkus et al., 2017) is multi-task/transfer learning. The motivation for these approach is to generalize over task specification (e.g., a floor map).\n\n# References\n\n[1] Subramanian, J., Sinha, A., Seraj, R., & Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1), 483-565.\n\n[2] Bacon, P. L., Schäfer, F., Gehring, C., Anandkumar, A., & Brunskill, E. (2019). A Lagrangian method for inverse problems in reinforcement learning. In Optimization in RL workshop at NeurIPS (Vol. 2019).", "broader_impact_concerns": "Broader impact seems sufficiently discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4qriLY9C3i", "reviewer": "Reviewer_aW56", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is very comprehensive in recent literature on decision-aware model-based reinforcement learning.\n\n2. This paper is well-written and free of grammatical errors. The presentation for the main surveyed algorithms is very clear and to the sufficient level of details.\n\n3. The summary and comparison for various different algorithmic categories are well motivated. \n\nWeaknesses:\n1. This paper primarily surveys only very recent papers, with most cited algorithms dating later than 2020. I believe there are several lines of classical literature on the subject that are worth mentioning. For example, there is a line of methods known as \"locally weighted learning\" or \"locally weighted regression\" in the 90s that have very similar flavor for model learning as the \"distribution correction\" approaches this survey studies. \n\nLocally weighted learning: https://www.ri.cmu.edu/pub_files/pub1/atkeson_c_g_1997_1/atkeson_c_g_1997_1.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a literature survey on the topic of objective mismatch in model-based reinforcement learning. Objective mismatch is the phenomenon that the dynamics model in MBRL is trained for a different objective and not aware of its role in the policy decision-making and learning; consequently, more accurate dynamics models often hurt policy performance. This paper's main contribution is a taxonomy of four categories of decision-aware MBRL approaches: Distribution Correction, Control-As-Inference, Value-Equivalence, and Differentiable Planning, which derive modifications to the model learning, policy optimization, or both processes for the purpose of aligning model and policy objectives and gaining better performance. Comparisons among different categories are presented and contextualized. Overall, this is a high-quality and sound survey of the literature on objective mismatch in MBRL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add some historical perspectives on objective mismatch in model learning, such as discussions on Locally Weighted Learning algorithms and their connections to the recent literature. While I believe this survey is high-quality and worthy of acceptance, I also do believe that proper historical contextualization and citations are important for a survey paper.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aDtrpzaYWZ", "reviewer": "Reviewer_ic6a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper strikes a good balance between the depth and breadth of the review\n- The identified clusters of approaches provide a valuable vocabulary and a map for navigating the field\n- The insights given at the end of each subsection and in the Discussion section provide a number of sharp and clear distinctions and concepts for delineating different approaches\n\n## Weaknesses\n### 1. Scope and Related Work\nThe chosen scope is somewhat restrictive and might need further clarification. The paper explicitly states that it collected the papers for review \"by examining articles citing (Lambert et al., 2020)\". However, the problem of learning a dynamics model and optimizing a policy has been studied quite extensively within *adaptive optimal control* or *dual control*. Perhaps stating in the Related Work section how the considered methods fit into the bigger picture would be helpful.\n\nThe paper understandably had to limit the amount of related work considered. However, I feel it should at least mention the omitted prominent approaches. For example, *active inference* is notorious for arguing for learning the model and the controller jointly, both of which are minimizing a single objective function — the variational free energy (see e.g. [1]). Furthermore, meta-learning or Bayes-adaptive MDPs are also approaching the same problem, e.g., [2].\n\n\n### 2. Framing\nCurrently, the paper relies on \"objective mismatch\" introduced by (Lambert et al., 2020) as the starting point — an observation that the model learning objective (e.g., maximum likelihood) is different from the policy learning objective (maximizing return). The referenced paper argues that the \"objective mismatch\" is the **cause** of the phenomenon that a better accuracy in one-step model prediction does not necessarily lead to better policies. However,\n1) it does not seem to be a non-controversial fact that it is the \"objective mismatch\" that is the cause of the observed phenomenon (e.g., see the discussion of the referenced paper on OpenReview https://openreview.net/forum?id=Sked_0EYwB (note that the accepted L4DC version http://proceedings.mlr.press/v120/lambert20a/lambert20a.pdf is very similar to the rejected ICLR https://openreview.net/pdf?id=Sked_0EYwB)).\n2) Indeed, in [3], it is shown that even a perfect model does not necessarily improve sample efficiency of RL.\n3) As the authors of the present paper note in Sec. 2.3, the issue may be due to compounding errors from using learned one-step prediction models.\n4) Additionally, the overall performance depends on what type of RL or planning method is used on top of the learned model.\n\nTherefore, it appears too strong a statement to say that \"objective mismatch\" is the **main reason** why a learned model is not helping the policy learning so much. But luckily, I think it is also not necessary for this paper to argue this way. It is sufficient to note that such behavior has been observed and that it can be attributed to multiple factors, one of which is the objective mismatch.\n\n\n### 3. Decision-Aware MBRL vs Multi-Task MBRL\nThe relation between decision-aware MBRL and multi-task learning is discussed in one paragraph in Sec. 5.4, but I think this could be discussed in a bit more detail and also mentioned in the introduction. Namely, a key advantage of learning a model is that it can be reused for other tasks. However, if one would learn a task-specific dynamics model, as the surveyed paper are trying to do, then what is the benefit of that model — i.e., why not directly learn model-free on that task? I think it would be worth providing some arguments addressing this question somewhere in the beginning of the paper.\n\n\n### 4. Value Optimization-Equivalence\n1) In Sec. 5.1, the paper immediately claims that \"The proposed taxonomy suggests a single principle to decision-aware model and policy objective design: *value optimization-equivalence*\". However, it is not shown how this is derived or where it follows from. For example, it is not clear how methods of \"Distribution Correction\" category (Sec. 4.1) and \"Control-As-Inference\" category (Sec. 4.2) are implementing the *value optimization-equivalence* principle.\n\n2) Furthermore, if it is indeed the underlying principle, then it is strange that the category of \"Value-Equivalence\" methods (Sec. 4.3) — which is explicitly implementing this principle — is placed at the same level as all other categories. Shouldn't it be at the root of the hierarchy in this case?\n\n\n### References\n[1] Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020). Reinforcement learning through active inference. arXiv preprint arXiv:2002.12636.\n\n[2] Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., & Whiteson, S. (2019, September). VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. In International Conference on Learning Representations.\n\n[2] Palenicek, D., Lutter, M., Carvalho, J., & Peters, J. (2022, September). Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning. In The Eleventh International Conference on Learning Representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a survey and taxonomy of model-based RL methods that learn a model of the system dynamics in a task-oriented manner. Four categories of approaches are identified, with their respective characteristics and objective functions. In total, 43 papers are surveyed. The proposed taxonomy suggests *value optimization-equivalence* as an important design principle, by which \"the model and the policy should be trained to optimize the expected return in the real environment\".", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my detailed comments above. Below I just repeat my requests in a short form.\n\n1) Scope and Related Work\n    \n    a) position the paper w.r.t. the work on adaptive optimal control with regards to the objective mismatch\n\n    b) [critical] either add active inference as a category (or sub-category of control-as-inference (e.g., [1])) or explain why it is out-of-scope in the Related Work section\n\n    c) mention meta-learning and explain why it is not included\n\n\n2) [critical] Objective Mismatch: consider reframing the paper, mentioning it as one possible problem instead of the \"root cause\". Or alternatively provide a theorem or strong empirical evidence that it is indeed the \"root cause\".\n\n3) Decision-Aware MBRL vs Multi-Task MBRL: in the introduction, discuss the motivation behind focusing on decision-aware MBRL for a single task, as the considered methods do.\n\n4) [critical] Value Optimization-Equivalence: show how this principle is derived from the taxonomy and how each category of methods obeys/implements this principle.\n\n### References\n[1] Imohiosen, A., Watson, J., & Peters, J. (2020). Active inference or control as inference? A unifying view. In Active Inference: First International Workshop, IWAI 2020, Co-located with ECML/PKDD 2020, Ghent, Belgium, September 14, 2020, Proceedings 1 (pp. 12-19). Springer International Publishing.", "broader_impact_concerns": "The paper poses no ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a comprehensive survey on an emerging sub-area of model-based reinforcement learning that concerns the problem of objective mismatch. The objective mismatch problem refers to the fact that many conventional model-based RL algorithms use different objectives for policy training (maximizing the return) and the model training (accurate prediction of the world, ignoring its role in the policy decision making). On the other hand, the more recent decision-aware model-based RL approaches learn a model that is aware of its role in the agent's decision making. This paper is a timely survey of this subarea of model-based RL.

The paper categorizes methods addressing this objective mismatch into four groups of Distribution Correction, Control-As-Inference, Value-Equivalence, and Differentiable Planning. It summarizes several papers in each group and discusses their relationship.

All reviewers are positive (three Accepts, one Weak Accept). They consider the paper a nice, timely, and comprehensive survey. The authors revised the paper based on the received feedback and it appears that the reviewers are happy about the changes. Therefore, I recommend the acceptance of the paper as is.

As a minor suggestion, the authors may consider citing the accepted paper of paper, instead of their arXiv version.
Decision: Accept (poster)