On-Policy Model Errors in Reinforcement Learning
Authors: Lukas Froehlich
Maksym Lefarov
Melanie Zeilinger
Felix Berkenkamp
Venue: ICLR 2022 Poster (2022)
Keywords: Model-based reinforcement learning
reinforcement learning
model learning

Abstract: Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data. In contrast, model-based methods can use the learned model to generate new data, but model errors and bias can render learning unstable or suboptimal. In this paper, we present a novel method that combines real-world data and a learned model in order to get the best of both worlds. The core idea is to exploit the real-world data for on-policy predictions and use the learned model only to generalize to different actions. Specifically, we use the data as time-dependent on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons. We motivate this method theoretically and show that it counteracts an error term for model-based policy improvement. Experiments on MuJoCo- and PyBullet-benchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.

Published as a conference paper at ICLR 2022
ON-POLICY MODEL ERRORS IN REINEORCEMENT LEARNING
Lukas P. "röhlich"
Maksym Lefarov
Institute for Dynamic Systems and Control
Bosch Center for Artificial Intelligence Renningen, Germany
ETH Zürich
Zurich, Switzerland
Maksym.Iefarove Lefarovvdeeeeeeeee boseeb.bcc..... com
lukasfro@ethz.co
Melanie N. Zeilinger
Felix Berkenkamp
Institute for Dynamic Systems and Control
Bosch Center for Artificial Intelligence Renningen, Germany
ETH Zürich
Zurich, Switzerland
Fel i x. Berkenkamp@de,
www.eeded.bscccooooo bosch.
mzeil ingerrehhttttccc
ABSTRACT
Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data. In contrast, model-based methods can use the learned model to generate new data, but model emrors and bias render learning unstable suboptimal. In this paper, we present a novel method that combines real-world data and a learned model in order to get the best of both worlds. The core idea is exploit the real-world data for on- policy predictions and use the learned model only to generalize to different actions. Specifically, we use the data as dependentt on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons. We motivate this method theoretically and show that it counteracts error term for model-based policy improvement. Experiments on MuloCo- and ByBuulee--enchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.
INTRODUCTION
Model-free reinforcement learning (RL) has made great advancements in diverse domains such as single- and multi-agent game playing (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019), robotics (Kalashnikov et al., 2018), and neural architecture search (Zoph & Le, 2017). All these model-free approaches rely on large numbers of interactions with the environment ensure successful learning. While this issue is less severe for environments that can easily be simulated, limits the applicability of model-free RI to (real-world) domains where data is scarce.
Model-based RL (MBRL) reduces the amount of data required for policy optimization by approximat- ing the environment with learned model, which (Sutton, 1990; Racanière et al., 2017; Moerland et al., 2020). While early approaches on low- dimensional tasks by Schneider (1997); Deisenroth & Rasmussen (2011) used probabilistic models with closed-form posteriors, recent methods rely on neural networks to scale to complex tasks on discrete (Kaiser al., 2020) and continuous (Chua al., 2018; Kurutach et al., 2018) action spaces. However, the learned representation of the true environment always remains imperfect, which intro- duces approximation errors to the RL problem (Atkeson Santamaria, 1997; Abbeel et al., 2006). Hence, a key challenge in MBRI is model-bias; small errors in the learned models that compound over multi-step predictions and lead to lower asymptotic performance than model-free methods.
to generate simulated state transitions
To address these challenges with both model-free and model-based RL, Levine & Koltun (2013); Chebotar et al. (2017) propose to combine the merits of both. While there are multiple possibilities to
Work done partially while the Bosch Center for Artificial Intelligence.

---

Published as a conference paper at ICLR 2022
combine the two methodologies, in this work we focus improving the model's 's predictive state distribution such that it more closely resembles the data distribution of the true environment.
The main contribution of this paper is on-policy conrections (OPC), novel hyperparameter-free methodology that uses on- on-policy transition data on top of a separately Iearned model enable accurate long-term predictions for MBRL.. A key strength of our approach is that it does not introduce any new parameters that need to be hand-tuned for specific tasks. We theoretically motivate our approach by means of a policy improvement bound and show that we can recover the true state distribution when generating trajectories on-policy with the model. We illustrate how OPC improves the quality of policy gradient estimates in a simple toy example and evaluate it on various continuous control tasks from the MuJoCo control suite and their PyBullet variants. There, we demonstrate that OPC improves current tfateoof-the-ard MBRL. algorithms in terms of data-efficiency, especially for the more difficult PyBullet environments.
Contributions
Related Work To counteract model-bias, several approaches combine ideas from model-free and model-based RL. For example, Levine & Koltun (2013) guide a model-free algorithm via model- based planning towards promising regions in the state space, Kalweit & Boedecker (2017) augment the training data by an adaptive ratio of simulated transitions, Talvitie (2017) use 'hallucinated" transition tuples from simulated to observed states to self-correct the model, and Feinberg et al. (2018); Buckman et al. (2018) use learned model to improve the value function estimate. Janner et al. (2019) mitigate the issue of compounding errors for long-term predictions by simulating short trajectories that start from real states. Cheng al. (2019) extend first-order model-free algorithms via adversarial online learning to leverage prediction models in a regret-optimal manner. Clavera al. (2020) employ a model to augment an actor-critic objective and adapt the planning horizon to interpolate between a purely model-based and a model-free approach. Morgan et al. (2021) combine actor-critic methods with model- -rreicctr rollouts to guarantee near-optimal simulated data and retain exploration on the real environment. A downside of most existing approaches is that they introduce additional hyperparameters that are critical to the learning performance (Zhang et al., 2021).
In addition to empirical performance, recent work builds on the theoretical guarantees for model-free approaches by Kakade Langford (2002); Schulman et al. (2015) to provide guarantees for MBRL. Luo et al. (2019) provide general framework to show monotonic improvement towards local optimum of the value function, while Janner et al. (2019) present a lower-bound on performance for different rollout schemes and horizon lengths. Yu et al. (2020) show guaranteed improvement in the offline MBRL setting by augmenting the reward with an uncertainty penalty, while Clavera et al. (2020) present improvement guarantees in terms of the model's and value function's 's gradient errors.
Moreover, Harutyunyan et al. (2016) propose a similar correction term as the one introduced in this paper in the context of oof-p policy evaluation and correct the state-action value function instead of the transition dynamics. Similarly, Fonteneau et al. (2013) consider the problem of off-policy policy evaluation but in the batch RL setting and propose to generate 'artificial' trajectories from observed transitions instead of using an explicit model for the dynamics.
A related field to MBRL that also combines models with data is iterative learning control (IL.C) (Bristow et al., 2006). While RL typically focuses on finding parametric feedback policies for general reward functions. ILC instead seeks an open-loop sequence of actions with fixed length improve state tracking performance. Moreover, the model in ILC is often derived from first principles and then kept fixed, whereas in MBRL the model is continuously improved upon observing new data. The method most closely related to RL and our approach is optimization- based ILC (Owens & Hätönen, 2005; Schöllig & D' Andrea, 2009), in which a linear dynamics model is used to guide the search for optimal actions. Recently, Baumgärtner & Diehl (2020) extended the ILC setting to nonlinear dynamics and more general reward signals. Little work is available that draws connections between RL and ILC (Zhang et al., 2019) with one notable exception: Abbeel al. (2006) use the observed data from the last rollout to account for a mismatch in the dynamics model. The limitations of this approach are that deterministic dynamics search procedure with rollouts on the true environment and that it was not combined with model learning. We build on this idea and extend it to the stochastic setting of MBRL by making use of recent advances in RL and model learning.
assumed, the policy optimization itself requires a line
2

---

Published as a conference paper at ICLR 2022
Algorithm 1 General Model-based Reinforcement Learning
1: for = 1,... do
2:
for = 1,..., do
3:
Rollout policy nn on environment and store transitions D {($}
4: (8t+11 St, a:): Learn a global dynamics model given all data 1.m U
U Dj
5:
aa+1 = On + a Vin Optimize the policy based on the model p with any RI. algorithm
PROBLEM STATEMENT AND BACKGROUND
We consider the Markov decision process (MDP) (S,A, P. are the continuous state and action spaces, respectively. The unknown environment dynamics are described by the transition probability p(8++1 8t, at), an initial state distribution p(So) and the reward signal r(s, a). The goal in RL is to find a policy Te 5t) parameterized by 0 that maximizes the expected return discounted by [0, over episodes of length T',
A.P. 7.P). p), where S C Rs and A S Rd
17
E SO/TPPOT
Σ
EL.
rr, az)
Se+1 ~ p(5++1 Se, ar), 80 P. a, ~ Ta 82).
(1)
The expectation is taken with respect to the trajectory under the stochastic policy mo starting from stochastic initial state Sp- Direct maximization of Eq. (1) is challenging, since we do not know the environment's transition model p. In MBRL, we learn a model for the transitions and reward function from data, P( Sr. ax) & p(S S;, ar) and F(S,, a²) & r(s;, at), respectively. Subsequently, we maximize the model-based expected return i7 as a surrogate problem for the true RL setting, where f) is defined as in Eq. (1) but with P and f instead. For ease of exposition, we assume a known reward function = r, even though we learn it jointly with ý in our experiments.
We let 7h, denote the return under the policy actions that are observed on the true environment. Algorithm summarizes the overall procedure for MBRL: At each iteration n we store B on on-policy trajectories D {$$ a, 1 TT obtained by rolling out the current policy nn the real environment in Line 3. Afterwards, approximate the environment with a learned model pr. (S,+1 St a,) based on the data Di... in Line 4, and optimize the policy based the proxy objective in Line 5. Note that the policy optimization algorithm can be off-policy and employ its own, separate replay buffer.
To,, at iteration n and use á and å for states and
an.6
Model Choices The choice of model á plays key role, since it is used to predict sequences of states transitions and thus defines the surrogate problem in MBRL. We assume that the model comes from distribution family P, which for each state-action pair (St, ax) models distribution over the next state st+1. The model is then trained to summarize all past data Dim = U.= U D by maximizing the marginal ligliiol L.
P smodel (s1+1 St, st) = arg max PEP
Σ
1( â.).
(2)
For a sampled trajectory index b ~ 11(11 B}), sequences T start from the initial state $p and are distributed according (m b) 5(so $ II pmodel st, aT, sz), where 5()) denotes the Dirac-delta distribution. Using model-data for policy optimization is in contrast to model-free methods, which only use observed environment data by replaying past transitions from recent on-policy trajectory b E {1...., B}. In our model-based framework, this replay buffer is equivalent to the non- paraameetr model
pr b) 5(s0
n.b
II
Ph t=u
adata (S:41 b,,), where pdlat [t,b) 5(5141 sHL)). (3)
where we only replay observed transitions instead of sampling new actions from TT.
ON-POLICY CORRECTIONS
In this section, analyze how the choice of model impacts policy improvement, develop OPC as a model that can eliminate one term in the improvement bound, and analyze its properties. In the following, drop the sub- and superscript when the iteration is clear from context.
3

---

Published as a conference paper at ICLR 2022
3.1 POLICY IMPROVEMENT
Independent of whether we use the data directly in pdata or summarize it in world model fuodel, our goal is to find an optimal policy that maximizes Eq. (1) via the corresponding model-based proxy objectíve. To this end, we would like to know how a policy improvement hh+1 - 7h1 2 O based on the model p. which what we optimize in MBRL. relates to the true gain in performance 7h++1 The on the environment with unknown transitions p. While the two are equal without model errors, in general the larger the model error, the worse we expect the proxy objective to be (Lambert et al., 2020). Specifically, we show in Appendix B.1 that the policy improvement can be decomposed as
+1 1m
1n+1 Tm
17++1 7
1/n
7/
(4)
True policy impoovement
Model policy improvement
Oll-policy model erroe
Oa-policy model error
where performance improvement on our model-based objective ij only translates to a gain in Eq. (1) if two error terms are sufficiently small. These terms depend on how well the performance estimate based on our model, fj, matches the true performance, 7). If the reward function is known, this term only depends on the model quality of p relative to p. Note that in contrast to the result by Janner et al. (2019), Eq. (4) is a bound on the policy improvement instead of a lower bound on 1-+11
The first error term compares Th++1 and M the performance estimation gap under the optimized policy T4+1 that we obtain in Line of Algorithm 1. Since at this point we have only collected data with nn in Line 3, this term depends on the generalization properties of our model to new data; what we call the off-policy model error. For our data-based model pdata that just replays data under nn independently of the action, this term can be bounded for stochastic policies. For example, Schulman et al. (2015) bound it by the average KL-divergence between T. and TT+11. For learned models gmodel it depends the generalization properties of the model (Luo et al., 2019; Yu et al., 2020). While understanding model generalization better is an interesting research direction, we will assume that our learned model is able to generalize to new actions in the following sections.
While the first term hinges on model-generallizatin,, the second term is the on-policy model error, i.e., the deviation between 7n and 7n. under the current policy T. This error term goes to zero for fdlata as we use more on-policy data B -> oc, since the transition data are sampled from the true environment, c.f., Appendix B.2. While the learned model is also trained with on-policy data, small errors in our model compound as we iteratively predict ahead in time. Consequently, the on-policy error term grows as O ?)² H/(1 - 7), H²)) )), c.f., (Janner et al., 2019) and Appendix B.3.
3.2
COMBINING LEARNED MODELS AND REPLAY BUFFER
The key insight of this paper is that the learned model in Eq. (2) and the replay buffer in Eq. (3) have opposing strengths: The replay buffer has low eтror on on-policy, but high error off off-policy since it replays transitions from past data, i.e., they are independent of the actions chosen under the new policy. In contrast, the learned model can generalize to new actions by extrapolating from the data and thus has lower error off-policy, but errors compound over multi-step predictions.
An ideal model would combine the model-free and model- -ased approaches in a way such that it retains the unbiasedness of on-policy generated data, but also generalizes to new policies via the model. To this end, we propose to use the model to predict how observed transitions would change for a new state-action pair. In particular, we use the model's mean prediction f' (s, a) = Eljmoodel construct the joint model
a)] to
opc Pn (8++1 St, ac, b) = (s5411 - S(+1) * 6(sc+1 - [[n
(sc, ar) (8P a7 1]),
(5)
pr (BL ()
Madel correction generalize s; R
where denotes the convolution of the two distributions and b refers a specific rollout stored in the replay buffer that was observed in the true environment. Given a trajectory-inder b, pope in Eq. (5) transitions deterministically according to S:+1 gn.b + (st, at) - (sp.b, an.)). resembling the equations in ILC (c.f., Baumgärtner & Diehl (2020) and Appendix E). If we roll out pope along a trajectory, starting from a state 87 and apply the recorded actions from the replay buffer, m..b the correction term on the right of Eq. (5) cancels out and we have poP ($x41 s7.b 5a.b . ,b) addaaa (SL+1 1, b) 0( sn.b)
Thus OPC. retrieves the true on-policy data
4

---

Published as a conference paper at ICLR 2022
guoddl (s1 (o0 .o)
gode (82 a1, b)
#0
So
Sig
pr (2 18g aj. b)
State
groodel (st 18 al)
T (eala1.
off- palicy
policy
pope (si Bis ao. 6)
guodel
Model (on-policy)
Thue enviramentt
Model (off-policy)
Model OPC
8 pr (81
18 a
Time Steps
(a) Multi-step predictions with OPC given n.
(b) State distribution over time.
Figure I: Illustration to compare predictions of the three models Eqs, (2), (3) and (5) starting from the same state s. In Fig. la, we see that on- on-policy, i.c., using actions (a.. af), - returns environment data, while moodel (blue) is biased. We correct this on-policy bias in expectation obtain pope. This allows us to retain the true state distribution when predicting with these models recursively (c.f., bottom three lines in Fig. lb). When using OPC for off-policy actions (ag. a1), pope does not recover the true off-policy state distribution since it relies on the biased model. However, the corrections generalize locally and reduce prediction errors in Fig. lb (top three lines).
distribution independent the prediction quality of the model, which is why we refer to this method on-policy corrections (OPC). This behavior is illustrated in Fig. la, where the model (blue) is biased on-policy, but OPC corrects the model's prediction to match the true data. In Fig. 1b, we show how this affects predicted rollouts on a simple stochastic double-integrator environment. Although small on-policy errors in jmmodel (blue) compound over time, the corresponding gope matches the ground truth environment data closely. Note that even though the model in Eq. (5) is deterministic. we retain the environment's 's stochasticity from the data in the transitions to $141, so that we recover the on-policy aleatoric uncertainty (noise) from sampling different reference trajectories via indexes b.
When our actions a are different from a,, pape still uses the data from the environment's transitions, but the correction term in Eq. (5) uses the learned model to predict how the next state changes in expectation relative to the prediction under a². That is. in Fig. Ia for a new a, the model predicts the state distribution shown red. Correspondingly, we shift the static prediction $/+1 by the difference in means (gray arrow) between the two predictions; i.c.. the change in trajectory by changing from a to a,. Since we shift the model predictions by a time-dependent but constant offset, this does not recover the true state distribution unless the model has zero error. However, empirically it can still help with long-term predictions in Fig. 1b by shifting the model off-policy predictions (red) to the OPC predictions (green), which closer to the environment's state distribution under the new policy.
3.3 THRORETICAL ANALYSIS
In the previous sections, we have introduced OPC to decrease the on- on-policy model error in Eq. (4) and tighten the improvement bound. In this section, we analyze the on-policy performance gap from a theoretical perspective and show that with OPC this error can be reduced independently of the learned model's 's error. To this end, we assume infinitely many on-policy reference trajectories, B -> oc, which is equivalent to variant of pope that considers 6/ as a random variable that follows the true environment's transition dynamics. While impossible to implement practice, this formulation is useful to understand method. We define the generalized opc-model
p
[s1+1
St, a,, b) = p(1++1 s/, a
8 S111 f(s,, a.) - f(s). ab)
(6)
Environment on-policy transition
OPC conrection term
which highlights that it transitions according to the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term. We provide a detailed derivation for the generalized model in Appendix B, Lemma 4. With Eq. (6), we have the following result:
Theorem 1. Let oe and vj be the expected return under the generalized oPC-model Eq. (6) and the true environment, respectively. Assume that the learned model's mean transition function f(s a.) = Elgmodel(ss.) (S:41 8t, at)a is i... -ip-hihch and the reward r(st, ar) is -.p--ipchittz Further,
5

---

Published as a conference paper at ICLR 2022
if the policy x(ar st) is Lx -xpissccitz with respect to St under the Wasserstein distance and its (co-)variance Var(r(ar 8t)l (s,) E $4 is finite over the complete state space, i.e.,
es trace S o. then with C₁ 2 1 1 Ly and C2 1 + L2z
In 17,7% σ. da VIC.T.
We provide a proof in Appendix B.4, From Theorem 1, we can observe the key property of OPC: for deterministic policies, the on-policy model error from Eg. (4) is zero and independent of the learned models' predictive distribution giuode,, so that v = -/= For policies with non-zero variance, the bound scales exponentially with T, highlighting the problem of compounding error. In this case, as in the off-policy case, the model quality determines how well we can generalize to different actions. We show in Appendix B.5 that, for one-step predictions, OPC's prediction error scales as the minimum of policy variance and model error. To further alleviate the issue of compounding errors, one could extend Theorem with branched rollout scheme similarly to the results by Janner et al. 22,,), such that the rollouts are only of length H & T.
In practice, pe cannot be realized as it requires the true (unknown) state transition model p. However, as we use more on-policy reference trajectories for pope in Eq. (5), it also converges to zero on-policy error in probability for deterministic policies.
Lemma 1. Let M be MDP with dynamics p(s:+1 St ar) and reward r <. Tmaxx Let M be another MDP with dynamics gmodel + p. Assume a deterministic policy π : : + A and a set of trajectories T- collected from M under и. If we use OPC Eq. (5) with data D, then with convergence rate (1/BB).
D = 1 at sb 11+1))7=0
lim Pr - 1 > 8) =0 Vs > B-x
0
(8)
We provide a proof in Appendix B.4. Lemma I states that given sufficient reference on-policy data, the performance gap due to model errors becomes arbitrarily small for any model guodel when using OPC. While the assumption of infinite on on-policy data in Lemma is unrealistic for practical applications, we found empirically that OPC drastically reduces the on-policy model error even when the assumptions are violated. In our implementation, we use stochastic policies as well as trajectories from previous policies, i.e., off-policy data, for the corrections in OPC (see also Section 4.2).
3.4 DISCUSSION
Epistemic uncertainty So far, we have only considered aleatoric uncertainty (noise) in our transition models. In practice, modern methods additionally distinguish epistemic uncertainty that arises from having seen limited data (Deisenroth & Rasmussen, 2011; Chua et al., 2018). This leads to a distribution (or an ensemble) of models, where each sample could explain the data. In this setting, we apply OPC by correcting each sample individually. This allows to retain epistemic uncertainty estimates after applying OPC, while the epistemic uncertainty is zero on-policy.
Limitations Since OPC uses on-policy data, it is inherently limited to local policy optimization where the policy changes slowly over time. As a consequence, it is not suitable for global exploration schemes like the one proposed by Curi et al. (2020). Similarly, since OPC uses the observed data and corrects it only with the expected learned model, pope always uses the on-policy transition noise (aleatoric uncertainty) from the data, even if the model has learned to represent it. While not having to learn a representation for alcatoric uncertainty can be a strength, it limits our approach to environments where the aleatoric uncertainty does not vary significantly with states/actions. is possible to extend the method to the heteroscedastic noise setting under additional assumptions that enable distinguishing model error from transition noise (Schöllig & D' Andrea, 2009). Lastly, our method applies directly only to MDPs, since we rely otatevobservttions. Extending the ideas to partially observed environments an interesting direction for future research.
EXPERIMENTAL RESULTS
We begin the experimental section with a motivating example a toy problem to highlight the impact of OPC on the policy gradient estimates in the presence of model errors. The remainder of the section focuses on comparative evaluations and ablation studies complex continuous control tasks.

---

Published as a conference paper at ICLR 2022
Without OPC
With OPC
0.4
Reference pullicy ar
Signed gradient crror
-0.5
-.5
0.2
-1.0
-10
0.0
-1.5
-1.
0.2
0.4
0.2
0.2
0.4
0.4
0.2
0.0
0.2
0.4
-044
Model Ervor
Bunsel
Biu
Model Error AB
Buse
Figure 2: Signed gradient error when using inaccurate models Eq. (9) to estimate the policy gradient without (left) and with (right) OPC. The background's opacity depicts the error's magnitude, whereas color denotes the sign of estimated and true gradient differ (red) or coincide (blue). OPC improves the gradient estimate in the presence of model errors. Note that the optimal policy is 8 = 1100.
4.1 ILLUSTRATIVE EXAMPLE
In Section 3.3, we investigate the influence of the model error directly on the expected return from theoretical perspective. From a practical standpoint, another relevant question how the policy optimization and the respective policy gradients are influenced by model errors. For general environments and reward signals, this question is difficult to answer, due the typically high- dimensional state/action spaces and large number of parameters governing the dynamics model as well as the policy. To shed light on this open question, we resort simple low-dimensional example and investigate how OPC improves the gradient estimates under misspecified dynamics model.
In particular, we assume a one-dimensional deterministic environment with linear transitions and linear policy. The benefits of this example are that we can 1) compute the true policy gradient based on a single rollout (determinism), 2) determine the environment's 's closed-loop stability under the policy (linearity). and 3) visualize the gradient emrr function of all relevant parameters (low dimensionality). The dynamics and initial state distribution are specified by p(s+41 St, ax) 5(As; + Ba, at) with p(so) 8(sp) where A. B E R and 5(-) denotes the Dirac-delta distribution. We define a deterministic linear policy (a S;) = ö(As, (s)) that is parameterized by the scalar 0 E E. The objective is drive the state zero, which we encode with exponential reward r(, a,) = exp /)) Further, we assume an approximate dynamics model
5(s s; ax) 5((A AA)s, + [B AB)a St, a:),
where AA, AB quantify the mismatch between the approximate model and the true environment. In practice, the mismatch can arise due to noise-corrupted observations of the true state or, in the case of stochastic environments, due finite amount of training data.
With the setting outlined above, we investigate how model errors influence the estimation of policy gradients. To this end, roll different policies under models with varying degrees of error AB. For each policy/model combination, we compute the model-based policy gradient and compare it to the true gradient. The results are summarized in Fig. 2, where the background's opacity depicts the gradient error's 's magnitude and its color indicates whether the respective signs of the gradients are the same (blue, 2 0) or differ (red, < 0). For policy optimization, the sign of the gradient estimate is paramount. However, we see in the left-hand image that even small model errors can lead to the wrong sign of the gradient. OPC significantly reduces the magnitude of the gradient error and increases the robustness towards model errors. See also Appendix C for more in-depth analysis.
4.2 EVALUATION ON CONTINUOUS CONTROL TASKS
In the following section, we investigate the impact of OPC range of continuous control tasks. To this end, we build upon the current state-of-fe of-the-rr model-based RL algorithm MBPO (Janner et al., 2019). Further, aim answer the important question about how data diversity affects MBRL, and OPC in particular. While having a model that can generate (theoretically) an infinite amount of data intriguing, the benefit of having more data is limited by the model quality in terms
7

---

Published as a conference paper at ICLR 2022
HalfCheetah
Hopper
Walker2d
AntTruncatedObs
Hvaltation return x x100 Evaluation retum x10
10
MuloCo
PyBullet
0.5
0.0
100
200
0
100
0
200
200
Steps x10?
Steps x10?
# Steps x
Steps x10"
Figure 3: Comparison of OPC ) MBPO(*) (=-=) and SAC (Doms) on four environments from the MuJoCo control suite (top row) and their respective PyBullet implementations (bottom row).
of being representative of the true environment. For OPC, the following questions arise from this consideration: Do longer rollouts help to generate better data? And is there a limit to the value of simulated transition data, i.e., is more always better?
For the dynamics model pmodel, we follow Chua et al. (2018); Janner et al. (2019) and use a proba- bilistic ensemble of neural networks, where each head predicts Gaussian distribution state and reward. For policy optimization, we employ the soft actor critic (SAC) algorithm by Haarnoja al. (2018). All learning curves are presented in terms the median (lines) and interquartile range (shaded region) across ten independent experiments, where we smooth the evaluation return with a moving average filter to accentuate the results of particularly noisy environments. Apart from small variations in the hyperparameters, the only difference between OPC and MBPO is that our method uses gopc, while MBPO uses gmandel We provide pseudo-cod for the model rollouts of MBPO and OPC in Algorithm 2 in Appendix A.1. Generally, we found that OPC was more robust to the choice of hyperparameters. The rollout horizon to generate training data is set H for all experiments. Note that when using pdlata generate data, we retain the standard (model-free) SAC algorithm.
the next
Our implementation based upon the code from Janner et al. (2019). We made the following changes to the original implementation: 1) The policy is only updated at the end of epoch, not during rollouts on the true environment. 2) The replay buffer retains data for fixed number of episodes, to clearly distinguish and off- policy data. 3) For policy optimization, MBPO uses small number of environment transitions in addition to those from the model. We found that this design choice did refrain
consistently improve performance and added another level of complexity. Therefore, from mixing environment and model transitions and only use simulated data for policy optimization. While we stay true the key ideas of MBPO under these changes, we denote our variant as MBPO(*) to avoid ambiguity. See Appendices D.6 and D.7 for comparison to the original MBPO algorithm.
Comparative Evaluation We begin our analysis with a comparison of method MBPO(*) and SAC on four continuous control benchmark tasks from the MuloCo control suite (Todorov et al., 2012) and their respective PyBullet variants (Ellenberger, 2018-2019), The results are presented in Fig. 3. We see that the difference in performance between both methods is only marginal when evaluated on the MuJoCo environments (Fig. 3, top row). Notably, the situation changes drastically for the PyBullet environments (Fig. 3, bottom row). Here, MBPO(*) exhibits little to no learning progress, whereas OPC succeeds at learning a good policy with few interactions in the environment. One of the main differences between the environments (apart from the physics engine itself) is that the PyBullet variants have initial state distributions with significantly larger variance.
Influence of State Representation In general, the success of RL algorithms should be agnostic to the way an environment represents its state. robotics, joint angles i are often re- e--armmeterized by sine/cosine transformation, & (sin((),cos(?))) We show that even for the simple CartPole envi-
8

---

Published as a conference paper at ICLR 2022
RoboSchool (original)
RobuSchool (transf.) coss [)]
PyBullet (original) 0
Pybullet (transf.) - [sin( c (#)
[sin( cos()(
<109
1.00
0.75
Evaluation
0.50
0.25
# Steps x 110
Steps =10
Steps x10
Steps x10'
Figure 4: Comparison of OPC (=) and MBPO(*) (===) on different variants of the CartPole environment. When the polc's angle is observed directly (center plots), both algorithms successfully learn policy, With the sine/cosine transformations (outer plots), MBPO(*) fails to solve the task.
20 X10
N x10
N 100 X10²
N 200 100
12
Evaluation X10
100
200 0
100
200
100
200 0
100
200
# Steps x 10
Steps 10
# Steps x 10
SSteps 10
Figure 5: Ablation study for OPC on the HalfCheetah environment. In each plot, we fix the number of simulated transitions N and vary the rollout lengths H {1( 5(
10(
ronment, the parameterization of the pole's angle has large influence on the performance of MBRL. In particular, we compare OPC and MBPO(*) on the RoboSchool and PyBullet variants of the CartPole environment, which represent the pole's 's angle with and without the sine/cosine transformation. The results are shown in Fig. 4. To rule out other effects than the angle's representation, we repeat the experiment for each implementation but transform the state to the other representation, respectively. Notably, OPC successfully learns policy irrespective of the state's 's representation, whereas MBPO(*) fails if the angle the pole is represented by the sine/cosine transformation.
Influence of Data Diversity Here, we investigate whether multi-step predictions are in fact beneficial compared to single-step predictions during the data generation process. To this end, keep the total number of simulated transitions N for training constant, but choose different horizon lengths H = {1, 5, 10). The corresponding numbers of simulated rollouts are then given by Prollout N/H. H. The results for N (20, 40, 100, 200) x 10 the HalfCheetah environment shown in Fig. 5. First, note that more data leads to a higher asymptotic return, but after a certain point more data only leads to diminishing returns. Further, the results indicate that one-step predictions are not enough to generate sufficiently diverse data. Note that this result contradicts the findings by Janner et al. (2019) that one-step predictions are often sufficient for MBPO.
5 CONCLUSION
In this paper, we have introduced on-policy corrections (OPC), novel method that combines observed transition data with model-based predictions to mitigate model-bias in MBRL. In particular, we extend replay buffer with a leamned model to account for state-action pairs that have not been observed on the real environment. This approach enables the generation of more realistic transition data to more closely match the true state distribution, which further motivated theoretically by tightened improvement bound on the expected return. Empirically, we demonstrated superior performance on high-dimensional continuous control tasks well as robustness towards state representations.

---

Published as a conference paper at ICLR 2022
REFERENCES
Pieter Abbeel, Morgan Quigley, and Andrew Y. Ng. Using Inaccurate Models in Reinforcement Learning. Proceedings of the International Conference on Machine Learning (ICML), pp. 1-8, 2006.
Christopher G. Atkeson and Juan Carlos Santamaria. A Comparison of Direct and Model-Based Reinforcement Learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 3557-3564, 1997.
K. Baumgärtner and M. Diehl. Zero-Order Optimization-Baser Iterative Learning Control. In Proceedings of the IEEE Conference Decision and Control, pp. 3751-3757, 2020.
Joseph K. Blitzstein and Jessica Hwang. Introduction to Probability. CRC Press, 2019.
Douglas A. Bristow, Marina Tharayil, and Andrew G. Alleyne. A Survey of Iterative Learning
Control. IEEE Control Systems Magazine, 26(3):96-114, 2006.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee, Sample-efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8224-8234, 2018.
Frank M. Callier and Charles A. Desoer. Linear System Theory. Springer, 1991.
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining Model-Based and Model-Free Updates for Trajectory-Centie Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), pp-
703-711, 2017.
Ching-An Cheng, Xinyan Yan, Nathan Ratliff, and Byron Boots. Fredictor-Correche Policy Op- timization. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1151-1161, 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning Handful of Trials Using Probabilistic Dynamics Models. In Advances in Neural Information Processing Systems (NeurIPS), 4754-4765, 2018.
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-Augmenteed Actor-Critie: Backpropagating Through Paths. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient Model-Based Reinforcement Learning Through Optimistic Policy Search and Planning. In Advances Neural Information Processing Systems (NeurIPS), pp. 44156-14170 2020.
Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A Model-Based and Data-Efficient Approach to Policy Search. In Proceedings of the International Conference on Machine Learning
(ICML), pp. 465-472, 2011.
Benjamin Ellenberger. PyBullet Gymperium.
https:://githabb.com/eeeee.
Pybull 1ellllltmmmmmmm 2018-2019.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. arX1v:1803.00101 [cs.LG), 2018.
Raphael Fonteneau, Susan A Murphy, Louis Wehenkel, and Damien Ernst. Batch Mode Reinforce- ment Learning Based the Synthesis of Artificial Trajectories. Annals of Operations Research, 208(1):383-4166 2013.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor- Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the International Conference Learning Representations (ICLR), pp. 1861-1870, 2018.
10

---

Published as a conference paper at ICLR 2022
Anna Harutyunyan, Mare G. Bellemare, Tom Stepleton, and Rémi Munos. Q(A) with Off-Policy Corrections. In International Conference on Algorithmic Learning Theory, pp. 305 325, 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to Trust Your Model: Model- Based Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 19519-12530, 2019.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-Basee Reinforcement Learning for Atari. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Sham Kakade and John Langford. Approximately Optimal Approximate Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 267-274, 2002.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Fihan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable Deep Reinforcement Learning for Vision-Basee Robotic Manipulation. In Proceedings of the Conference on Robot Learning (CoRL), pp. 651-673, 2018.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-Drivver Imagination for Continuous Deep Reinforcement Learning. In Proceedings of the Conference on Robot Learning (CoRL), pp. 195-206, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieezadeh-Abadeh Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning. In Operations Research & Management Science in the Age of Analytics, pp. 130-166. INFORMS, 2019.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-Ensemble Trust-Region Policy Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. Proceedings of the Annual Learning for Dynamics and Control Conference (L4DC), pp- 761-770, 2020.
Sergey Levine and Vladlen Koltun. Guided Policy Search. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1-9, 2013.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic Framework for Model-Based Deep Reinforcement Learning with Theoretical Guarantees. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Ester Mariueci and Markus ReiB. Wasserstein and Total Variation Distance Between Marginals of Lévy Processes. Elecronic Journal of Statisties, 12(2):2482-2514, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533 533, 2015.
Thomas M Moerland, Joost Brockens, and Catholijn M Jonker. Model-Based Reinforcement Learning: A Survey. arXiv:2006. 16712 [cs.LG], 2020.
Andrew Morgan, Daljeet Nandha, Georgia Chalvatzaki, Carlo D'Eramo, Aaron Dollar, and Jan Peters. Model Predictive Actor- Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 6672-6678, 2021.
11

---

Published as a conference paper at ICLR 2022
David H. Owens and Jari Hätönen. Iterative Learning Control - An Optimization Paradigm. Annual Reviews in Control, 29(1):7777,, 2005.
Victor M. Panaretos and Yoav Zemel. Statistical Aspects of Wasserstein Distances, Annual Review of Statistics and Its Application, 6:405-431, 2019.
Sébastien Racanière, Theophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pas- canu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-Augment Agents for Deep Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5694-5705, 2017.
R. Tyrrell Rockafellar. Integral Functionals, Normal Integrands and Measurable Selections. In Nonlinear operators and the calculus of Variations, pp. 157-207. Springer, 1976.
Jeff G. Schneider. Exploiting Model Uncertainty Estimates for Safe Dynamic Control Learning. Advances in Neural Information Processing Systems (NeurIPS), pp. 1047-1053, 1997.
Angela P. Schöllig and Raffaello D' Andrea. Optimization-Baae Based Iterative Learning Control for Trajectory Tracking. In Proceedings of the European Control Conference (ECC), pp. 1505-1510, 2009.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In Proceedings of the International Conference on Machine Learning (ICML., pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms.
[cs.LG), 2017.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, $29:484-489 2016.
Richard S. Sutton. Integrated Architectures for Learning, Planning, and Reacting Based on Approx- imating Dynamic Programming. In Proceedings of the International Conference on Machine Learning (ICML), pp.
1990.
Erik Talvitie. Self-Correctin Models for Model-Based Reinforcement Learning. In Proceedings of the AAAI National Conference on Artificial Intelligence, pp. 2597-2603, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: physics Engine for Model-Based Control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033, 2012.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev. Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatuma, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575(7782): 350-354, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-Based Offline Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 14129-14142, 2020.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the Importance of Hyperparameter Optimization for Model- Based Reinforcement Learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 4015 2021.
12

---

Published as a conference paper at ICLR 2022
Yueqing Zhang, Bing Chu, and Zhan Shu. A Preliminary Study the Relationship Between Iterative Learning Control and Reinforcement Learning. PFAe-PapersOnLine, 22299)314-319, 2019.
Barret Zoph and Quoe V. Le. Neural Architecture Search with Reinforcement Learning. In Proceed- ings of the International Conference on Learning Representations (ICLR), 2017.
13

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
In the appendix we provide additional details on our method, ablation studies, and the detailed hyperparameter configurations used in the paper. An overview is shown below.
Table of Contents
A Implementation Details and Computational Resources
15
A.1 Detailed Algorithm for Rollouts with OPC
15
A.2 Hyperparameter Settings
16
A.3
Implementation and Computational Resources
16
B Theoretical Analysis of On-Policy Corrections
B.I General Policy Improvement Bound
17
B.2 Properties of the Replay Buffer
17
B.3 Properties of the Learnd Model
18
B.4 Properties of OPC
18
B.5 Model Errors OPC
26
Motivating Example In-depth Analysis
C.I Sctup
................................
27
C.2 Reward Landscapes
................................
................................
28
C,3 Influence of Model Error
................................
28
C.4 Influence of Off-Policy Error
29
C.5 Additional Information
30
Additional Experimental Results
D.1 Comparison with Other Baseline Algorithms
................................
32
D.2 Ablation Retain Epachs
................................
33
D.3 Influence of State Representation: In- Depth Analysis
had.............................
33
D.4 Improvement Bound: Empirical Investigation
35
D.5 Off-Policy Analysis
36
D.6 Implementation Changes to Original MBPO
37
D.7 Results for Original MBPO Implementation
38
E Connection between MBRL and ILC
41
E.1 Norm-optimal ILC
41
E.2 Model-hased RI. to Norm-optimal ILC
42
E.3 Extensions
43
14

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS AND COMPUTATIONAL RESOURCES
A.1 DETAILED ALGORITHM FOR ROLLOUTS WITH OPC
In Section 3.2 and Fig. la, have introduced the OPC transition model and how roll out trajectories with the model. Here, we will give more details on the algorithmic implementation for the generation of simulated data. Algorithm 2 follows the branched rollout scheme from MBPO (Janner et al., 2019). Differences MBPO are highlighted in blue.
Generally, OPC only requires deterministic transition function f to compute the corrective term in Line 8 in Algorithm 2. For models that include aleatoric uncertainty, we choose f(st. as) Es, [p(s, St. a.)] If, in addition, the model includes epistemic uncertainty, we refer to comment in Section 3.4 in the main part of the paper.
In practice, rollouts on the true environment are terminated early if, for instance, particular state exceeds user-defined boundary. Consequently, not all trajectories in the replay buffer necessarily of length T' Since the prediction in Line requires valid transition tuples for the correction term, we additionally check in Line 10 whether the next reference state is terminal state. Thus, in contrast to MBPO, for OPC we terminate the inner loop in Algorithm early if either simulated or reference state is terminal state.
Algorithm 2 Branched rollout scheme with OPC model (differences to MBPO highlighted in blue)
Input: Required parameters:
and an.b 12= for b E {1,.... B} Set of trajectories D {(8) a, (841)
Environment model p(se 18, ar). Define f(s, a) Eag (ā(s 8, a)..
Policy ma
Prediction horizon
Number of simulated transitions N
1: Deim <- 0: Initialize empty buffer for simulated transitions
2:
while |Dsim| N do
3:
b~ 111, B): Sample random reference trajectory
4:
10,00 T H): Sample random starting state
5:
h 0,8; <- sh: Initialize starting state
6:
while h <
7:
arth (a Sc+n): Sample action from policy
8:
St+++1 $ f(s arta) f( )) Do one-step prediction with OPC
9:
Deim Deim (St+ a $14441); Store new transition tuple
10: if (S is terminal)) or (s) ($++++1 is te terminal) then
break
12:
-- h 1: Increase counter
return Demm
15

---

Published as a conference paper at ICLR 2022
A.2 HYPERPARAMETER SETTINGS
Below, we list the most important hyperparameter settings that were used to generate the results in the main paper.
Table I: Hyperparameter settings for OPC (blue) and MBPO(*) (red) for results shown in Fig. Note that the respective hyperparameterss for each environment are shared across the different implementations, i.c., MuJoCo and PyBullet.
HalfCheetah
Hopper
Walker2D
AntTruncatedObs
epochs
200
150
300
300
env steps per epoch
1000
retain epochs
50/5
50
50
policy updates per epoch model horizon
40
20
20
10
model rollouts
1000000
per epoch
mix-in ratio
0.0
model network
ensemble of 7 with 5 elites
policy network
MLP with 2 hidden layers of size 64
A.3 IMPLEMENTATION AND COMPUTATIONAL RESOURCES
Our implementation is based on the code from MBPO (Janner et al., 2019), which is open-sourced under the MIT license. All experiments were run on an HPC cluster, where each individual experiment used one Nvidia V100 GPU and four Intel Xeon CPUs. All experiments (including early debugging and evaluations) amounted to total of 84"713 hours, which corresponds to roughly 9.7 years if the jobs ran sequentially. Most of this compute was required to ensure reproducibility (ten random seeds per job and ablation studies over the effects of parameters). The Bosch Group is carbon-neutral. Administration. manufacturing and research activities do no longer leave a carbon footprint. This also includes GPU clusters which the experiments have been performed.
16

---

Published as a conference paper at ICLR 2022
B THEORETICAL ANALYSIS OF ON-POLICY CORRECTIONS
In this section, we analyze OPC from a theoretical perspective and how it affects policy improvement.
Notation In the following, we drop the superscript for states and actions for ease of exposition. That is, gn.b and a n.b
B.I GENERAL POLICY IMPROVEMENT BOUND
We begin deriving inequality Eq. (4), which serves motivation for OPC and is the foundation for the theoretical analysis. Our goal is to bound the difference in expected return for the policies before and after the policy optimization step, i.e., 7h+11 - T/2- Since are considering the MBRL setting, is natural to express the improvement bound in terms of the expected return under the model ij and thus obtain the following
Tn 1
1/1
Mn+1
1/1
Th1+1 1/1+1
1nn 0m+11
'm1
Mn+1
1 1/1 + 11+1 1m1+1
+ 17n 1/n
7n
77
Mn+1
17n
Mm+1
1770 1m
True policy improvement
policy improvement
Off-policy model errrr
On-policy model crror
According to this bound, the improvement of the policy under the true environment governed by the three terms on the LHS:
Model policy improvement: This term is what we are directly optimizing in MBRL offset by the return of the previous iteration 7h. which is constant given the current policy Tn. Assuming that we are not at an optimum, standard policy optimization algorithms guarantee that this term is non-negative.
Off-policy model error: The last term is the difference in return for the true environment and model under the improved policy mn+11 This depends largely on the generalization properties of our model, since it is not trained on data under Tn+11
On-policy model error: This term compares the on-policy return under Th between the true environment and the model and it is zero for any model transitions from the true environment under the Ta the replay buffer Eq. (3) fulfills this condition under certain circumstances and the on-policy model error vanishes, Lemma 2.
p. Since we have access to
Note that the learned model Eq. (2) is able to generalize to unseen state-action pairs better than the replay buffer Eq. (3) and accordingly will achieve lower off-policy model error. The motivation behind OPC is therefore to combine the best of the learned model and the replay buffer to reduce both on- and off off-policy model errors.
B.2 PROPERTIES OF THE REPLAY BUFFER
The benefit of the replay buffer Eq. (3) is that it can never introduce any model-bias such that any trajectory sampled from this model is guaranteed come from the true state distribution. Accordingly, if we have collected sufficient data under the same policy, the on-policy model error vanishes.
Lemma 2. Let M be the true MDP with (stochastic) dynamics bounded reward r < Tmax and let pdata be the transition model for the replay buffer Eq. (3). Further, consider set of trajectories
D Uh ((sh at; ( collected from M under policy If we collect more and more on-policy training data under the same policy, then
lim Pr (nn--roppayy | =>)=0 VE> B-500
where sreplay is the expected return under the replay buffer using the collected trajectories DD
Proof. First, note that the corresponding expected return for the replay- buffer model is given by
preplay
1
ΣΣ rss ab),
1=0
17

---

Published as a conference paper at ICLR 2022
which is sample-based approximation of the true reward 1). By the weak law of large numbers (see e.g., Blitzstein & Hwang (2019, Theorem 10.2.2)), the Lemma then holds.
B.3 PROPERTIES OF THE LEARNED MODEL
Following Janner et al. (2019, Lemma B.3), a general bound the performance gap between two MDPs with different dynamics can be given by
H ty
m1 72l S 2r OTmaafn Em
(10)
where cm 2 max; Eg (s) (L (s', P2((,, a)) bounds the mismatch between the respective transition models. Now, the final form Eq. (10) depends on the horizon length H. For + 0o, we obtain the original result form Janner et al. (2019) with Σi>1 tt (1-)2 For the finite horizon case one can obtain tigther bounds when H smaller than the effective horizon, H < 7). encoded in the discount factor:
H(H +1
H
ty
min
(11)
2 "1-7"
2)
which can be verified by upper bounding s H to obtain /1/(1 7) or by bounding S obtain (H12).
Note that this bound is vacuous for deterministic policies, since the KL divergence between two distributions with non-overlapping support is infinite. In the following we focus on the Wasserstein metric under the Euclidean distance.
B.4 PROPERTIES OF OPC
In this section, we analyze the properties of OPC relative to the true, unknown environment's 's transition distribution p and learned representation p. In general, the OPC-model mixes observed transitions from the environment with the learned model. The resulting transitions are then a combination of the mean transitions from the learned model, the alcatoric noise from the data (the environment), and the mean-error between our learned model and the environment.
B.4.1 PROOF OF LEMMA
In this section, we prove Lemma by showing that the OPC-model coincides with the replay-buffer Eq. (3) in the case of deterministic policy and thus lead to the same expected return ñj.
Lemma 3. Let M be the true MDP with (stochastic) dynamics p and let M be a MDP with the same reward function and initial state distribution po, but different dynamics gmodel respectively. Further, assume deterministic policy S + A and set of trajectories D = Um i (sh at, s collected from M under we extend the approximate dynamics guodel by OPC with data D, then
speplay 1p.
where groplay and ipos are the model-based returns following models Eqs. (3) and (5), respectively.
Proof. For the proof, it suffices to show that the resulting state distributions of the two transition models pdata and pope under the deterministic policy я are the same for all 5 with < < B. We show this by induction:
18

---

Published as a conference paper at ICLR 2022
Theorem
Lemma 5
Lemma
Lemma
Lemma 10
Lemma
Lemma
Lemma
Lemma
Corollary
Lemma 13
Theorem
Lemma 12
Figure 6: Overview about the supporting Lemmas for the proof of Theorem 1.
For t 0 the states are sampled from the initial state distribution, thus we have So S by definition. Assume that = S as induction hypothesis. Then
popo (8141 St, ((ss), b) = 8 St+1
1/+1)
0 =s(s1+1 [((), T(:))) f(s), a?}
$441 + f(s,, (S:)) - [( ab)
8 St+1
8 S141
6441 - f((). a() /(5) at)
8 S1+1
$/+1
(
1,
where the second step follows by the induction hypothesis and due to the deterministic policy. Thus, for any index b we have To ope and the result follows.
Now, combining Lemmas 2 and 3 proofs the result in Lemma 1.
B.4.2 PROOF THEOREM
this section we prove our main result. An overview lemma dependencies shown in Fig. 6.
Remark on Notation In the main paper, we unify the notation for state sequence probabilities of the different models Eqs. (2), (3) and (5) as (T h t, b) with (replay, model, opc). This allows for consistent description of the respective rollouts independent of the model being learned representation, a replay buffer or the OPC-model. For that notation, the index denotes the sampled trajectory from the collected data the real environment. Implicitly, therefore condition the state sequence probability on the observed transition tuples i..,
gh af),.... ($ + 44H-1
px (TMCH tt = (TEL-H (5h41 a/) a.)..., (sh S -
(12)
where we omit the explicit conditioning for the sake of brevity in the main paper. Similarly, we can write the one-step model Eq. (5) for OPC in an explicit form as
Spp((s.+ SS, ar, b) (S [St, ar. S sp. ag).
(13)
Note that with the explicit notation, the relation between the opC-model Eq. (5) and generalized OPC- model Eq. (6) becomes clear:
p (Sp (S141
St, ar, b) = (s44 St, a,, S/. aç. (s st, a,, Sh. a+) dsh (14)
19

---

Published as a conference paper at ICLR 2022
For the following proofs, we stay with the explicit notation for sake of clarity and instead omit the conditioning on
Generalized orC-Model In this section, have closer look at the generalized OPC-model Eq. (6). The main difference between Eqs. (5) and (6) is that the former is fact transitioning deterministically (the stochasticity arises from the environment's aleatoric uncertainty which manifests itself in the observed transitions). The two models can be related via marginalization of $ see Eq. (14). The resulting generalized OPC-model can then be related to the true transition distribution according to the following Lemma.
Lemma 4, For all s6, àß E x A with 6//1 p(S++1 18) aQ) it holds that
(sp0 St, ac, s/ ab) pl 8141 [](s,, ar) f($) ab)
St,, 8,,
(15)
Mean correction
where f(s ar) EEipmodolsss St. ax)] is the mean transition of the learned model Eq. (2) and gopr(ss+) St, a, S9. ab) denotes the OPC-model if we marginalize over the distribution for $ instead of using its observed value.
Proof. Using the explicit notation Eq. (13), the OPC-model from Eq. (5) is defined as
p (s:-1 Sz, ar, b) = p (51+1
St, a, s al, s(+1)
(16)
= =5(s $141 - -b, * 0(s - [((),aa)) - F(s). at) a
(17)
5 8141 - |/4 + [(st, a.) j(8)/ [(8). a
(18)
With 6441 p(S++1 s/, a)), marginalizing $1+1 yields (note that we use 8 instead of s to denote the random variable for the next state in order to distinguish it from the random variable for (SPr(s ss a, b) under the integral)
(s+41 st ar, a?) = / 6( 6(mm1 - [Bu4 + f(se, a.) - F(s), a(s),aa)) p( | s a?) d$
p s111 (f() - f(s), ab)] (/)
Remark 1. An alternative way of writing the general OPC-model is the following.
(sp4 st, ar, a?) = p(s st. 8 (
[f(s, a.) f(s), (B))
(19)
On-policy transition
Mean correction
which highlights that the pape transitions according the true on-policy dynamics conditioned on data from the replay buffer, combined with a correction term. We can further explicitly see why an implementation of this model wouldn't 't be possible due to its dependency the true transition probabilities. Thus, in practice we're limited to the sample-based approximation shown in the paper.
The fundamental idea for the proof of Theorem I lies in the following Lemma 5, which is the foundation for bounding the on-policy error. The Wasserstein distance naturally arises in bounding depends the expected return under two different distributions. The
this type of errrr model final result is then summarized in Theorem 1.
Lemma 5. Let pope be the generalized OPC-model (cf. Lemma 4) and ippe be its corresponding expected return. Assume that the return r(st, ax) is Lip-schhhttz and the policy T(a, s;) L, LipLichhtitz with respect to under the Wasserstein distance, then
1 --pppc- I
12 Σ (Wa(p(ss).
(20)
120
20

---

Published as a conference paper at ICLR 2022
Proof.
17
kpppe]
7(((s,, ax)
E Tmpppe
Σ 120
r an)
(21)
Trup
120
E
(r($c,
E 81 (ampom
([(s,,8))]
(22)
((
E
(r(Se, às]
[r(s),
(23)
a. Cxanppore (e)
120
Applying Lemma 8:
L, We(((s: âz), (op ar))
(24)
120
Writing the joint distributions for state/action in terms of their conditional (i.e., policy) and marginal distributions p(se, at) = -Taa ss)(s::::
Σ We S.)p(S:), x(a sp)p 120
(25)
Under the assumption that the policies are L. Lipschittz under the Wasserstein distance, application of Lemma 10 concludes the proof:
x L, 1 (a po
(26)
10
Lemma (Wasserstein Distance between Marginal State Distributions).. Let (PP"(s:) and p(š:) be the marginal state distributions at time t when rolling out from the same initial state under the same policy with the OPC-model and the true environment, respectively, Assume that the underlying learned dynamics model is T.p-Lipschitz continuous with respect to both its arguments and the policy T (a s) is L, Lx-Lipschitz with respect to under the Wasserstein distance. If it further holds that s)] = Ex(s) E S is finite over the complete state space, i.e., max.s trace (E, (s)) S 02 then the discrepancy between the marginal state distributions of the two models is bounded
the policy's 's (co-) Jvariance Var[I(a
t-1 (3(PP)(()) p(s)) S T. (13
(27)
r=0
Proof. We proof the Lemma by induction.
Base Case: / -=1 For the base case, need show that starting from the same initial state So the following condition holds:
(2)l))))22////)) 2V
For case of readability, we define z = (s, a) and use the notation y) p(x,y) dx dy whenever no explicit assumptions are made about the distributions.
(s()
(28)
=Wa [ F (P
20, zo) dp(20, 20), / p(s1 20) dp(2w)
(29)
Recall that we can write both pope and p as convolution between Pt and Dirac delta (see Lemma 4). Together with the identity Eq. (54), the Wasserstein distance for sums of random variables Eq. (53) and noting Wp (((() 0:
S W2 (ʃʃ 6(s (5())) + f(zo) f((n))) dp(20, 20), / 5($1 I ((20)) dp(2w)
(30)
21

---

Published as a conference paper at ICLR 2022
Squaring and using Lemma 9:
W (ʃʃ 8(s1 - (f((2) + f(zo) f(2))) dp(2)))(zo), / 8($1 - f(20)) dp(2o)
$ // (20 + f(zo) f(20) - f(2o) dp(2u. 2o)
(31)
// llf(zo) f(20)11 dp(žu.20 zo)
(32)
S L3 / |ssu - sol1? + ||ay - 2oll dp(Sy, åg, Su, an)
(33)
We are assuming that the initial states of the trajectory rollouts coincide. The joint state/action distri- bution can then be written as p(So, åo, So, an) = (( 6n)d with respect to So leads to:
So) (aa so). Integrating
= / |a 2 ((5)) (T (ān 86) (ao 80) d&a dâg dan
(34)
This term describes the mean squared distance between random actions. Since condition the same state So. the policy distributions coincide. Define Aa = au ay.
1 E 8o
E Aa| []]
(35)
1.3 E
trace Var[Aa]]]
(36)
Now Var[Aa] = Varl [o)] + Var[n(ao So)] 2 Var(Iaaao) [o)
2 E tracee (ar [o)]}]
(37)
This term in fact less than Eq. (27) for | 0, thus proofing the base case.
(38)
Inductive Step We will show that if the hypothesis holds for then it holds for + 1 as well. We explicitly write the following intermediate bound such that its application in the proof is more apparent, i.e.,
< / [1((111) ((()-1)11dp(2111
W2(p(S),
(39)
< S2Vdn0LL3 Ee1++
(40)
P=
where the first inequality immediately follows from the same reasoning as in the base case Eq. (28) Eq. (32).
(41)
< / [If(z.) - f(2x))1 dp(22,, 2t)
(42)
s 13 / ||st - dil(e, 2,) + 13 / ||a,-aal' dp(2, 2,)
(43)
Applying Lemma 11 to the second integral
< ( + 122) / |ss - &ell dp(2t, 2z) + 2 d.
(44)
We predict along a consistent trajectory, i.e., St = S, + j(st- 1, a.-1) - (aa-))--(ss-_))) 18 a:-1)
$ {(1 + I22) / [1f(21-1) f(2) 2p + 22 2 1.1.7022
(45)
22

---

Published as a conference paper at ICLR 2022
Assume that the hypothesis Eq. (39) holds for
[-1
s (L3 x 2V 22 L3
[Lg+
+
(46)
-=
2 2 1
1
(47)
t==
2 E(1 L2)
(48)
Theorem 1. Let 11=" and 1j be the expected return under the generalized OPC-model Eq. (6) and the true environment, respectively. Assume that the learned model's mean transition function ((s a1) = Blj ($441 St,, ax)] is Li-Lpschhitz and the reward r(s,, a,) is L,-Lisschitz. Further, if the policy x(a, st) is Lx-Lipschitz with respect to St under the Wasserstein distance and its (co-)variance Var(r(ar | [x)] = Ex(s)) E 84 is finite over the complete state space, ie.
es (s)) S o then with C₁ = /2 + 12,, and C2 = 1 + L2
O d C VT.
In
PopE S
Proof. From combining Lemmas 5 and 6 follows that
|/ - -..c < V (1 Σ 7t
Σ
(1) 12)
(49)
120
1'=U
with the shorthand notations C₁ = /2 + L, and C2 = L2 + L2-
= = π
Ch
(50)
Since / < T, we have that Et Ch' < <TC]
$ S C2 VT
(51)
Since T' S co, we have with the geometric series Eno r < 11/(1-1)
s C1 1- C VTo.
(52)
B.4.3 DEFINITIONS, HELPFUL IDENTITIES AND SUPPORTING LEMMAS
Here we briefly summarize some basic definitions and properties that will be used throughout the following.
The Wasserstein distance fulfills the properties of a metric: Wp(p1,P3 S Wp P2) +
Wp Wp(p2. Ps).
Wasserstein distance of sums of random variables (see, e.g., Mariucci & ReiB (2018. Corollary for a proof):
We(pi * Pn. 91 * ...* qn) S Σ Wp(p..))
(53)
23

---

Published as a conference paper at ICLR 2022
For any function g(z. 2) we have
II D( g(2, 2)) dz p*
-
/| 8(5 g(z, 2)))((z (2, 2) dz d2
(54)
For any multivariate random variable and with probability distributions p(z1) pi (x)a(y) and p(z2) = p2 p2(x))(y), respectively, we have that (Panaretos & Zemel, 2019)
(2 (xx)((()))
px),
(55)
Further, the following Lemmas are helpful for the proof of Theorem 1.
Lemma 7 (Kantorovich Rubinsseninnniin (cf. Mariucci & Reiß (2018) Proposition 1.3)). Let X and be integrable real random variables. Denote by μ. and p their laws [...]. Then the following characterization of the Wasserstein distance of order I holds:
W)(v,
sup [o (oma())
E (
[6(T)]] - E [d(v))
(56)
where the supremam is being taken over all o satisfying the Lipschitz condition (0())-(()) S x- for all x, y E R.
Lemma 8. Let f be Lipischhit with respect to a metric d. Then
x>v(+) [[+))] you()) y-m())
[((i]] S L.Mi(u,. s L,Wa(u.))
(57)
Proof. The first inequality is a direct consequence of Lemma and the second inequality comes from the well-known fact that if S p q, then (,, I W, (1, v) (cf. Mariucci & Reiß (2018. Lemma 1 1.2)).
Lemma 9. For any two functions (s) and g(s) and probability density p(s) that govern the distri- butions defined by
= 6(x== - ((s)((s ds and p2(x2) = f 6(x² - g(s))p(s) ds,
pi(x1) =
(58)
holds for any q that
W (pL, / [If(() g(s))l(((sdds
(59)
Proof. We have
// |15: (E, de d&2
W/ (pi()) = inf
(60)
rET(p1m2))
Enforcing the following structure on y(E,, E2) reduces the space of possible distributions: 7(E1, (2) I 8({ - f()))8(E2 (ps)) ds, so that (E1, E2) e I(p1, p2) and thus
< f
l151 -
E211
8(E1
f(s))) g(s))p(() ds d5 d&2
(61)
Integrating over & and &2 yields
= |IIF((s) p(s)) ds
(62)
Lemma 10. the policy raa s:) is -ipsccht with respect to St under the Wasserstein distance, then with p(8, â) = T(ā 8))($) and p(s, a) = [(a (})(s).
()) < < T (p(s,,
(63)
24

---

Published as a conference paper at ICLR 2022
Proof.
(((s,)
(64)
/ |||2- + ||s -=|l] >(, S, då da da ds
inf -ETT)((
(65)
Enforcing the following structure on 7 reduces the space of possible distributions: 7(S, a, a) 78, (), a 8, with ((, s) E ())(S), p(s)) and 7( als,s) E [ à 8), a(a s)).
/ /
inf (( ((
a all (a, 8. dåda) + ||s =s| 7 ds
dsds
(66)
Interchange infimum and the integral: Rockafellar (1976, Theorem 3A)
inf / { T(a s)) + ||s s||² 77($ s) dåds
Ky inf ((atats.s)
/lla- |la- ||á-all?7(ā..) all
s) dá + s|| }r((5 s) ds
(67)
((8,3)
inf (P(3)
(68)
Using the assumption that the action distribution is L -Lisccicc continuous under the Wasserstein metric with respect to the state s:
inf (( / (1 1ss - s/ s||3]>(8, 8) d8 ds
<
(69)
(1 (2)
(70)
Lemma (Average Squared Euclidean Distance Between Actions), the policy T(a s) is L
Lipschitz with respect under the Wasserstein distance and the policy's (co-) (co-)variance Vari
s)] Mn (s) adA finite over the complete state space, i.e., maxses trace(2i, (s) S o², then
[|l;- - ar11]] 11 s:ll + d
(71)
(B)) (Be)
Proof. Straightforward application of Corollary and Lemma 13
Lemma 12 (Average Squared Euclidean Distance). Consider two random variables x, y with distributions Px: Py. mean vectors pxx, Hy E IR and covariance matrices Ex, Ex ST, respectively. Then the average squared Euclidean distance between the two
[||x x,y
y1 = |11x 1yy12
trace { Ex + Eyy
(72)
Proof.
Define z = x with mean (lz = fxx My and variance Eg = =x Ey.
E ||z||
E
z
Σ E [2]"
Σ E[z,]² + Var[z:]
Mg Mz + trace {Ez}
l1,x 1y + trace [Ex Eyy }.
25

---

Published as a conference paper at ICLR 2022
Theorem 2 (Gelbrich Bound (from Kuhn al. (2019))). If || is the Euclidean norm, and the distributions Px and Py have mean vectors ux Hy E R and covariance matrices Ex, By E s respectively. then
Wz(Px.Py Py) 2 V Ilux - Myll2 + trace { - E 1
(73)
The bound is exact if Px and Py are elliptical distributions with the same density generator. Corollary 1. Consider the same setting in Lemma 12, then the average squared Euclidean distance is bounded by
E [|x - y113] S W2 + trace 2 2., /3/3)//23
(74)
Proof. Straightforward application of the results from Lemma 12 and Theorem
Lemma 13. If the policy's 's (co-) vvariance Var[=(( s)] [s)(s) su is finite over the complete state space, i.e., maxges trace (s)} S oz. then
1/2 S vd
trace
(75)
Proof.
trace
(76)
The trace of matrix is the same the of its eigenvalues, and the square root of trix has cigenvalues that are square root of its eigenvalues. From Jensen's inequality we know
that DA V I Vdda A and consequently it holds for matrix M E IR xda that trace 2 S dd trace(MM) so that
V A (9))
(77)
The trace is invariant under eyclic permutation
v de ($),,
(78)
Since both matrices positive semi-definite, follows from the Cauchy-Schwartz inequality that
<v 2M, ($)] trace (s)}
(79)
By assumption, the covariance matrices' traces are bounded
(80)
B.5 MODEL ERRORS OPC
While Theorem highlights that OPC counteracts the on-policy error in predicted performance, for stochastic policies we use the Lipschitz continuity of the model to pouer- errors. In this section, we look at the impact of model errors in combination with OPC. Specifically we focus the one-step prediction case from a known initial state $p. There, while for the model guodel without OPC the prediction
only depends the quality of the model, with OPC it is instead the minimum of the model error and the policy variance. This is advantageous, since typical environments tend to have more states than actions, so that the trace of the policy variance can be significantly smaller than the full-state model error.
Lemma 14. Under the assumptions of Theorem starting from an initial state So the following condition holds:
min (((traco( /Bs)ss))))))))))) .0
(I((0,)) f(So, a)ll dr(a Bo)
Onc-stop model
26

---

Published as a conference paper at ICLR 2022
Proof. The first term in the minimum follows directly from the base-case of Lemma 6. For the second
term, follow the same derivation, but note that under the distribution p( â. So, an) = p( / (å. (s) (a) So) have f p(š, 2o) dp(2o) ʃ p(s) Zo) dp(zo). Inserting this into the r.h.s. of Eq. (28) and following the same steps we obtain
W (/ʃ 6(s) -[f(20) f(zo) - f(2))) dp(2o)p(2o), / 8(s) f(zo)) dp(zo)
(81)
< // (lf(2)) + f(zo) - f(20) f(zo) 112 d///ppp dp(20: zo)
(82)
// (1f(2)) f(20))12 + llf( f(zo))1² dp///.p(2200 oo)
(83)
2 / llf(2)) - f(20)1l dp(zo)
(84)
2 / [1(60, a) i(S0, a)ll dp(a so)
(85)
Note the additional factor of two in front of the upper bound on the model error, which comes from using the model 'twice': once with 2 and once with In practice do not see any adverse effects of this error, presumably because either the variance of the policy is sufficiently small, or due to the upper bound being lose in practice.
C MOTIVATING EXAMPLE IN-DEPTH ANALYSIS
In this section, we re-visit the motivating example presented in Section 4.1 of the main paper. For completeness, we re-state all assumptions that lead the simplified system hand. We continue with an analysis of the reward landscape and how OPC influences its shape. Next, investigate how an increasing mismatch the dynamics model impacts the gradient error. In addition to the result presented in the main paper, here show the influence of different model errors, i.e., AA well as AB. While OPC motivated for the use case of on-policy RL algorithms, we further show that the resulting gradients are robust with respect to differences data-generating and evaluation policy, iee. the off-policy setting. Lastly, we state the the signed gradient distance that we use for evaluation of the gradient errors, state the relevant theorem for determining the closed-loop loop stability of linear systems, as well as all numerical values used for the motivating example.
C.I SETUP
Here, we assume a linear system with deterministic dynamics
p(s++1 S; a,) 8 6(As, + Ba, s, a.). (o( 5(sp)
(86)
with A, B E and 8(-) denoting the Dirac-delta distribution. The linear policy and bell-shaped reward are given by the following equations
(a | s:) 8(0s, s;) with 0 R and r(s,, a.) exp {-
2
(87)
or
Further, assume have access approximate dynamics model ş 5(5 St a+) 5((A AA)s, + (B AB)a, St, at),
(88)
where AA, AB quantify the mismatch between the approximate model and the true system. For completeness, the (deterministic) policy gradient is defined as
0
T
r(s at)
(89)
E=
27

---

function of Published as a conference paper at ICLR 2022
1.00
1.00
Cumulative reward
Curnu Nunullltive reward
0.95
0.95
0.90
0.90
0.85
-1.5
1.0
0.0
-1.5
-10
0.0
Policy parumeter B
Policy parameter 0
True system
Model
Model OPC
Reference policy x"
Figure Cumulative reward for different systems as a function of the policy parameter. The reference trajectory that is used for OPC was generated by To (denoted by the black dashed line). The model mismatch between the true system and the approximated model is AA = 0.5, AB = 0.0.
where the state/action pairs are obtained by simulating any of the above models for T time-steps and following policy zh resulting in the trajectory f" {(s] a.) af)) Because both the model and policy are deterministic, we can compute the analytical policy gradient from only one rollout.
C.2 REWARD LANDSCAPES
In a first step, we will look the cumulative rewards as a function of the policy parameter for the different systems at hand: 1) the true system, 2) the approximate model without OPC and 3) the approximate model with OPC. Further, let's assume that the model mismatch is fixed to some arbitrary value. The resulting reward landscapes are shown in Fig. 7. We would like to emphasize several key aspects the plots: First, as one would expect the model mismatch leads to different optimal policies well misleading policy gradients for large parts of the policy parameter space. Second, the reward landscape for the model with OPC depends on the respective reference policy x" that was used generate the data for the corrections. Consequently, the correct reward recovered 0= 6". More importantly, the OPC reshape the reward landscape such that the policy gradients point towards the correct optimum (left plot). Lastly, even when using OPC the policy gradient's ''s sign is not guaranteed to have the correct sign (right plot). The extent of this effect strongly depends on the model mismatch, which we will investigate in the next section.
C.3 INFLUENCE OF MODEL. ERROR
As shown in the previous section, the estimated policy gradient depends on the current policy well as the mismatch between the true system and the approximate model. Fig. 2 depicts the (signed) differences between the true policy gradient well the approximated gradient model mismatch and the reference policy. Here, the opacity of the background denotes the magnitude of the error and the color denotes if the true and estimated gradient have the same (blue) or oppposite (red) sign. In the context of policy learning, the sign of the gradient is more relevant than the actual magnitude due to internal re-scaling of the gradients in modern implementations of stochastic optimizers such as Adam (Kingma& & Ba, 2015). In our example, even for negligible model errors (either in A or AB), the model-based approach can lead to gradient estimates with the opposite sign, indicated by the large red areas for the left figures in Fig. 2. On the other hand, applying OPC to the model, we gradient estimates are significantly more robust with respect to errors the dynamics.x
28

---

Published as a conference paper at ICLR 2022
Without OPC
With OPC
0.4
Reference policy T
Signed gradient errer
0.5
0.5
0.2
-1.0
1.0
0.0
1.5
-.5
0.4
0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
-0.4
Model Error AA Amadel
Ammo
Model Error AA
Amoda
(a) Gradient error when varying model error AA
Without OPC
With
0.4
Reference policy x
Signed gradient error
-0.5
-05
0.2
-1.5
1.5
0.2
0.2
0.0
0.4
-04
Model Error AB
moool
Bine
Model Error AB
muokd
Bine
(b) Gradient error when varying model error
Signed gradient error (see Equation equation 90) when using the approximate model to estimate the policy gradient without (left) and with (right) on-policy corrections (OPCs). Using OPCs increases the robustness of the gradient estimate with respect to the model error.
Figure 8:
C.4 INFLUENCE OF OFF-POLICY ERROR
Until now we have considered the case in which the reference trajectory used for OPC is generated with the same policy the used for gradient estimation, i.c., the on-policy setting. In this case, we have observed that the true return could be recovered (see Fig. 7) when using OPC and that the gradient estimates are less sensitive model errors (see Fig. 8). The off- off-policy case corresponds to the policy gains in Fig. that are different from the reference policy T" indicated by the dashed line. Fig. summarizes the results for the off-policy setting. Here, varied the policy error and the reference policy itself for varying model errors. Note that for the correct model, we always recover the true gradient. But also for inaccurate models, the gradient estimates retain good quality in most cases, with the exception for some model/policy combinations that are close to unstable.
29

---

Published as a conference paper at ICLR 2022
0.20
AA = 0.10
AA = 0.00
AA 0.10
AA 0.20
Reference policy x
-0.5
-1.0
-1.5
0.25
0.00
0.25
0.25
0.00
0.25
0.25
0.00
0.25
0.25
0.00
0.25
0.25
0.00
0.25
Policy Error
Policy Eror
Policy Error
Policy Eroo
Policy Error
A/
Reai
x"
Ax Rewl
AR
Aenul x²
A Nevol n
AN Reval X
(a) Model error A
AB 0.20
AB 0.10
AB 0.00
AB = -0.10
AB -0.20
Reference policy xe
-1.
0.25
0,00
0.25
-0.25
0.00
0.25
-0.25
0.00
0.25
-0.25
0.00
0.25
-0.25
0.00
0.25
Policy Error
Policy Enor
Policy Errr
Policy Enor
Policy Error
Tenii x"
Newl
x
AR
Tenul x
AT Mevol
n²
AN Teval
(b) Model error AB.
Figure 9: Signed gradient error due to off-policy data when using OPC. Note that we retain the true gradient in case of no model error.
C.5 ADDITIONAL. INFORMATION
Next, we provide some additional information about how we compute gradient distances, properties of linear systems, and exact numerical values used.
C.5.1 COMPUTING THE SIGNED GRADIENT DISTANCE
Ag
92
Figure 10: Sketch depicting the signed gradient distance Eq. (90). In this particular case, gradient 91 positive and g2 negative.
In order to compare two (1-dimensional) gradients in terms of sign and magnitude, we use the following formula
sign(g2) Ag.
if gi
d(gi. 92)
sign(g1) Ag. 92 with Ag Jarctan @1 arctan gal- (90)
sign(g) 92) Ag,
otherwise
The magnitude of this quantity depends the normalized difference between the tangent's 's angles Ag and is positive for gradients with the same sign and vice versa is negative for gradients with opposing signs. See also Fig. 10 for sketch.
30

---

Published as a conference paper at ICLR 2022
C.5.2 DETERMINING THE CLOSED-LOOP STABILITY FOR LINEAR SYSTEMS
For linear and deterministic systems, we can easily check if the system is (asymptotically) stable for a particular linear policy using the following standard result from linear system theory:
Theorem (Exponential stability for linear time-invariant systems (Callier Desoer, 1991)). The solution of X++1 Fx, is exponentially stable if and only if o DO, 1), i.e., every eigenvalue of F has magnitude strictly less than one.
our setting, this means that the closed-loop systems fulfilling the following are unstable,
A AA + (B + A
(91)
Le., the state and input grow exponentially. We therefore refrain from including unstable system in the results to avoid numerical issues for the gradients' computation. The respective areas in the plots not colored, e.g., bottom left corner in Fig. 8b.
C.5.3 NUMERICAL VALUES
The numerical values for all parameters used in the motivating example are given as follows:
True system dynamics: A = 1.0, B = 1.0
Initial condition: So = 1.0
Reward width parameter: a, 0.05
Optimal policy gain: 0" -1.0
Rollout horizon: T = 60
31

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide additional experimental results that did not into the main body of the paper.
D.1 COMPARISON WITH OTHER BASELINE ALGORITHMS
Fig. 11 shows our method compared to range of baseline algorithms. The results for all baselines were obtained from Janner et al. (2019) via personal communication. Note that all results are presented in terms of mean and standard deviation. The comparison includes the following methods:
MBPO (Janner et al., 2019),
PETS (Chua et al., 2018),
SAC (Haarnoja et al., 2018),
PPO (Schulman et al., 2017),
STEVE (Buckman et al., 2018),
SLBO (Luo et al., 2019).
HalfChectah
Hopper
Evaluation return x10²
Evaluation return 10
10
Steps x10
Steps 10"
Walker2d
AnTTruncatedObs
Evaluation 100
Evaluation return x10
50
100
150
200
300
50
150
200
250
300
Steps 2
Steps 1
OPC
MBPO(+)
MBPO
PETS
STEVE
SAC
PPO
SI.OO
Figure 11: Comparison of OPC against range of baseline methods three MuJoCo environments. We present the mean and standard deviation across independent experiments (10 for OPC and MBPO(*)). The original data for MBPO and the other baselines were provided by Janner et al. (2019). Solid lines represent the mean and the shaded areas correspond to mean + one standard deviation.
32

---

Published as a conference paper at ICLR 2022
D.2 ABLATION RETAIN EPOCHS
One of the hyperparameters that we found is critical to both OPC and MBPO(*), is retain epochs, i.e., the number of epochs that kept in the data buffer for the simulated data generated with the j with x = {OPC, model]. The results for a comparison are shown in Fig. 12. For MBPO(*), we found that for some environments (HalfCheetah, AntTruncatedObs) smaller values for retain epochs are helpful, i.c., simulated data is almost only on- on-policy, and for other environments larger values are beneficial (Hopper, Walker2d). For OPC on the other hand, we found that retain epochs = 50 almost always leads to better results.
HallCheetah
Hopper
Walker2d
AntTruncatedObs
Evaluation return x10>
MuloCo
5
Evaluation return x10>
1.5
PyBullt
0.5
100
100
200
200
Steps 10
Steps x 100
Steps x10
Steps X10
OPC (50)
OPC (5)
MBPO(*) (50)
MBPO (5)
Figure 12: Ablation study for OPC and MBPO(*) on four environments from the MuJoCo con- trol suite (top row) and their respective PyBullet implementations (bottom row). We vary the retain epochs hyperparameter (indicated by the number in the parentheses behind the legend entries), i.e., the number of epochs that are kept in the data buffer for the simulated data.
D.3 INFLUENCE OF STATE REPRESENTATION: IN-DEPTH ANALYSTS
In the following, we will have a closer look at the surprising result from Fig. 4 (left). In there, the results indicate that MBPO(*) is not able to learn stabilizing policy within the first 7'500 steps the RoboSchool variant of the CartPole environment. We hypothesize that the failure of MBPO(*) is due to mismatch of the simulated data and the true state distribution. As result, the policy that is optimized with the simulated data cannot stabilize the pole in the true environment.
To validate our hypothesis, perform the following experiment: First, we train policy "" that we know performs well on the true environment, leading to the maximum evaluation return of 1000. With this policy, roll out reference trajectory the true environment 7e {(,,, a,) To perform branched rollouts with the respective methods, OPC and MBPO(*), we use the learned transition models after 20 epochs (5'000 time steps) that were logged during full learning process for each method. We then perform 100 branched rollouts of length H = 20 starting from randomly sampled initial states of the reference trajectories.
Fig. 13 shows the difference between the true and predicted state trajectories (median and 95th percentiles) for each state in [z,, cos((), sin(())sin(() (in) it. v)). Since we start each branched rollout from a state on the real environment, the initial error is always zero. Across all states, the errors are drastically reduced when using OPC. Fig. 14 shows the predicted trajectories for the cosine of the pole's angle, cos( For MBPO(*), we observe that the trajectories often diverge and attain values that are clearly out of distribution, i.e., cos(0) 1.
33

---

Published as a conference paper at ICLR 2022
Lateral Position
cos(Angle)
sin(Angle)
State difference. ghue ped
0.1
0.0
-0.1
5
10
15
20
5
10
15
20
5
10
15
20
steps
time steps
time steps
Lateral Velocity
Angular Velocity
State difference: pod
0.0
-0.5
10
20
5
10
15
20
time steps
time steps
Figure 13: Difference in state trajectories strue - s pred from branched rollouts using fixed policy of length H 20 on the RoboSchool environment (cf. Fig. 4 (left)) with = [z, cos ()) sin( à, új.. The solid lines show the median across 100 rollouts and the shaded areas represent the 95th percentiles. With OPC (=), the simulated rollouts follows the true state trajectories much closer, whereas with MBPO(*) (===) the prediction errors quickly accumulate over time.
1.03
Transforned Angle: cos (8)
1.4
1.02
1.2
1.01
1.00
0.99
0.8
0.98
5
10 20
10 15
20
Figure 14: Trajectories of the second state (cos(0)) from 100 branched rollouts using fixed policy of length H 20 on the RoboSchool environment (cf. Fig. 4 (left)). Both plots present the same data but the differ in terms of scaling of the ordinate. With OPC (-), the respective trajectories remain around values close to which corresponds to the upright position of the pendulum. When using the standard predictive model from MBPO(+) (---), the state trajectories often diverge and the rollouts are terminated prematurely.
34

---

Published as a conference paper at ICLR 2022
1000
True retum
On-Policy Model Error
17h - 'Tal
200
100
300
Model Error
[7++1 - -f++11
Off Po--Policy
200
100
10
Iteration r of Pulicy Optimization
Figure 15: Empirical evaluation of the error terms in the policy improvement bound in Eq. (4) for the CartPole environment (PyBullet). We evaluate the respective terms for sequence policies that were obtained during different iterations n from full policy optimization run. The respective returns ih+1, The M 1: 7 are approximated the mean from 100 rollouts on the true environment and the respective models, pape (-) and pmodel (---). For the return on the true environment Th. (top), we additionally show the sample distribution the rollouts' returns. This nicely demonstrates how the policy smoothly transitions from failing consistently (n 8) to successfully stabilizing the pole (n 2 12). Additionally, note that the on-policy model error is almost always smaller for OPC compared to MBPO(*). which supports the theoretical motivation that method is build upon.
D.4 IMPROVEMENT BOUND: EMPIRICAL INVESTIGATION
this section, we empirically investigate to what extent OPC able to tighten policy improvement bound Eq. (4) compared to pure model-based approaches. As mentioned in the main paper, the motivation behind OPC is to reduce the on- on-policy еттоr and we assume that the off off-policy error is not affected too badly by the corrected transitions. Generally speaking, it is difficult quantify a-priori how OPC compares to purely model-based approach as this depends on the generalization capabilities of the learned dynamics model, the environment itself and the reward signal.
Here, we analyze the CartPole environment (PyBullet implementation) and estimate the respective error terms that appear in the policy improvement bound Eq. (4). To this end, roll out sequence of policies on the true environment (to estimate T)n and 1) and on the learned model with and without OPC (to estimate f, and 7m+ The sequence of policies
obtained during full policy optimization run (the policy was logged after each update) and we roll out each policy 100 times on the respective environnen//models. The corresponding learned transition models were similarly logged during full policy optimization. For OPC, we estimate the off- off-policy return fh using the learned model from iteration n and reference trajectories from the true environment that were collected under Tx. but then roll out the model with Tx+1- The results are shown in Fig. 15.
35

---

Published as a conference paper at ICLR 2022
OPC
Off po-pyl Return
750
500
MBPO(*)
1000
Off-pollicy Retum
750
500
250
2.9
Policy Stochasticity Multiplier B
Sample distributions of the return the CartPole environment (PyBullet) with in- creasing stochasticity of the behaviour policy Tolloot when rolling out with /OPC (top) and prodel /MPOO(* (bottom). The multiplier 8 quantifies the stochasticity of Trollout relative to the reference policy n, (Eq. (92)) such that higher values lead to more 'off- off-policy-nes:.
Figure 16:
D.5 OFF-POLICY ANALYSIS
In this section, investigate the robustness of OPC towards 'off-policy-ness' of the reference trajectories that are simulated under the data-generating policy тn. To this end, we manually increase the stochasticity of the behavior policy Trollout by a factor 8 such that
V(Trollout (- [)] = (- ss]]
(92)
and we keep the mean the same for both policies. Fig. 16 shows the distributions of the returns from 100 rollouts with varying degree of 'off-policy-ness' that the data-generating policy consistently leads to the maximum return of 1000. For 3 close to one, both OPC and MBPO(*) lead almost ideal behavior and correctly predict the return in more than 95% of the cases. As we increase the policy's 's stochasticity (from left to right), the respective performance the policy decreases until all rollouts terminate prematurely the extent of this effect is almost identical between OPC and MBPO(*). We conclude that OPC is, at least empirically, robust towards 'off- saff-policy-nest observe a more pronounced degradation of the policy's performance with increased stochasticity of the behavior policy, since this leads to observing more off-policy state/actions.
the CartPole environment (PyBullet). Note
2.3). Notably,
the reference trajectories. Otherwise, we should
36

---

Published as a conference paper at ICLR 2022
Algorithm 3 Original MBPO algorithm
1: Initialize policy TΦ. predictive model environment dataset D Deav, model dataset DD model
2: for N epochs do
3: Trrin model Pe on Denv via maximum likelihood
4: for E' steps do
5: Take action in environment according to zai add to Denv
6:
for M model rollouts do
7:
Sample 8, uniformly from Deny
8:
Perform s-step model rollout starting from 8, using policy Thi add to model
9:
for G gradient updates do Update policy parameters on model data: - >xx VoJa (0) D Dmndel)
Algorithm 4 Our version of MBPO algorithm, denoted MBPO(*)
1: Initialize policy a.. predictive model Pa, environment dataset D model dataset D model
2: for N epochs do
3: Train model po Donv via maximum likelihood
4: for E' steps do
5:
Take action in environment according to Tgi add Denv
6: for M model rollouts do
7: Sample s, uniformly from Denv
8: Perform kssttep model rollout starting from s, using policy Tai add to D model
9: for G gradient updates do
Update policy parameters on model data: - Φ A V Doddel
D.6 IMPLEMENTATION CHANGES TO ORIGINAL MBPO
Here, we provide details to the changes we made to the original implementation of MBPO. We denote our variant MBPO(*).
Episodic Setting MBPO updates the policy during rollouts on the real environment. Algorithms 3 and show the original and our version of MBPO, respectively. While the original version might be more sample efficient because it allows for more policy gradient updates based on more recent data, it is not realistic to update the policy during a rollout on the real environment.
Mix-in of Real Data As mentioned in the main paper, one of the key problems with MBRL is that the generated data might exhibit so-called model-bias. Biased data can be problematic for policy optimization the transition tuples possibly do not come from the true state distribution of the real environment, thus misleading the RL algorithm. In the original implementation of mix of simulated data from the model well observed data from the
MBPO, the authors true environment for policy optimization (https: : / i thub. com/ jar///jnnneen hwtpmm/m///////ooo hoabbo/bbooo ob/ 12cab517clbe 74124 74112ec33 fbe5c51 0e018d5 81 3ebf f /mbpo/ algoritthhit /mbpo. pyy# L430) to alleviate the issue of model bias. While this design choice might help in practice, it adds another hyperparameter and the refrain from mixing in data from the true environment, but instead only use simulated transition tuples staying true the spirit of MBRL.
influence of this parameter is difficult interpret. We therefore
Fix Replay Buffer The replay buffer that stores the simulated data Dindel for policy optimization allocated a fixed size of ro rollout, length xrollout batch si dizexres retain epochs, meaning that per epoch ro lout length rollout batcc si ze transition tuples are simu- lated and only the data from the last retain epochs are kept in the buffer (nttps: /github. com/janne rm/mbpp hhhopo/blbb//e http://bbbb/2cbb1111eeeaeaa http:///eeeeeeeeapoo////////aaaa 7412eee 3 0e 8d5813eb1 mbpo/ alggriiiiii sitritthmss /mbpo. py L3311.. As the buffer is implemented as a FIFO queue and rollouts might terminate early such that the actual number of simulated transitions is less than rollout length, data from older episodes as specified by retain epochs are retained the buffer. We assume that this is not the intended behavior and correct for that in implementation.
37

---

Published as a conference paper at ICLR 2022
InvertedPendulum
Hopper
HalfCheetah
Evalitation return x10
1.0
Evaluation return x10
Evaluation return x10
10
0.5
0.0
2.5
5.0
7.5
0
50
100
150
0
100
200
# Steps x x10
# Steps x 10
Steps x
OPC (ours)
MBPO 5%
MBPO 3 0%
SAC
Figure 17: Results based on original implementation Appendix D.7: Comparison of OPC to the model-based MBPO and model-free SAC algorithms. The two MBPO variants differ in terms of the mix-in ratio 3 of real off-policy transitions in the training data- a critical hyperparameter. All model-based approaches outperform SAC in terms of convergence for the high- disensinnni tasks. Moreover, on the highly stochastic Hopper environment, OPC. outperforms both MBPO variants and does not require additional real off-policy data.
D.7 RESULTS FOR ORIGINAL MBPO IMPLEMENTATION
The following results were obtained with the original MBPO implementation without the changes described Appendix D.6. Consequently, the results for OPC in this section also do not include these changes.
Comparative Evaluation
We evaluate the original implementation three continuous control benchmark tasks from the MuloCo control suite (Todorov et al., 2012). The results for OPC, two variants of MBPO and SAC are presented in Fig. 3. Both OPC and MBPO rollout horizon of H 10 generate the training data. The difference between the two MBPO variants lies in the mix-in ratio /3 of real off-policy transitions the simulated training data. Especially for highly stochastic environments such as the Hopper, this mix-in ratio is critical hyperparameter that requires careful tuning (see also Fig. 18). Fig. indicates that OPC is par with MBPO on both the InvertedPendulum and HalfCheetah environments that exhibit little stochasticity. On the Hopper environment, OPC outperforms both MBPO variants. Note that the mix-in ratio for MBPO is critical for successful learning (the original implementation uses 3 = 5%). OPC on the other hand, does not require any mixed-in real data. SAC learns slower on the more complex Hopper and HalfCheetah environments, re-iterating that model-based approaches are significantly more data-efficient than model-free methods.
Large Ablation Study - Hopper
The full study investigates the influence of the following hyperparameters and design choices:
Rollout length H and total number of simulated transitions N.
Mix-in ratio of real transitions into the training data B.
Deterministic or stochastic rollouts (for MBPO): Current state-of-the-ate methods in MBRL rely on probabilistic dynamic models to capture both aleatoric and epistemic uncertainty. Accordingly, when rolling out the model, these two sources of uncertainty are accounted for. However, we show that in terms of evaluation return, the stochastic rollouts do not always lead to the best outcome.
Re-setting the buffer of simulated data after a policy optimization step: We found that re-setting the replay buffer for simulated data after each iteration of Algorithm can have a large influence. In particular, the replay buffer is implemented as a FIFO queue with fixed size. Hence, the buffer is not emptied after each iteration, it still contains (simulated) off-policy transitions.
Fig. 18 presents the results of the ablation study. We want to highlight few core insights:
38

---

Published as a conference paper at ICLR 2022
1. When choosing the best setting for each method, OPC improves MBPO by large margin (bottom row, right). Generally, for long rollouts H = 20 (bottom row), OPC improves MBPO.
2. Across all settings, OPC performs well and is more robust with respect to the choices of hyperparameters (e.g., bottom row left and center, third row left). Only for few exceptions, re-setting the buffer can have detrimental effects (c.g., top right, third row right).
3. Mixing in real transition data can be highly beneficial for MBPO (second row left and center) but it can also have the opposite effect (bottom row).
4. Using deterministic rollouts be beneficial (third row right and left), detrimental (third row center) or have no influence (bottom row left) for MBPO.
5. It is not clear, if re-setting the buffer after each iteration should overall recommended or not. This remains open question left for future research.
39

---

Published as a conference paper at ICLR 2022
Evaluation return x10*
Evaluation return xI0*
Exaluation return x 100
50
150
50
150
# Steps x10
Steps
x10
Steps x 100
Evaluation return x10
Evaluation retur x103
Evaluation return x103
50
100
150
50
100
150
50
100
150
# Steps x 10
# Steps x x00
# Steps x <00
Evaluation retur 110
Evaluation return X10
Evaluation returm x <00
3
100
50
150
150
# Steps 10
# Steps x10*
Steps x 10
Evaluation return x10*
Evaluation return x10*
Evaluation return x10*
50
100
150
50
100
150
50
150
# Stops x10"
Steps
x10
Steps x x100
OPC reset buffer
OPC not reset butfer
MBPO + reset buffer + deterministic
MBPO + not reset buffer deterministic
MBPО reset buffer stochastic
MBPO reset buffer stochastic
Figure 18: Results based on original implementation Appendix D.7: Large ablation study on the
Hopper environment, investigating the influence various hyperparameters and design choices. Mix-in ratio of real data & (columns): 0%, 5%, 10% from left to right. Rollout length H (rows): 1, 5, 10, 20 from top bottom.
40

---

Published as a conference paper at ICLR 2022
E CONNECTION BETWEEN MBRL AND ILC
In this section we compare optimization- based or so-called norm-optimal ILC (NO-ILC) with MBRL. In particular, show that under certain assumptions can reduce the MBRL setting to NO-ILC. This comparison is structured as follows: First, we review the basic assumptions and notations for NO-ILC. While there are many variations on NO-ILC, we will only consider the very basic setting, i.e. linear dynamics, fully observed state and deterministic state evolution. Then, based on the lifted state representation of the problem, we derive the solution the optimization problem that leads to the input sequence of the next iteration / rollout. Next, we will state the general MBRL problem and pose the simplifications that we need make in order to equivalent to the NO-ILC problem. Last, we show that the solution to the reduced MBRL problem is equivalent to the one of NO-ILC.
E.1 NORM-OPTIMAL ILC
The goal of NO-ILC to find sequence inputs a [ag ....., a...,,a-1) -] TT with length + such that the outputs = [y ........... simplest setting we assume that the system evolves according the following linear dynamics
y yy] follow desired output trajectory y = ........ .11 In the
S As; + Ba + d,
(93)
Cs:,
(94)
with state S, E Rds,, action a, E Rdd, output y: E IR and disturbance d, E Rdds One of the major assumption in ILC that the disturbances d, are repetitive, meaning that the sequence d == .d......ddd dd] does not change (or only varies slightly) across multiple rollouts. While these disturbances can be considered come from some exogenous error source, one can also interpret these as unmodeled effects of the dynamics, e.g., nonlinearities stemming from friction, aerodynamic effects, etc.
For the derivation, let's assume C I so that we re operating an MDP. Consequently, the goal of ILC to track a sequence of desired states ò instead of outputs y. order to find an optimal input sequence, we minimize the squared 2-norm of state's deviation from the reference for each time-step & of the sequence, i.c., e, = š; St. As is common in the ILC literature, we make use of conveniently write the state evolution for one
the so-called lifted system formulation such that Assuming zero initial state (which can always rollout using a single matrix/vector multiplication. be done in the linear setting by just shifting the state by constant offset), obtain the following formulation
0
B
0
AB
B
O
== +d,d, with F
E (Td
(95)
A
AB
B
Thus, we can write the state-error the j-th iteration of ILC in the lifted representation (recall that the disturbance d does not change across iterations)
( = ((() c (++1) = (i)) -F(a(++1) (1)).
- d,
(96)
(97)
Now, the resulting optimization problem then becomes
(++1) arg min J (a (a(ttt)) with J (a (+(+11) 1 2
ee
1
In order be less sensitive to noise and the inherent stochasticity of real-world problems, one typically also adds a regularizing term that penalizes the changes in the input sequence. While this
additional penalization term can slow down the learning process it makes more robust by avoiding overcompensation to the disturbances. The full objective then becomes
1 |a (1+1)
(i+1)
e
(6+1)
2 + M
a (0) W1
(98)
2
41

---

Published as a conference paper at ICLR 2022
where we additionally added positive semi-definite cost matrices M, W for the respective norms in order to facilitate tuning of the corresponding terms in the objective. Given the regularized cost function obtain the optimal sequence of inputs at the next iteration
a (i+1) (8) + (F + W) "P)''''" FF Me!
-
(99)
E.2 MODEL-BASED RL TO NORM-OPTIMAL ILC
Assumptions
In order to show the equivalence of MBRL and NO-ILC we need to make some assumptions to the above stated optimization problem.
No aleatoric uncertainty in the model, i.e., no transition noise We = O Vt.
Typically in RL assume the policy to be stationary, however, in this setting will allow for non-stationary policies that are indexed by time (the result will not necessarily depend on the state such that we essentially just obtain feedforward control sequence. To include feedback one can always just combine the feedforward signal with a local controller that tracks the desired state/action trajectory).
The reward is given as the negative quadratic error of the state w.r.t. desired state trajectory, r(s,, a.) = - 2 ||
Typically in RL. we have constraints on the policy such that it does not change too much after every iteration, see e.g. TRPO, REPS, etc. While in the mentioned approaches, the policy are often constrained in terms of their parameterization vectors, we constrain it as | (<((+1)) - ( S 6 Vt.
Assume that the model is given by f((s = As + Ba + d,, where and B are fixed system matrices and d, is time-dependent offset that we learn.
The learned dynamics model Given state/input pairs of the i-th trajectory 7(0 from the true system, we can now improve our dynamics model. In particular, if we minimize the prediction error over (()), we obtain
{(s]") a
(8) Ho
(9) s11 a =
(As
(3) + Ba,"
(6)
(100)
The resulting dynamics for the optimal control problem in Eq. (1) become
f1(0) (s, a) = 1 + A(s - s( + B(a-a()) - (())
(101)
which are the error dynamics around the trajectory. Now in the noisy case, taking the last trajectory is not necessarily the best thing one can do. F.g.. (Schöllig & D' Andrea, 2009) instead integrate the information of all past trajectories via Salman-filtering In the fully observed case, one way to think of this as low-pass filtering de in order to account for the transition noise w.
The resulting MBRL problem
Now, let's plug in all assumptions into the MBRL problem and
have a look at how to solve
(++1)
arg min url
Σ
|| - s1l12
(=0
S41 A(s ((sss)))
(6)
((0)) B(a (.)))
(102)
S
a mt
|
12 < Cn
Vt
where we have flipped the reward's sign to transform it into minimization problem. In the presented form, this optimization problem has well-defined unique solution, however, it is a-priori not clear if the trust-region constraint is active for some time-steps. To facilitate an analytical solution to equation 102, we incorporate the constrain on the policy's 's stepsize by soft-constraint such that
(++1)
arg min
Σ
||,
st 11
C
-
प
(6)
- 1el
2
AT
t=0
(103)
S41 + A(s (00) B(a 20))
(1)
8
a We
42

---

Published as a conference paper at ICLR 2022
with OF being constants that weight the respective trust-region terms.
Generally, the MBRL problem cannot be solved analytically due to the (possibly highly non-linear, non-differenniaall, etc.) reward signal and the need for propagating the (uncertain) dynamics model forwards in time. However, using the simplifying assumptions above, the reduced MBRL problem equation 103 can in fact be solved analytically. We circumvent the equality constraints by predicting the state trajectory using the error corrected dynamics such that we obtain lifted dynamics formulation similar the analysis for ILC,
Fa ( + a
with d g(()
Fa (9)
(104)
with s, a denoting the stacked states and actions of the recorded trajectory. Using this notation, the optimization problem reduces to
(++1)
arg min
s0
1/ (1)
x I
(105)
with C = diaglCo -.... Cil-- By inserting equation 104 into equation 105 obtain the closed- form solution as
(++1) ( (F F (s-d(")
= (F +
F (0) (106)
which, clearly, is equivalent to equation 99 for M I and W = C
E.3 EXTENSIONS
In the previous section, we analyzed the most basic setting for NO-ILC, i.e., linear dynamics, no transition noise in the dynamics and state/input constraints. Some of these assumptions can easily be liftd to genearlized the presented framework.
Nonlinear Dynamics Instead of the system being defined by fixed matrices A, B. can just linearize a non-linear dynamics model [s, a) around the last state/input trajectory such that
0ʃ
0ʃ
A
(107)
Os
da
8.=8 a
8,=5 a.=a
and the corresponding dynamics matrix for the lifted representation becomes (dropping the superscript notation indicating the iteration for clarity)
B
0
A
B1
0
F
(RdS(T+I)xdaT
(108)
A
AqB
B₂
State/Input Constraints Recall that we could solve the quadratic problems equation 98 and equation 103 analytically because we assumed no explicit inequality constraints on the state and inputs. practice, however, both states and actions are limited by physical or safety con- straints typically given by polytopes. While closed-form solution is not readily available for this case, one can easily employ any numerical solver to deal with such constraints. See, e.g., https: //n. wi k pedia. wwt.////////.... http///winn../////// a comprehensive list of available solvers.
wtrptttccppoorrammmmmmmm for
Stochastic Dynamics Schöllig & D' A'Anrea (2009) generalize the ILC setting by considering two separate sources of noise: 1) transition noise in the dynamics, i.e., S f (St, ar) w/ well varying disturbances, i.e., ((+1)) ( wh Based on this model, Kalman-Filter in the iteration-domain is developed that estimates the respective random variables and the ILC scheme is adapted account for the estimated quantities.
43


Reviews:
[{"review_id": "hiY019eDyrg", "reviewer": "Reviewer_xLi5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper well structured and easy to read. I liked in particular the deep analysis of the illustrative example in section 4.1 and appendix B. The authors provide also an extensive ablation on what happens when varying the rollout length. \n\nThe intuition I got is that this method is able to retain very low error on the last term of equation (5), while at the same time it should also have a low off-policy model error since a model is being used. However, it is not clear to me how OPC behaves compared for instance to MBPO, when it is evaluated on all three terms on the right side of equation (5). Could it happen that while the on-policy error of OPC is low, the off-policy error becomes much bigger? Could it happen that both errors are low, but because of the model choice in OPC the model policy improvement is extremely lower? Perhaps this is something that can be analytically derived for a very simple MDP like the one in section 4.1", "summary_of_the_paper": "This paper addresses on-policy errors in model-based Reinforcement Learning. The authors decompose the policy improvement bound in terms that depend on the off-policy and on-policy model errors. They first present two major techniques, one purely based on the replay buffer, which asymptotically obtains zero on-policy error, and one based on a learned model, which leverages off-policy data. Then they propose On-Policy Corrections (OPC), a combination of the two techniques which can use a model and still maintain low on-policy error using the replay buffer. This method has asymptotically (with infinite data) zero on-policy error when using deterministic policies. If the policy is stochastic, the error grows exponentially with the trajectory length T. The authors implement their algorithm on top of MBPO and show experimentally that OPC-MBPO has superior performance in some environments.", "summary_of_the_review": "This paper combines the use of a replay buffer and a model to lower the on-policy model error. The experiments seem convincing and the method is very well analyzed.\n\nWhile I am not familiar with the related literature and I might have missed something important (hence I lowered my confidence accordingly), I have the impression that this work is solid and would bring a nice contribution to the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xr_7XJZVKhX", "reviewer": "Reviewer_A4Tf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The idea of the paper is clear.\n\n2. The presence of theoretical results and a fitting empirical side.\n\nWeaknesses:\n\n3. The idea of correcting samples with the model sounds immediate, which means more is required of the analysis: \n\n(a) Since there is a trade-off between model based and sample based, how to implement\\calculate this trade-off in the proposed correction? \n\n(b) How do the theoretical results compare to model based and replay based results?\n\n4. I don't understand why include the MujoCo experiments if you can't see any improvement there. The improvements in PyBullet are mostly due to the initial state randomization, how does that fit the general theme of the paper?\n\n5. The comparison is made against SAC and MBPO, but why does the first method proposed (replay buffer based simulation) not compared? \n\n6. Consider adding the reference to the highly related (though not very new) work:\n\nBatch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories\n\nRaphael Fonteneau, Susan A. Murphy, Louis Wehenkel, and Damien Ernst\n\n\n", "summary_of_the_paper": "The paper proposes a way to combine observed data with a learned model to mitigate the cost of each choice separately - the observed data is over-fitted to a specific policy while the learned model has approximation errors.\nThe authors mix the two options using probabilistic corrections based on the learned model applied to observed transitions. \nThe authors also provide convergence results for their algorithm,", "summary_of_the_review": "The paper considers a straightforward combination of two methods which does not seem highly significant. \n\nThe paper is also somewhat lacking in its analysis and empirical claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9tnsaTCmYjt", "reviewer": "Reviewer_XAKT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I'd say this is a relatively thorough examination of a simple but seemingly effective technique. As far as I'm aware, the approach is novel in the context of MBRL, and I can see it being widely used by the community, so I recommend acceptance. \n\n**Issue 1:** Issues with Section 3.1/the bound in Eqn (5). \n\nClarity: The off-policy model error and on-policy error are just measuring the accuracy of $\\hat \\eta_{n+1}$ and $\\hat \\eta_n$. If we assume the data is generated by some $\\pi_0$ then both $\\pi_{n+1}$ and $\\pi_n$ will be \"off-policy\", so really this relies on $\\pi_n$ being the data-generating policy, which isn't stated until a few paragraphs later. \n\nMisleading: The bound also implies that our ability to estimate $\\eta_n$ is an important part of policy improvement, but in reality... it isn't. Ultimately the only term we care about is the final performance $\\eta_{n+1}$ (i.e. eqn (1)), and we cannot, generally, say that the ability to estimate the performance of the data-generating policy will impact the final performance. \n\nThe problem is that the bound is presented as a decomposition of policy improvement, but it's actually a bound on the model's estimate of how much better the current policy is better than the data-generating policy. So for example, if we instead bound $\\eta_{n+1}$ (as done in Janner et al.), by adding $\\eta_n$ to both sides of the equation, we arrive at $\\eta_{n+1} \\geq \\hat \\eta_{n+1} - \\text{off-policy model error}$, which does not include any on-policy term. \n\nOf course, this doesn't invalidate the technique, as I would still expect that improving an estimate of the on-policy policy would impact the model's ability to estimate an off-policy policy but I'm not sure this is as fundamental as presented by the authors.\n\n**Issue 2:** Theoretical results are nice but don't say anything comparatively. In other words, it's not clear if the presented bound is improved by including OPC. \n\n**Issue 3:** Issues when using OPC off-policy / suggested analysis. \n\nIn any MBRL policy improvement context there will be some amount of off-policy evaluation. If we are estimating the on-policy then this cancellation is convenient, however, when estimating a trajectory which is quite different than the on-policy trajectory, i.e. where $s_t$ and $\\hat s^b_t$ are not similar, then $\\hat s^b_{t+1} - \\hat f(\\hat s^b_t, \\hat a^b_t)$ seems like just noise added to the estimate $\\hat f(s_t,a_t)$, and it seems like this technique would be more harmful than beneficial. This situation also arises naturally when rolling out the off-policy trajectory for a few time steps, as we would expect that it will be quite different from the on-policy trajectory. \n\nAn interesting experiment would be to test the model's ability for off-policy evaluation (OPE) over varying levels of \"off-policy-ness\" (i.e. adding noise to the on-policy/behavior policy). This is a common experiment in OPE literature. I would expect that when near on-policy OPC would be helpful, but it's not clear OPC will be helpful when acting more off-policy. \n\n**Strengths**\n* Empirical results are strong/convincing. 10 seeds are used.\n* Reproducibility is high. The method is simple/easy to implement.\n* The appendix is thorough and includes a lot of additional experiments and details. \n* I can see the method being widely adopted by the MBRL community. \n\n**Minor Comments**\n- I think Figure 1.b would be clearer if the on-policy and off-policy estimates were separated into two separate graphs (in other words, cutting the figure in half). \n- Figure 5 is seemingly used to suggest that higher performance = higher data diversity, but I'm not sure how the authors made that jump. I can see that longer rollouts provides a higher performance, but there's no guarantee that rolling out is necessarily more diverse than the data gathered from the initial policy. \n- I find Figure 2 unclear. The meaning of dotted lines is not defined and it's not clear how the figure should be interpreted.  \n- A very similar technique has been proposed in value-based RL [1] (ironically named off-policy corrections instead) and should be cited.\n- I think an interesting experiment would be applying OPC to different models (rather than just the models used by MBPO).\n- Do the authors have any insight as to why this technique matters much more on the PyBullet domains? \n\nReferences:\n- [1] Harutyunyan, Anna, et al. \"Q ($\\lambda $) with Off-Policy Corrections.\" (2016).\n", "summary_of_the_paper": "The authors introduce a model-agnostic technique for improving estimates of on-policy data and show improvements on MuJoCo and PyBullet domains. \n", "summary_of_the_review": "Overall, I'd say this is a relatively thorough examination of a simple but seemingly effective technique. As far as I'm aware, the approach is novel in the context of MBRL, and I can see it being widely used by the community, so I recommend acceptance. I have concerns regarding Section 3.1 which is the main motivation of the method, which limits my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4g2z7J-Bn9J", "reviewer": "Reviewer_8GLt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thought this was a nice paper. The writing was generally clear and the theoretical justifications were interesting. The connection to ILC was also an interesting addition.\nWhile the idea in itself feels rather incremental, it is saved by the thorough analysis, empirical results and extensive ablations. \n\nI enjoyed Sections 1 and 2 and throught they were well written. Although the jump to Section 3 feels rather abstract. I think Section 2 discusses MBRL too broadly when the paper should really focus on the MBPO-style of MBRL since OPC is limited to this section.\n\nSection 3 proposes an improvement bound that can be decomposed onto on-policy and off-policy model-based improvement. These quanitities feel quite abstract to reason about, since they ultimately rest on the environment, choice of approximator and training protocol. However the paper focuses on compounding model biases which is reasonable. \nOne slighly strange aspect of the OPC model is its deterministic treatment, such that the aleatoric uncertainty is accessed by sampling different buffer indices, which to me indicates a rollout is deterministic after the choice of index b?\n\nThe paper assumes that the model error is constant and time-dependent, but there is no empirical study of how well this assumption holds. Given that the true ground-truth simulator is available, the model bias could be evaluated empirically to see how well this assumption holds in practice. \n\nWhile OPC appears superior to MBPO* in Figure 3, my understanding this that this MBPO sees no real data from the replay buffer. While Figure 11 shows the standard MPBO where it is comparable if not superior. There appears to be many discrepencies between Fig. 3 and FIg. 11, for example\nAre these figures meant to report the same results? (i.e. median vs mean?).\nHalf Cheetah: OPC mean is noisy and below 10 in Fig. 11, but smooth and above 10 in Fig. 3 \nAnt: MBPO* mean clearly goes up to 6 in Fig. 11, but in Fig. 3 it is at more like 4.\n\nMoreover ,OPC demonstrates signifiant improvement over MBPO on the pybullet environments. However, I am not sure how to feel about these experments, since the dramatic performance shift speaks to the evaluation assumptions in general rather than the performance of OPC. If the algorithm performance does not translate between simulations, it begs the question how reflective these simulated settings are of these algorithms performance. \n\nThe feature representation experiment in Figure 4 is also disconcerting, since the sine cosine features contain an inductive bias that should aid generalization.\nIs the idea that the model predicts theta, but uses the features as an intermediate transform for the model, or is the model also predicting the featurespace? In which case, predicting sine features over-complicates prediction since you have the manifold constraint.\nIs there not a bug in theta being whitened before the feature transform for MBPO?\nI dont see how the OPC addition relates to this issue of feature space, so I don't see how this experiment demonstrates the benefits of OPC. Also, as the code was not submitted I cannot check for bugs.\n\nThe paper is missing a reference to \nModel Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning, Morgan et al. ICRA 2021\nwhich does MBPO with path integral MPC rollouts. \n", "summary_of_the_paper": "The paper considers model-based reinforcement learning (MBRL) of the MBPO flavour, where model-free RL methods are accelerated using rollouts of a learned model. \nTHis paper proposes to alleviate (off-policy) model bias per rollout using the (on-policy) data and calls this approach on-policy correction (OPC). \nWith this perspective, a comparison is made between MBRL and iterative learning control (ILC).\nThis method is compared to MBPO on standard mujoco tasks and pybullet versions.  ", "summary_of_the_review": "While this looks like it is a solid paper on what is arguably a somewhat incremental idea, I have some issues with the experimental results that need ironing out before I can accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study of on-policy data in the context of model-based reinforcement learning and proposes a way to ameliorate the resulting model errors.

This is a timely and interesting contribution, and all reviewers agree on the quality of the manuscript.
Please incorporate all the remaining feedback from the reviewers.

Minor comment: There might be interesting points of contact between this work and the concept of objective mismatch (https://arxiv.org/abs/2002.04523)
Decision: Accept (Poster)