When to Update Your Model: Constrained Model-based Reinforcement Learning
Authors: Tianying Ji
Yu Luo
Fuchun Sun
Mingxuan Jing
Fengxiang He
Wenbing Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model-Based Reinforcement Learning
monotonic improvements
event-triggered mechanism

Abstract: Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL. Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. These discoveries encourage us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns. Motivated by these analyses, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model.  Experiments show that CMLO surpasses other state-of-the-art methods and produces a boost when various policy optimization methods are employed.

When to Update Your Model: Constrained Model-based Reinforcement Learning
Tianying Ji', Yu Luo', Fuchun Sun*.,, Mingxuan Jing", Fengxiang He², Wenbing Huang
Deparrment of Computer Science and Technology, Tsinghua University
2 Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences JD Explore Academy, JD.com Inc
Gaoling School Artificial Intelligence, Renmin University China Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China [jity20, (ooyyl1)@maalss e eeiighhaaadu...... edu. cn; fcsun@teinghua edu. cn; jingmingxuan@issa ac. en; fengxiang.. f he@gmail. com; hwenbing@126 com
Abstract
Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose novel and general theoretical scheme for non-decreasing perfor- mance guarantee of MBRL. Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. These discoveries encour- age us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL. A further example demonstrates that learning models number of explorations benefit the eventual returns.
from a dynamically-varyyie Motivated by these analyses, design simple but effective algorithm CMLO (Constrained Model-shift Lower-bound Optimization), by introducing an event- triggered mechanism that flexibly determines when update the model. Experi- ments show that CMLO surpasses other state-of-the-ar methods and produces a boost when various policy optimization methods are employed.
Introduction
Reinforcement learning (RL) has driven impressive advances in many complex decision-making problems in recent years [36, 48]. Many of these advances are obtained by model-free (MFRL) meth- ods, whose desirable asymptotic performance yet comes at the cost of sample efficiency. Hence their applications are mostly limited to simulation scenarios [45, 33, 35]. In contrast, Model-Based RL (MBRL) methods, which learn transition model directly from orders-of-magninuue fewer samples and then derive the optimal policy from the learned model, have become an appealing alternative in small-data and
practical cases [37, 9, 41, 18).
In general, MBRL methods altemnate between the two stages: model learning and policy optimiza- tion (e.g. the general Dyna-style [50, 51]). A more accurate model will lead to better policy. Vari- ous attempts have been proposed to improve model accuracy by investigating high-capacity models (the model ensemble technique [27, 8] and better function approximators [15, 38)) amending the policy optimization stage based on model bias [24, 39, 28, 20, 7, 57]. Even so, their resultant models
"Corresponding authors: Fuchun Sun.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

are just accurate in local and relative sense, since the learning is conditional on fixed number of state-action tuples explored by the policy the current step. other than the full transition dynamics of the environment. Indeed, it is tricky to determine how much we should explore. Insufficient exploration would trap the model and the following policy optimization, whereas excessive newly- encountered itateacction pairs would confuse the model and later cause policy chattering. To derive a "truly" accurate model, we need a smarter scheme to choose different numbers of explorations at different times, instead of the unchanged setting in current methods.
From an optimization point of view, the thinking of how to derive an accurate model for MBRL in a global sense also motivates us to investigate the monotonicity of the optimization target (i.e. the return of the learned model and policy in MBRL), which, unfortunately, is less explored and not well guaranteed in current works. However, discussing the monotonicity guarantee for MBRL is challenging by any means, arising from the coupling of the model learning and policy optimization processes. Although there has been recent interest in related subjects, most of the theoretical works seek to characterize the monotonicity in terms of a fixed model of interest [49, 34, 20, 12, 28), which does not naturally fit our case when the model is dynamically shifted.
In this paper, we study how to guarantee the optimization monotonicity theoretically, upon which we then develop an event-triggered strategy that learns the model from a dynamically-varying num- ber of explorations. In particular, we interestingly find that the lower bound of the performance improvement between two adjacent alternation steps in MBRL is dependent on the one-step model accuracy plus the constraint of the model shifts under certain mild assumptions. This discovery encourages us to formulate constrained optimization problem, in order to permit positive perfor- mance improvement and thus the optimization monotonicity for MBRL.. We also give feasible solution example to show that dynamical alternation between model learning and policy exploration does benefit performance monotonicity. To resolve the constrained optimization problem, we de- sign simple but effective algorithm CMLO (Constrained Model-shift Lower-bound) equipped with an event-triggered mechanism. This mechanism first estimates whether the model shifts meet the constraint and then decides when to train the model.
We evaluate CMLO on several continuous control benchmark tasks. The results show that CMLO learns much faster than other state-of-the-an MBRL methods and yields promising asymptotic per- formance compared with the model-free counterparts. Note that our optimization framework is general and can be applied to different backbones of policy optimization algorithms, which is also ablated in our experiments.
2
Related works
Model-based reinforcement learning methods have shown great potential for sequential decision- making both in simulation and in the real world due their sample efficiency [9, 21]. Generally, these MBRL algorithms can be grouped into several categories to highlight the range of uses of predictive models [55]. And our work falls into the Dyna-style category. In Dyna-style algorithms. training alternates between model learning under policy iterations with the real environments, and policy optimization using the model rollouts [50-52, 13]. Many attempts have been devoted to improving these two stages.
For model learning stage. previous main concerns are function approximators and training objectives. The dynamics approximator has advanced from Gaussian processes [26, 9]. time-varying linear dynamics [29, 30] to neural network predictive models [15, 38). And training objectives vary from Mean Square Error (MSE) [38, 34), Negative Log Likelihood (NLL) [8, 20]. etc. Moreover, the deep ensemble technique appealing for improving the robustness model error. Our method adopts an ensemble of probabilistic networks similarly as in [8, 20].
The policy optimization stage allows Dyna-style algorithms to leverage various off-the-shee model- free methods, such as SAC [16], TRPO [44], and TD3 [14]. Much owing to the progress of model- free methods, our method is to invoke any reasonable optimization oracle for the empirical models, rather than entangle a particular policy optimization algorithm.
A consensus of MBRL. is that a smart policy requires an accurate model. However, model bias cannot be eliminated because the state-action distribution of the samples in the model learning stage and policy optimization stage is quite different. Many prior works attend to this distribution mismatch

---

issue and then tailor the data used for policy optimization according to the model bias. Janner [20], Buckman al. [7] encourage truncated rollout lengths. Besides, the ratio of real to model-generated data can be dynamically tuned according to the model uncertainty [24, 39, 28). Yu et al. [57] penalizes rewards by the model uncertainty. Our method incorporates the truncated model rollouts mechanism. Moreover, we further explore how real interactions affect overall performance which is rarely studied before. We construct an event-triggered mechanism to cope with overfitting in small-data regime and suffering generalization error when facing a drastic distribution shift.
Monotonic improvement guarantee has been fundamental concern both model-free and model- based avenues. In MFRL methods, both CPI [22] and TRPO [44] can be understood as approximat- ing and optimizing the performance gap by forcing the new policy to be not too far away from the current policy. However, in Model-based settings, their trust-region constraints cannot directly be satisfied because the policies highly depend on the randomness of the models. While constructing such a bound for performance gap is straightforward. it has not been explored in previous MBRL theorectical analyses, instead they [49, 34, 20, 12, 28] turn to bound the discrepancy between re- turns under model and those in the real environment. Although they could guarantee that the lower bound of policy performance improves under a certain model, this guarantee may face several issues regarding model updating. In contrast, we construct the performance difference scheme for MBRL algorithms and perform monotonicity analysis under this scheme.
Another line of theoretical works focus on regret bounds [10, 23] or sample complexity proper- ties [1, 5, 47], focusing on the convergence performance and sample complexity for model-based approaches.
3 Preliminaries
Markoy Decision Process A discounted Markov Decision Process (MDP) is a quintuple M AS, PM: T,, 7), where S is the state space, A represents the action space, PM denotes the transi- tion function, r : x A + [-R, R) stands for the reward function, and y E (0, 1) is the discount factor. For fixed policy and model M, we define V (u) the return of the model M with the starting state distribution μ4, and (=( denotes the returns under the real environment,
(M)
E 4L Car
Σ T'rM(s (e)l so T=0
(3.1)
We make a mild assumption that the model M to be identical to the real MDP except the transition function. Let dii (s, (1) denote the visitation probability s, when starting at su ~ J and follow- ing Te under the dynamics PMG,- We will omit it as dM, henceforth for brevity. Besides, we denote M as a (parameterized) family of models of interest, and let II be a family of policies.
Generative Model
Many previous works [31, 25, 1] focus assuming an access to the generative model, we collect samples for each state-action pair (s,a)
stylized generative model. By
A: sy iid P(-(s,a) a) which allows us construct an empirical model defined as follows:
N = s
E E S, p(s'(s,
Σ
1fs s,a
(3.2)
N 1=1
where 1(-) is the indicator function. This leads an empirical MDP M = -S,AA,,
4 Monotonic improvement under model shifts
This section provides theoretical analyses for monotonic improvement of MBRL, factoring in the interdependence between policies and models. We first construct a general scheme for a non- decreasing performance guarantee and follow it up by characterizing the lower bound when shifting the model. Towards a non-negative lower bound, we restrict the model shifts and then obtain a re- fined bound. These discoveries encourage to translate the bound maximization to a constrained optimization problem to permit monotonicity. By deriving an instance solution under the generative model setting, demonstrate the merits of the dynamic model learning interval.
3

---

4.1 Monotonic improvement with policy optimization oracle
Our goal is to construct a general recipe for a monotonicity guarantee. Naturally, we seek build performance difference scheme for bodel--bsed algorithms.
Definition 4.1 (Performance Difference Bound Scheme). Vxilds(p) (p) denotes the return of the policy n; in the real environment, whereas n, is derived from the dynamical model M, E M. Then, the lower bound on the true return gup of ιı and T2 can be stated in the form,
V V (n) 2 C.
(4.1)
Such a statement guarantees that the policy allows non-decreasing performance in the real environ- ment when C is non-negative.
Although there has been interest in non-decreasing performance guarantee, previous works [34, 20] commonly derive under "discrepancy bound" scheme disregarding the model shifts (i.e. M1 M2). Their results imply that once a policy update л - #2 increases the returns under the same model Mr VM, ), the lower bound the policy performance evaluated in the real environment improves accordingly, inf(V") 3 > (n(
When the model shift is introduced, establishing the performance difference bound turns out to be rather difficult, mainly due to the coupling of the policy optimization and model learning: the estimated model is generated from the policy explorations, while the policy derives from the model rollouts. Hence, we need to consider the performance gap arising from the integration of the two stages, which, unfortunately, has never been explored before. Since the performance of the policy with fixed model is already well guaranteed, is natural to make the following assumptions.
Assumption 4.2 (Policy Optimization Oracle). The policy optimization oracle is defined as the one that takes as inpul a model M and returns a co Optimal policy π satisfying: Vūs(H) Copt S (u) < Vīr (J).. We assume that our policy optimization stage always meets the policy optimiza- tion oracle given ity corresponding estimated model M.
Such assumption usually holds in practice when we explore existing off-the-shelf model-free algo- rithms [43, 1, 16, 44]. With this assumption, we can focus directly on the eventual performance difference under the real environment encountering the model updating.
The bound C' we seek can expressed in terms two kinds of gaps: the inconsistency gap between the model and the environment, and the optimal returns gap between the two models. When policy F samples in a model M,, it will encounter states not consistent with those generated by the real environment. We denote this inconsistency by <nt aa (Prv(P(1ss PM. (-1s, a))]- Besides, for each model M, E .M, remarkable property is that there always exists a policy that maximizes the value function VT, (μ) [53]. Hence, define the ceiling performance of model M as M, (j4) en Vž, (n). We will omit μ henceforth for simplicity unless confusion exists. With these two terms well-defined, we now present our bound.
Theorem 4.3 (Performance Difference Bound for Model-based RL). Let Mc M be the esti- 2Rty mated models and ni be the oppr-attimal policy for M4. Recalling (1-7)² where R. is the bound of the reward function, we have the performance difference of गı and σ2 evaluated under the real environment be bounded as below,
22 V=ilMs 2 K- -EM, - + Vin - VM. - Eoppt
(4.2)
Proof. See Appendix B, Theorem B.I.
This bound implies that if the model update M1 + M2 can (1) shorten the divergence between the estimated dynamics and the true dynamics and (2) improve the ceiling performance on the model, it may guarantee overall performance improvement under the true dynamics.
4.2 Lower-bound optimization with model shift constraints
Theorem 4.3 provides general performance difference bound that suggests models with higher ceiling performance and lower bias would raise the overall performance. However, finding a non- negative lower bound may face several issues regarding a drastic model shift in practice. On the one

---

hand, performing an abrupt model update could potentially lead to tumble in ceiling performance and then fail to make incremental improvements. On the other hand, huge distribution divergence between model rollouts would confound the policy optimization stage and hamper its access to optimal policies. Thus, we shoot for refining the bound upon adding model shifts constraint.
First, we seek to further unfold the ceiling performance gap, which serves as the main building block toward the desired bound. During derivation, we additionally introduce the L-Lipschitz assumption.
Assumption 4.4 (L-Lipschitzness of Value Function). We call a value function VM on the esti- mated dynamical model M is L- L-Lipschitz w.E. to some norm || || in the sense that
vs, (,IVV (s) ( S L 10-51 s'l.
(4.3)
We assume that estimated model M M satisfies the Lipschiz character, inspired from [4, 34]. Under the I-LLipschitzeess assumption, we can derive a bound for the ceiling return gap.
Theorem 4.5 (Ceiling Return Gap under Model Shift). For an estimated model Mi E M, the ceiling return gap is bounded as:
VMa
VM 2 1-?
L. sup E, E, saa [Pas Pas, (-1s,
(4.4)
xEll
Proof. See Appendix B, Theorem B.2.
This conclusion reveals the connection between the ceiling return gap and models' disagreement. In a benign scenario, the term of ceiling performance gap the RHS of Eq. 4.2 should dominated by the term x (eM, M2 when the model shift is sufciently small. A sharp model shift, however, risks a massive reduction in ceiling performance that hardly bridged by the other parts in Eq. 4.2, and therefore corrupts monotonicity. This inspires us introduce the constraint of the model shift. We further refine our performance difference bound to better characterize the relationship between performance gap and model shift.
Theorem 4.6 (Refined Bound with Constraint). policy E II denotes the Copt optimal policy under the dynamical model M; E M, and a M, be the constraint threshold for M, and My. Note that T, 2 I-q" constraint as,
Then we can refine performance difference lower bound under the model shifts
T M2 - V 2 n (E ( (M,
(4.5)
- -a Dry (P(ss (1),,))]
7 2 T. (20
Copt:
11-
S.1. Drv PM, (.18, aM) (+18, S NoN, E(
(4.6)
Proof. See AppendixB, Theorem B.3.
Theorem 4.6 implies that policy स2 is guaranteed to outperform policy क, once it makes the RHS in Eq. 4.5 greater than under the constraint Eq. 4.6. More specifically, guarantee non- decreasing performance, M2 and л2 should meet the following two requirements,
IE 5.
Σ PP(s'ss.)) -M (5' a)) S'ES
2
(1-7)L
(20M,
2 Eopt. (R1)
R.
Drv (-Is, P) (-Is S O,, ((s.a)
(R2)
RI encourages us to alleviate model bias as much as possible. Moreover, when the policy w samples too many states not encountered by the model Mj. it may lead to excessive generalization errors during model learning, i.e. the estimated model will suffer extrapolation emror these unexperienced regions, which further causes instability or crashes in the following derived sub-optimal policies. Thus, the introduction of the R2 constraint can help solve the above problem. Finally, we abstract these two requirements to constrained optimization problem.
5

---

Proposition 4.7 (Constrained Lower-Bound Optimization Problem). We reduce the issue find- ing non-negative C the following constrained optimization problem. Here, n, still the sub- optimal policy under model M, The minimal objective in E.q. 4.7 leads to the maximum of C. Then the overall optimization problem can be formalized as.
min MgEM eg
E aaadd
[P(5'|s, - PMo (s E'ES
Σ
(4.7)
S.I.
sup sES,
Drv a)l (x S &M.,M
4.3 A feasible example for constrained optimization problem
We first remark that, Proposition 4.7 provides a useful guide for acquiring performance improvement through restricting the upcoming model shift into safe zone. In this section, we give an instance for feasible solution of the constrained optimization problem under the generative model setting [1, 31]. Specifically, we construct an empirical model Mi from the N samples per state-action pair that stems from the generative model. Upon encountering another k samples on each state-action pair, we segue into the model updating stage, which outputs M2 given these newly collected samples as input. Towards obtaining non-decreasing performance, we yield the following feasible solution for model training interval through satisfying requirements RI and R2.
Corollary 4.8. Here, vol(S) denotes the volume of the state coverage simplex. For simplicity, denote dM, (-Is, a) = (P(+ss,)~ P0) PM, (-|s, for each model M; E M. Under the generative model setting, with a probability larger than E, we can provide non-negative C when given,
2vo((S) -2 NN.
=
log
(4.8)
2
Here, = =ō (-|s, (1-7)D (20 - (1-2)2 R
Be Copt and & E (0, is constant.
Proof. See Appendix B, Corollary B.4.
One can deduce from Corollary 4.8 that dynamically adjusting the model training interval according to the model bias and the model shifts constraints threshold does benefit monotonicity. Along with the model bias d., (-1s, a) decaying, the model training interval l requires to be scaled up to obtain adequate newly encountered samples for training M2. Besides, once gathering excessive samples, we risk violating the model shifts constraint, thus impairing performance. This instance supports the insight that determining "when to update your model"" is vital for performance improvement and motivates a smarter scheme to choose different numbers of explorations at different times instead of the unchanged setting in current methods.
5 CMLO framework
It is nontrivial to tackle the proposed constrained optimization problem in Proposition 4.7, since one cannot directly assign a value to the optimization variable M2. We decouple the optimization objective and the constraint "how to train the model" and "when to train the model" through an event-triggered mechanism towards dynamic alternation.
Objective minimization. Minimizing the objective function involves improving the model aceu- racy. Specifically, we adopt the model-ensemble technique to reduce model bias. For practical implementation, the probabilistic models {f fe....., , } are fitted on shared but differently shuffled replay buffer De, and the target to optimize the Negative Log Likelihood (NLL).
Constraint estimation. The unobserved model M2 makes the constraint function incalculable. and here we seek to design an estimator for Recall that Drv (Pr, (-1s, a) EEves [PM, (s'ls, the state space coverage, and the models' disagreement. We estimate the policy coverage (state- space coverage) by computing the volume (Sl(Sp) of the convex closure Sp {E.eD X,S,
PMr (s,a))))))))))))))())))) a)) (s')s, (s]]s,,),, a)). then we can find that the distance arises from two parts,

---

We exploit the average prediction error on these new samples data estimate the disagreement on newly encountered data (AD) and get ((D = ((s,a,"')Edn & E ||s' - (s. Combining these two components, we can ob- tain an estimation for the model shift, i.e, vol(Sp) ((AD). This practical overestimation for the model shifts makes the constraint stay satisfied during objective optimization.
A 2 ,, = constructed on the replay buffer D.
Event- -ringeerd mechanism. We design an event-triggered mechanism to determine the occasion to pause collecting and turn to solve the optimization objective. Remark that although diverting to train models as long as not to violate the constraint theoretically reasonable, we refrain from doing update data with minor shift in coverage and distribution
so in practice because performing wasteful and may risk overfitting. Therefore, we trigger when the constraint boundary is touched reduce computational cost and escape overfitting. The event-triggering mechanism is based the following condition:
vol(SDUAD(T)) L(AD(T)) a a. (ol(S
(5.1)
Here, J is the event- weint-rigggring time, and a is given constant.
Policy optimization oracle. Clearly, TRPO [44], PPO [46] etc.) as our policy optimization oracle. Besides, we adopt truncated short model rollouts technique mitigate compounding error while encouraging model usage. Based on the fresh model rollouts, we perform the policy optimization oracle, employing SAC as an example.
leverage many model-free RL. methods (SAC [16],
Algorithm Overview. We briefly give an overview of our proposed CMLO in algorithm 1. No- tably, the weent-triggerd mechanism subtly determines the occasion to perform model updating, promoting performance monotonicity and reducing computation load.
Algorithm 1: CMLO
initialize policy Tg. ensemble models fer: Soa.... ... fox environment buffer D. and model buffer Dm
repeat
Sample AD,
from real environment; add to De Estimate model shifts by (S ((DD.)
if Event-triggered condition (F.3) is reached then
Train all models fåL, Jō2........ fōx D.
Perform h-step model rollouts using policy कe; add to Dm
Update me on Dm through SAC [16]
until the policy performs well in the environment;
6 Experiments
Our experimental evaluation aims to investigate the following questions: (1) How well does our algorithm perform on standard reinforcement learning benchmarks compared prior state-of-the- art model-based and model-free algorithms? (2) Does the performance with or without the constraint consistent with previous theoretical analyses?
6.1 Comparative evaluation
To illustrate the effectiveness of our method, we contrast several popular model-based and model- free baselines. Model-free counterparts include: (1) SAC [16], the state-of-the-art terms of asymptotic performance. (2) PPO [46] that explores monotonic improvement as well. Model-based baselines include: (3) PETS [8], which employs models directly for planning, different from the Dyna-style. (4) SLBO [34], that explores monotonicity under the discrepancy bound scheme. (5) MBPO [20], that employs similar design model ensemble technique (ensemble of probabilistic dynamics networks) and policy optimization oracle (SAC) as we do. (6) AutoMBPO [28]. a variant of MBPO, that uses an automatic hyperparameter controller to tune the model-training frequency but suffers from high pre- pre-training cost and lacks theoretical analysis parameters rationality.
7

---

Ha HChhetahhhhhh
Hopper
Walker2d
4003
6000
10000
2300
average
#000
2000
0
300K
Swimmeer
Humanoid
6000
Srerage retr
SHerag a reur
6000
4000
2000
100k
300k
B0Ck
steps
steps
steps
Ours MBFO
AutOMB
SLBO
PETS
SAC
PPO
convergencs
Figure I: Comparison of learning performance on continuous control benchmarks. We evaluate each algorithm the standard 1000-step versions. Solid curves indicate the average performance among seven trials under different random seeds, while the shade corresponds to the standard deviation over these trails. The dashed lines the asymptotic performance of SAC (at 5M steps) and MBPO.
We evaluate CMLO and these baselines on six continuous control tasks in OpenAl Gym [6] with the MuJoCo [54] physics simulator, including HalfCheetah, Hopper, Walker2d, Swimmer, Ant, Hu- manoid. For fair comparison, adopt the standard full-length version of these tasks and align the same environment settings.
Figure shows the learning curves of all compared methods, along with the asymptotic performance. These results show that our algorithm is far ahead of the model-free method in terms of sample effi- ciency, coupled with an asymptotic performance comparable that of the ffte-off-the-an model-free counterparts SAC. Compared to model-based baselines, our method gains faster convergence speed and better eventual performance. Notably, credit to the event-triggered mechanism, our method en- joys a more stable training curve. The better monotonic property of the learning curve agrees with previous analyses.
6.2 Ablation studies
Next, we make ablations and modifications to our method to validate the effectiveness and general- izability of the mechanism we devised.
The necessity of event-triggered mechanism. To verify the necessity of event-triggered mech- anism, we compare three unconstrained cases (given fixed model training interval) under two environments. The training curve and triggered times are shown in Figure 2. Clearly, our mecha- nism improves the performance while reducing the total times of model training. Besides, we notice that the performance is comparable to other MBRL baselines (MBPO etc.) when fixing our model training interval at 250. Still it performs worse than that equipped with a smart mechanism to decide whether to train the model at current exploration step.
To better understand why the event-triggered mechanism brings up our outperformance, we asses its main bricks. Model shift, which reflects the current ability to digest new data, is the basis of triggered condition. And we estimate model shift from two parts, policy coverage and prediction er- ror. In Figure 3 we observe that, gradually, as the training progresses, the policy coverage increases, which reflects our policy has new explorations at every stage without falling into a local optimum prematurely. Also, the prediction error gradually decreases, which implies that our estimated dy- namics come closer to true dynamics in the explored region. Figure implies that the number of samples required to hit the constraint tends to grows and the model training frequency then goes down. This is also consistent with our intuition that, as the model refines and the exploration novelty fades, then the sample size required to reach a certain level of model shifts grows up. The result

---

HallCheetah
Ant
Ho HiiTChaeee hah
Art
(a) average return
(a) policy coverage
laves
(b) average prediction error
(b) triggered times
Figure 3: Visualization of the policy coverage
Figure 2: Ablation event-triggered mecha-
and prediction error on HalfCheetah and Ant. Each stage contains 60k steps. (a) implies that policy coverage expands over stages. (b) shows the prediction error over newly collected data.
nism. These experiments are average over 5 random seeds. (a) shows the average return with or with-out event-triggered mechanism in HalfCheetah and Ant benchmarks. (b) shows the number of triggered times per 10k step.
The bars are average values of each stage.
ANNNINAL I.MNNNNN MONNNNA
(a) Halfcheetah
(b) Ant
Figure 4: Visualization of our event-triggered mechanism HalfCheetah and Ant benchmarks. Solid lines show the model shifts estimation and dotted lines reveals the threshold of triggered condition. Note that here apply log value.
agrees with our theoretical analyses which describe that a dynamic alternation subject to the model shifts constraint does benefit monotonicity rather than an assigned one.
The generalizability of event-triggered mecha- nism. We further investigate the generalizabil- ity of our proposed mechanism through ablation on policy optimization oracle, and results are shown in Figure 5.
HalfChe
20
1) under Dyna-style. We adopt TRPO [44] the policy optimization oracle and test the per- formance with or without event-triggered mech- anism in Halfcheetah and Ant benchmarks. Ob- servably, our mechanism can effectively guaran- tee the overall performance improvement and al- leviate the local optimization issue.
Aaps
policy optimization oracle: TRPO
DKiltySSaadd
1300
1300
23:0
SCE
1008
15.2
2) jumping off Dyna-style. We utilize iLQR [3] and then conduct ablation experiments in the DKitty-Stand [2] and Panda-Reachin, [19] scenarios. The result shows that adding event-triggered mechanism has a comparable asymptotic performance as the most-frequently triggered case but enjoys lower computation cost.
(b) policy optimization oracle: iLQR
Figure 5: Ablation on generalizability via vari- ous policy optimization oracles. Line plots re- flect the average return and the standard devia- tion over 5 trails. Bar plots indicate the total triggered times and the y-axis is scaled to [0.1].

---

7 Conclusion
We have investigated the role of the decision on "when update the model" in joint optimization procedures through theoretical and empirical lens. We devise a general novel scheme for exploring the monotonicity of MBRL methods, distinguished from the existing discrepancy bound scheme. We then derive lower bounds under the scheme, suggesting that models with higher ceiling perfor- mance and lower bias guarantee non-decreasing performance evaluated in the real environment. An effective constrained optimization problem comes from the follow-up refined bound to seek a non-negative lower bound. Further, the instance under the generative model setting further verifies the effectiveness of learning models from dynamically varying number of explorations. The algo- rithm CMLO, stemming from these analyses, has asymptotic performance rivaling the best model- free algorithms and boasts better monotonicity. Further ablation studies reveal that the proposed mechanism scales to various policy optimization oracles and benefits computation cost reduction. Currently, observed empirically that the event-triggered condition is usually related to specific environments, which will cost little time for tuning. Thus, one direction that merits further investi- gation is construct the dual problem of our constrained optimization problem for better exploring optimality and monotonicity.
Acknowledgments and Disclosure of Funding
This work was jointly supported by the
Collaborative Research Project "Crossmodal Learning" (NSFC 62061 136001/ DFG TRR169), the CAS Project for Young Scientists in Basic Research (Grant No. YSBR-040), the National Natural Science Foundation of China (No.6200613) and Beijing Outstanding Young Scientist Program No.B.JWZYJH011 2019 201910000000000000000 100020098). The authors would also like to thank the anonymous reviewers for their careful reading and their many insightful comments,
References
[1] Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, 2020.
[2] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Confer- ence robot learning, 2020.
[3] Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. Differentiable mpc for end-to-end planning and control. In Advances in Neural Information Processing Systems, 2018.
[4] Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning, 2018
[5] Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning, 91 (3):325-334 349, 2013.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 1606.0/6060015400 2016.
[7] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems.
[8] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems.
[9] Mare Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control low- cost manipulator using data-efficient reinforcement learning. Robotics: Science and Systems VII, 7:57-64, 2011.
10

---

[10] Yonathan Efroni, Nadav Merlis, Mohammad Ghavamzadeh, and Shie Mannor. Tight regret bounds for model-based reinforcement learning with greedy policies. Advances Neural Information Processing Systems.
[11] Ying Fan and Yifei Ming. Model-based reinforcement learning for continuous control with posterior sampling. In International Conference on Machine Learning. 2021.
[12] Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, 2017.
[13] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, arXiv preprint arXiv: 2018.
[14] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, 2018.
[15] Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In ICML Workshop on Data-Efcient Machine Learning Workshop, 2016.
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
[17] Wilhelmus PMH Heemels, Karl Henrik Johansson, and Paulo Tabuada. An introduction to event-triggered and self-triggered control. In 2012 ieee 51st ieee conference on decision and control (cde), pages 3270-3285, IEEE, 2012.
[18] Todd Hester, Michael Quinlan, and Peter Stone. Rtmba: A real-time model-based reinforce- ment learning architecture for robot control. In International Conference on Robotics and Automation, 2012.
[19] Stephen James, Mare Freese, and Andrew J. Davison. Pyrep: Bringing v-rep to deep robot learning, arXiv preprint urXiv:1906.11170 2019.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
[21] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[22] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002.
[23] Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Infor- mation theoretic regret bounds for online nonlinear control. In Advances in Neural Information Processing Systems, 2020.
[24] Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, 2017.
[25] Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for near- optimal planning in large markov decision processes. Machine learning. 49(2):193-200 2002.
[26] Jonathan Ko and Dieter Fox. Gp-bayesfiltess Bayesian filtering using gaussian process predic- tion and observation models. Autonomous Robots, 27(1):75-99 90, 2009.
[27] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. International Conference on Learning Representations, 2018.

---

[28] Hang Lai, Jian Shen, Weinan Zhang, Yimin Huang, Xing Zhang, Ruiming Tang, Yong Yu, and Zhenguo Li. On effective scheduling of model-based reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
[29] Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, 2013.
[30] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133 2016.
[31] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with generative model. In Advances Neural Information Processing Systems, 2020.
[32] Huiping Li and Yang Shi. Event-triggered robust model predictive control of continuous-time nonlinear systems. Automatica, 5065):1507-1513, 2014.
[33] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[34] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2018.
[35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In Advances in Neural Information Processing Systems, 2013.
[36] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51897540)559-533, 2015.
[37] Jun Morimoto and Christopher Atkeson. Minimax differential dynamic programming: An application to robust biped walking. In Advances in Neural Information Processing Systems, 2002.
[38] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. Inter- national Conference on Robotics and Automation, 2018.
[39] Feiyang Pan, Jia He, Dandan Tu, and Qing He. Trust the model when is confident: Masked model-based actor-critic. In Advances in Neural Information Processing Systems.
[40] Luis Pineda, Brandon Amos, Amy Zhang, Nathan O. Lambert, and Roberto Calandra. Mbrl- lib: A modular library for model-based reinforcement learning. Arxiv, 2021. URL https: //arxiv..a ort.://./gaaaabbeea abw/aaa/....... 10159.
[41] Athanasios Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learn-
ing: Applications robotics. Journal of Intelligent & Robotic Systems, 16(2):153-173, 2017. 201,
[42] pranz24.
pytorch-soft-cctor-e pytorch-soft- shtrrc--oftt-ccto--o critic, 2018.
[43] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programmingg John Wiley & Sons, 2014.
[44] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. 2015.
12

---

[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017.
[47] Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving discounted markov decision process with generative model. arXiv preprint 1806.0004 2018.
[48] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam,, Mare Lanc- tot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 (7587):484-48, 2016.
[49] Wen Sun, Geoffrey Gordon, Byron Boots, and Bagnell. Dual policy iteration. In Advances in Neural Information Processing Systems.
[50] Richard $ Sutton. Integrated architecture for learning, planning, and reacting based on approx- imating dynamic programming. In International Conference on Machine Learning, 1990.
[51] Richard Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4))160-1633 1991.
[52] Richard Sutton. Planning by incremental dynamic programming. In International Confer- ence on Machine Learning, 1991.
[53] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, Cambridge, MA, 2018.
[54] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
[55] Tingwu Wang. Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shun- shi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model- rein- forcement learning, arXiv preprint arXiv: 100.:01977.00077 2019.
[56] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the deviation of the empirical distribution. Hewlett-Puckann Labs, Tech. Rep, 2003.
[57] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [No] No poten- tial negative societal impacts have been found yet.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? (Yes) See Section and Appendix B.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B and Appendix C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URLY? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix E and our provided code.
(c) Did you report error bars (e-g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See in corresponding figure captions.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Appendix E
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [N/A] Our work does not use existing data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] No crowdsourcing/humal subjects are used.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] No crowdsourcing/human subjects are used.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/]] No crowdsourcing/human subjects are used.
14


Reviews:
[{"review_id": "gpEG3gTGBoC", "reviewer": "Reviewer_iXge", "summary": "This paper studies how to ensure optimization monotonicity of learning an accurate dynamics model for MBRL. They derive a lower bound for the derived policy performance improvement that depends on the one-step dynamics prediction error of the current model, constraint by the model shift. \n\nInspired by the theory, they propose an algorithm to dynamically alternate between policy exploration and model learning, with the aim to improve optimization monotonicity. They evaluate the proposed algorithm on a series of MoJoCo control tasks and compared the results against a few model-free baselines, and show that their model-based method is able to reach the same SOTA asymptotic performance while being more stable and sample efficient. ", "questions": "1. In Figure 1, the performances on different tasks are capped at different timesteps. In several cases the learning curves have not stabilized yet, e.g. Walker2d and Swimmer. Could you please report the final performance where the learning curves stabilize or at 5M steps (as used to determine the “asymptotic performance” of the SAC and MBPO baselines).\n2. What is the y-axis in Figure 3(b)? Could you please include the policy coverage and prediction error results for some of the baselines for a comparison?\n3. Figure 4 looks a bit confusing to me. Do the MBRL baselines start with an initial 30k steps exploration followed by model learning (inferred from Figure 4)? Why only showing 4k steps per stage instead of the whole 60k? What is the y-axis?\n4. Figure 4(b) shows that the model shifts estimation jumps drastically from stage 1 to stage 2, and that it doesn’t hit the threshold at all in stage 1 (first 4k steps). Could the authors please explain this unexpected observation?\n5. The absolute performance of TRPO on HalfCheetah and Ant tasks shown in Figure 5(a) are much lower than the other baseline results shown in Figure 1. The variance also seems quite high. Is it expected?\n6. Why choosing these specific training step caps in Figure 5(b)?\n7. Could you briefly discuss the costs induced by estimating the volume of the convex closure, especially when applying to high-dimensional state space?", "limitations": "The paper mentioned briefly in the conclusion that finding the optimal threshold for each specific environment may be time consuming. Could you please elaborate on that? For example, how different are the thresholds for different environments and the amount of steps it takes for tuning. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe problem studied in the paper is important and relevant to practice. The paper is well-written and easy to follow. The proposed method is theoretically motivated.\n\nWeakness:\n\nMy main concerns are around the experiments and potential limitations. Please see the section below for detailed comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X2m43Ei1ucL", "reviewer": "Reviewer_r1Xm", "summary": "This paper first demonstrates that the model shifts---the difference between the updated model and the model before updating---hinder the monotonic improvement of model-based RL. To tackle this problem, the paper proposes CMLO, which introduces an event-triggered mechanism to determine when to alleviate the model shifts. Experiments show the effectiveness of the proposed method.", "questions": "My suggestions are as follows.\n\n1. The advantages of the proposed event-triggered mechanism are unclear. The authors may want to explain why the event-triggered mechanism is more effective than previous methods [1, 2].\n\n2. The authors claim that the proposed constraint estimation is “the practical overestimation for the model shifts” in Lines 247-248. The authors may want to provide the theoretical analysis of the overestimation.\n\n3. Several definitions are missing, such as $\\sigma_{M_1, M_2}$ in Theorem 4.6 and $\\Delta \\mathcal{D}$ in Line 245.\n\n4. The formulation for computing the proposed volume $vol(\\mathcal{S}_\\mathcal{D})$ in Lines 239-244 is missing.\n\n5. The authors may want to provide the detailed motivation of Equation 5.1.\n\n6. The experiment settings in Section 6.2 are missing, such as the settings of all methods in Figure 2 and that of the visualization in Figure 3. The authors may want to provide the detailed settings of all ablation studies.\n\n[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.\n\n[2] Hang Lai, Jian Shen, Weinan Zhang, Yimin Huang, Xing Zhang, Ruiming Tang, Yong Yu, and Zhenguo Li. On effective scheduling of model-based reinforcement learning. In Advances in Neural Information Processing Systems, 2021.\n", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The authors propose the theoretical analysis to show that the model shifts hinder the monotonic improvement of model-based RL, which provides a useful perspective in model-based RL.\n\n2. Experiments demonstrate that the proposed method improves the performance and generalization of existing methods.\n\n\nWeaknesses:\n1. The authors propose a constraint estimation in their method to estimate the model shifts in Proposition 4.7. However, in Lines 245-247, the authors use the data from the current ensemble models and real environment to compute the constraint estimation, which is not consistent with Proposition 4.7.\n\n2. The authors propose the event-triggered mechanism to determine when to update the model instead of the frequent model updating in MBPO [1]. However, the motivation to alleviate the model shifts by the event-triggered mechanism is unclear. The authors may want to explain why they introduce the event-triggered mechanism based on their proposed theoretical analysis.\n\n[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Uyxxl36GoST", "reviewer": "Reviewer_99tJ", "summary": "The paper focuses on the monotonic improvement for model-based reinforcement learning which is an extremely important problem due to the inherently entangled nature of the multi-level optimization problem - policy optimization and model learning. Earlier research has not specifically considered the model shift which is considered in this research while proving the monotonic improvement. The primary objective is to show that $||V^{M1|\\pi_1} - V^{M2|\\pi_2}|| \\geq C$ which can guarantee monotonic improvements under the updating dynamics. The primary reason for the model bias is a mismatch between the samples in the model learning stage and the policy optimization stage. To tackle the same, they formulate a constrained bi-level optimization framework for the MBRL problem and design an event-triggered strategy to decide when to update the model to guarantee monotonic improvement under changing dynamics. Empirical results show some improvements in sample efficiency from prior stable model-based RL methods and the ablation study supports the main hypothesis of the paper. \n", "questions": "1. Can you please illustrate how exactly is the volume of the of convex closure computed from the replay buffer? \n\n2. Can you please point me to the equation where you derive an estimation for the model shift as a product of volume $\\times$ avg prediction error?\n\n3. Instead of going by the complicated event-triggered mechanism, if I simply try to obey the constraint in the optimization problem with a TRPO type regularized update in the model parameter space that should also do right? Given that we know that under the gaussian dynamics assumption, the total variation distance simply can be upper-bounded by the difference b/w the means and variances as well and hence it should be easy to be in the feasibility region.\n\n4. Can you please show the derivation on how you obtain line 47 in Appendix where you mention that with Lipschitzness of $V^{\\pi}_{M_1} V^{\\pi}_{M_2}$ you derive the bounds for $|G^{\\pi}_{M_1, M_2} (s,a)| \\leq L |P_{M_2}(\\cdot|s,a) - P_{M_1}(\\cdot|s,a)|$. \n\n5. Additionally in line 48 in Appendix, you derive an upper bound for value function difference in-terms of the $|P_{M_2}(\\cdot|s,a) - P_{M_1}(\\cdot|s,a)|$. However, it has been shown in [1] that there is a dependence of the maximum reward $R$ (and horizon $H$ which will be replaced by some factor of $\\gamma$ here) intermingled/ associated with the Lipschitz constant. Basically, they prove that in the Lipschitz assumption of $||V1-V2|| \\leq L ||P1- P2||$, $L$ has a dependence on $CHR_{max}$. I think that should be applicable here as well, then how does the analysis gets affected? As now there is an additional dependence on the max reward that will be added. Can you please share your thoughts on the same?\n\nReferences : \n[1]. Ying Fan and Yifei Ming. Model-based reinforcement learning for continuous control with posterior sampling. In Marina Melia and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3078–3087. PMLR, 18–24 Jul 2021.", "limitations": "The author mentions some points regarding the current applicability to certain environments and wants to scale with improved optimization methods which are sensible.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary strength of the paper lies in the formulation of the bi-level constrained optimization objective from the lower bound objective with an event-triggered mechanism under a generative model assumption to update the model which is quite novel according to me. Earlier research has addressed the bias in the model learning and has also tried to guarantee monotonic improvement, but this paper explicitly talks about the monotonic improvement of the policy under shifted model dynamics and shows guarantees of monotonic improvements (with certain assumptions) which is interesting and novel.  The biggest challenge in guaranteeing a monotonic improvement in the policy is due to the changing dynamics which might occur due to a potential mismatch between the true trajectories and model-generated trajectories and is indeed an important challenge to overcome. The paper breaks down the source of error into 2 components with the assumption that we can always get an epsilon optimal policy under a given model which is a bit optimistic but a very common and frequently used assumption in RL. The components being 1. inconsistency gap between the model and the environment : $E_{s,a \\sim d_{\\pi}} TV(P(\\cdot|s,a) || P_M(\\cdot|s,a))$ and 2. Optimal returns under the true model and they hypothesize that the performance difference $||V^{M1|\\pi_1} - V^{M2|\\pi_2}||$ is lower bounded by the above two aspects. Finally, with the above notions, the authors formulate a constrained lower bound optimization problem as stated in Theorem 4.6 which is quite novel as the formulation of model-based RL as a bi-level optimization problem is quite natural as done in this paper. The strength of the paper lies in designing an event-triggered mechanism and providing a high probability bound for the value of $k$ for guaranteeing monotonic improvement under the generative model assumption. In Corollary 4.8, the authors derive a relation between the model training interval $k$, model bias, and state-space coverage which is quite unique to my knowledge and given by $k \\propto \\frac{2 \\times vol(S)}{\\epsilon^2}$ (just approximated and ignore other terms). In other words, when the model bias is less one needs to increase $k$ to increase state action coverage which is quite intuitive. Overall the ablation study seems interesting and the experimental results show some improvements over the past SOTA model-based RL methods.\n\nThe primary weakness lies in various assumptions that have been made to derive some of the primary results that are not very general. To begin with, the main result in Corollary 4.8 is shown with a generative model assumption which is quite restrictive. The primary novelty of the research from a solution perspective lies in designing the event-triggered mechanism and the theory is shown with a generative model assumption also with linear\nquadratic regulator (which I still believe can be relaxed) and not for a general scenario. Secondly, it is not very clear how minimization of the constrained objective function in Proposition 4.7 turns out to be simple negative likelihood minimization. In Proposition 4.7, there is $P(s'|s,a)$ involved in the expression which can't be ignored since the $(s,a) ~ d^{\\pi_2}$ and optimization variable involve $\\pi_2$ as well. So, it's not clear how this is handled as we will never know $P(s'|s, a)$, and how this boils down to simple NLL minimization is not explicitly described. Also, I see in the derivations on lines 90, 97, 128, etc. the proofs involve summation over s,a which are relevant for Tabular or Discrete settings but the experiments are for continuous state-action space which causes a mismatch and some of the proofs might break when everything is expressed for continuous state-action spaces. The ablation study is interesting and the experimental results show improvement but not significant improvements. However, I feel the work addresses an important question and it might be helpful to understand this aspect for model-based RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ACK1veiIHN4", "reviewer": "Reviewer_3kr9", "summary": "This paper studies the relationship between the shift brought by model updates and policy performance. The authors proposed a model shift constraint and the CMLO algorithm for a monotonic improvement guarantee.", "questions": "1. It would be better to have more comparisons between CMLO and previous works (e.g. DPI). Although they have different proof structures (from the local and global view?), the monotonic improvement results are similar in my view. \n\n2. Why do we care about the *optimal* value under a model and what is the advantage compared to the local view? I am assuming that it is the primary reason that leads to different algorithms between CMLO and previous works.\n\n3. The definition of state coverage and the event-triggered equation is not so clear.\n\nMy concerns are closely related. So it might be the case that I missed something important. I would like to change my score depending on the authors' rebuttal.", "limitations": "No.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "pros: 1. The authors consider an important problem in RL, the inconsistency of policy updates under shifted models. The main results and proofs are largely correct in my view.\n\n2. The ablations verify the proposed algorithm.\n\ncons: 1. The monotonic results are not novel in MBRL, for example, the authors cite DPI [1], which established similar monotonic results. But the authors claim that previous works \"characterize the monotonicity in terms of a fixed model of interest\". I don't see the reason for such a claim and the exact advantage of CMLO.\n\n2. From the introduction and related work, the partial reasons for cons 1 might include that the authors derive the theorems from a global optimal view. But I can't judge the necessity of doing this. E.g., in theorem 4.5, why do we care about the *optimal* value under a model? Notably, the monotonic property rather than global optimality is of interest. Therefore, according to the performance difference lemma, shouldn't we focus more on the value-under-model evaluated using the *current policy*?\n\n3. Another confusion I have is the notation of state coverage. As it is introduced from a global point of view, I can't judge its necessity before understanding the necessity of the global view. Besides, it is better to include the exact definition of state coverage in the context of RL, so that we can see why it is estimated with the replay buffer. \n\n4. The intuition behind the event-triggered equation 5.1 is also not very clear. What will the fraction of state coverage give?\n\nMinor: how much additional time does it cost for estimating model shifts?\n\n[1] Dual Policy Iteration, Wen Sun et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept