Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function
Authors: Ruijie Zheng
Xiyao Wang
Huazhe Xu
Furong Huang
Venue: ICLR 2023 poster (2023)
Keywords: Model-based Reinforcement Learning
Probabilistic Dynamics Model Ensemble
Lipschitz Regularization

Abstract: Probabilistic dynamics model ensemble is widely used in existing model-based reinforcement learning methods as it outperforms a single dynamics model in both asymptotic performance and sample efficiency. In this paper, we provide both practical and theoretical insights on the empirical success of the probabilistic dynamics model ensemble through the lens of Lipschitz continuity. We find that, for a value function, the stronger the Lipschitz condition is, the smaller the gap between the true dynamics- and learned dynamics-induced Bellman operators is, thus enabling the converged value function to be closer to the optimal value function. Hence, we hypothesize that the key functionality of the probabilistic dynamics model ensemble is to regularize the Lipschitz condition of the value function using generated samples. To validate this hypothesis, we devise two practical robust training mechanisms through computing the adversarial noise and regularizing the value network’s spectral norm to directly regularize the Lipschitz condition of the value functions. Empirical results show that combined with our mechanisms, model-based RL algorithms with a single dynamics model outperform those with ensemble of the probabilistic dynamics models. These findings not only support the theoretical insight, but also provide a practical solution for developing computationally efficient model-based RL algorithms.

Published as a conference paper at ICLR 2023
IS MODEL ENSEMBLE NECESSARY? MODEL-BASED RL VIA A SINGLE MODEL WITH LIPSCHITZ REGULAR- IZED VALUE FUNCTION
Ruijie Zheng
Xiyao Wang 1.8
Huazhe x
Furong Huang'
(rzheng12, xywang, furongh (anighh@ummme @uudd. edu 2 Tsinghua University huazhe wuemail.tsinghuaa... tsinghha. edu. en
University of Maryland, College Park
3 Shanghai Qi Zhi Institute
ABSTRACT
Probabilistic dynamics model ensemble is widely used in existing model-based reinforcement learning methods as it outperforms a single dynamics model in both asymptotic performance and sample efficiency. In this paper, we provide both practical and theoretical insights on the empirical success of the probabilis- tic dynamics model ensemble through the lens of Lipschitz continuity. We find that, for a value function, the stronger the Lipschitz condition is, the smaller the Bellman opera-
gap between the true dynamics- and learned jyynamics-inducee tors is, thus enabling the converged value function to be closer to the optimal value function. Hence, we hypothesize that the key functionality of the proba- bilistic dynamics model ensemble is to regularize the Lipschitz condition of the value function using generated samples. To test this hypothesis, we devise two practical robust training mechanisms through computing the adversarial noise and regularizing the value network's ' spectral norm to directly regularize the Lipschitz condition of the value functions. Empirical results show that combined with our mechanisms, model- based RL algorithms with single dynamics model outper- form those with an ensemble of probabilistic dynamics models. These findings not only support the theoretical insight, but also provide a practical solution for developing computationally efficient model-based RL. algorithms.
INTRODUCTION
Model-based reinforcement learning (MBRI. improves the sample efficiency of an agent by learning a model of the underlying dynamics in a real environment. One of the most fundamental questions in this area is how to learn model to generate good samples that it maximally boosts the sample efficiency of policy learning. To address this question, various model architectures are proposed such Bayesian nonparametric models (Kocijan et al., 2004; Nguyen-Tuong et al., 2008; Kamthe & Deisenroth, 2018), inverse dynamics model (Pathak et al., 2017; Liu et al., 2022), multi-step model (Asadi et al., 2019), and hypernework (Huang al., 2021).
Among these approaches, the most popular and common approach is to use an ensemble of proba- bilistic dynamics models (Buckman et al., 2018: 8; Janner et al., 2019; Lai al., 2020; Clavera al., 2020; Frochlich et al., 2022; Li et al., 2022). It is first proposed by Chua et al. (2018) to capture both the aleatoric uncertainty of the environment and the epistemic uncertainty of the data. In prac- tice, MBRL methods with an ensemble of probabilistic dynamics models can often achieve higher sample efficiency and asymptotic performance than using only single dynamies model.
However, while the uncertainty perspective seems reasonable, previous works only directly apply probabilistic dynamics model ensemble in their methods without an in-depth exploration of why this structure works. There still lacks enough theoretical evidence to explain the superiority of probabilistic neural network ensemble. In addition, extra computation time and resources are needed to train an ensemble of neural networks.
$ 'Equal contribution

---

Published as a conference paper at ICLR 2023
In this paper, we provide new perspective on why training probabilistic dynamics model en- semble can significantly improve the performance of the model-based algorithm. We find that the ensemble model-generated state transition samples that used for training the policies and the critics are much more "diverse" than samples generated from a single deterministic model. We hy- pothesize that it implicitly regularizes the Lipschitz condition of the critic network especially over local region where the model is uncertain (i.e., a model-uncertain local region). Therefore, the Bell- man operator induced by the learned transition dynamics will yield an update to the agent's value function close to the true underlying Bellman operator's ss update. We provide systematic experimen- tal results and theoretical analysis to support our hypothesis.
Based on this insight, we propose two simple yet effective robust training mechanisms to regular- ize the Lipschitz condition of the value network. The first one spectral normalization (Miyato et al., 2018) which provides global Lipschitz constraint for the value network. It directly follows our theoretical insights to explicitly control the Lipschitz constant of the value function. However, based on both of our theoretical and empirical observations, only the local Lipschitz constant around a model-uncertain local region is required for a good performance. So we also propose the second mechanism, robust regularization, which stabilizes the local Lipschitz condition of the value net- work by computing the adversarial noise with fast gradient sign method (FGSM) (Goodfellow et al., 2015). To compare the effectiveness of controlling global versus local Lipschitz. systematic experi- ments are implemented. Experimental results on five MuJoCo environments verify that the proposed Lipschitz regularization mechanisms with a single deterministic dynamics model improves SOTA performance of the probabilistic ensemble MBRL algorithms with less computational time.
Our contributions are summarized as follows. (1) We propose new insight into why an ensemble of probabilistic dynamics models can significantly improve the performance of the MBRL algorithms, supported by both theoretical analysis and experimental evidence. (2) We introduce two robust training mechanisms for MBRL algorithms, which directly regularize the Lipschitz condition of the agent's ' value function. (3) Experimental results on five MuloCo tasks demonstrate the effectiveness of our proposed mechanisms and validate our insights, improving SOTA asymptotic performance using less computational time and resources.
2 PRELIMINARIES AND BACKGROUND
Reinforcement learning. We consider a Markov Decision Process (MDP) defined by the tuple A, P, Po, r), where S and A are the state space and action space respectively, P(s'|s, a) is the transition dynamics, Pu is the initial state distribution, r(s, a) is the reward func- tion and y is the discount factor. In this paper, we focus on value-based reinforcement leamn ing algorithms. Define the optimal Bellman operator T such that T*Q(s.a) a) y P(d'ls, a) arg max.' Q(s',a')ds' The goal of the value-based algorithm to learn a Qo to approximate the optimal state-action value function Q", where Q' = T"* is the fixed point of the optimal Bellman operator. For model- based RL (MBRL), we denote the approximate model for state transition and reward function as Py and fo respectively. Similarly, we define the model-induced Bellman Operator 7" such that (*0 a) = f(s, + Po (ss1(s,)) arg max, Q(s', a' yds''.
r(s,a) +
Probabilistic dynamics model ensemble. Probabilistic dynamics model ensemble consists of K neural networks Pe = (Po., Poz:.., P..)) with the same architecture but randomly ini- tialized with different weights Given a state-action pair as, the prediction of each neural network is Gaussian distribution with diagonal covariances of the next state: Ps, (s'ls, a) = N(He (s, a); Lo, (s,a)) a)). The model is trained using negative log likelihood loss (Janner et al., 2019): [(Ox) = E (st, - st+1] " az [p) (st, az) st+1] + log det Ee, (st, ae) During model rollouts, the probabilistic dynamics model ensemble first randomly selects a network from the ensemble and then samples the next state from the predicted Gaussian distribution.
Local Lipschitz constant. in the following discussion.
We now give the definition of local Lipschitz constant, key concept
Definition 2.1. Define the local (2, e)--pschhistt constant of a scalar valued function f RR + R over set X as:
[f -((2)
[(p] (f, = -upp xEX
sup D1 (lt)
(11 * v2).
(1)
11 121lp
2

---

Published as a conference paper at ICLR 2023
Throughout the paper, consider p unless explicitly stated. If X, C. is finite, say that f is (X, clocally Lipschitz with constant C.
In particular,
view the global Lipschitz constant L(f,R* ,c), which upper bounds the local Lipschitz constant I(,,Xe)) with A' C RRV.
3 LOCAL LIPSCHITZ CONDITION OF VALUE FUNCTIONS IN MBRL
3.1 KEY INSIGHT: IMPLICIT REGULARIZATION ON VALUE NETWORK BY PROBABILISTIC ENSEMBLE MODEL
In this section, we provide insight into why a value-based reinforcement learning algorithm trained from simulated samples generated by probabilistic ensemble model performs well in practice. We first conduct an experiment on the MuloCo Humanoid environment with MBPO (Janner et al,, 2019), the SOTA model-based algorithm using Soft Actor-Critic (SAC) (Haarnoja et al., 2018) as the backbone algorithm for policy and value optimization. We run MBPO with four different types of trained environment models: (1) a single deterministic dynamics model, (2) single probabilistic dynamics model, (3) ensemble of seven deterministic dynamics models, and (4) an ensemble of seven probabilistic dynamics models. As we can see from Figure la, in the Humanoid environment, the agent trained from an ensemble of the probabilistic dynamics model indeed achieves much better performance. similar observation found across all the other MuloCo environments.
Prolshilistic Ensanlic
Deseministic Essemide
Siugle Pobiuiilittic
Singl Deteminnitic
Edtisand Eipasto Cossuut Log Sote
Medd Mea Squardd Enner
12-
Liwohit
2,00
1.00
Number Intercioons
Number Interactions
Number Interactions
Number
(a)
(b)
(c)
Figure (a) Performance of MBPO algorithm trained with single deterministic, single probabilistic. determin- istic ensemble, and probabilistic ensemble model respectively on Humanoid. (b) Model mean squared error. (c) Value-aware model error in log scalc. (d) Upper bound of Lipschitz constant of value network. Results are averaged over 8 random seeds.
Value-aware model error. Why does MBPO with ensemble of probabilistic dynamics models perform much better? We try to answer this question by considering the effects of the learned transi- tion models on the agent's value functions. In MBPO algorithm, fits an ensemble of M Gaussian environment models p5(0) (+1a,a) = (
(+) (s. a), x:ft) (s, a)). = 1,... M. With a deep neural net- work powerful function approximator for the mean and variance of the Gaussian distribution, the mean squared error LE gued in many previous MBRL works (Grimm et al., 2020; 2021), the mean squared error is often good measurement for the quality of the learned transition model. Instead, the value-aware model error (Farahmand et al., 2017a) as defined below plays crucial role here, which connects directly to the suboptimality of the MBRL algorithm.
Ills 8' is often small. However, as ar-
Liaam (Po; P) = [[tsa)~< (TTTI((, ((, ( (*,(),)))
( P() P [
= =>?
(V(()) (V(8') v(s')) V(s) max Q(+ a) This value-aware model error measures the difference between the simulated and true Bellman op- crator acting on a given Q function. other words, even though the learned transition model Pa approximately accurate, the model prediction error still be amplified by the value function so that the two Bellman operators yield completely different updates on the value function.
< (t..aa)a)pp PP(ls)))
Value function Lipschitz regulates value-aware model error. When the value function's 's local- Lipschitz constant is under control, the value-aware model error can be made small. As we see from Figure 1b, Ic, and 1 although the probabilistic ensemble model and single deterministic model

---

Published as a conference paper at ICLR 2023
achieve similar mean squared errors, value functions trained from ensemble of probabilistic dy- namics model has a significantly smaller Lipschitz constant and value-aware model error. (Note that the value-awarr model error plotted in Figure 1d is in log scale.)
To see why this the case, we view the learned Gaussian transition models P/ as 16 + 90
where (() is deterministic model and (6) (-Is, a) = NO, _ (s,a)) is noise distribution with zero mean. When training the value network of the MBRL agent, the target value is computed as r(s, a) +) s-C(M, # )e (+)s,a) [V (F1 (s, (s,a))+ te:)]. Here view data generated from the prob- abilistic ensemble model as an implicit augmentation. The augmentation comes from two sources: (i) variation of prediction across different models in the ensemble and (ii) the noise added by each noise distribution gg By augmenting the transition with a diverse set noise and then training the value functions with such augmented samples, it implicitly regularizes the local Lipschitz condition of the value network over the local region around the state where the model prediction is uncertain.
(0)
In the next section, we provide some thcoretical insights into how the local Lipschitz constant can play a role in the suboptimality of the MBRL algorithms. Later in the algorithm and experiment section, we will provide practical mechanisms regularize the Lipschitz condition of the value network and demonstrates the effectiveness of such mechanisms, which further validates our claim.
3.2 ERROR BOUND OF MODEL-BASED VALUE ITERATION WITH LOCALLY LIPSCHITZ VALUE FUNCTIONS
In this section, we formally analyze how the Lipschitz constant of the value function affects the learning dynamics of the model-based (approximate) value iteration algorithm, the prototype for most of the value-based MBRL algorithms. For simplicity. following previous works (Farahmand et al., 2017b; Grimm et al., 2020), we assume that we have the true reward function r(s,a) but extending the results to reward function upproximation should be straightforward.
We consider the following value iteration algorithm. At th-iteration of the algorithm, we obtain a dataset Dj è {(s, Qg } where (s,, as) is sampled i.d.d from p E A(S x A), the empini- cal state-action distribution, and s P(-|se, az) is the next state under the environment transition.
Based on this dataset, we first approximate the transition kernel P to minimize the mean L2 differ- ence between the predicted and true next states.
Px arg min PEM
f
Σ
N
i=1
Then at k th-iteration, we update the value function by solving the following regression problem:
Q arg min Creg (Q: Qu-1, Pa) = arg min (s))~ (((ss))((((ss))))) --.
(3)
QEF
QEF
Empirically, we update the value function such that
Q+ f- arg min QEF
Σ (( -
(4)
N
where V = max (8-1
To simplify the analysis, we assume that we are given a fixed state-action distribution p such that for every iteration, we can sample i.i.d. from this data distribution. However, in practice, we may use a different data collecting policy at different iterations. As argued in (Farahmand et al., 2017a), a similar result can be shown in this case by considering the mixing behavior of the stochastic process.
Now we list the assumptions we make. Some assumptions are made only to simplify the finite sample analysis, while others characterize the crucial aspects of model and value learning.
First, we make the deterministic assumption of the environment. This is only for the purpose of the finite sample analysis. When the environment transition is stochastic, we will not have the finite sample guarantee, but our insights still hold. We will provide more discussion on this later when we present our finite sample guarantee.
Assumption 3.1. The environment transition is deterministic.
4

---

Published as a conference paper at ICLR 2023
Next, apply the concentration inequality in analysis, have to make the following technical assumption of the state space and reward function.
Assumption 3.2. (Boundedness of State Space and Reward Function) There exists constants D, Rmax such that for all E S, E A. || S D and r(s,a) < Rmax-
In addition, make the following mild approximate realizability assumption the model class of approximated transition kemnlss so that in the model space, at least one transition model close to the true underlying transition kernel should exist.
Assumption 3.3. ((E, p\-pproxzimaee Realizability)
/
inf C2(P) inf
(s -8' 1111,dp(s, Se PEM
PEM
We also make critical assumption of the local Lipschitz condition of the value function class. In particular, for the state-action value function Q S A + R, we define that Q (e. pllocallly Lipschitz with constant T. if for every a E A. the function Qa + ((,,a) is (c, (c,p)-locally Lipschitz with constant less than or equal to L..
Assumption 3.4. (Local Lipschitz condition of value functions) There exists finite L, such that for every Q E F. it is (X, 2c) -locally Lipschitz with constant less than or equal to L, where X' is the support of the distribution p.
Note here we only need to assume that value functions are all (X, (1 + 3)e)-locally Lipschitz with for simplicity. Finally, same as the previous work (Farahmand et al., 2017a),
O. We set / to provide finite sample guarantee of model learning, we make the following assumption on the complexity of the model space of the approximated transition kernels
Assumption 3.5. (Complexity of Model Space) Let R 0, J Ma - (0) be a pseudo-norm, where Mo is a space of transition kernels. Let M MR = {P J(P) S R]. There exists
constants and 0 < < 1 such that for any c, R. 0 and all sequence of state-action pairs 21:m 21, .... 2n S A, the following metric entropy condition is satisfied:
R 2x
(og,((,,,,LL S
(6)
where Me, L2(31m)) is the covering number of M with respect to the empirical norm (2(21m)
such that | 1112.
1
N
Under these assumptions, we can now present our main theorem, which relates the local Lipschitz constant the suboptimality of the approximate model-basee based value iteration algorithm. In partic- ular, we provide a finite sample analysis of model learning in Theorem B.2 and value-aware model error in Theorem B.4. Then we apply the error accumulation results from (Farahmand et al., 2017a), connecting the local Lipschitz constant with the suboptimality of the algorithm.
Theorem 3.6. Suppose Qo initialized such that Qo(s,a) S Remax
1-7 for V (8,a). Under the as- sumptions of 3.1, 3.2, 3.3, 3.4, and 3.5, after K iterations of the model-based approximate value iteration algorithm, there exists constant s((x)) which depends solely on 10,1) such that,
(P.a))~P (00 Qx((s,a)| S 7)3
2y
[Ce, Po)
max òR 7 OSkSK
(1 - R)
(1 7)- )) 27 KRuax
where 0% LuegQus Qu-1, is the regression error defined in Equation (3), E 1
eN 1to ))), and C(p, is the concentrability constant defined in Definition A.I.
exp(-
(2R)DRRTT
Remarks.
(1) As the number of samples N + 0o, E + 1. Consequently, the value-aware model error [[[T"Qu(s,, T'Qu a) will be governed by the term 4e2 LE which is controlled by the local Lipschitz constant L. In addition, if the number of iterations k oc, the left hand side of
5

---

Published as a conference paper at ICLR 2023
Equation (7) will be bounded by 2rC(p, Po)
max 6% + + From here, we can see the OSASK
(1 - 7)2
role that the local Lipschitz constant has played in controlling the suboptimality of the algorithm. (2) Although we assume that the environment transition deterministic, our insights should still hold when it is stochastic. In particular, if the learned transition model has low prediction error (2( (Eq. 5) (which is often the case with neural network as the function approximator), and the local Lipschitz constant of the value function is bounded, then we can still have a small value-award aware model error and get similar error propagation result as Theorem 3.6.
Tradeoff between regression error and value-aware model error. essential trade-off between the regression error and value-aware model error through the Lipschitz condition of the value function class, i.c., the constant L. As L gets smaller, the model-induced Bellman operator gets closer the actual underlying Bellman operator. However, with smaller L, we also impose stronger condition on the value function class. Therefore, the value function space will shrink, and the regression error 6% is expected get larger.
This theorem also reveals an
To further visualize this trade-off, we conduct experiment on the Inverted- Pendulum environment, where we run the analyzed model-based approximate value iteration algorithms for 1000 iterations. We plot (Figure 2a) the best evaluation performance for 20 episodes at every iteration, (Figure 2h) the maximum regression eгror across every iteration, well as (Figure 2c) the maximum value- aware model error. To upper bound the (local) Lipschitz constant, constrain the spectral norm of the weight matrix for each layer of the value network.
(ogg Statd)
Rqporsice
20,01 4000
6.30C
1,00
2,001
4060
6.303
5.300
2.30 4.540
0.000
K/00
Upper Bound of Nerwock Lipachitz Coastant
Upper Bound of Network Lipachite Constaat
Upper Bound of Network Lipschitz Constare
(a) best evaluation performance
(b) max regression
(c) max value-aware model error
across value networks with different Lipschitz constraints. All the results the median over 5 random seeds.
Figure 2: Model-based value iteration
As from the results in Figure 2b, indeed, the regression error dramatically drops the Lipschitz constant grows from 100 to 1000 and then levels off, which indicates that perhaps the Lipschitz constant of 1000 is rich enough for the value function class be Bellman-complete for this environment. Meanwhile, the value-aware model error also increases with bigger Lipschitz constant. However, we can achieve a good balance between these two errors when the Lipschitz constants are between 1000 (left dashed line) and 4000 (right dashed line), where the algorithm is observed to perform the best.
METHODS
In this section, we present two different approaches to regularize the (local) Lipschitz constant of the value function.
4.1 SPECTRAL NORMALIZATION
First, to explicitly control the upper bound of the Lipschitz constant, adapt the technique of Spec- tral Normalization, which is originally proposed to stabilize the training of Generative Adversarial Network (GAN) (Miyato et al., 2018). By controlling the spectral norm of the weight matrix at every layer of the value network, its Lipschitz constant is upper bounded. In particular, during each forward pass, approximate the spectral norm of the weight matrix ||W]] || with step of power iteration. Then, we perform a projection step so that its spectral norm will be clipped to / if bigger than 8, and unchanged otherwise. See Appendix C for more computational details.
4.2 ROBUST REGULARIZZATINN
Spectral Normalization directly bounds the global Lipschitz constant of the value function. How- ever, argued in Section 3.2, only require the local Lipschitz conditions around the model-

---

Published as a conference paper at ICLR 2023
uncertain local region. Such strong regularization is not necessary and can even negatively impact the expressive power of value function. Now we present an alternative approach to quickly regular- ize the local Lipschitz constant of the value function based robust training mechanism.
To regularize the local Lipschitz condition of the value network, we minimize the following loss:
Clog (Qoi 6, Te {8,, ag $1 Σ max
(Qo (p(s)) Qo(ss,
(8)
18--&llaSt
This robust loss is to guarantee that the variation of the value function locally is small. Then we combine it with the original loss of the value function Lore Tiy {5t, at s() by minimizing [(Qe: Tpp (si, dz; s (1) Long (Qoi Ta, {&i, 0i 221) + (o) (Qoi Thp (8r, ai s()p))
Here, and c are two hyperparameters. A larger is required for the dynamics model which has larger prediction errors. In terms of X, a bigger makes the value network varies smoother over the local region and gives better convergence guarantees, but it also hurts the expressive power of the value network. In practice, we find that using e = 0.1 is often enough for good performance. So we fix and search for the best A. For a detailed discussion on the choice of s, see Appendix D.4.
To solve the perturbation & in the constrained optimization within the robust loss, gradient sign method (FGSM) (Goodfellow et al., 2015). Here, let (s, be the state action pair. Then we compute = proj $0 + E sign (all==oo (5, - as)) (so))) e)).
the fast
where 80 ~ (Ns 2). Here the purpose of random initialization of so is because the gradient a (Qo (8)) - (s, a))²) vanishes 8. In our experiments, we focus on the Lo norm, but our proposed robust loss should be applicable to any Ip norm.
In addition, we can apply more advanced constrained optimization solvers such as Projected Gradi- ent Decent (PGD). But here, using FGSM together with a single deterministic environment model is mainly for efficiency purposes. In practice, we find that using PGD does not give much performance gain over FGSM method, and we refer the readers to Appendix D.3 for the experimental results.
5
EXPERIMENT
5.1 EMPIRICAL EVALUATIONS OF PROPOSED MECHANISMS
We evaluate our two proposed training mechanisms on five MuJoCo tasks, including Walker, Hu- manoid, Ant, Hopper, and HalfCheetah. We compare the two training mechanisms with MBPO (Janner et al., 2019) using an ensemble of probabilistic transition models. We do not compare our methods with MBPO using an ensemble of deterministic transition models mainly because it is ensemble-based and across all five tasks, it is outperformed by probabilistic ensemble models. In addition, since our regularization mechanisms are only trained on top of single deterministic tran- sition model, we compare with both MBPO using a single deterministic transition model and a single probabilistic model. We implement our methods and the baseline methods based on a PyTocch implementation of MBPO (Lin, 2022). More implementation details are provided in Appendix C.
Improved asymptotic performance. Figure 3 presents the learning curves for all methods. These results show that using just a single deterministic model, MBPO with our two Lipschitz regular- ization mechanisms achieves a comparable and even better performance across all five tasks than MBPO with a probabilistic ensemble model. In particular, the proposed robust regularization tech- nique shows a larger advantage on three more sophisticated tasks: Humanoid, Ant, and Walker. For example, on Humanoid, it achieves the same final performance as that a probabilistic ensemble with only about 60% of the environment interaction steps. In contrast, spectral normalization shows little improvement over a single deterministic model on these three tasks, showing the limitation of constraining the global Lipschitz constant. We provide more discussion on this in Section 5.3.
Time Efficiency. Compared with MBPO using an ensemble of probabilistic models, our proposed training mechanisms, especially robust regularization, more time efficient. In Table I we record the wall clock time of the algorithms trained for 200,000 environment interaction steps on Walker and Ant, averaged over 8 random seeds. We see that robust regularization is much faster than spectral normalization. While spectral normalization takes slightly less the amount of time than probabilistic ensemble, robust regularization only takes about 70% of the computational time.
7

---

Published as a conference paper at ICLR 2023
Probubilistic Ensemble
Single Probabilistic
Single Deterministic
Spectral Normalization (Ours)
Robust Regulatization (Ours)
Walkeer2d-22
Ant-v2
Humanoid-v22
0,300
4,000
1100
4,200
Porfortaace
Performance
4,000
2,000
8,000
1,00
7,000
2:000
2000
1,000
1100
1,300
Sael
15ad
30ad
Se-1
10ked
Moal
Nbet
Sart 10wd Hest Nhee Me-t
Newd
Number Interactions
Number of Interactions
Number of Interactions
Hopper-v2
HalfChectab.v2
Performance
Performance
1.000
0.2
2004
4644
60-4
S04
3024
5.44
10pm
15044
3004
Number of Interactions
Number of Intoractions
Figure 3: Performance of the proposed value function training mechanisms against baselines. Results are averaged over 8 random seeds and shaded regions correspond to the 95% confidence interval among seeds.
The significance of the experimental results twofold. First, it further validates insights and shows the importance tion of value functions in MBRL. Second, it demon- strates that having an ensemble of transition models not necessary. We can save the computational time and cost of training an ensemble of transition models with simple Lipschitz regularization mechanism. In practice, we can even use single probabilistic model combined with our proposed mechanisms to get the best performance. See Appendix D.2 for the extra experimental results.
Table 1: Comparison of computational time.
Walker
Ant
the local Lipschítz condi-
Algorithm
Time (h)
Time(h)
Single Deterministic
29.6
34.0
Probubilistic Ensemble
60.0
62.3
Robust Regularization 45.4
48.3
Spectral Normalization
54.2
57.1
5.2 VISUALIZING THE VALUE- MODEL ERROR
Given the excellent performance of robust regularization, we now verify its effectiveness in con- trolling the value-aware model error. On Walker, compare it with variant: computing the perturbation with uniform random noise instead of adversarially choosing the perturbation.
from the first figure in Figure 4, robust regularization has much smaller value-awar model error than all other methods. Adding uniformly random noise can somewhat reduce the value- aware model error compared with single deterministic model (without any noise added). However, it is still much less effective than robust regularization, which computes the error adversarially. In the second figure of Figure 4, see that with uniform random noise, it achieves slightly better performance than a single deterministic model but is still far worse than robust regularization. The results further verify that by adversarially choosing the noise, robust regularization is extremely effective at controlling the value-aware model eтror, resulting in great empirical performance. In Appendix D.I, we provide additional experimental results of value-aware model error in the rest of the four environments.
As
5.3 ROBUST REGULARIZATION SPECTRAL NORMALIZATION
In Table 1, we already see that robust regularization achieves much better time efficiency than spec- tral normalization. Now we analyze their performance difference through the lens value-aware model error. To reduce the value-aware model error, only need control the local Lipschitz con- stant of the value network over the model-uncertain region, and thus controlling the global Lipschitz constant of the value function with spectral normalization is not necessary. To see this, in the last two figures of Figure+ 4, we visualize the value-aware model error and performance of the algorithm on Walker with varying spectral radius defined in Section 4.1.
8

---

Published as a conference paper at ICLR 2023
Vilc-oou Mooet Stay
Iettekinnaaccaee Balae Bopdeinnks
Spaata Necackaticc i
130
Spatal (a 13)
Kalka Nase
nont Nerw homee
1.38
2.00
Presncberi Paceaic
ar Rap
1300
Sowt
Dbeed
Steek
Jleak
2044
Number of Imeractions
Number of Intoractions
Namber
of Interactions
Number of Internationn
Figure 4: (Ieff Two) Robust Regularization with comparison of uniform random noise on Walker, (Right Two) Spectral Normalization with different spectral radius / on Walker.
As we see from the plot, to reduce the value-aware model error so that it is about the same bust regularization, we need to choose the spectral radius 3 as small as 10. However, under this constraint, it achieves significantly worse performance than robust regularization, indicating that this constraint is perhaps too strong. To get the best performance, spectral normalization has to use larger spectral radius trade value-aware model error for expressive power and thus has worse empirical performance. In addition, we observe from Figure 3 that the performance discrepancy between the two methods is even greater in more complicated environments such Ant and Hu- manoid, which further shows the limitation of constraining the global Lipschitz constant.
6 RELATED WORK
Dynamics ensemble in model-based reinforcement learning. Dynamics model ensemble is first introduced in MBRL by Kurutach et al. (2018) avoid the learned policy exploit the insufficient data regions. Then Chua et al. (2018) proposed a probabilistic dynamics ensemble to capture the aleatoric uncertainty and epistemic uncertainty, and achieves significant performance improvement compared to the deterministic dynamics ensemble. Probabilistic dynamics ensemble is now widely used MBRL methods (e.g. Lai al. (2020); Clavera et al. (2020); D'Oro & Jaskowski (2020); Lai et al. (2021); Froehlich et al. (2022); Li et al. (2022)). However, despite its popularity, there still no clear explanation why the use of probabilistic dynamics ensemble brings such a large improvement the policy. In this paper, we fill this gap and propose novel theoretical explanation on why probabilistic dynamics ensemble works well.
Lipschitz continuity in reinforcement learning. In the realm of model-free RL, a few recent works also apply the technique of spectral normalization to regularize the Lipschitz condition of the RL agent's value function, which results in more stable optimization dynamics (Gogianu et al., 2021: Bjorck et al., 2021). Ball & Roberts (2021) investigate the effects of added Gaussian noise of exploration on the smoothness of value functions. In addition, Shen et al. (2020); Zhang et al. (2020) propose to use a similar robust regularizer as our robust regularization method, aiming at an adversarially robust policy. In our work, we focus on MBRL, fundamentally different setting. As explained in Section 3.1 and Appendix F, if no restrictions are imposed on the value function class, then the value-aware model error could explode, whereas the model error vanishes the model-free setting. In MBRL, Osband & Van Roy (2014) connects Lipschitz constant of the value function to the regret bound of posterior sampling for RL. Asadi et al. (2018) proposed learn a generalized Lipschitz transition model with respect to the Wasserstein metric, resulting in a bounded multi-step model prediction error and Lipschitz optimal value function induced from the learned dynamics. In contrast, paper focuses the local Lipschitz condition of the value function instead of the underlying transition model, studying its relation with the suboptimality of MBRL algorithm.
7 CONCLUSION AND DISCUSSION
In this paper, we provide insight into why the probabilistic ensemble model can achieve great empir- ical performance. We demonstrate the importance of the local Lipschitz condition of value functions in MBRL algorithms and justify our hypothesis with both theoretical and empirical results. Based on our insight, we propose two training mechanisms that directly regularize the Lipschitz condition of the value function. Empirical studies demonstrate the effectiveness of the proposed mechanisms. single model instead of ensemble, then cannot use many
One limitation is that if we existing methods of uncertainty estimation based on model ensemble, and we need to redesign the uncertainty estimation mechanism based on single model. We leave this future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by National Science Foundation NSF-IIS-FA program, DOD-ONR-Oftice of Naval Research, DOD Air Force Office of Scientifie Research, NOD-DARPA-Delenss Advanced Research Projects Agency Guaranteeing Al Robustness against Deception (GARD), Adobe, Capital One and JP Morgan faculty fellowships.
REFERENCES
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based rein- forcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Re- search, pp. PMIR, 10-15 Jul 2018. URL / /ttp://pooeeeeeeeee ngs ml ..rrsss///// v8C/asadi18a.ttm
Kavosh Asadi, Dipendra Misra, Seungehan Kim, and Michel L Littman. Combating the compounding problem with a multi-step model. arXiv preprint arXiv:1905.13222 2019.
Philip Ball and Stephen J. Roberts. Offcon What is state of the art anyway? abs/2101.11331, 2021. URL https://arae wrg. hts//bb//2100 http:/bs/2111.11333 11331.
CoRR,
Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The
Annals of Statistics, 33(4): 2005. ISSN 00905364. URL http:/ /www. jstor.
org/stable/3att 3448611
Nils Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learn- ing with spectral normalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp- 8242-8255. Curran Associates, Inc., 2021. URL https: / http///poceennnnneeeeeeeppe.. neurips.com hewee///aaaeeedddddeeeeeeeeeeeee Paper...d.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: httpss////ooo/eeopeeeee wew.innnnsnieeeeeeeeeeeeeee wwwperes.../ppppeeeeeeeeeeeeeeee cccppaper/211///1ee 302200007700ee11cc22222aeeeeeee
pdf.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.
Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. In International Conference on Learning Representations, 2020.
Pierluca D'Oro and Wojciech Jaskowski. How to learn a useful critic?
model-based action-gradient- estimator policy optimization. Advances in Neural Information Processing Systems, 33:313-324, 2020.
Farahmand, Andre Barreto, and Daniel Nikovski. Value-Award Loss Function for Model-based Reinforcement Learning. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 1486-1494. PMIR, 20-22 Apr 2017a. URL https:// proceedings. wwr. .ww.ppess./55. Tateess///5///aaaamaaa farahmandl 7a html.
Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, pp. 1486-1494,
PMLR. 2017b.
Lukas Frochlich, Maksym Lefarov, Melanie Zeilinger, and Felix Berkenkamp. On-policy model errors in reinforcement learning. In International Conference Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2023
Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Raz- van Pascanu. Spectral normalisation for deep reinforeement learning: An optimisation per- spective. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 3734-3744. PMLR, 18-24 Jul 2021. URL https: /ttp:://proceeeee ngs ml. press/v199 gogianu2la. html.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence princi- ple for model-based reinforcement learning. Advances in Neural Information Processing Systems, 33:5541-5552, 2020.
Christopher Grimm, Andre Barreto, Greg Farquhar, David Silver, and Satinder Singh. Proper value equivalence. M. Ranzato, Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 7773-7786. Cur- ran Associates, Inc., 2021. URL https: httttts::/prrceeeeee http///oocceddeeoeeooo ngs neurips.ccmm https/cccppppe///00.. file/ 20w0000000eee.mmc 7ce0 225544422285555557888773333
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Yizhou Huang, Kevin Xie, Homanga Bharadhwaj. and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In 2021 IEEE International Conference on Roboties and Automation (ICRA), pp. 799-801 805. IEEE, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32:12519- 12530, 2019.
Sanket Kamthe and Mare Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. In International conference on artificial intelligence and statistics, pp. 1701- 1710. PMLR, 2018
Juk Kocijan, Roderick Murray-Smith, Carl Edward Rasmussen, and Agathe Girard. Gaussian pro- cess model based predictive control. In Proceedings of the 2004 American control conference, volume 3, pp. 2214-2219, IEEE, 2004.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, pp. 5618-5627. PMLR, 2020.
Hang Lai, Jian Shen, Weinan Zhang, Yimin Huang, Xing Zhang, Ruiming Tang, Yong Yu, and Zhenguo Li. On effective scheduling of model-based reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Chongehong Li, Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, and Tin Liu. Gradient in- formation matters in policy optimization by paok-propagating through model. In International Conference on Learning Representations, 2022.
Xingyu Lin. m pytorch. https://githin com/ Xiigyu Lin Ltt/mbpo_pytoreeee 2022.
Minghuan Liu, Zhengbang Zhu, Yuzheng Zhuang, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Plan your target and learn your skills: Transferable state-only imitation learning via de- coupled policy optimization. In Proceedings of the 39th International Conference on Machine Learning, 2022.
11

---

Published as a conference paper at ICLR 2023
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL https:: stttps://openreveee. wwt. oet.e/eeeeetrrrro1oooo0011121111
Rémi Munos. Error bounds for approximate value iteration. In Proceedings the National Confer- ence on Artificial Intelligence, volume 20, pp. 1006. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
Duy Nguyen-Tuong, Jan Peters, and Matthias Seeger. Local gaussian process regression for real time online model learning. Advances in neural information processing systems, 21, 2008
lan Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL httt:s:s//proceedingg neurips wwtppaperr211//iileeeeee Paper pdf.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. PMLR, 2017.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8707-8718. PMLR, 13-18 Jul 2020. URL htttps://proceedinng mlr press / v 1 s0b html.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state obser- vations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21024-21037. Curran As- sociates, Inc., 2020. URL http:: htttps://proceedinng www.inngg....u.eeeeeeeeeeeeeeeee #0eb656 68ea a 1 4bwae223ff903c34da7444 Paper. pdf.
12

---

Published as a conference paper at ICLR 2023
Supplementary Material
A ADDITIONAL DEFINITIONS AND NOTATION
We start with the definition the Concentrability constant
Definition A.1. (Concentrabillty Constant) (Farahmand al., 2017a)
Given P.V & (S)), an integer k > O, and an arbitrary sequence of policies (T.) the distribution o . denotes the future state distribution obtained when the state in the first step is distributed according to p and the agent follows the sequence of policies T1, ... Rk- Define:
dpp "
(p.u(k)
sup
dv
2,2
dpp p is the Radon-Nilyydom Derivative of two probability measures, which is well-defined up to a set of measure zero by if pP ...p%. absolutely continuous with respect to In case it's not absolutely continuous, we set be oo. Then, for constant 0 S 7 < 1, define the discounted weighted average concentrability coefficient as
Here, = J The derivative
dy
C(p,v) = 7)²
Σ k=1
(k)
Throughout the proof, f S such that
|] denote the (p)( 1 S oo of measurable function I S.
(f()))
In addition, we define the empirical norm. Given collection points {.1, } in S, define the empirical norm Lp(s1..... 8,,) such that
1 N
If(s))
s.)
Finally, we define Rademacher complexity as in (Bartlett et al., 2005). Same as Farahmand et al. (2017a), we will use a local variant of Rademacher complexity to derive the rate of estimation error. Definition A.2. (Rademacher Complexity) Let di, on be independent Rademacher random variables, Le. Paa; = -1} P(a; = 1} 2- Given a collection of measurable functions F from X to R and probability distribution μ over X, sample points Ii, In i.i.d. from μ. Define
1 n
R sup Σ a.f(x.)
JEF N
Then define the Rademacher complexity of as E(R,FF
Besides, same as (Bartlett et al., 2005), we define the sub-root function as non-negative and non- decreasing function [0; oo) [0, oo) such that x(r) is non-increasing for > 0
vr
B PROOF OF THE THEOREM
We begin by citing the following theorem.
Theorem B.1. (Bartlet et al., 2005) Let F be a class of functions with values in range [a, b) and
assume that there some functional T TR. and some constant B such that for every f E F.
Varif] < T(f) < BE(f)
(9)
Let 1b be sub-root function and let = +r*(F) be the fixed point of tb. Assume for any < a satisfies
(br) E T() sr)
(10)
13

---

Published as a conference paper at ICLR 2023
Then, with 01 704 and 02 for any K and every with probability at least 1 for any F. we have
e=
K < K-1
Emlsl + a r" x(11(b - a) og
E
(11)
B
Also with a probability at least e for any f E F, we have
En - - ciK
B T x(11(b - o2
(12)
will then use (F) to denote the fixed point of a sub-root function & that sutisfies 10
Now we prove the following theorem which provides a finite sample bound on the value-aware model error.
Theorem B.2. Under the four assumptions 3,1,3.2, 3.3, 3.5, with the probability model learned based on Equation 2, there exists a constant k(a)) which depends solety on E (0, 1), such that with probability 1 1
20 L(P) Set x(a)D2RTE) In(
(13)
NIia
where N is the number samples from the data-collection distribution p. D is the size the state- space desined in assumption 3.2, and R is defined in the metric entropy condition of the model class in assumption 3.5.
Proof. Given batch of state-action transition triples {(si, ai, +1)) with (8;, a:) sampled i.i.d from data distribution we denote the empirical loss
/ (((ls, </|1d6
L. (P N
(14)
We also denote the underlying loss over the data distribution p as
L(P) = B(s.a>~p / P(8' (s, a)lls' s "||ds'
(15)
In addition, let P E M be the best model in the transition kernel class M defined in 3.5.
Now we would like to apply Theorem B.2 bound the difference between the empirical and the true underlying loss. First, let F {(s a, s') 1(s a, s'; P) I(s x a, s'; P); P .M be the class of functions in Theorem B.1, where [(s x s'; P) is the single datapoint version of the empirical loss I., (P). So Exxa<pl [(s x a, s L(P). Now by assumption 3.2, S I(s x a, a; S 2D for every x a E S x A. S, and P M. Therefore, the value of f is bounded between -2D and 2D for every f F.
As consequence, for every E F, Var([) S B S 4D2 So we can set
T() AD*E / P(5'|s,a)|1'' ss s'|(ds))
B 4D2
Now, we can apply Theorem B.2 to conclude that with probability 1 - 8 (let K =2),
2 704
(11 4D 26 4D2) In(
L(P) < ((L. (P)
4D r' (F)+
(16)
N
Since P. is the minimizer of the empirical loss L, (P),
352 L(P) - (())) <
(44D + 208D2 )) In(
r (F) +
(17)
D
N
14

---

Published as a conference paper at ICLR 2023
We can provide upper bound of the local Rademacher complexity r" (F):: there exists a finite
constant T > 0 such that for a given 0 S a $ 1, we have
(F) < a((a)DRRi TD
(18)
NT
N
where c(a)
The proof follows the exact same steps of Proposition 10 in Farahmand
(1 a) a
पे.
et al. (2017a). Now back to Equation 17, by the realizability assumption 3.3, the best model P in the model class satisfies that L(P) S Therefore, with probability - 8,
352c, (2) 352 In N (44D + 208D2) In(3)
L(P) Se
(19)
N
N
N
Finally, there should exist a constant rt(a) sufficiently large such that with probability 8,
L(P) set =(a)D2 In(3)
(20)
NI
Corollary B.3. Under the five assumptions 3.1,3.2, 3.3, and 3.5 with the probability model learned
based on Equation 2, there exists a constant k(()) which depends solely on E (0, 1), such that with probability -exp(-
eN
D(
L(P) S 26,
(21)
Proof. This is a straightforward application of Theorem B.2, where we could just let = D2 20 ln())
Next, consider the local Lipschitz condition of the value function and provide finite sample bound of the value-aware model error.
Theorem B.4. Under the five assumptions 3.1,3.2, 3.3, 3.4, and 3.5, with the probability model learned based on Equation 2, there exists a constant r((a) which depends solely on a E (0, 1), such that for any m >
R (1 7)2 (1 - -E)],
(T) do(s, ~ +
(22)
ito
where & = -
exp(-
D2
Proof.
IT'Q -16) r(s,a) -? P(a5')s, dp(s.
7/ /l P( (V))))) - v(6')) 2 dp(s,a)
5/2 / P(ds' 18 (V(5)) v(5') dp(s,a)
2>² / / |[lls' s'll < 2e) - v(s')
+*² / L (V((())) v(s') ' di' a.)ps(sss
R2 R
S 2722212PP(11' - 5 7
P(lls 81 > 2c
(1 71
15

---

Published as a conference paper at ICLR 2023
Now apply Corollary B.
eN EN
- S7246 eepp(
K(a)) k(a) D2R
R max
eN I+a
+ ?-
exp(
(1 11-7) 7)2
Finally, with the value-aware model error bounded, could apply the error propagation results from (Munos, 2005; Farahmand et al., 2017a) and prove our main theorem, which relates the local Lipschitz constant to the sub-optimality of the approximate model-based value iteration algorithm.
Ranx for V (s, Under the Theorem B.5. Suppose Qu is initialized such that Qu(s,a) 1-77
sumptions of 3.1, 3.2, 3.3, 3.4, and 3.5, after iterations of the model-based approximate value iteration algorithm, there exists constant k(()) which depends solely on E (0,1) such that,
[(s,a)/P [ Qx(s,a)
S
27
[[(p, Po)
max òū 7 OSkSk
(1 7)
- [-
) 27 Rmaxx
+ (1 7)²
(23)
where 0% =
is the regression error defined in Equation (3), E =
eN exp(- k(a)D2RITT ). and C(p, C() the concentrability constant defined in Definition A.I.
Proof. This follows directly from the Theorem B.4 and also Theorem 4 from (Farahmand et al.,
R²
2017a), where the value-aware model error emodel 72 4e2 L2
(1 -t)
(1 - 7)²
C IMPLEMENTATION DETAILS
In this section, We implement our methods based on a PyTorch implementation of MBPO (Lin, 2022). The dynam- ies model architecture is MLP with four hidden layers of size 200. In Ant and Humanoid, the hidden size is 400 because these two environments are more complex than others. For the probabilistic dy- namics model ensemble, we set the ensemble size 7 which is the setting used in the original paper of MBPO (Janner et al., 2019). The policy is optimized with Soft Actor-Critic (SAC) (Haarnoja al., 2018). The actor network architecture and the critic network architecture MLP with two hidden layers of size 256.
going to introduce the implementation details of our two proposed methods.
For robust regularization, we fix e to 0.1 as discussed in Section 4.2, and we do a grid search of over [0.01, 0.1, 1.0]. with the result presented in Table 2 For spectral normalization, we add the nor- particular, every forward pass, we approximate
malization on each layer of the critic network. the spectral radius of the weight matrix with one step of power iteration. The algorithm is sketched below with u and v being the right and left singular vector of the weight matrix W.
< (t + ||a|| ( + a- --v
(24)
u + (()) p d w plu
(25)
Then we perform a projection of the parameters: W == max max(a p) - So the spectral
3
norm will be clipped to if it's bigger than X, unchanged otherwise. We do grid search of B over [15, 20,25, 30, 35], with results shown in Table 3. For a fair comparison, we set the rollout horizon during model rollouts in all environments.
16

---

Published as a conference paper at ICLR 2023
Table 2: Robust Regularization with different regularization weights X
Environment
Probubilistic Model Ensemble
Deterministic Ensemble
0.01
0.01
À 1.0
Walker
3236
4385
4138
4515
4112
HalfCheetah
10519
10648
10871
10982
10763
Ant
1684
4290
4190
4592
4011
Humanoid
468
4235
4210
4531
5123
Hopper
2432
3278
3213
3110
3010
Table 3: Spectral Normalization with different spectral radius B
Environment
20
25
]
30
[ 35
Walker
3870
4447
4124
3982
3645
HalfCheetah
10452
10763
10313
10423
Ant
3870
4447
4124
3982
3645
Humanoid
427
662
704
712
684
Hopper
2876
3125
3364
3243
2972
D ADDITIONAL EXPERIMENT RESULTS
D.I ADDITIONAL EXPERIMENTAL RESULTS VALUE-AWARE MODEL ERROR
Ant-v2
Humanoid-v2
Value aware Model Ernur (Log Scale)
Prohabilistic Ensemble
Value- Salee-aware Error (Log Scale)
Probabilistic Ensemble
6
Robust Regularization
Robust Regularization
Spectral Normalization
Spectral Normalization
5
4
3-
5e+4
10c+4
15c44
20c+4
5044 10c44 15c44 20c+4 0
25c44 30c44 Number Interactions
Number of Interactions
Hopper-v2
HalfCheetah-v2
Value aware Model Error (Log Scale)
Probabiistic Ensemble
aware (Log Scale)
Robast Regulatization
Spectral Normalization
2.
3
Probabilistic Ensemble
Robust Regolarization
Spectral Normalization
2c+4
4e+4
6c+4
Sc+4
10c=4
Set4
10e-4
15e+4
20e+4
Number of Interactions
Number of Interactions
Figure 5: Comparison between the proposed mechanisms and the probabilistic ensemble baseline in terms of Value-aware Model Error (Log Scale)
In Section 5.2 and 5.3, we demonstrate the effectiveness of robust regularization in controlling the value-aware model eтror on Walker and also show the limitation of constraining the global Lips- chitz constant by spectral normalization. Here, provide the results of value-aware model error in the rest of the four environments. We used the same best hyperparameters reported in Table 2 and Table 3, which have the best performance under grid search. Once again, observe that robust
17

---

Published as a conference paper at ICLR 2023
regularization effectively reduces the value-aware model error by constraining the local Lipschitz condition with computing adversarial perturbation. In addition, we also see that global Lipschitz constraints strong for spectral normalization. In two more complicated environments, Ant and Humanoid, it has to sacrifice value-aware model error for the expressive power of the value function. Therefore, spectral normalization does not achieve good performance in these two vironments. However, in two easier environments, Hopper and HalfCheetah, spectral normalization could still effectively reduce the value-aware model error and has good empirical performance.
D.2
PROPOSED MECHANISMS WITH SINGLE PROBABILISTIC MODEL.
Prukubilistic Ensemble
Valle--ware Model Etror (Log Scale)
Prokubilistic Ensemble
6,000
RR (Prahahilistic)
RR (Prabuhilisttic
(Deterministic)
(Deterministcc)
4,000
SN (Probabilistie)
SN (Probabilistic)
Porfornance
SN (eterministic)
SN (Deterministir
3
2,000
1,000
0
50+4
10c+4
15c+4
20c+4
5c+4
10c+4
15c+4
20c+4
Number of Internctions
Number of Internctions
(a) Performance
(b) Value-aware Model Error (Log Scale)
Figure 6: Spectral Normalization and Robust Regularization with single probabilistic model Ant. RR short for Robust Regularization, SN is short for Spectral Normalization
In the experiment section, combine proposed mechanisms with single deterministic model and compare it against MBPO using an ensemble of probabilistic models. The purpose is to verify that regularization of the local Lipschitz constant is critical in MBRL algorithms and propose computationally efficient MBRL algorithm without model ensemble. In practice, complementary our proposed Lipschitz regularization mechanisms, we can also use a single probabilistic model to further regularize the local Lipschitz condition of the value function. In addition, training probabilistic environment model would be better suited for environments with stochastic transitions.
Here in Figure 6, we combine our two proposed mechanisms with both probabilistic and determin- istic model on Ant, comparing them with the probabilistic ensemble baseline. From Figure 6b, we see that although spectral normalization with single deterministic model has large value-aware model error, it is significantly reduced when combining it with a probabilistic dynamics model. Therefore, find that spectral normalization with a probabilistic model achieves much better per- formance and even outperforms MBPO with an ensemble of probabilistic models. For robust regu- larization, using a probabilistic model also helps improve the algorithm's value-aware model error and performance. This observation suggests that the two Lipschitz regularization approaches, ex- plicit regularization by spectral normalization or robust regularization and implicit regularization by probabilistic models, are complementary. In practice, we can combine the two approaches to get the best performance of the MBRL algorithm.
D.3 ROBUST REGULARIZATION WITH FGSM VS. PGD
In Figure 7. we compare the performance of robust regularization with 20 steps of Project Gradient Descent (PGD-20) against the Fast Gradient Sign Method (FGSM) on Walker. In particular, al- though PGD-20 is much more computationally expensive, we do observe the improvement with this more powerful constrained optimization solver.
D.4 ROBUST REGULARIZATION WITH DIFFERENT REGULARIZATIIN WEIGHTS
Figure 8 further visualizes how the regularization weight À of robust regularization influences the algorithm performance. Similar to the findings of the experiments that the algorithm's 's performance first increases and drops as the regularization weight gets larger. This verifies theoretical insights from Theorem B.5 that with small X, with algorithm gets less
spectral normalization,
18

---

Published as a conference paper at ICLR 2023
5,000
Performance
3,000
2,000
1,000
PGD 20 FGSM
0
20e+4
Number Intoractions
Figure Robust Regularization with 20 steps of Project Gradient Descent (PGD-20) against Fast Gradient Sign Method (FGSM).
4,500
1,000
Performance
3,500
3,000
Robust Regularization
Figure 8: Robust Regularization with diffcrent regularization weights X's on Walker. Experiments are all with 8 random seeds.
regularization and thus has a big value-aware model error. But meanwhile, it also has a small re- gression error since the regularization has little effect on the expressive power of the value function. When À goes up, the regularization will have stronger negative effect on the expressive power of the value function, but the value-aware model error will also get smaller. We observe that the algorithm performs the best with À = 0.1, achieving the balance between aware model error and the value function's 's expressive power.
19

---

Published as a conference paper at ICLR 2023
E ADDITIONAL DETAILS ON THE INVERTED-PENDULUM EXPERIMENT
In Section 3.2, we provide an experiment of model-based value iteration on the Inverted Pendulum further verify the validity our theorem. Below we provide the pseudocode for it.
Algorithm Model-based Approximate Value Iteration on Inverted Pendulum
K Total number of iterations for the algorithm
H Number of gradient steps to solve the inner optimization
M,9 Space of transition probability kernels and reward functions
F: Space of value function
A(S x A): a data-collecting state-action distribution
Sample i.i.d from j to generate the training dataset D = {(s;, ag 8 r;)) N
N
/ P(ds') a)
P +- arg min
||
PEM
arg min
Σ
(ra f(8;, a:))²
FEG
i=1
Initialize the value function Qo- repeat
for = to K do
Sample i.. N state-action pairs from p Compute & ~ ((-ss, a,), f. f(s,, o,)
0:))
for to H 1 do
N
Update policy using gradient ascent with
Velo
N
end for for = 0 to И 1 do
Update value function using gradient descent with
V (A:+ + 12o mo( a))))
end for end for until end of training Output: Qo
20

---

Published as a conference paper at ICLR 2023
F LIPSCHITZ REGULARIZATION OF MODEL-FREE RL ALGORITHMS
Original
Spectral Normalization
Robust Regularization
Ant-v2
Walker2d
6,000
5,000
4,000
4,000
Performance
Performance
3,000
2,000
2,000
1.000
0
0.2c+6 0,4e46 0.6e+6 0.8c+6
lc+6
0.2o+6 0.4c+6 0.6c+6 0.8c+6 le+6
Number Interactions
Number Interactions
Figure 9: Combination of robust regularization with model-free SAC
In this paper, provide both theoretical and empirical insights into why Lipschitz regularization is crucial in model-based RL algorithms through the lens of value-aware model error. On the con- trary, the model error vanishes in the model-free setting, so our theoretical insights no longer hold. However, fitting value function with smaller Lipschitz constant may still be beneficial for policy optimization and the value prediction of -fiittidiitiiitoi acteon pairs.
So does the improvement shown in Section 5 indeed come from the controlled value-aware model error, which unique in the model-based setting? We conduct an experiment on the model-free setting, adding our proposed spectral normallization and robust regularization mechanisms to the model-free SAC respectively with the same hyperparameter settings used in the paper. As shown in Figure 9, combining the Lipschitz regularization mechanisms slightly improves the performance of model-free SAC algorithm. Still, the improvement
more limited compared with the im- provement of the model-based algorithm shown in Figure 3. The results suggest that although some additional aspects of value learning could be affected by Lipschitz regularization, our insights into the value-aware model error for model-based scenarios should still hold. The exploration of how Lipschitz regularization impacts other aspects of value learning the model-free setting is beyond the scope of our work, but it would be an interesting direction for future work.
21


Reviews:
[{"review_id": "uQNtKa_Nf1k", "reviewer": "Reviewer_S2Gs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Is model ensemble necessary\" presents an investigation of the interplay between value function smoothness (in the form of a Lipschitz constant) and model errors in model based reinforcement learning. They present a theoretical analysis and show empirical performance gains based on their insights in a practical algorithm.", "summary_of_the_review": "The authors present a well written paper that carefully examines the impact of a Lipschitz regularization on the performance of model based reinforcement learning. They highlight both theoretically and empirically how the Lipschitz bound of the value functions impacts the performance of model based value estimation and how the insight can be transformed into a strong algorithm. Overall, I think the paper is a strong submission to ICRL. However, I believe there are issues in the presentation that could be improved to strengthen the submission further, with the details outlined above.", "strength_and_weaknesses": "Detailed review:\n\n== Strengths ==\n\n- The paper combines theoretical insight and empirical evidence in a very convincing fashion.\n- The insight seems broadly useful for many related model based algorithms that use a Dyna style update rule.\n- The practical algorithm seems to be a strong and simple addition to MBRL algorithms\n- The paper is easy to understand, the mathematical results are well explained both with rigorous derivation and intuitive explanations.\n\n== Weaknesses ==\n\nMain main concern is the presentation and contextualization of the result. It seems that the impact of the Lipschitz regularity is independent of the concrete choice of an ensemble model. The theoretical derivation does not hinge on the fact that the model is represented by an ensemble and while the empirical results do show that the performance of the regularized algorithm is superior even when using a single model, this does not necessarily suggest that ensembles cannot contribute separately. For example the model can be used to construct certainty estimates or to drive exploration, which the underlying MBPO algorithm does not leverage.\n\nI would strongly suggest that the authors decouple their core contribution, the impact of Lipschitz bounds of the value functions, from the insights into ensemble models in the presentation. This is not because I think that any of the claims are wrong (although I think the statement that \"model ensembles might not be necessary\" is slightly to general to be supported by the paper), but because I think their insight is more valuable as an independent insight, not just as an auxiliary insight to ensemble performance.\n\n---\n\nEmpirically, I would like to see a comparison of the impact of the Lipschitz regularization on a model free baseline. This simple ablation would highlight the fact that it is the interplay of Lipschitz smoothness and model based learning that produces the effect, not an independent mechanism that improves value function learning in general.\n\nIn the related work section, the authors acknowledge previous work in the area of Lipschitz regularization in RL, but claim that they work in the \"fundamentally different\" field of MBRL. While I agree that the authors insights are important and novel in MBRL, I do not think that the field is \"fundamentally different\" enough from MFRL that the alternate hypothesis that spectral normalization simply improves RL (MF or MB) can be dismissed out of hand.\n\n---\n\nThe regularization/spectral norm creates an important hyperparameter, which the authors themselves highlight as a tradeoff between value function flexibility and model optimization. I would have liked to see more discussion or experimental evaluation what happens with a sub-optimal choice for this hyperparameter. If the algorithm is not robust to this choice, it creates yet another potential pitfall in a field which already struggles with complicated algorithm which are very expensive to tune. The results in the appendix showcase at least some robustness to the concrete choice of regularization, but it is unclear whether this holds over all benchmark tasks. Since the grid-search was performed already, would it be possible to append the results?\n\n---\n \n\n== Minor corrections ==\n- Page 6, top: For In particular -> In particular?\n- Page 18, bottom: computationally expansive -> expensive?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uFCAr0g4Sdr", "reviewer": "Reviewer_pnXv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to regularize the value function approximation in MBRL in order to learn policies without having to use an ensemble of models (as is done in MBPO bu Janner).\n\nAfter the authors present evidence that the ensemble of models is indeed regularizing the value function by using the value-aware model error, they propose their own scheme to regularize the value function.\n\nThis scheme works quite well in experimental results, beating MBPO while only using one model, not an ensemble of models.", "summary_of_the_review": "This paper provides empirical evidence for a strong new avenue of research in MBRL: value function regularization. Although I find the theory unconvincing, I think the message of this paper is something the RL community would benefit from, providing empirical evidence that will invite theories to follow.", "strength_and_weaknesses": "I think the experimental results are quite good, and I really like that the authors provide a good path to get sample efficient results without an ensemble. I also think that value function regularization is a good path to follow.\n\nThe main weakness is the theoretical contributions.\n- For a continuous function over a convex state space, the local $(\\epsilon,p)$-Lipschitz constant is the same as the Lipschitz constant (see proof below). It would seem to be easier to simply assume the state space to be convex, and use a global Lipschitz constant. So why all the complexity with the Local-Lipschitz property. Or am I missing something here?\n- I don't feel that Theorem 3.6 is particularly interesting, I would hope for something where the error goes down as N increases, but that's not the case. I fail to see why a bound that doesn't decrease as more environment interaction iterations occur is somehow useful, since it will never get better than the performance after just one iteration.\n\nI think a good theoretical avenue would be to look at [1], which lays out a relationship between model error and RL performance, and also shows that a bounded Lipshcitz is important. \n\nProof that $L(\\epsilon,p)(f)=L(\\infty,p)(f)$. The authors already state that $\\epsilon_1\\leq\\epsilon_2\\Rightarrow L(\\epsilon_1,p)(f) \\leq L(\\epsilon_2, p)(f)$, so I will show the $\\epsilon_1\\geq\\epsilon_2\\Rightarrow L(\\epsilon_1,p)(f) \\leq L(\\epsilon_2, p)(f)$. \n\nThis follows simply from a property of continuous functions over a convex state space, if there are states $s_1,s_2$ with\n$f(s_1)<f(s_2)$ and $s':=\\lambda s_1 + (1-\\lambda) s_2$ is some point between $s_1$ and $s_2$, then \n\n$$\\left\\vert\\frac{f(s_1)-f(s_2)}{\\Vert s_1-s_2\\Vert}\\right\\vert\\leq\\max\\left(\\left\\vert\\frac{f(s_1)-f(s')}{\\Vert s_1-s'\\Vert}\\right\\vert,\\left\\vert\\frac{f(s')-f(s_2)}{\\Vert s'-s_2\\Vert}\\right\\vert\\right)$$\n\nso over a smaller distance (either from $s_1$ to $s'$ or from $s'$ to $s_2$) we get a larger lipschitz constant, showing my claim\n\n\n[1] Ian Osband, Model-based Reinforcement Learning and the Eluder Dimension", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WzFyyjqh49J", "reviewer": "Reviewer_4iuU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies 'probabilistic dynamics model ensemble' method in reinforcement learning. It proposes that (1) an important factor to the convergence of RL is the Lipschitz condition of the value function and (2) model ensemble helps to regularize the Lipschitz condition in the training. The paper provides both theoretical analysis and experiment results to support such claims.", "summary_of_the_review": "The paper is not ready for publication yet as the contribution of the paper is rather vague to readers.", "strength_and_weaknesses": "Strength: The paper aims to provide some insights for designing more efficient RL algorithms.\nWeakness: The contribution of the paper is rather vague to readers. It is not very clear what the exact problem the paper wants to solve and what can we learn from the results. In particular, the concept of model ensemble in the title is not fully explained in the paper, and the later main results are based on very strong conditions, e.g., deterministic transition. In more details, I have the following questions:\n\n1. What is 'model ensemble' in this paper exactly referring to? Is it an ensemble of multiple independent fitting or sequential adaptive fitting? Should the algorithm stated in section 3.2 be considered ensemble algorithm or a single model algorithm as mentioned in the title? One thing make me confused is that the number of ensemble models and number of value iterations are both $K$. Is this a correct? \n\n2. Does the upper bound in Theorem 3.6 converges to zero as $K\\rightarrow\\infty$? If it does (under any condition), please clearly show it. If it does not, I can only see it decreases as the Lipschitz constant and $C$ getting smaller, which does not provide any new insight.\n\n3. What is exactly hidden when we only consider deterministic transition rather than stochastic transition? Please be more clear about whether the analysis framework will go through and what will be different.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SoBteG-Jr-4", "reviewer": "Reviewer_24kG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies why ensembles of dynamics models are useful for model-based RL. The main result is that ensembles effectively regularize the value function to be more smooth, and explicitly regularizing the value function can achieve similar performance without requiring an (expensive) ensemble.", "summary_of_the_review": "Overall, this is a strong paper that makes progress on an important problem. I think the paper could be improved in a few ways, but I nevertheless think that the paper should be accepted.", "strength_and_weaknesses": "Strengths\n* I think this question studied in this paper is really important. Both in model-based RL and in other subareas of RL, ensembles are used to great effect, but _why_ they're useful has remained an open problem.\n* The empirical results are quite strong. It's great that the paper not only identifies the underlying mechanism that explains why ensembles are useful, but also uses that finding to propose a simpler approach.\n\nWeaknesses\n* Writing. The writing could be improved/clarified in some places (see below). The paper has a number of grammar errors. I'd recommend copy-pasting the paper into a Google Doc and running the grammar checker.\n* It seems like the proposed method requires careful hyperparameter tuning, with different hyperparameters used for each environment. While some prior work does this, it is generally frowned upon. I would recommend highlighting this limitation in the main paper.\n\n\nQuestions/concerns\n* Is this really a paper about model-based RL, or is it showing that smooth Q-functions are useful, especially when doing many gradient steps on each example? I.e., if this same regularizer is applied to a model-free RL method, can it match the performance of model-based methods?\n* Stochastic dynamics (top of page 6) -- Do the results hold for stochastic dynamics? The discussion at the top of page 6 seems to indicate that they do, but the results on page 5 say that they do not.\n* Eq. 9 -- Would a 1-sided penalty work, too? I.e., is the main purpose of the regularizer to make the Q-function smooth, or to prevent it from making significantly _larger_ predictions for nearby states?\n* Why isn't the proposed method faster? I would have guessed that removing the ensemble would make the method 2 - 4x faster.\n\nMinor writing comments\n* \"to validate\" -> \"to test\" -- \"validate\" seems less rigorous\n* Perhaps cite this paper [1], which provides a nice discussion of how noise in the actor can provide smoothness in the critic (w.r.t. the actions, rather than the states).\n* The passive voice is to be avoided where possible.\n* \"than using only a single dynamics model\" -- Cite.\n* \"we hypothesize\" -> \"We hypothesize\"\n* \"Therefore, the Bellman operator ...\" -- I didn't understand this sentence.\n* \"Ian J Goodfellow\" -- This is an odd citation format.\n* \"Plug-and-play modules\" -- Where is this shown in the experiments?\n* \"time and resources\" -- what type of resources?\n* \"value function Q*\" -- add a comma at the end\n* \"we define the model-induced Bellman Operator\" -- Should this use $\\hat{r}$ rather than $r$?\n* Eq 1 -- This could be cut or moved to the appendix.\n* \"Define local\" --> \"Define the local\"\n* Eq. 2 -- What is the first \"sup\" over? Is that a typo?\n* Sec 3.1, first paragraph -- the line spacing here looks odd.\n* All figures -- Make the xlabel and ylabel the same size as the surrounding text. Use the Matplotlib built-in log-scale and scientific notation, rather than doing it manually (it is easier to read).\n* In other words, even ...\" -- This sentence is great!\n* \"achieve similar mean squared errors\" -- This is fairly subjective; the ensemble does achieve lower errors. Perhaps just put numbers to it: \"The mean squared errors are within x%\"\n* Assumption 3.1 seems very strict. I'd recommend adding more discussion of why this is needed and when this can be relaxed.\n* Assumption 3.2 -- This is a very weak assumption, and perhaps could simply be noted when defining the MDP (and not stated as another assumption in this section)\n* Eq 6 -- Is the norm here the L2 norm? If so, indicate that using \"$\\|\\|_2$\"\n* \"it's\" -- It's generally better to avoid contractions in technical writing.\n* \"Assumption 3.5, $J(\\mathcal{P})$ -- Where is $J(\\cdots)$ defined?\n* \"Tradeoff between\" -- This section is really great!\n* \"We plot (1) ...\" -- It might be clearer to replace \"(1)\" with \"(Fig 2a)\"\n* \"between 1000 and 4000\" -- Where does the 4000 number come from?\n* \"more advanced constrained optimization solvers such as PGD\" -- To me, PGD seems no more complex than FGSM\n* Fig 3 -- I found the color for \"single probabilistic\" very hard to read. Consider using colorblind-friendly colors, or adding different markers to the lines\n* \"then many existing methods ...\" -- I didn't understand this sentence.\n\n[1] https://arxiv.org/pdf/2101.11331.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster