Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms
Authors: Shenao Zhang
Boyi Liu
Zhaoran Wang
Tuo Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Model-Based Reinforcement Learning
Policy Gradient

Abstract: ReParameterization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. However, recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes with exploding gradient variance, which leads to slow convergence. This is in contrast to the conventional belief that reparameterization methods have low gradient estimation variance in problems such as training deep generative models. To comprehend this phenomenon, we conduct a theoretical examination of model-based RP PGMs and search for solutions to the optimization difficulties. Specifically, we analyze the convergence of the model-based RP PGMs and pinpoint the smoothness of function approximators as a major factor that affects the quality of gradient estimation. Based on our analysis, we propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls. Our experimental results demonstrate that proper normalization significantly reduces the gradient variance of model-based RP PGMs. As a result, the performance of the proposed method is comparable or superior to other gradient estimators, such as the Likelihood Ratio (LR) gradient estimator. Our code is available at https://github.com/agentification/RP_PGM.

Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms
Shenao Zhang'
Boyi Liu'
Zhaoran Wang
Tuo Zhao
Northwestern University Georgia Tech
shenaodu shenao@u.northwesterr www.tneeteeee.eeeeeee.
Abstract
RcParameterizzaioo (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. However, recent studies have revealed that, when applied to long-term reinforcement learn- ing problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes with exploding gradient variance, which leads slow convergence. This is contrast to the conventional belief that reparameterization methods have low gradient estimation variance in problems such training deep generative models. To comprehend this phenomenon, we conduct a theoretical examination of model-based RP PGMs and search for solutions the optimization difficulties. Specifically, we analyze the convergence the model-based RP PGMs and pinpoint the smoothness of function approximators as major factor that affects the quality of gradient estimation. Based on our analysis, we propose a spectral normalization method mitigate the exploding variance issue caused by long model unrolls. Our experimental results demonstrate that proper normalization significantly reduces the gradient variance of model-basee RP PGMs. As result, the performance of the proposed method is comparable superior to other gradi- ent estimators, such as the Likelihood Ratio (LR) gradient estimator. Our code is available at https: // thub. com/ coom/aentiff cnantifiiaiiio/RR. __PGM
Introduction
Reinforcement Learning (RL) has seen tremendous success in variety of sequential decision- making applications, such as strategy games [51, 59] and robotics [15, 61], by identifying actions that maximize long-term accumulated rewards. As one of the most popular methodologies, the policy gradient methods (PGM) [56, 31, 52] seek to search for the optimal policy by iteratively computing and following a stochastic gradient direction with respect to the policy parameters. Therefore, the quality of the stochastic gradient estimation is essential for the effectiveness of PGMs.
Two main categories have emerged in the realm of stochastic gradient estimation: (1) Likelihood Ratio (LR) estimators, which perform zeroth-order estimation through the sampling of function evaluations [64, 32, 31], and (2) ReParameteriiztioo (RP) gradient estimators, which harness the differentiability of the function approximation [17, 48, 12, 53J. Despite the wide adoption of both LR and RP PGMs practice, the majority of the literature on the theoretical properties of PGMs focuses on LR PGMs. The optimality and approximation error of LR PGMs have been heavily investigated under various settings [3. 60, 7]. Conversely, the theoretical underpinnings of RP PGMs remain to be fully explored, with dearth of research the quality of RP gradient estimators and the convergence of RP PGMs.
RP gradient estimators have established themselves as reliable technique for training deep generative models such as variational autoencoders [17]. From a stochastic optimization perspective, previous
Equal advising.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

studies [48, 40] have shown that RP gradient methods enjoy small variance, which leads to better convergence and performance. However, recent research [42, 37] has reported an opposite observation: When applied to long-horizon reinforcement learning problems, model-based RP PGMs tend to encounter chaotic optimization procedures and highly non-smooth optimization landscapes with exploding gradient variance, causing slow convergence.
Such an intriguing phenomenon inspires us delve deeper into the theoretical properties of RP gradient estimators in search of a remedy for the issue of exploding gradient variance in model-based RP PGMs. To this end, we present a unified theoretical framework for the examination of model-based RP PGMs and establish their convergence results. Our analysis implies that the smoothness and accuracy of the learned model are crucial determinants of the exploding variance of RP gradients: (1) both the gradient variance and bias exhibit a polynomial dependence on the Lipschitz continuity of the learned model and policy w.r.t. the input state, with degrees that increase linearly with the steps of model value expansion, and (2) the bias also depends on the error of the estimated model and value.
Our findings suggest that imposing smoothness on the model and policy can greatly decrease the variance of RP gradient estimators. To put this discovery into practice, we propose a spectral normal- ization method enforce the smoothness of the learned model and policy. It's worth noting that this method can enhance the algorithm's is efficiency without substantially compromising accuracy when the underlying transition kernel smooth. However, if the transition kernel is not smooth, enforcing smoothness may lead to increased егтоr in the learned model and introduce bias. In such cases, a balance should be struck between model bias and gradient variance. Nonetheless, our empirical study demonstrates that the reduced gradient variance when applying spectral normalization leads to significant performance boost, even with the cost of a higher bias. Furthermore, our results highlight the potential of investigating model-basee RP PGMs, as they demonstrate superiority over other model-based and Likelihood Ratio (LR) gradient estimator alternatives.
2 Background
Reinforcement Learning. We consider learning to optimize an infinite-horizon y-discounted Markoy Decision Process (MDP) over repeated episodes of interaction. We denote by S C RR., and A C Rd. the state and action space, respectively. When taking an action a E A at a state E S, the agent receives reward r(s, a) and the MDP transits a new state s' according s' ~ f()
We aim to find a policy π that maps a state to an action distribution to maximize the expected cumulative reward. We denote by V S - R and Q" : S A + E the state value function and the state-action value function associated with т,, respectively, which are defined as follows,
((-)))) E.T.
Σ r(si,,,) so s, ao =
=] a]
V*((s) = Eawt Q" (3)
Here E S. A, and the expectation Ex . is taken with respect to the dynamic induced by the policy и and the transition probability/ f. We denote by Ç the initial state distribution. Under policy x, the state and state-action visitation measure v. (s) over S and a as a) over S A are defined as
M (s) = (1 7) 7f PPs =),
(x = (1 (1-7) Σ 7" IP(x= = 8,04 = =a),
0
where the summations are taken with respect to the trajectory induced by su ~ S, a; ~ x(: |ss), and 8141 [- 81, a.). The objective (TT) of RL is defined as the expected policy value as follows,
J(T) = Esg - (so)] E(,,,)_,, [)(s.
(2.1)
Stochastic Gradient Estimation. The underlying problem of policy gradient, i.e., computing the gradient of expectation with respect to the parameters of the sampling distribution, takes the form V Vp [y)).. To restore the RL objective, we can set p(r; f) as the trajectory distribution conditioned on the policy parameter and (αx) as the cumulative reward. In the sequel, we introduce two commonly used gradient estimators.
Likelihood Ratio (LR) Gradient KeeotthOOddety By leveraging the score function, LR gradients only require samples of the function values. Since Vo log (z; 0) Vop(a; ()pp(;; f), the LR gradient is
Vo pp [u(x)] = Ep(a,o) (x Vo logp(x;;)
(2.2)

---

ReParameterization (RP) Gradient (First-Order) RP gradient benefits from the structural characteris- ties of the objective, i... how the overall objective is affected by the operations applied to the sources of randomness as they pass through the measure and into the cost function [40]. From the simulation property of continuous distribution, we have the following equivalent sampling processes:
E ~ p(z;)) 0) + 2 g O), é~ p(e),
(2.3)
which states that an alternative way to generate sample T from the distribution p(a; 0) is sample from a simpler base distribution p(e) and transform this sample using a deterministic function g(e, 0). Derived from the law of the unconscious statistician (LOTUS) [21], i... Ep(r:0) [u(x)]
Eptely(9te ())], the RP gradient can be formulated VoEpp(,,0) [=x
Epte) [V Voyvgg( [)]]
3 Analytic Reparameterization Gradient in Reinforcement Learning
In this section, we present two fundamental analytic forms of the RP gradient in RL. We first consider the volicy-Value Gradient (PVG) method, which is model-free and can be expanded sequentially to obtain the Analytic Policy Gradient (APG) method. Then we discuss potential obstacles that may arise when developing practical algorithms.
We consider a policy (s(s,<) ())s with noise ç in continuous action spaces. To ensure that the first-order gradient through the value function is well-defined, we make the following continuity assumption.
Assumption 3.1 (Continuous MDP). We assume that f(s') s, a). (s s)', r(s, and (a)(s,a) are continuous in all parameters and variables s'.
Policy-Value Gradient. The reparameterizatioo PVG takes the following general form,
V Bertsrp
V V " (.(.)s)))) {
(3.1)
In sequential decision- making, any immediate action could lead to changes in all future states and rewards. Therefore, the value gradient V possesses a recursive structure. Adapted from the deterministic policy gradient theorem [52, 34] by considering stochasticity, we rewrite (3.1) as
V ( E. VoTo(S,s) V.Q (s, a) (==
Here, VaQue can be estimated using a critic, which leads to model-free frameworks [25, 4]. Notably, as a result of the recursive structure of Q=Q"0, the expectation is taken over the state visitation Vy instead of the initial distribution Ç.
By sequentially expanding PVG, obtain the analytic representation of the policy gradient.
Analytic Policy Gradient. From the Bellman equation (s) = E, - (o(ss)
+
yE.- [VT (f(s, (), E")]], obtain the following backward recursions:
V T (s) Eg
[1 - 7) TT + Bs. [V.. T (s' a TT VOV
(3.2)
V. (5) = E. (1 7)
m + T.g. [V T (5)
V /. m (3.3)
See $A for detailed derivations of (3.2) and (3.3). Now we have the RP gradient backpropagated through the transition path starting at s. By taking expectation over the initial state distribution, we obtain the Analytic Policy Gradient (APG) V (Ta) = E.~c (VoV mm (s)].
There remain challenges when developing practical algorithms: (1) the above formulas require the gradient information of the transition function f. In this work, however, we consider a common RL setting where f is unknown and needs to be fitted by a model. It is thus natural to ask how the properties of the model (e.g., prediction accuracy and model smoothness) affect the gradient estimation and the convergence of the resulting algorithms, and (2) even if we have access to an accurate model, unrolling over full sequences faces practical difficulties. The memory and computational cost scale linearly with the unroll length. Long chains of nonlinear mappings can also lead to exploding or vanishing gradients and even worse, chaotic phenomenons [9] and difficulty in optimization [43, 36, 58, 38J. These difficulties demand some form of truncation when performing RP PGMs.
Due to the simulation property of continuous distributions in (2.3), we interchangeably write [T with To s) and s' ~ f[(1 a) with s' f(s, a, E"), where e" is sampled from unknown distribution.
3

---

4 Model-Based RP Policy Gradient Methods
Through the application of Model Value Expansion (MVE) for model truncation, this section unveils two RP policy gradient frameworks constructed upon MVE.
4.1 h-Step Model Value Expansion
To handle the difficulties inherent in full unrolls, many algorithms employ direct truncation, where the long sequence is broken down into short sub-sequences and backpropagation is applied accordingly, e.g., Truncated BPTT [63]. However, such approach over-prioritizes short-term dependencies, which leads to biased gradient estimates.
In model-based RL (MBRL), one viable solution is to adopt the ht-tep Model Value Expansion [16], which decomposes the value estimation (s) into the rewards gleaned from the learned model and a residual estimated by critic function Qu that is,
h-1 Σ
x r + a)) (..
TT) = (1-7)-
where 8o a. (T 5), and 3141 (3, E). Here, the noise variables and & can be sampled from the fixed distributions or inferred from the real samples, which we now discuss.
4.2 Model-Based RP Gradient Estimation
Utilizing the pathwise gradient with respect to 0, we present the following two frameworks.
Model Derivatives Predictions (DP). A straightforward way compute the first-order gradient to link the reward, model, policy, and critic together and backpropagate through them. Specifically, the differentiation is carried out on the trajectories simulated by the model fo. which serves as tool for both the prediction states and the evaluation of derivatives. The corresponding RP-DP estimator of gradient (=)( is denoted as 6 (TTO) which takes the form of
N
PDP J(TO) = N
r( Ce,)) + + (uh (Bh.n. @h..):
(4.1)
1=
where 80.n ~ Mm āzn T Ss.n Sn), and 81+1, (F a En) with noises Sn ~ p(s) and &n ~ p(E). Here, pro is the distribution where the initial states of the simulated trajectories are sampled. In Section 5, we study a general form of Mma that a mixture the initial state distribution Ç and the state visitation Vao.
be instantiated from (4.1) with different choices of h. When h 0, the framework reduces to model-free policy gradients, such as RP(O) [4] and the variants of DDPG [34], e.g., SAC [23]. When h co, the resulting algorithm is BPTT [22, 13, 6] where only the model is learned. Recent model-based approaches, such as MAAC [12] and related algorithms [42, 4, 33], require a carefully selected h.
Various algorithms
Model Derivatives on Real Samples (DR), An alternative approach is to use the learned differentiable model solely for the calculation of derivatives, with the aid of Monte-Carlo estimates obtained from real samples. By replacing V, V.f in (3.2)-(3.3) with V.fw. w Vafe and setting the termination of backpropagation at the h-th step as Vy (Sn..) = v (Bn,,), we are able to derive dynamic representation of Va V=0, which we defer to $A. The corresponding RP-DR gradient estimator is
1 N Σ (Ban).
DDR
(4.2)
where 8p,m ~ PT.- Equation (4.2) can be specified as (A.9), which is in the same format as (4. 1), but with the noise variables Sn. En inferred from the real data sample (si, a, 8141) via the relation
a; = ( Sa) and 8:+1 Fø (8i, (ss an,Sa) (see ŠA for details). Algorithms such SVG [25] and its variants [1. 5] are examples of this RP-DR method.

---

4.3 Algorithmic Framework
The pseudocode of model- RP PGMs is presented Algorithm where three update procedures are performed iteratively. In other words, the policy, model, and critic are updated at each iteration generating sequences of (TT, he|r+1 (fs. hejry. and hewre respectively.
t e [T],
Algorithm 1 Model-Based Reparameterization Policy Gradient
Input: Number iterations T, learning rate v), batch size N, empty dataset
1: for iteration & (7) do
2: Update the model parameter the by MSE or MLE
3: Update the critic parameter by performing Temporal Difference
4: Sample states from Pr, and estimate VoJ(
(DPJ(Ta)) (4.1) (T..) (4.2)
5: Update the policy parameter Og by 0c+1 + 02 + 17 (TT(
6: Execute T0.41 and save data to D to obtain D++1
7: end for
Policy Update. The update rule for the policy parameter 0 E θ with learning rate rj is as follows,
Oct1 + 02 + v (oJ(T)))
(4.3)
where (Te.) can be specified ( VD (TT,).
Model Update. By predicting the mean of transition with minimized mean squared error (MSE) or fitting a probabilistic model with maximum likelihood estimation (MLE), e.g., argmax.ev ED, [log fo (s1+1/54, a:)]. canonical MBRL methods learn forward models that predict how the system evolves when an action is taken at state.
However, accurate state predictions do not imply accurate RP gradient estimation. Thus, define (t) to denote the model (gradient) error at iteration
Øs;
88
0s,
0s
(t) = max (P(s..)))(E..)) SE[b]
+
(4.4)
0s
05
da.-1
06;-1
1
where P(si, as) is the true state-action distribution at the i-th timestep by following 80 ~ VTO, aj ~ no, (-1,,), 8141 ~ [(-1 85,a;), with policy and transition noise sampled from a fixed distribution. f((1
Similarly, P(St, (.) is the model rollout distribution at the i-th timestep by following So V/T,'" aj no, (1)) 85+1 ~ Fex (-13,, az). where the noise is sampled when we use RP-DP gradient estimator and inferred from real samples when we use RP-DR gradient estimator (in this case P(S,, a,) = P(ss a.)).
In MBRL, common learn state- predicrreee model that can make multi-step predictions. However, this presents a challenge in reconciling the discrepancy between minimizing state prediction error and the gradient error of the model. Although it is natural consider regularizing the models' directional derivatives to be consistent with the samples [33]. we contend that the use of state- predictive models does not cripple our analysis gradient bias based on Ey: For learned models that extrapolate beyond the visited regions, the gradient error can still be bounded via finite difference. In other words, cf can be expressed as the mean squared training error with an additional measure of the model class complexity to capture its generalizability. This same argument can also be applied to the case of learning a critic through temporal difference.
Critic Update. For any policy π, its value function Q" satisfies the Bellman equation, which has unique solution. In other words, Q = if and only Q = Q". The Bellman operator T is defined for any (s, a) E S x A as
(*)(s,a) E..f [11 7) r(s,a) + Q(x,a')).
We aim to approximate the state-action value function Q" with a critic Q... Due the solution uniqueness of the Bellman equation, it can be achieved by minimizing the mean-squared Bellman error W, argmin Ep, (s..(s.)) Qm a)) via Temporal Difference (TD) [55, 10). We define the critic error at the t-th iteration as follows,
aQ**
aQu.
aQ=M
aQ
(t) = =a² -P(Cn,
+
(4.5)
0s
05
Oa
Fa
5

---

where (1 ///// and P(s): ob), P( @n)) are distributions at timestep h with the same definition as in (4.4). The inclusion of a² ensures that the critic enror remains in alignment with the single-step model error ef: (1) the critic estimates the tail terms that occur after h steps in the model expansion, therefore the step-average critic error should be inversely proportional to the tail discount summation Pi=A Y = 1/a, and (2) the quadratic form shares similarities with the canonical MBRL analysis the cumulative error of the model trajectories scales linearly with the single-step prediction emror and quadratically with the considered horizon (i.e., tail after the h-thh step). This is because the cumulative error is linear in the considered horizon and the maximum state discrepancy, which is linear in the single-step error and, again, the horizon [28].
5 Main Results
In what follows, we present our main theoretical results, whose detailed proofs are deferred to $B. Specifically, we analyze the convergence of model-based RP PGMs and, more importantly, study the correlation between the convergence rate, gradient bias, variance, smoothness of the model, and approximation error. Based on our theory, we propose various algorithmic designs for MB RP PGMs.
To begin with, we impose a common regularity condition on the policy functions following previous works [68, 46, 69, 3]. The assumption below essentially ensures the smoothness of the objective J (To), which is required by most existing analyses of policy gradient methods [60, 6, 2].
Assumption 5.1 (Lipschitz and Bounded Score Function). We assume that the score function of policy Te Lipschitz continuous and has bounded norm V(ss, a) E ES x A, that is,
log Te, (al s) - log Te, (a s) S L1 1|01 - 82l12, log (a 8) S Bg.
We characterize the convergence of RP PGMs by first providing the following proposition. Proposition 5.2 (Convergence to Stationary Point). We define the gradient bias b, and variance 1/ as vs = [ - E [ )
by = (TTe) - [ (Te,
Suppose the absolute value of the reward r(s, a) is bounded by [r(s, a)l S rm for (s. a) S A. Let 8 = sup |0]||. = rm Ln/(1 - 7² + (1+
.m rm B3/( - 7)3 and e = (n Lnd)) then
holds for T 2 4L2 that
T-1 Σ c(26 b, + 7 2 v.) + 62 + Uz
4c
min (E[T]
[To.)))
I B[J(TTT) (TT, + T T
Proposition 5.2 illustrates the interdependence between the convergence and the variance, bias of the gradient estimators. In order for model-based RP PGMs to converge, it is imperative to maintain both the variance and bias sublinear growth rates. Prior to examining the upper bound of b, and ut, make the following Lipschitz assumption, which has been implemented in a plethora of preceding studies [46, 12, 33].
Assumption 5.3 (Lipschitz Continuity). We assume that r(s, a) and f(s, a,,",, E*) are L, and Ls Lipschitz continuous, respectively. Formally, for any $1, 82 E S, a1, 02 E A, and
|r(s), 01) r(82, az) s L, [[(s] $2, (011-a2)1 [2)]|2= (1,1),,1,")) f(82, a2, 50)l lll S Ly [|(s) 82, 01 - a2 1|2
Let T. max(Lg, Lg, 1}, where Lg is the Lipschitz of function g. We have the following result for gradient variance.
Proposition 5.4 (Gradient Variance). Under Assumption 5.3, for any E [T]. the gradient variance of the estimator (T)()))) which can be specified as PDP J(TO) or EDR J (TT)), can be bounded by
2 = o h2
2
2
ak T /N + 2h i
I /N)
where Li- SuDwev..,.agSS....ne [[) (a5,,,,,, a1, +) - (32, 02, 1) |[(1 82, aj - a2))l2 and
Ln supace, (T). #e(32, <) / ||s1 - 321l2.

---

We observe that the variance upper bound exhibits polynomial dependence on the Lipschitz continuity of the model and policy, where the degrees are linear in the model unroll length. This makes sense intuitively, as the transition can be highly chaotic when I, > and L > 1. This can result in diverging trajectories and variable gradient directions during training, leading to significant variance in the gradients.
Remark 5.5. Model-based RP PGMs with non-smooth models and policies can suffer from large variance and highly non-smooth loss landscapes, which can lead to slow convergence failure during training even in simple toy examples [42, 37, 53). Proposition 5.4 suggests that one can add smoothness regularization to avoid exploding gradient variance. See our discussion at the end of this section for more details.
Model-based RP PGMs possess unique advantages by utilizing proxy models for variance reduction. By enforcing the smoothness of the model, the gradient variance is reduced without a burden when the underlying transition is smooth. However, in cases of non-smooth dynamics, doing so may introduce additional bias due increased model estimation error. This necessitates trade-off between the model error and gradient variance. Nevertheless, our empirical study demonstrates that smoothness regularization improves performance in robotic locomotion tasks, despite the cost of increased bias.
Next, we study the gradient bias. We consider the case where the state distribution fx,, which is used for estimating the RP gradient, is mixture of the initial distribution Ç of the MDP and the state visitation V= other words, we consider μ/= = B V/ + (1 B8 C. where 3 E [o, 1.. This form is of particular interest as it encompasses various state sampling schemes that can be employed. such as when and h - oc: When not utilizing a model, such as in SVG(O) [25, 4] and DDPG [34],
states are sampled from Ma; while when unrolling the model over full sequences, in BPTT, states are sampled from the initial distribution.
Given that the effects of policy actions extend to all future states and rewards, unless we know the exact policy value function, its gradient V cannot be simply represented by quantities in any finite timescale. Hence, the differentiation of the critic function does not align with the true value gradient that has recursive structures. tackle this issue, we provide the gradient bias bound that based on the measure of discrepancy between the initial distribution and the state visitation Vz-
Proposition 5.6 (Gradient Bias). We denote ε = sup, [(dC/ du, (s)) where dc/dv /dvy is the Radon-Nikodym derivative of Ç with respect to Vr. Let n²' = B + for any & [T), the gradient bias is bounded by
B). Under Assumption 5.3,
b, = (1 (1 (//(11///1)/ n n' T (011--79 71)))
where f and cn.e are the shorthand notations of (() defined in (4.4) and en (1) (4.5), respectively.
The analysis above yields the identification of an optimal model expansion step h that achieves the best convergence rate, whose form is presented by the following proposition.
Proposition 5.7 (Optimal Model Expansion Step). Given L, S 1, if we regularize the model and policy so that L5 S. and L < 1, then when 7 A 1, the optimal model expansion step hi at iteration & that minimizes the convergence rate upper bound satisfies h- = max(h'* 0}, where ft* ((e,,//(1) (E + Ev,)))) scales linearly with c,,t/(Ef, + Ev,t) and the effective task horizon 1/(1 - 7)
In Proposition 5.7, the Lipschitz condition of the underlying dynamics, i.e., L, S 1, ensures the stability of the system. This can be seen in the linear system example, where the transitions are determined by the eigenspectrum of the family of transformations, leading to exponential divergence of trajectories w.r.t. the largest eigenvalue. In cases where this condition is not met in practical control systems, finding the best model unroll length may require trial and error. Fortunately, we have observed through experimentation that enforcing smoothness offers a much wider range of unrolling lengths that still provide satisfactory results.
Remark 5.8. As the error scale (v,t/(ff,t 6v,z) increases, so too does the value of It" This finding can inform the practical algorithms to rely more on the model by performing longer unrolls when the model error e is small, while avoiding long unrolls when the critic error Ev,t is small.
7

---

A Spectral Normalization Method. To ensure a smooth transition and faster convergence, we propose using a Spectral Normalization (SN) [39] model-based RP PGM that applies SN to all layers of the deep model network and policy network. While other techniques, such as ad- versarial regularization [50], exist, we focus primarily on SN as it directly regulates the Lips- chitz constant of the function. Specifically, the Lipschitz constant I.g of a function g satisfies = sup, 0 max [z)x)), where max (W) denotes the largest singular value of the matrix W, defined as F (W) = max | Sa ||W For neural network f with linear layers g(z) = Wa and 1- Lipschitz activation (e.g., ReLU and leaky ReLU), we have Lg = max (W;) and L S II, T mmx (W;).
By normalizing the spectral norm of W, with W = o (W.), SN guarantees that the Lipschitz of f is upper-bounded by 1.
Finally, we characterize the algorithm convergence rate.
Corollary 5.9 (Convergence Rate). Let <(T) = Et= be. We have for T 2 4L that
min E VoJ ||]]] & 160-- -o))/e(TV/V/ +T E
The convergence rate can be further clarified by determining how quickly the errors of model and critic approach zero, i.e.. Σ/= (t) Such results can be accomplished by conducting a more fine-graince investigation of the model and critic function classes, such utilizing overparameterized neural nets with width scaling with 7 to bound the training error, as done in [10, 35), and incorporating complexity measures of the model and critic function classes to bound e (t) and (t). Their forms, however, are beyond the scope of this paper.
Related Work
Policy Gradient Methods. Within the RL field, the LR estimator is the basis of most policy gradient
algorithms, e.g., REINFORCE [64] and actor-critic methods [56, 31, 30, 14]. Recent works [3, 60, 7. 35] have shown the global convergence of LR policy gradient under certain conditions, while less attention has been focused on RP PGMs. Remarkably, the analysis in [33] is based on the strong assumptions on the chained gradient and ignores the impact of value approximation, which oversimplifies the problem by reducing the h-step model value expansion to single-step model unrolls. Besides, [12] only focused on the gradient bias while still neglecting the necessary visitation analysis. Despite the utilization in our method of Spectral Normalization on the learned model to control the gradient variance, SN has also been applied in deep RL to value functions in order to enable deeper neural nets [8] or regulate the value-aware model error [72].
Differentiable Simulation. This paper delves into the model- setting [11, 28, 29, 70, 24], where a learned model is employed to train a control policy. Recent approaches [41. 53, 54, 67] based on differentiable simulators [18, 27] assume that gradients of simulation outcomes w.r.t. actions are explicitly given. To deal with the discontinuities and empirical bias phenomenon in the differentiable simulation caused by contact dynamics, previous works proposed smoothing the gradient adaptively with contact-aware central-path parameter [71], using penalty-based contact formulations [20, 66] or adopting randomized smoothing for hard-contact dynamics [53, 54]. However, these are not in direct comparison to our analysis, which relies on model function approximators.
7
Experiments
7.1 Evaluation of Reparameterization Policy Gradient Methods
To gain deeper understanding and support the theoretical findings, evaluate several algorithms originating from our RP PGM framework and compare them with various baselines in Figure I. Specifically, RP-DP-SN is the proposed SN-based algorithm: RP-DP, as described in (4.1), is imple- mented as MAAC [12] with entropy regularization [4]; RP-DR, as described in (4.2), is implemented as SVG [25]; the moodel-free RP(() is described in $4.2. Details and discussions are deferred to $C.
The results indicate that RP-DP consistently outperforms or matches the performance of existing methods such as MBPO [28] and LR PGMs, including REINFORCE [56], NPG [31], ACKTR [65], and PPO [49]. This highlights the significance and potential of model-based RP PGMs. Due to space limitations, we refer the readers to $C.5 for larger versions of the figures in the experiment section.
8

---

RR-DP-SN
ACKTR
DDPG
RP-DF
RP(O)
REINFORCE
A2C
PFO
SAC
Convergence
Figure 1: Comparisons between RP PGMs (the green labels) and MF/MB baselines (the black labels) in the MuloCo [57] tasks.
7.2 Gradient Variance and Loss Landscape
Our prior investigations have revealed that vanilla MB RP PGMs tend to have highly non-smooth landscapes due to the significant increase in gradient variance. We now conduct experiments to validate this phe- nomenon in practice. In Figure 2, we plot the mean gradient variance of the vanilla RP-DP algorithm dur- ing training. To visualize the loss landscapes, we plot in Figure 3 the negative value estimate along two directions that are randomly selected in the policy parameter space of a training policy.
wolkerdd
BP-
Unvel Langth
Model Lange
Figure 2: Gradient variance of the vanilla RP- DP explodes while adding spectral normaliza- tion solves this issue.
We can observe that for vanilla RP policy gradient algorithms, the gradient variance explodes in expo- nential rate with respect to the model unroll length. This results in a loss landscape that is highly non- smooth for larger unrolling steps. This renders the im- portance of smoothness regularization. Specifically, incorporating Spectral Normalization (SN) [39] in the model and policy neural nets leads to a marked reduction in mean gradient variance for all unroll length settings, resulting in a much smoother loss surface compared to the vanilla implementation.
(a) RP-DP h 3. h 15.
(b) RP-DP
(c) RP-DP-SN h 15.
Figure 3: The loss surface of hopper.
7.3 Benefit of Smoothness Regularization
In this section, we investigate the effect of smoothness regularization to support our claim: The gradient variance has polynomial dependence the Lipschitz continuity of the model and policy, which is a contributing factor to training. Our results in Figure 4 show that SN-based RP PGMs achieve equivalent or superior performance compared to the vanilla implementation. Importantly, for longer model unrolls (e.g., 10 in walker2d and 15 in hopper), vanilla RP PGMs fail to produce reliable performance. SN-based methods, on the other hand, significantly boost training.
Bork
Figure 4: Performance of vanilla and SN-based MB RP PGMs with varying h. The vanilla method only works with a small and fails when increases, while the SN-based method enables a larger
Additionally, we explore different choices of model unroll lengths and examine the impact of spectral normalization, with results shown in Figure 5. We find that by utilizing SN, the curse of chaos can be mitigated, allowing for longer model unrolls. This is crucial for practical algorithmic designs: The most popular model-based based RP PGMs such as [12, 4] often rely on a carefully chosen (small) h (e.g.. h = 3). When the model is good enough, a small may not fully leverage the accurate gradient information. As evidence, approaches [67, 41] based on differen- tiable simulators typically adopt longer unrolls compared to model-based approaches. Therefore, with SN, more accurate multi-step predictions should enable more efficient learning without making the
mulaar2
hoppor
Figure 5: Performance with different h.

---

underlying optimization process harder. SN-based approaches also provide more robustness since the return is insensitive to h and the variance of return is smaller compared the vanilla implementation when h is large.
Ablation on Variance. By plotting the gradient variance of RP-DP during training in Figure 6, can discern that for walker h = 10 and hopper / = 15, a key contributor to the failure of vanilla RP-DP is the exploding gradient variances. On the contrary, the SN-based approach excels training performance as a result of the drastically reduced variance.
Figure 6: Gradient variance of RP PGMs. The variance is significantly lower with SN when is large.
Ablation on Bias. When the underlying MDP is itself contact-rich and has non-smooth or even discontinuous dynamics, explicitly regularizing the Lipschitz of the transition model may lead to large error y and thus large gradient bias. Therefore, is also important to study if SN causes such negative effect and if it does, how to trade off between the model bias and gradient variance. To efficiently obtain an accurate first-order gradient (instead of via finite difference in MuJoCo), we conduct ablation based on the differentiable simulator dFlex [26, 67], where Analytic Policy Gradient (APG) described in Section 3 can be implemented.
Figure 7: Performance and gradient bias in differentiable simulation. The third and last columns are the full training curves of APG, which need 20 times more steps than RP-DP-SN to reach a comparable return in the hopper task and fail in the half-cheetah task, respectively.
Figure 7 illustrates the crucial role SN plays in locomotion tasks. It worth noting that the higher bias of the SN method does not impede performance, but rather improves it, indicating that the primary obstacle in training RP PGMs is the large variance in gradients. Therefore, even if the simulation differentiable, learning a smooth proxy model can be beneficial when the dynamics have bumps or discontinuous jumps, which is usually the case in robotics systems, sharing similarities with the gradient smoothing techniques [53, 54, 71] for APG.
8 Conclusion & Future Work
In this work, we study the convergence of model-based reparameterization policy gradient methods and identify the determining factors that affect the quality of gradient estimation. Based on our theory, we propose spectral normalization (SN) method to mitigate the exploding gradient variance issue. Our experimental results also support the proposed theory and method. Since SN-based RP PGMs allow longer model unrolls without introducing additional optimization hardness, learning more accurate multi-step models to fully leverage their gradient information should be a fruitful future direction. It will also be interesting explore different smoothness regularization designs and apply them to a broader range of algorithms, such as using proxy models in differentiable simulation to obtain smooth policy gradients, which would like to leave future work.
Acknowledgements
The authors would like to thank Yan Li for his valuable insights and discussions during the early stages of this paper. Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports.
10

---

References
[1] Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference Machine learning, pages 1-8, 2006.
[2] Alckh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approxi- mation with policy gradient methods in markov decision processes. In Conference Learning Theory, pages 64-66. PMLR, 2020.
[3] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1 2021.
[4] Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, pages 6-20. PMLR, 2021.
[5] Christopher G Atkeson. Efficient robust policy optimization. In 2012 American Control Conference (ACC), pages 5220-5221 IEEE, 2012.
[6] Osbert Bastani. Sample complexity of estimating the policy gradient for nearly deterministic dynamical systems. In International Conference on Artificial Intelligence and Statistics, pages 1858-3869 PMLR, 2020.
[7] Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.0) 0006/99060778 2019.
[8] Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning. arXiv preprint 1xXiv:20660015// 2021.
[9] Erik M Bollt. Controlling chaos and the inverse frobenius-perron problem: global stabilization of arbitrary invariant measures. International Journal of Bifurcation and Chaos, 10(05): 1050, 2000.
[10] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural lemporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
[11] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[12] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model- oodel-augmented actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005. 08068, 2020.
[13] Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, page 6, 2019.
[14] Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic, arXiv preprint arXiv:1205.48.0 4839, 2012.
[15] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning. pages 1329-1338. PMLR, 2016.
[16] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph Gonzalez, and Sergey Levine. Masel-based value expansion for efficient model-free reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 2018.
[17] Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterizzatoon gradients. Advances in neural information processing systems, 31, 2018.
[18] C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physies engine for large scale rigid body simulation. arXiv preprint 2XX.:2106.1328/ 2.21.
11

---

[19] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[20] Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact, ACM Transactions on Graphics (TOG), 39(6):115,, 2020.
[21] Geoffrey Grimmett and David Stirzaker. Probability and random processes. Oxford university press, 2020.
[22] Radek Grzeszezuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural network emulation and control of physics-based models. In Proceedings of the 25th annual conference Computer graphics and interactive techniques, pages 9-20, 1998.
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[24] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 04000004,.4 2023.
[25] Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Advances in neural informa- fion processing systems, 28, 2015.
[26] Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. Disect: A differentiable simulation engine for autonomous robotic cutting. arXiv preprint arXiv:2105. 12244, 2021.
[27] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9474-9481. IEEE, 2021.
[28] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
[29] Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint 1rXiv:1903000374 2019.
[30] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
[31] Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
[32] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[33] Chongehong Li, Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma. and Tie- Yan Liu. Gradient information matters in policy optimization by back propagatiin through model. In International Conference Learning Representations, 2021.
[34] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 2509.02971, 2015.
[35] Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy, Advances in neural information processing systems, 32, 2019.
12

---

[36] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyperparameter opti- mization through reversible learning. In International conference on machine learning, pages 2113-2122. PMLR, 2015.
[37] Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.00 0050003, 2021.
[38] Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pages 4556-4565, PMLR, 2019.
[39] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: D..Xiv:8802.05955 2018.
[40] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(132):1-66 62, 2020.
[41] Miguel Angel Zamora Mora, Momchil P Peychev, Schoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning, pages 7805-7817. PMLR, 2021.
[42] Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4065-4074 PMLR, 2018.
Flexible model-
[43] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. International conference on machine leurning, pages 1310-1318. PMLR, 2013.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[45] Luis Pineda, Brandon Amos, Amy Zhang, Nathan O Lambert, and Roberto Calandra. Mbrl-lib: A modular library for model-based reinforcement learning. arXiv preprint 1rXiv:2104.10159 2021.
[46] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning, 100(2):255 2015.
[47] Emmanuel Rachelson and Michail G Lagoudakis. On the locality of action domination in sequential decision making. 2010.
[48] Francisco R Ruiz, Titsias RC AUEB, David Blei, et al. The generalized reparameterization gradient. Advances in neural information processing systems, 29, 2016.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint arXiv:
[50] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718 3718. PMLR, 2020.
[51] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.008855 2017.
[52] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pages 387-395. PMLR, 2014.
[53] HJ Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake, Do differentiable simulators give better policy gradients? arXiv preprint arXiv:2202.00817. 2022.
13

---

[54] Hyung Ju Terry Suh, Tao Pang, and Russ Tedrake. Bundled gradients through contact via randomized smoothing. IEEE Robotics and Automation Letters, 7(2):4000-4007, 2022.
[55] Richard S Sutton, Learning to predict by the methods of temporal differences. Machine learning. 3(1):9444, 1988.
[56] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[57] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In 2012 TEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[58] Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pages 10553-10563. PMLR, 2021.
[59] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii DeepMind blog, 2, 2019.
[60] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint 19XXvv1909.000 150.00111000000 2019.
[61] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprini arXiv: 1907:::907.00000 .2w.000077...0 2019.
[62] J Weng et al. A highly modularized deep reinforcement learning library. arXiv preprint arXiv:2107.14177 2021.
[63] Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 18(10)1550-15500 1990.
[64] Ronald J Williams. Simple statistical followng algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-255 1992.
[65] Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using fronecker-factored approximation. Advances in neural information processing systems, 30, 2017.
[66] Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojeiech Maussik, Shinjiro Sueda, and Pulkit Agrawal. An end-to-end differentiable framework for contact-aware robot design. arXiv preprint 21X.v:2107.07501, 2021.
[67] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint 27X40:2204..7713, 2022.
[68] Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. arXiv preprint arXiv:1909.0886 2019.
[69] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6):356-3366 06166,6612220 2020.
[70] Shenao Zhang. Conservative dual policy optimization for efficient model-based reinforcement learning. Advances in Neural Information Processing Systems, 35:25450-25663, 2022.
[71] Shenao Zhang, Wanxin Jin, and Zhaoran Wang. Adaptive barrier smoothing for first-order policy gradient with contact dynamics. In Proceedings of the 40th International Conference on Machine Learning, pages 41243. PMLR, 2023.
[72] Ruijie Zheng, Xiyao Wang, Huazhe Xu, and Furong Huang. is model ensemble necessary? model-based via a single model with lipschitz regularized value function. In Deep Reinforce- ment Learning Workshop NeurlPS 2022, 2022.
14

---

A Recursive Expression of Analytic Policy Gradient
In what follows, interchangeably write Va and dx//a as the gradient and denote by 0z/0a the partial derivative.
Derivation of Analytic Policy Gradient. First of all, we provide the derivation of (3.2) and (3.3), ie., the backward recursions of the gradient in APG.
Following [25], we define the operator
da
dsj a d0
(A.1)
daz
asj j>i
J2
We begin by expanding the total derivative operator by chain rule
d
da;
0
ds; 0
Σ
do
d0
0a,
d0
0s;
120
1>0
dag
Ô
d81
0
da;
0
ds; a
(A.2)
d0
Oag
do
081
do
Oa;
do
0s;
Here, the expansion holds when d/d0 operates on policies and models that are differentiable with respect all states 8; and actions dg.
Plugging (A.2) into (A.1), we obtain the following recursive formula for V
da; a da, ds a
Vi+i
(A.3)
d0 Da d0 da,
0sit1
By the Bellman equation, we have v (s) = Eeg (1 - 7) r(s, ())), + r Bg. (FTo
(A.4)
Combining (A.3) and (A.4) gives
VOV = dv (8)
E, (1 7) rr. (o(s ty. Ee. [yTO (F(8,
d0 = d0
Dr da
da
as
av))(s) (
= Eg (1 -) 7)
+
Eg-
+ d0
(A.5)
da
do
do da ds'
which corresponds to (3.2).
For the dV (s)/ds term on the right-hand side of (A.5), we have the following recursion,
(s) = dV (s)
Ex
(1 -1-18 7) r(s, Te(s, 5) + 7
Eg.
( Te(s, s). (f(s,,e(ss (e))
|x (f(s,
ds ds
Gr
+ Ör
da
Os
d
+ Os' da do (s'))
= (1 - 7)
+ 7 Eg
Is
Da
Is
Os ds'
du Os do
(A.6)
which corresponds (3.3).
Therefore, we complete the derivative of (3.2) and (3.3).
Derivation of RP-DR Policy Gradient. By the same arguments in (A.5) and (A.6) with V (or 0s' /0a) and V.f (or 0s' /ds) replaced with Va and V.fg, we obtain
VaV=o (Bi..) (1 7) V (B.,,, a. VV (B Sn)
+ 7 v T. (8.+1, V Q Vo Po (
(A.7)
6 VT = (1 - 7) ( / (8), a..)) (ar Vo
- 1 (T (3)+1,.) (V fe (3 a En) + V Vafe di,n: En) V.
To ($,, 5m)
(A.8)
15

---

where the termination of backpropagation at the h-th step is (Sh.m) (S
Combining (A.7) and (A.8), obtain the RP-DR policy gradient estimator as follows,
1 DR J(To) = N Vov (Fo.n)
(Br(Bi, T.,n) + 7 ( (Sh,m (n,))
(A.9)
where 80,n Ms 0i,n To (3i,n: Sn), and 1,4 ( (iim Ci,n, En). Here, Sn. and En inferred by solving a; = (8; Sn) and 8++1 = (8i, ai, Ea), respectively, where (8i, ai 81+1) is the real data sample. For example, for state 81+1 sampled from one-dimensional Gaussian transition model 8;1 ~ (o(s;, az), o²), where the variance is a and the mean d(s;, a.) is the output of some function parameterized by d, the noise &n can be inferred &n = (si+1 ((8i, 0;))/o.
B
Proofs
B.1 Proof of Proposition 5.2
As preparation before proving Proposition 5.2, we first present the following lemma stating that the objective in (2.1) is Lipschitz smooth under Assumption 5.1.
Lemma B.1 (Smooth Objective, [69]mema 3.2). The objective (TTO is L-smooth in 8, such that [[Vo)(Fo,) VoJ( I L| 02l12, where
Tm L1
(1 + 7) V'
. B
+
7) (1 7)3
Then we are ready to prove Proposition 5.2.
Proof of Proposition 5.2. See the proof of Theorem 4.2 [71].
B.2 Proof of Proposition 5.4
Proof. Since the RP-DP gradient VoJ(TT) P J(Te) in (4.1) and the RP-DR gradient (oo()))) D J(To) (4.2) share the same state transition 8411, f(S... Ea), where re- call that the only difference lies in the source of noise &n, our subsequent analysis holds for both RP-DP and RP-DR.
To upper-bound the gradient variance et = [ norm inside the outer expectation.
we characterize the
We start with the case where the sample size N = 1, which naturally generalizes to N Specifically, consider arbitrary h-step trajectory obtained by unrolling the model under policy .T.. We denote the pathwise gradient (To,) of this trajectory as g'. Then we have
U/ max g/ IE[
| FoJ(Fo)))
where g is the pathwise gradient (TT()))) of a fixed (but unknown) trajectory (50, ao,n $1,m, "i,m. such that the maximum is achieved.
Using the fact that ||B|-]||2 < Elll [|2], we further obtain
v [|g (o.(To)))
(B.1)
Let Zi,n = (Bi,n ). By the triangular inequality, have
h-1
[llg (od(
r² Ez, || Vor(fe..) (or
<
1=0
nh Ez, (Q(Th.n) -Ql
(B.2)
16

---

By the chain rule, we have for any i 2 that
dā
Fūg,
d8gn
Jüs.n
+
(B.3)
d0
0iin
d0
00
dzin
0sin
1
Fsin
dā3-1,n
(B.4)
d6
-8
d0
dū;-1.n
d0
Denote by Lg suppeo, (8) in (B.3) into (B.4), we get
Plugging
= 1
( 18
-0
d3
F0;-1,m
d0
dā.-1,
18:
d0
Ja,-1,n
00
d3 d 1.1
S Liglt
+ T.gTt:
(B.5)
d0
where the inequality follows from Assumption 5.3 and the Recursively applying (B.5), obtain for any i that
inequality.
d LiLo
1-1 LugT si.
<
Lo
(B.6)
dØ
2
1=0
where the first inequality follows from the induction
2 = a + a (az-2 + + a'- -2-20 +
Σ a
(B.7)
In (B.7), {2ylosis is the real sequence satisfying 2j = azj-1 + b. For d /o defined in (B.3), we further have
dā
d
S
Lx
+ Lo S Li 1t1 Li+1 1 + Lg.
(B.8)
d0
d0
2
Combining (B.6) and (B.8), we obtain
d
d8i.
dā
+
< 2i
Lo LoL ++1 1 + Lo
(B.9)
d0
d0
d0
R(i)
Therefore, we bound the second term on the right-hand side of (B.2) as follows,
Ez. (iQmn. Voin. (Q(I)) Vorall
< Er, [ VoTh. VQ(T.))
VoTh.
+ ET,
(Q(T))
(Q(TA) VoTA [|]]
d8in
ds
dā
dē.
< 52L R(1) + L0
Es.
Ex
(B.10)
do
do
d0
db
where the last incquality follows from the Cauchy-Schward inequality and Assumption 5.3, and (=1, - Qu (82, //)l
La =
82, a1 - a2)
By the chain rule, we bound the first term on the right-hand side of (B.2) as follows,
Ez.
|oro(a Vor
Vr Vr V VoSte - Vr(o,) Vori
||2]
<
Ez.
Vr VoLin - Vr(āin V g.. [1g
+ IE
Vr(a. Vo. Vr(z,) VoT.
< Ly
Ez,
d3 dš
d3,
+ Ex.
dā,,
da,
1) 1
+ 2L, -
(B.1l)
d0
d0
d0
d0
17

---

Plugging (B.10) and (B.11) into (B.2) and (B.1), we obtain
2
( 4
h-1
+ 7tth
L0)
Ezz,
dāh,
dsh
+
Eg
d
d
2 )
+
2R())
de
d#
2 d0
d0
i=0
2hh a
= (1
aa
+ + h² Lah
(B.12)
where the inequality follows from Lemma B.2 and by plugging the definition of Ri) in (B.9).
Note that the variance U; scales with the batch size the rate L/N. Since the analysis above is established for N = 1, the bound of the gradient variance is established by dividing the right-hand side of (B.12) by N, which concludes the proof of Proposition 5.4.
Lemma B.2. Denote sup Esg (ll - d2)/ which is constant that only depends on the initial state distribution For any timestep 2 1 and the corresponding state, action, we have the following results,
dōi,n
d8;
S T. (e (e+4i 4i T.yTor R(i - 1) 2i 28. T.gT.0).
Ex
d6
d0
Ez.
d
da;
$ &TyLi
iitl
(e 4i LgIt R(i - 1) 2i
2i Ł,L + +L, + 2Lg.
d0
d0 2
Proof. Firstly, from (B.4), we obtain for any 2 that
d
ds;
E7.
de
d0
d8
88
dā;-1.a 1.a
85:
d3;-1
0s,
dØ,-1
=
1
Ts
de
Jā,-1.m
d0
03
d0
00;-1
According to the triangle inequality, we further have
E
18
d8;-1,n
03,
d8
1,
+ IE
05
d3 1-1-
03,
d3;-1
1,m
d0
05,-1
d0
03;-1
d0
08;-1
d0
03
1 1,n
03:
dā 1,m
05
dā,
1,m
05:
dā,
+ E
+ E
Fa,
da--1
Da,
d0
Da.
d0
2.
d8;-1. 1,1
dā
d3
d8;
$ 2
+
+ Lg
Ex,.1
d0
d0
d0
d0
2
2
dā,
dØ;-1
+ L ET,
(B.13)
d0
d0
Similarly, have from (B.3) that
dā
da,
Em,
do
d0
Dā.,
d
0ō,,n
00,
d3:
E
d0
00
0s:
d8
00
d
0a,
d8i,
0a,
d8i,n
00: d3;
oā.n
oa,
<
E
+ E
+ E
08,,
d0
05:
d0
03:
d0
03,
d0
00
00
2
ds
d3
ds;
<
2Lx
IE
+
L
E
+ 2
(B.14)
d0
d0
d0
We define account for the stochasticity of the initial state distribution. = 0 when the initial state is deterministic.
18

---

Plugging (B.14) back to (B.13), we obtain
d8i.
ds;
Ex,
do
d0
(1
(1
dā 1,n
d
d3;-1
x AligTer
+
+
T.gTix
1xx,-
+ 21 2T L.
d0
d0
2
d8
d0
$ AL.II K(i - 1) + L L
B
d8;
11m
d3,-1
+
22LLLo.
d0
d#
where the last inequality follows from the definition of K in (B.9). Applying this recursion gives
da de
d3; d0
i-1
Ex
= (((gIx)) (AI R(i - 1) +L1.Lo))
(I
2
1=0
+ 4i Typtax R(i-1) - + 2i T.pTT))))
where the first equality follows from (B.7) B. As consequence, have from (B.14) that
dō
dā, d0
| I
i+ (et 4i LgLn R(i - 1) + 2i LpLl) + 2L, + 2Lg
Es.
do
This concludes the proof.
B.3 Proof of Proposition 5.6
Proof. The analysis of gradient bias differs from that of gradient variance it involves not only the distribution of approximate states but also the recurrent dependencies of the true value on future timesteps, which must be given extra attention.
In the following analysis, we will first apply similar techniques as those outlined in the previous section establish an upper bound the decomposed reward terms in the gradient bias. Afterward, we will address the distribution mismatch issue caused by the recursive structure of V and the non-recursive structure of the value approximation V.,.
Step 1: Bound the cumulative reward terms the gradient bias.
To begin with, we decompose the bias of the reward gradient at timestep i 2 as follows,
dr(a,.n) d0
dr(z;) d0
(PPaa))((s P(G..).
2
dr
d dr
dr;
E
d d0
dx;
d0
dr
d
dr
d
dr
dē
dr
dz;
< E
dāin
d0
dx;
d0
12
dr
d0
dx
d0
2
d3...
ds;
dā
do;
|.)
< 2Tp R(i) Lyp
E
E
(B.15)
d0
d0
d0
d0
where P(s a;) and P,,,, a,) are defined in (4.4) with respect Su Vx. So Vn
We have from (B.3) that for any i 2 1,
dā
da;
do
d0
2
Jā.,
d3,,n
Fo,,
0a,
ds;
da,
E
08.n.
d0
30
0s: d0
00
19

---

By the triangle inequality and the Lipschitz assumption, it then follows that
Jāg.
d
da,
diin
da.
d
Da,
ds;
Jā
da,
E
+
E
+ E
d0
0s:
d0
12
0s;
d0
0s:
d0
00
00
2
d$i,n
d3
ds;
2Lx
E
+
Lx
EB
+ 2
(B.16)
d0
d0
d0
22
Similarly, we have from (B.4) that for any i 2 1,
d
ds;
E
d0
d0
2.
d8;-1.a
Us
dā,- 1.n
0s;
d8;-1
0s:
do;-1
E
d0
-1,
d0
08;-1
d0
da,-1
d0
Applying the triangle incquality to extract the Efa term defined in (4.4), proceed by
E
88
ds,
1m
Øs:
d
1.m
+
E
0s:
d 1m
Üs;
ds;_1
08
1,1
d0
0s
d0
as:-1
d0
ds,-1
d0
08
d 1.m
as:
dē, 1,m
Os,
d
Os:
da;-1
+
E
0a 1,
Da:-
Da;-1
d0
0a;-1
d0
2
S Eff E
d3 1n
dā 1,
+
Ly
d 1.n
ds
1
d0
d0
d0
d0
2
2
d0;-1.n
da:-1
+ T.y
d0
d0
d8;
ds;-1
dā
da;-1
C (i - Ly
E
+ L,
(B.17)
d0
dl
2
d0
d0
where the last inequality follows from the definition of Rii - 1) in (B.9). Combining (B.16) and (B.17), we have
d
ds;
d8
d8
E
S (ffa 2..) R(i + T. T.m E
+ 2T, T.o
de
de
d0
d0
2
=(le,r + b,) R(i 1) +L,
1174.
1=
< ((eJ, + 2L, (ii-1) Lo) L,
(B.18)
where the last inequality follows from (B.7) and the fact that 8p, 80,n are sampled from the same initial distribution, and the equality holds by applying the recursion.
Plugging (B.18) into (B.16), obtain
dā.. de
da.
|,]
E
< [[er + LL Riii - 1) + Lo] . 11 21, 2L,
2Lg-
d0
(B.19)
Step 2: Address the state distribution mismatch issue.
The next step is to address the distribution mismatch issue caused by the recursive structure of the value function and the non-recursive structure of the value approximation, i.e., the critic.
We define σ1 (s, a) P(s s. a. a) where so Vn. 0; ( ss). and si+1 f(: s; a:). In a similar way, we define ã (ss = PP(r, 8, a. a) where So ~ Un. ā ~ o(. Bi), and 3 f(,1,,,,0,) a.).
20

---

Now we are ready to bound the gradient bias. From Lemma B.4, we know that
h-1
h-1
b. S KR' Bespru, V
(81,
Vo
(3
Tin)
0Q
dap
aQ
d8h
+ n''h (e(s C0, Jw
+
(on @n.n)
dah
d0
dsh
do
(B.20)
where recall that k' = B x- B).
From Lemma B.5, have for any policy me that the state-action value function is Lo-Lipschitz continuous, which gives for any 8 E S and a E A that
aQ da
1Q
S LQ
$
Lq.
(B.21)
Os
1
The bias brought by the critic, i.e., the last term on the right-hand side of (B.20), can be further bounded by
aQ
dap
aQ
dsh
(s as aaa 10
+
( Cn,n)
Oap
d0
0sh
db
2
8Q
dan
uQ
dsh
aQ
dō
aQ.
d
E,
dap
d0
Osh
d0
Dān.
d0
d0
2
da 8Q
aQ
d
aQ dā
aQ,
d
<
E
Dap
do
da
de
Bap
d0
Joh.n
d0
0Q
dsh aQ
d
OQ
dēp aQ,
d
+
+
Osh
d0 ash
d0
8sh
d0
Foh,
do
|| de
|dan
dō
dsn
do
1,1)
2
S TQQ
+
R(h) Cp,ts (B.22)
de
de
de
where the equality follows from the chain rule and the fact that the critic Q has non-recursive structure, the last inequality follows from (B.21), (B.9) and the definition of c in (4.5).
Plugging (B. and (B.22) into (B.20), we obtain
be S KK' h - (Lr |
d
dsh
+ EZ i
dō
daa
1) + 2L, R(h)
d0
d0
d0
d0
2
+ xhah ( liQ
d5n,n
dsh,
cdp,
da,
() R(h).
Eg, C.
(T-2)
62,t
d0
df
d8
dd
(B.23)
Plugging (B. (B.19), and (B.9) into the (B.23), we conclude the proof by obtaining
b. o (kn'n2
h22 1- 1h
I3I ef2 - R'hn
(B.24)
1 1-7
Lemma B.3. The expected value gradient over the state distribution P(s)) can be represented by
TaQ da aQ= ds
Ex Vo E (s.a)~o
da
do
Ōs
d0
where P(sx) is the state distribution at timestep h when so C,a M( ss), and s+1
21

---

Proof. At state sn. the value gradient can be rewritten as
Volk r(sh: an) +x Is (8h4118h,,a)) V (8h+1 )dsa+1
VoVTu
VoL[r(sn an)] +7 E Vo
(Sn4ilss, an) V (on+1))dnn+
Brm,
dap
0rk dsa
E
+
Dan
db
ssh d0
/ (S 1ssh,
v (* + (sh+1lsh: an) VoV ( dsn+1
0rk
da
Örk
dsa
+7 Is / (VaJ(
dap
E
V (*(+++1)
Oan
d0
Osh d0
d0
dsh
V.f(sh+1l sh, an)
(T((s+1) + (s an) (VVT())+1)) dsh+1
d0
(B.25)
where the first equation follows from the Bellman equation and the last two equations hold due to the chain rule. Here, it is worth noting that when h 2 1, both дд and sy, have dependencies on all previous timesteps. For any h 2 we have from the chain rule that Ver(s,, an) ar /da, /d./d0 + /rs d /d0. This differs from the case when h = 0, e.g., in the deterministic policy gradient theorem [52], where we can simply write Vor(sh, an) = Ör / 0a,/00.
Rearranging terms in (B.25) gives
r(sh, an) +7 J [Sn41lsh, an) V (sn+ (sh+1)dsn+)) (don+1
da
VnVe = E V.
do
r(sh an) + / f(sht1ls2,as) an) V (sh+1)dssnt)
dsh
V.
d0
f(sn+1 an) Vo VoV (sh+1))))))
TQQ
dan
0Q7e dsh
[ f(nnt1lsh, VoV" VoV + js
E
Dan
dl
Osh
do
(B.26)
where the last equation holds since QT an) r(h, +7 Ss [8 an). (on. ddsa+ By recursively applying (B.26), we obtain
[8 = //g00 h (s141181,aa)
(QQ
da;
aQ**
ds;
d8i+1
(B.27)
da:
d0
0s:
d0
Let 02 a) (1 E ,ー P(si 8, 04 = a), where so ~ S, a m(-18:). and
8i41 {( 8;, 0;). By definition we have
h-1
(s, a) = (1 7))
7 P(s, = =,,;; = =,a;=a) +) ,/ (1
1=0
h-1
(1 -
P(s, =,a; = + ,0 0z (8)
1=0
Therefore we have the equivalence d (1, 02 a).
By taking the expectation over 8y in (B.27), we have the stated result, i.c.,
aQue da + da do
aQ ds
UQQ da
aQ* ds
Exg P(an) VoVT = E(
(e,a)oo
0s d0
da
d0
0s
d0
22

---

Lemma B.4. Recall that the state distribution μa where So.n is sampled from is of the form fl (s)
1/7 + +11 B) ((s). The gradient bias b, at any iteration satisfies
h-1 V
h-1
b, S (33 k (1- E Bu Nive
r(s az) aQ
Vo
Σ i=0
x
r rSain (
1=0
+8+ + k. (1- 8))
aQ
dan
dsh
Voh @nmna
B (i,
San
d0
Ush
do
Proof. To begin, we decompose the gradient bias by
b. VoJ(TO, - E [[[oJ (
[[VoO[(TO)) - (TO
(B.28)
A-1
h-1
Esp 50, ~ ~p
Vo Σ
r(s (oVT - V
2 r n ar,))
1=0
2m0
(on.)))
where we note that so and Sn,, are sampled from Ç and μlr following the definition of the RL objective and the form of gradient estimator, respectively.
For Un (s) = 3 (s) + (1- 3) ((s), let Z be the random variable satisfying PP = 0) = 3 and P(Z 1) - B, i... the event Z = 0 and Z = 1 corresponds to that the state sampled from Vx and Ç. respectively. For any random variable Y, following the law of total expectation, we know that
[Y] = B| 2 = E| = ()F = 0) + E = 1]P(Z = Y Z = 0] + (1 8) 1]
3E., Y|| + (1- [Y]
(B.29)
Therefore, we have from (B.28) that
h-1
be 5 Ego. mpz
Boons
V
7'
((8,))))+ (s) V.
Σ
x
111]
VoVe
h-1 1
h-1
< 3E BE
80,n -M+
Boone
V
xd r(,, 0;) + voV"o(s)) - Vo Σ
r -.i,
1=0
1=0
-gh.
+ +1 - (11-3))2aace
Vo Σ
2 r a:) + +
VoV=o
1=0
7d r( ann)) nh. (
(B.30)
where the first inequality holds since ||||]|| < E(| [|2] and the second inequality holds due to (B.29).
23

---

Using the result from Lemma B.3, we know that
h-1
h-1
Bsawe
V
r(5), a;) + +7" VoVT
V Σ r r(Bim, -/ (oP.(Sn.))
h-1
h-1
= Eso
V
2' (8), a;) Va 2¹ T (3 asn)
==0
==
B.
aQn
dap
oQ
dsh
Vo
+ +7 (n
da,
d0
dsk
d0
B.
Here, the shorthand notation B, denotes the bias introduced by the h-step model expansion and Bg denotes the bias introduced by using a critic for tail estimation. Then we may rewrite (B.30) as
be S 3 Eio, m-2 Br + B [ (1 B) Esan ~ - [| + B Bull2]
1 ( BBs Nove [I] By
2 + +1 (3) Eio n s [1] Brl
+ ( Exo.nmu Bu + (1 - 3) BBo.. ~c [I| Bell2]
(B.31)
For the first term on the right-hand side of (B.31), we have
3. Egg... mu, B, [2] + (1 - 3) Eso..
~ [r
4-1
B Eso., war
E Expne
V
rs; a.)
V.
Σ
r
(B.m:
(1 - 9).
1=0
h-1
h-1
Brss ( (5l (s) 2
}}} 1/2
Bio,. ~Mh
Brows
Vo
r ((8) 0;) - Vo
7 r
(Bi,m Tin))
2
h-1
h-1 1 V
$ (8 + x- 1.-11115) (1- Eio Bio.n ~M
Brows
V
x² r(i, 0;)
(B (Bin):ann)
2=0
1=0
I 5x(B + (1- - 3)) Esgree 20, wote
Vo
a.)
V
7 r ( ai.n)
==0
==0
(B.32)
where the first and second inequalities follow from the definition of 6 in Proposition 5.6. Similarly, for the second term on the right-hand side of (B.31), we have
B Esg...e, .u, [1] B |B + (1-8) Ego.. ~ur
[]] B
[
(0. (/h Bionaawwos
aQQu daz
aQ*
dsk
||]]
=B Eig Eiionner
VoVe(Sinn))
dan
d0
Osh d0
aQQ dah
aQ*s dsh voVi(Sn.n)
+ (1 -Eo Ses
Dap
d0 dsh d0
2.
aQ da aQ* dsh Dan d0
(8 + K (1 (1)
EEsn 30,n
VoVi(5m.))
Seh d0
(8 + K (1- (1-B)) ph)
0Q dap
aQ dsh
V (.0m.,))))))) 11 (B.33)
E E )NJ. (es..,,) az )~ā
+
Oan
do
Osh
d0
Plugging (B.32) and (B.33) into (B.31) completes the proof.
24

---

Lemma B.5 (Lipschitz Value Function [47] Theorem 1). Under Assumption 5.3, for y Lx) < then the state-action value function -L-Lischttzz continuous, such that for any policy To, state $1, 82 E S and action 01, 02 A, _ (1, Q (82, 02) - LQ [[ 82, a1 02) 12= and
LQ = ( (1+ + Lx)).
B.4 Proof of Proposition 5.7 Proof. When 1, we have
- 2 R
h-1
7
1 1- 4
1
h.
1 0=
1
1 1-7
We denote by H = 1/(1 -
Ei=
7 the effective task horizon.
To find the optimal unroll length h" that minimizes the upper bound of the convergence, we define g(h) as follows,
((h) = c. (26 b/g +
17 x U22 + of?
Here, uf² and b9 are the leading terms in the variance, bias bound (i.c., (B. and (B.24)) when Ly. Li. and Ly are less than or equal 1. Formally, 1/g = h and by hº h(H e., We consider the terms that are only dependent on h, H, Cja, and Cn to simplify the analysis and determine the order of h".
Our first problem is find the optimal model unroll h that minimizes g(h). We notice that ghh) increases monotonically with respect to bú and of when they are non-negative. This further simplifies the problem to find
h argmin b/ + d'ef = argmin (eg, c)) + h(H - N)
(B.34)
b
h
9h (h)
where c' is some constant that does not affect the order of h²*
By taking the derivative of the right-hand side of (B.34) with respect to h and setting it to zero, obtain
2h 91 (h) = 3h² (Eg,e + c)) + (3h² - 4Hh + H+) Ev,t = 0.
(B.35)
Solve the above quadratic equation with respect to h, we have the two non- -oggatite roots h and h2 as follows,
h 4H + / (4H 12cIEV
Ev H
4HE,, (AHe,,)) 12c, E 112
h
6c1
6c1
where we define e = Cf.t + cu.t + d.
Now study the resulting two cases. If (AHen,))² 12c H1 2 0, we have
="= hit = O + cv,)) H).
We can verify that hf is indeed the minimum by calculating the second-order derivative at hi* as follows.
02 (h(")
4Hln.
(AHHema)) 4e H2 * 6(( + tv,t +c) - 4He
0h
6c1
(AHe,, 4c₁ H² > 0.
The other case is (2 120 H2 < 0. When this happens, (B.35) does not have real solution h² and we set h² to 0. This concludes the proof of Proposition 5.7.
25

---

B.5 Proof of Corollary 5.9
Proof. We let the learning rate 1/vT. Then for T 2 4I2,, we have c = (n 1-- 1 s 2VT
and Ln - 1/2. By setting N = (VT)), we obtain
min E [|Vo 112] tE]T]
4
T 1
c. 220 by + "1 uz) + 1 + v/
+ 4c
B[((())r) - J [
T
T
1=0
4
8
Σ
T8 b, bi + +20
T -B(((TTO))) [
7
VT
T6 b, +b² b,+bb)+ +0(1VTT)
168 VT (T)
4
T +T))+((I//T))
This concludes the proof.
C Experimental Details
C.1 Implementations and Comparisons with More RL Baselines
For the model-based haseline Model-Based Policy Optimization (MBPO) [28], we use the imple- mentation in the Mbrl-lib [45]. For all other model-free baselines, we use the implementations in Tianshou [62] that have state-of-the-art results.
We observe that the RP-DP has competitive performance in all the evaluation tasks compared to the popular baselines, suggesting the importance of studying model-based RP PGMs. In experiments, we implement RP-DR as the on-policy SVG(1) [25]. We observe that the training can be unstable when using the off off-policy SVG implementation, which requires carefully chosen policy update rate as well as a proper size of the experience replay buffer. This is because when the learning rate is large, the magnitude of the inferred policy noise (from the previous data samples in the experience replay) can be huge. Implementing an on-policy version of RP-DR can avoid such an issue, following [25]. This, however, can degrade the performance of RP-DR compared to the off ooflcolicy RP-DP algorithm in several tasks. We conjecture that implementing the off- aff-policy version of RP-DR can boost its performance, which requires techniques to stabilize training and we leave it as future work. For RP-DP, we implement it as Model-Augmentee Actor-Critic (MAAC) [12] with entropy regularization [23], as suggested by [4]. RP(O) represents setting h = 0 in the RP PGM formulas [4], which is a model-free algorithm that is a stochastic counterpart of deterministic policy gradients.
For model-free baselines, we compare with Likelihood Ratio (LR) policy gradient methods (c.f. (2.2)), including REINFORCE [56]. Natural Policy Gradient (NPG) [31]. Advantage Actor Critic Trust Region (ACKTR) [65], and Proximal Policy Optimization (PPO) [49]. We also evaluate algorithms that are built upon DDPG [34], including Soft Actor-Critic (SAC) [23] and Twin Delayed Deep Deterministic policy gradient (TD3) [19].
(A2C), Actor Critic using Kronecker-Factores
C.2 Implementation of Spectral Normalization
In experiments, we use Multilayer Perceptrons (MLPs) for the critic, policy, and model. Besides, we adopt Gaussian dynamical models and policies as the source of stochasticity. To test the benefit of smooth function approximations in model-based RP policy gradient algorithms, spectral normalization is applied all layers of the policy MI.P and all except the final layers of the model MI.P The number of layers for the policy and the dynamics model is 4 and 5, respectively.
Our code is based on PyTorch [44], which has an out-of-the-shelf implementation of spectral normal- ization. Thus, applying SN to the MLP is pretty simple and no additional lines of code are needed. Specifically, we only need to import and apply SN to each layer:
from torch. nn utils. parametrizations import spectral_norm layer = [spectral_ norm (nn. Linear (in_dim, hidden_dim) )), nn ReLU
26

---

C.3 Ablation on Spectral Normalization
In this section, we conduct ablation studies on the spectral normalization applied to model- -ased RP PGMs. Specifically, we aim to answer the following two questions: (1) What are the effects of SN when applied to different NN components of model-based RP PGMs? (2) Does SN improve other MBRL algorithms by smoothing the models?
What are the effects of SN when applied to different NN components of model-based RP PGMs? We study the following NN components that spectral normalization is applied to: both the model and the policy (default setting suggested by our theory); only the model; only the policy; SN is applied (vanilla setting). The results are shown in Figure 8.
Tepper 2w15
valke
400
300
Figure 8: Ablation on the effects of spectral normalization when applied to different NN components of based RP PGMs.
We observe in Fig. 8 that for both the hopper and the walker2d tasks, applying SN to the model and policy simultaneously achieves the best performance, which supports our theoretical results. Besides, learning a smooth transition kernel by applying SN to the neural network model only is slightly better than only applying SN to the policy. At the same time, the vanilla implementation of model-based RP PGM fails to give acceptable results.
Does SN improve other MBRL algorithms by smoothing the models? We have established that smoothness regularization, such SN, in model-based RP PGMs can reduce the gradient variance and improve their convergence and performance. However, it is not necessarily the case for other model- based RL methods. In this part, we investigate whether SN can improve previous MBRL algorithms due to a smoothed model. Specifically, we evaluate MBPO [28], a popular MBRL algorithm when SN is added to different numbers of layers in the model neural network. The results are shown in Figure 9. We observe that SN has a negative influence when applied, which is in contrast to our findings that SN is beneficial to RP PGMs.
hopper
half-cheetah
Haro eyerr
5000
MOFO 3R,
HOrO 3N, epers
MOFO SH Sapos
20.00
40000
Retur
1000
1000
3.0
1000
305
345
Figure 9: Ablation on the effect of SN in other MBRL algorithms. Spectral normalization has negative effect when applied to MBPO, indicating that SN by smoothing the model, does not necessarily lead to better gradient estimation or improve the performance for MBRL methods other than model-based RP PGMs.
27

---

C.4 Ablation on Different Model Learners
Our main theoretical results in Section 5 depend on the model error defined in (4.4), which, however, cannot directly serve as the model training objective. For this reason, we evaluate different model learners: single- and multi-step (h-step) state prediction models, as well as multi-step predictive models integrated with the directional derivative error [33]. The results are reported in Figure 10. We observe that enlarging the prediction steps benefits training. The algorithm also converges faster in walker2d when considering derivative error, which approximately minimizes (4.4) and supports our analysis. However, calculating the directional derivative error by searching nearest points in the buffer significantly increases the computational cost, for which reason use /tstep state predictive models as default in experiments.
hopper
walker2d
5000
3000
4000
Return
2000
3000
2000
Single Stop State Prediction
Step
1000
w/ Directional Derivative Emar
Steps
le5
Steps
1e6
Figure 10: Ablation on different model learners: single-step and multi-step state prediction models, and multi-step state prediction models trained with additional directional derivative error.
C.5 Figures in the Main Text in Larger Sizes
Here, we provide identical figures that are larger size. Figure 11, 12, 13, 14 correspond Figure 1, 4, 6, in the main text, respectively.
haspar
halichesta
3343
2000
24
3503
330
Staps
Sapx
RP:DP-SN
ACKTR
RP-DP
RP(0)
REINT ORCE
A2C
PPO
SAC
Convergence
Figure 11: Comparisons between RP PGMs (the green labels) and MF/MB baselines (the black labels) in the MuloCo [57] tasks.
28

---

wallierdd h
wa walkerr2 h-6
walker2d h-10
2000
2000
2000
1000
1200
Naturn
0.0
6.2
0.4
5.5
3.0
3-0
500
0.2
0.5
3.6
1.0
3.0
0.5
1.0
3.2
2.0
23
30
Steps
Stops
Staps
hopper h-8
hopper h-10
hopper h-15
2002
2003
1032
1003
3.2
05
3.0
1.
2.0
3.0
05
2.0
Steps
Staps
345
Steps
Figure 12: Performance of vanilla and SN-based MB RP PGMs with varying h. The vanilla method
only works with small and fails when increases, while the SN-based method enables larger
walker2d
h=6
walker2d h=10
209
Goedeet Vertance
2014
2014
1.0
-6
loc
162
160
hopper h=8
hopper h=10
hopper h=15
Grabeet Veluencc
10t
207
201
20
105
165
165
Figure 13: Gradient variance of RP PGMs. The variance significantly lower with SN when is large.
dFlex hopper
dFicx happer
dFlex hopper
6033
12t0
B9 w SN Dh.P whb SN
APG
107
mall Phet
201
3032
dFlex half-choctal
dFlex half-cheetah
dFlcx half-chentah
2014
1020
2619
Betum
4000
RF-P SN
Gredd Bas
Ietum
2000
Steps
Steps
Steps
Figure 14: Performance and gradient bias in differentiable simulation. The last column is the full training curves of APG, which need 20 times more steps than CP- to reach comparable return in the hopper task and fail the chee-ch task, respectively.
29


Reviews:
[{"review_id": "yNDEcHxvbx", "reviewer": "Reviewer_epDC", "summary": "The paper theoretically analyzes the reparameterization policy gradient estimator’s bias and variance in reinforcement learning optimization and provide results characterizing the optimization convergence using such gradient estimators. It then proposes to apply spectral normalization (dividing the linear weight matrix by its largest singular value to make the Lipschitz constant upper bounded by $1$) on the learned world model and learned policy network. Empirically, they observe the spectral normalization improves the variance of the RP gradient estimator and improves the performance of RP to be comparable/better than other RL methods (including likelihood ratio methods).", "questions": "1. **Correlated randomness over time** Line 129 and 138 are reusing the same random variable $\\zeta$ and $\\xi$ for all time steps’ action sampling and environment transition in the same rollout. However, the randomness in action sampling and the randomness in state transition shouldn’t be correlated over different time steps. Can the authors clarify why these variables are shared for a given rollout trajectory?\n\n2. **Are there other ways to trade off bias for variance?** The authors propose to use spectral normalization to reduce the variance at the cost of bias. Another way to potentially trade off bias for variance is to use something similar to truncated back propagation through time. In this case, one could imagine using a very small time horizon and completely ignore the discounted rewards after a certain time step (currently captured by the learned critic function). How would such a (high bias, low variance) remedy perform in comparison to the SN methods on the RL tasks considered in this paper?\n\n3. **Would there be sufficient incentive to use longer expansion step + spectral normalization?** Looking at the experiment figures 4, 5, and 6, it seems that the performance of RP gradient only starts to deteriorate for longer unroll length (h > 5). When h ≤ 5, it seems that using SN doesn’t really improve the performance. In this case, why wouldn’t researchers just choose to use a short model expansion step (together with a learned critic) and not to use the spectral normalization approach proposed by the authors?", "limitations": "The paper doesn’t have much discussion on the limitations. I would encourage the authors discuss what they think are the limitations in their theoretical and experimental results. I don’t think negative societal impacts are relevant to this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper does a good job setting up the background and context about the policy gradient estimation methods in reinforcement learning.\n\n2. The experimental results confirm the benefits of applying spectral normalization in reparameterization policy gradient methods when the model expansion steps are large.", "weaknesses": "1. Despite having an extensive background discussion on policy gradient, the paper is very brief on describing the formula and implementation specifics of model derivatives on predictions (DP) and model derivatives on Real Samples (DR): Equation (4.1) and (4.2) are plain tautology (RHS only expands the reward function $J$) and provides no information the backward recursive structure of the gradient estimators.\n\n2. Assumption 5.3 on Lipschitz Continuity. In the paper, the authors assume a Lipschitz constant on the learned world model $f_\\psi$. However, this neglects the fact that the learned world model is also changing as the learning progresses. Without spectral normalization of the world model, it is conceivable that the world model might have a growing Lipschitz constant over the update steps $T$. It’s not clear to me that this assumption has already taken into account of this update time aspect and deserves further explanation in the paper.\n\n3. The paper conducts experiments on Mujoco tasks.  To my understanding, Mujoco tasks only have randomness in the initialization state, but doesn’t have randomness in the state transition. As a result, the assumptions on stochastic transitioning environment made in the paper don’t seem to hold. It would be necessary for the authors to clarify whether this is the case or why they haven’t experimented with environments with greater randomness (or make Mujoco random).\n\n4. Convergence theory is non-informative. \n    - In proposition 5.2, to have the learned model converge to a stationary point, we want the LHS (minimum gradient’s squared 2-norm encountered so far) to be decreasing as a function of $T$. However, in the upper bound on the right hand side, the first term contains $\\mathbf{E}[J(\\pi_{\\theta_T}) - J(\\pi_{\\theta_1})] $ which should increase as $T$ increases if the optimization is making progress in maximizing the value function. The relationship between this term and its denominator $T$ should be further discussed. Besides, the second term on the right hand side has a term $O(\\frac{\\sum_{t=0}^{T-1} v_t}{T})$. In the Proposition 5.4, the authors provide a $O(1)$ bound for $v_t$ for a fixed number of gradient estimates $N$. Thus the term $O(\\frac{\\sum_{t=0}^{T-1} v_t}{T})$ would only be $O(1)$. Hence it’s not clear to me this theory can capture the empirical observation that training using reparameterization gradient can converge to approximate local (or even global) maxima (which are stationary points).\n    - In Proposition 5.7, the author gives a big O notation for the optimal model expansion step $h^*$ for update iteration $t$. This bound depends on the gradient errors $\\epsilon_{v, t}$ and $\\epsilon_{f, t}$, both of which are unknown values that depend on the ground truth value function and world model. As a result, it’s not clear whether this theory can offer any practical guidance. Besides, this optimal expansion step is update-iteration dependent (depends on $t$) and it’s not clear how to practically instantiate such an h-schedule.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. Below are our specific responses to the questions raised by the reviewer:\n\n**Weakness 1: Eq. 4.1 and 4.2 are plain tautology and provides no information the backward recursive structure of the gradient estimators.**\n\n- Eq. 4.1 and 4.2 (or A.9) depict the two variants of RP gradients where the sole distinction lies in the incorporated noise variables $\\zeta$ and $\\xi$, which are sampled from predefined distributions and inferred from real samples, respectively. \n- These equations first compute the value and then obtain its first-order gradient w.r.t. $\\theta$, whose recursive expression is given in Appendix A. In practice, the gradient is computed using modern deep learning libraries by automatic differentiation.\n\n **Weakness 2:  The model might have a large Lipschitz constant over the update steps.**\n \n- The Lipschitz constant of the *learned* model is a global coefficient that can be enforced by constraints or model designs like SN. \n- In fact, what we aim to address is the case when the model has a large Lipschitz constant that can result in significant variance in gradients. In such cases, it becomes crucial to employ smoothness regularization techniques to effectively tackle the optimization challenges that arise.\n\n**Weakness 3: Randomness in the state transition.**\n\n- Our results hold for MDPs with both stochastic and deterministic transitions. For the latter case, the noise $\\xi^*$ would be $0$ for the dynamics $s'=f(s, a, \\xi^*)$. However, the issue of exploding variance still exists due to the randomness in the initial state and the stochastic policy.\n- In **Figure 1 in the PDF**, we report the results in Mujoco tasks with stochastic state transitions.\n\n**Weakness 4.1: The relationship between $\\mathbb{E}[J(\\pi_{\\theta_T}) - J(\\pi_{\\theta_1})]$ and $T$ in Proposition 5.2. Besides, the RHS has a variance term that would be $O(1)$.**\n\n- For MDPs with bounded reward, e.g., if $|r(s, a)| \\leq r_m$ as stated in Proposition 5.2, then $\\mathbb{E}[J(\\pi_{\\theta_T}) - J(\\pi_{\\theta_1})]\\leq 2r_m$, which is a universal upper bound. This is because $|V^\\pi(s_0)|=(1-\\gamma)|\\mathbb{E}_\\pi[\\sum_i\\gamma^i r(s_i, a_i)]|\\leq r_m$.\n- Proposition 5.4 shows that $v_t$ scales as $O(1/N)$. It suffices to choose a reasonably large value for $N$ for convergence, as evidenced by Corollary 5.9. Even so, we emphasize that Proposition 5.2 primarily serves to justify our SN method by characterizing the roles of gradient bias and variance in convergence. To serve this purpose, we explicitly express the bias and variance terms while imposing minimal assumptions. As a result, the upper bound in Proposition 5.2 may appear looser compared to the typical bounds in model-free RL analysis, which involve fewer error sources and stronger assumptions (e.g., the bounded variance Assumption 4.4 in [1].). \n\n\n**Weakness 4.2: Optimal model expansion step in practice.**\n\nProposition 5.7 aims to shed light on the factors influencing $h^*$ and its dependence on the horizon and model, critic errors. The result indicates that $h^*$ should increase when the model is more accurate and decrease when the critic is more accurate. However, this is only a rough guidance in practice since accurately quantifying these errors poses a significant challenge, making it difficult to determine an optimal $h^*$ schedule during training.\n\n\n**Question 1: Correlated randomness over time.**\n\nAt each timestep $i$, the random variables $\\zeta$ and $\\xi$ are independently sampled and are not shared within a rollout trajectory. We will revise the notations in Line 129 to $\\zeta_i, \\xi_i$, and in Line 138 to  $\\zeta_{i,n}, \\xi_{i,n}$. We sincerely appreciate the reviewer for bringing this to our attention.\n\n\n**Question 2: Are there other ways to trade off bias for variance?**\n \n- The reviewer is correct that one way to reduce gradient variance is to use Truncated BPTT. However, this approach can lead to a huge gradient bias, as it over-prioritizes short-term dependencies. This has been observed in some previous works [2].\n- In order to further investigate this, we performed additional experiments and report the performance results in the **Figure 2 in the PDF**. These experiments demonstrate that Truncated BPTT struggles to achieve high returns.\n\n\n**Question 3: Would there be sufficient incentive to use longer expansion step + SN?**\n \n- Our experimental results demonstrated that longer expansion steps (e.g., $h=8$ in hopper and walker2d) with SN offers a better performance than choosing a short expansion steps (e.g., $h\\leq 5$ as commonly used in previous methods).\n- The observed decrease in performance with larger $h$ is in line with our theoretical findings from Proposition 5.7 as the compounding model error becomes more pronounced. Therefore, it is advisable to decrease $h$ and rely more on the critic. Notably, SN is proposed to mitigate the optimization challenges, applying which the above tradeoff between model and critic error is valid.\n- Learning more accurate models to address the bias issue holds significant promise for future research. Once we develop such models, larger $h$ can become advantageous, as the model surpasses the critic in accuracy.\n\n**Limitation: The paper doesn’t have much discussion on the limitations.**\n\nWhile SN addresses the challenge of exploding gradient variance and enables longer model unrolls, learning more accurate longer-horizon models to fully exploit the gradient information remains an open problem. Besides, our experiments focus solely on SN as a smoothness regularization technique, and we acknowledge the need for further exploration of alternative designs.\n\nWe hope the above response resolves your questions and would be happy to have further discussions if you have any additional comments.\n\n[1] Wang et al. ''NPG methods: Global optimality and rates of convergence.''\\\n[2] Xu et al. ''Accelerated policy learning with parallel differentiable simulation.''"}, {"review_id": "BjuCPEZV1k", "reviewer": "Reviewer_yXDQ", "summary": "This paper examines reparameterization policy gradient methods in model-based reinforcement learning. It investigates the relationship between the convergence rate, bias and variance of reparameterization policy-gradient, smoothness of the model, and approximation error. Based on the theoretical analysis, it further proposes a spectral normalization method to enforce smoothness on the model and policy.", "questions": "No", "limitations": "The authors have adequately addressed the limitations", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. It is interesting to note how different parts of MB PR PGMs interact with each other, and how the smoothness of the model affects the model expansion steps.\n3. The experimental results demonstrate that applying spectral normalization to regularize the model and policy leads to improved performance and enables longer model expansion.", "weaknesses": "1. The application of spectral normalization to deep networks is not new. \n2. Applying spectral normalization limits both the model and policy capacity, which can result in an increasing gradient bias and a slower convergence rate. The trade-off between gradient variance and bias has not been thoroughly studied in experiments. (e.g. Section 7.3 only studies variance and bias on two disjoint environment suites)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for identifying our work's soundness and technical contributions. The valuable comments have helped us improve our manuscript. Below are our specific responses to the questions raised by the reviewer:\n\n---\n**Weakness 1: The application of spectral normalization to deep networks is not new.**\n\n- Previous studies have primarily focused on utilizing spectral normalization to tackle training instability problems associated with deep neural networks, such as [1, 2]. \n- In contrast, our spectral normalization method is rooted in a theoretical analysis of model-based reparameterization policy gradient methods and aims to address the issue of exploding variance that can arise when employing lengthy model unrolls.\n- Additionally, our investigation in Appendix D.3 reveals that applying spectral normalization to other model-based reinforcement learning algorithms can yield adverse effects. This suggests that the applicability of smoothness regularization is not universal, and we specifically employ spectral normalization based on theoretical motivations, tailored for model-based reparameterization policy gradient methods.\n\n---\n**Weakness 2: The trade-off between gradient variance and bias has not been thoroughly studied in experiments. (e.g. Section 7.3 only studies variance and bias on two disjoint environment suites).**\n\n- In our ablation studies on the gradient bias, we demonstrated a consistent result: even in cases where the bias is minimal (e.g., RP-DP without spectral normalization) or completely absent (e.g., analytic policy gradient approach), the amplified variance can still lead to poor performance outcomes. \n- The task designs and agent configurations are the same in the Mujoco and dFlex environment. We opted to use the *differentiable* dFlex simulator in order to implement the analytic policy gradient approach. This choice also enhances the precision and computational efficiency of our analysis as the bias can be calculated by directly comparing the model-based RP gradient and the analytic gradient.\n\n---\nWe hope the above response resolves your questions and we would be happy to have further discussions if you have any additional questions or comments.\n\n---\n[1] Miyato et al. ''Spectral normalization for generative adversarial networks.''\\\n[2] Bjorck et al. ''Towards deeper deep reinforcement learning.''"}, {"review_id": "1XFEgEKlRA", "reviewer": "Reviewer_eACF", "summary": "This paper studies differentiable model-based reparametrized policy gradient methods (RP-PGMs) with a particular focus on mitigating policy gradient variance and bias under longer horizon model rollouts to benefit agent learning and convergence. The paper introduces theorems on the bounds of policy gradient variance and bias in terms of the rollout horizon length and the lipschitz constant (``smoothness'') of the model and policy networks. Building upon the theory foundations, the paper then proposes to use Spectral Normalization on network parameters to minimize policy gradient variance and bias. Experiments on various control tasks demonstrate the effectiveness of the proposed approach.", "questions": "See \"weaknesses\".", "limitations": "Some further discussions on limitations could be added. For example, future works can explore tasks with higher dimensional observations (e.g., visual input), higher-dimensional action / control outputs (e.g., Humanoid), and alternative networks like CNNs.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written. The theoretical and empirical findings are organized in a structured and smooth manner that is comfortable to read.\n\n- The proposed spectral normalization approach to mitigate policy gradient variance and bias under longer model unroll horizons is both theoretically sound and empirically justified.\n\n- The main theoretical results provide a solid theory underpinning for the convergence of reparametrized policy gradient methods.", "weaknesses": "- In this paper, authors set the overall spectral norm of networks to be 1. It would be interesting to investigate the impact on gradient variance, bias, and policy return when the network's spectral norm is set to a value lower than 1 during training. In particular, is there a tradeoff between policy return and gradient variance & bias?\n\n- From Fig. 4, setting a longer model rollout horizon ($h=8/10/15$) seems to harm the sample complexity of policies, though they still converge to similar returns as the shorter horizon cases ($h=3$). It would be interesting to further empirically explore the impact of even longer $h$ on agent sample complexity (e.g., $h=20/50$). It would also be interesting to explore tasks where longer horizon model rollouts offer performance advantages over shorter horizon model rollouts. Investigating such scenarios would enhance the applicability and generalizability of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for identifying our work's soundness and technical contributions. The valuable comments have helped us improve our manuscript. Below are our specific responses to the questions raised by the reviewer:\n\n---\n**Weakness 1: It would be interesting to investigate the impact on gradient variance, bias, and policy return when the network's spectral norm is set to a value lower than 1 during training. In particular, is there a tradeoff between policy return and gradient variance & bias?**\n\n- According to our theory and experiments, adjusting the spectral norm to a value of 1 effectively addresses the issue of exploding gradient variance and the associated optimization problems.\n- Enforcing the spectral norm to a smaller value has the potential to further reduce the gradient variance at the expense of the networks' approximation capacity. This tradeoff indicates that setting the spectral norm to a smaller value near $1$ has the potential to reach higher policy returns in certain tasks. For instance, as demonstrated in the **Figure 3 in the PDF**, we observe that setting the spectral norm to both $1$ and $0.8$ yields comparable results in the half-cheetah locomotion tasks. These settings outperform the vanilla RP-DP when spectral normalization is not applied, as well as RP-DP with a spectral norm of $0.5$. The latter exhibits increased gradient variance and gradient bias, respectively, contributing to the inferior performance.\n- Notably, the optimal spectral norm value can be highly task-dependent. For example, when the Lipschitz of the system dynamics is less than $1$, models with smaller spectral norms are still able to approximate the dynamics effectively. In contrast, in stiff systems, enforcing a small spectral norm may result in significant model error and gradient bias due to the decreased approximation capacity. Therefore, in most of our experimental tasks, we recommend using standard spectral normalization as it strikes a good balance between these considerations.\n\n\n---\n**Weakness 2: It would be interesting to further empirically explore the impact of even longer $h$ on agent sample complexity (e.g., $h=20/50$). It would also be interesting to explore tasks where longer horizon model rollouts offer performance advantages over shorter horizon model rollouts.**\n\n- Based on our experimental observations, we found that utilizing longer expansion steps (e.g., $8$ in the hopper and walker2d tasks) combined with spectral normalization leads to improved performance compared to selecting shorter expansion steps (e.g., $h\\leq 5$ as commonly used in previous methods [1, 2]). \n- However, setting an even larger $h$ value, such as $20/50$, can negatively impact training due to the compounding effect of model errors and the significant bias. This observation is in line with our theoretical results from Proposition 5.7, where we demonstrated that when model error is large, the optimal unroll step should decrease to rely more on the critic.\n- Addressing the compounding error and bias issue by learning more accurate multi-step models holds great potential as a promising avenue for future research. Once such models are developed, larger values of $h$ can potentially become advantageous, as the model surpasses the critic in accuracy. This assertion finds partial support in the results presented in Appendix D.4, where we demonstrated that learning more accurate models (e.g., by incorporating directional derivative error) leads to further performance improvements.\n\n\n---\n**Limitation: Some further discussions on limitations could be added.**\n\nWe thank the reviewer for the insightful suggestions. We would add the following discussions on the limitations of our work in a later version of our manuscript:\n\nWhile the proposed framework and analysis are applicable to general MDP settings, our current experiments do not cover control tasks with high dimensional inputs, such as visual observations. Additionally, our use of multi-layer perceptron networks as dynamics models restricts us from tackling more complex image input tasks. Exploring alternative model designs, such as CNN and latent models, would be a fascinating avenue for future research that we intend to pursue.\n\n---\nWe hope the above response resolves your questions and we would be happy to have further discussions if you have any additional questions or comments.\n\n---\n[1] Clavera et al. ''Model-augmented actor-critic: Backpropagating through paths.''\\\n[2] Amos et al. ''On the model-based stochastic value gradient for continuous reinforcement learning.''"}, {"review_id": "XoaRhed6jL", "reviewer": "Reviewer_iodf", "summary": "When using re-parameterization (RP) gradient estimators for policy gradient methods (PGM) the\noptimisation landscape becomes chaotic and non-smooth, with exploding gradient variance.\nThis paper examines RP gradient estimators for policy gradient methods.\nFirst, the authors theoretically examine RP PGM.\nTheir theoretical examination highlights that the smoothness of the dynamic model's and the policy's function approximators has a large impact on the quality of the gradient estimator.\nFollowing this theoretically guided insight, they propose to enforce smoothness by applying spectral normalisation to all layers of the dynamics and policy's neural networks.\nTheir results suggest that this simple modification mitigates the exploding gradient issue associated with RP PGM.\nOverall, I think this is a nice paper as it presents a practically significant result backed by theory.\n", "questions": "I will increase my rating if the authors address the issues raised in the Weaknesses section.", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides an interesting insight into how the smoothness of function approximators influences the performance of reparameterisation gradients.\nIn particular, the paper builds a practical algorithm on top of theoretical insights and I think this is its greatest strength; as it provides a strong contribution to the community.\nThe paper's contributions are clear and the work is positioned well against the literature.\nThe theoretical claims appear sound but some of the derivations in the appendix are out of the scope of my expertise.", "weaknesses": "The paper's biggest weakness is its clarity. First of all, the figures are of very low quality and in some cases they are illegible.\nThere are also no figures until the results section on page 8. I think the paper would benefit by moving some of the figures earlier in the paper and making them bigger.\nI also have concerns regarding reproducibility as no hyperparameters are reported in the paper/appendix.\n\nIn general, the figures are of very low quality.\n- Figure 1\n  - I can't read the title or axis labels. These should be made bigger.\n  - The figure is also very crowded.\n    - I cannot see the variance for several algorithms.\n    - Consider running more random seeds to smooth the curves.\n    - Consider having fewer plots in a row so that they can be bigger. Put the rest in the appendix.\n- Figures 4/6/7\n  - I can't read the title or axis labels. These should be made bigger.\n  - Do you need both the walker2d and hopper figures in the main paper? The figures would be easier to read if they were bigger, for example, if there were only three plots per row. Perhaps the hopper results could be moved to the appendix?\n- What does the shading represent? variance/std? How many seeds were used in the experiments to calculate the variance?\n- In Figure 7 the authors refer to the third/last columns. Consider labelling the columns a-f.\n\nReproducibility\n  - How many seeds were used for each experiment?\n  - What hyperparameters were used? Learning rates, optimiser, activation functions, batch size, width of hidden layers, number of epochs, discount factor, early stopping callback, etc. I cannot see any of these details in the paper or appendix\n\nSome citations use arXiv versions instead of conference publications.\n - 11 is published at ICLR 2020\n - 36 is published at ICLR 2018\n\nMinor comments:\n- Line 127: Gleaned seems an odd word to use here.\n- Line 149: $\\nabla_{a} \\hat{f_{\\Psi}}$ is repeated twice. Should one of them be $\\nabla_{s} \\hat{f_{\\Psi}}$?\n- Line 221/222: this sentence doesn't read very well.\n- Line 295: should $g(x)$ be $g_{i}(x)$?\n- Section 7 leads straight into 7.1\n  - Consider adding an overview of Section 7 before Section 7.1 so that the reader knows what to expect.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for identifying our work's soundness and technical contributions. The valuable comments have helped us improve our manuscript. Below are our specific responses to the questions raised by the reviewer:\n\n---\n**Weakness 1: Clarity of the figures in experiments.**\n\n- We sincerely appreciate the valuable suggestions provided by the reviewer. Due to space limitations, we chose to defer the larger-size versions of the experiment figures, namely Figure 1, 4, 6, and 7, to Appendix D.5. \n- We will make the following changes according to the reviewer's suggestions: We will modify Figure 1 by splitting it into two rows and move the hopper ablation results in Figure 4, 6, 7 to the Appendix. By implementing these changes, each row within a figure will contain only three plots, improving the overall clarity and readability of the figures.\n\n---\n**Weakness 2: What does the shading represent? How many seeds were used in the experiments to calculate the variance?**\n\nThe shaded regions in the figures correspond to standard deviation using $6$ different random seeds. We will make this clear in our paper.\n\n---\n**Weakness 3: In Figure 7 the authors refer to the third/last columns. Consider labelling the columns a-f.**\n\nWe thank the reviewer for the suggestion and will revise our manuscript accordingly.\n\n---\n**Weakness 4: What hyperparameters were used?**\n\nThe hyperparameters used in our experiments are listed in the following table, which we will also add  to our manuscript. \n\n---\n| Hyper-parameter | Optimizer | Actor learning rate | Critic learning rate | Model learning rate | Reward learning rate | Replay buffer capacity | Mini-batch size | Discount factor | Target update rate | Activation function | Actor hidden dim | Actor hidden depth | Critic hidden dim | Critic hidden depth | Model hidden dim | Model hidden depth | Reward hidden dim | Reward hidden depth |\n|----------------|-----------|---------------------|----------------------|---------------------|----------------------|------------------------|-----------------|-----------------|--------------------|---------------------|------------------|--------------------|-------------------|---------------------|------------------|--------------------|-------------------|---------------------|\n|          | Adam      | 1e-4                | 1e-4                 | 1e-3                | 1e-3                 | 1e6                    | 512             | 0.99            | 5e-3               | ReLU                | 512              | 4                  | 512               | 4                   | 256              | 5                  | 512               | 2                   |\n\n---\n**Weakness 5: Some citations use arXiv versions instead of conference publications.**\n\nWe will fix this in our manuscript.\n\n---\n**Minor 1: Line 127: Gleaned seems an odd word to use here.**\n\nWe will change \"gleaned\" to \"unrolled\".\n\n---\n**Minor 2: Line 149: $\\nabla_a \\hat{f}_\\psi$ is repeated twice.**\n\nThe reviewer is correct that latter one of them should be $\\nabla_s \\hat{f}_\\psi$. We thank the reviewer for pointing this out and will fix it in our manuscript.\n\n---\n**Minor 3: Line 221/222: this sentence doesn't read very well.**\n\nWe will revise the text in Line 221/222 to \"Proposition 5.2 reveals how the convergence rate changes with the variance and bias of the gradient estimators.\"\n\n---\n**Minor 4: Line 295: should $g(x)$ be $g_i(x)$?**\n\nYes. We thank the reviewer for pointing this out and will fix it in our manuscript.\n\n---\n**Minor 5: Consider adding an overview of Section 7 before Section 7.1 so that the reader knows what to expect.**\n\nWe appreciate the valuable suggestion from the reviewer. We will incorporate the following paragraph before Section 7.1 into our manuscript:\n\n\"In this section, we provide empirical studies to support our theoretical findings. Firstly, in Section 7.1, we present a comprehensive evaluation of multiple algorithms derived from the proposed RP PGM framework. Using the Mujoco control tasks as our experimental domain, we compare these algorithms against various baselines to assess their performance. Secondly, in Section 7.2, we thoroughly examine the optimization challenges inherent in vanilla RP PGMs, which are characterized by issues such as exploding gradient variance and highly non-smooth loss landscapes. Subsequently, in Section 7.3, we demonstrate the effectiveness of employing smoothness regularization techniques to address these issues. Additionally, we conduct ablation studies to reveal the distinct roles played by the gradient variance and bias during training.\"\n\n\n---\nWe hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments."}]

Author rebuttal:
We conduct additional experiments to address Weakness 3 and Question 2 raised by **Reviewer epDC**, and Weakness 1 raised by **Reviewer eACF**. The results can be found in the attached PDF file.

Meta-review: This paper had strong support from a majority of its reviewers. The reviewers appreciated the theoretical and empirical justifications of the spectral normalization scheme in confining bias on long unrolls of the model, unified synthesis of model-based and policy gradient methods (highlighting the role of smoothness) and introduction of a practical algorithm. Therefore, we are happy to recommend the paper for acceptance.
Decision: Accept (poster)