Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes
Authors: Joe Watson
Jan Peters
Venue: CoRL 2022 Oral (2022)
Keywords: model predictive control
policy search
reinforcement learning
approximate inference

Abstract: Monte Carlo methods have become increasingly relevant for control of non-differentiable systems, approximate dynamics models, and learning from data.
These methods scale to high-dimensional spaces and are effective at the non-convex optimization often seen in robot learning. We look at sample-based methods from the perspective of inference-based control, specifically posterior policy iteration.
From this perspective, we highlight how Gaussian noise priors produce rough control actions that are unsuitable for physical robot deployment.
Considering smoother Gaussian process priors, as used in episodic reinforcement learning and motion planning, we demonstrate how smoother model predictive control can be achieved using online sequential inference.
This inference is realized through an efficient factorization of the action distribution, and novel means of optimizing the likelihood temperature for to improve importance sampling accuracy.
We evaluate this approach on several high-dimensional robot control tasks, matching the sample efficiency of prior heuristic methods while also ensuring smoothness.
Simulation results can be seen at monte-carlo-ppi.github.io.

Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes
Joe Watson Jan Peters!231
Department of Computer Science, Technical University of Darmstadt Centre for Cognitive Science, Technical University of Darmstadt GGrmann Research Center for AI "Hessian @rrboot-eerning..d
Abstract: Monte Carlo methods have become increasingly relevant for control of non-differentiable systems, approximate dynamics models and learning from data. These methods scale to high-dimensional spaces and are effective at the non-convex optimizations often in robot learning. We look sample-based methods from the perspective of inference-bart control, specifically posterior policy iteration. From this perspective, produce rough control actions that are unsuitable for physical robot deployment. Considering smoother Gaussian process priors, as used in episodic reinforcement learning and motion planning, we demonstrate how smoother model predictive control can be achieved using online sequential inference. This inference is real- ized through an efficient factorization of the action distribution and novel means optimizing the likelihood temperature racy. We evaluate this approach on several high-dimensional robot control tasks, matching the sample efficiency of prior heuristic methods while also ensuring smoothness. Simulation results can be seen at monte-carlow
highlight how Gaussian noise priors
improve importance sampling accu-
gi thub. io.
Keywords: approximate inference, policy search, model predictive control
Baseline (iCEM, coloured noise)
Ours (LBPS, SE kernel)
20
20
-1
-1
Action prior sample
T
Action sample
-1
H
Figure 1: High- fiigh-dimessiona,, contact-rich tasks such as manipulation (left) be performed effec- tively using sample-base model predictive control. While prior work uses correlated actuator noise to improve sample-efficiency and exploration. these methods do not preserve the smoothness in the downstream actuation a, resulting in aggressive control (center). We use smooth Gaussian process priors to infer posterior actions (right), which preserves smoothness while maintaining performance and sample efficiency, as both are using only 32 samples. Rewards r show quartiles over 25 seeds.
1 Introduction
Learning robot control requires optimization to be performed on sampled transitions of the envi- ronment [1]. Monte Carlo methods [2] provide a principled means approach such algorithms, bridging black-box optimization and approximate inference techniques. These methods have been adopted extensively by the community for their impressive simulated [3, 5, 6] and real-world
[7, 8, 9, 10, 11, 12, 13, 14, 15] robot learning results. Their appeal includes requiring only function evaluations of the dynamics and objective, so can be applied complex environments with min- imal overhead (Figure 1). Moreover, their stochastic nature also avoids issues with local minima
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

that occur with gradient-basee based solvers [16. 17]. Finally, while Monte Carlo sampling is expensive, shooting methods can be effectively parallelized across processes and the advent of simulations on GPUS also provides a means of acceleration [18, 13]. However, some aspects of black-box opti- mization are open to criticism. Sample-based solvers such as the cross-entropy method (CEM) [19] appear 'wasteful', ignoring computation by throwing away the majority of samples, while others enforce high-entropy search distributions to avoid premature convergence [18]. Moreover, many design decisions and hyperparameters are heuristic in nature, which is undesirable from both the user- and research perspective when interpreting, tuning or advancing these methods.
In this work, we consider Monte Carlo optimal control through the broader perspective of inference- based control [20, 21, 22, 23, 24, 25, 26, 27]. where optimization achieved through importance sampling [28]. This approach covers settings such as policy search [29], motion planning [8, 30] and model predictive control (MPC) [18]. From this view point, we highlight two key design decisions: the likelihood temperature and the distribution over action sequences. An adaptive temperature scheme is crucial for controlling the optimization behavior across objectives and distributions, but in many methods this aspect is ignored or opaque. Moreover, correlated action sequences are equally crucial for performing effective exploration and control in practical settings. Smoothness, arising from such correlations, is an aspect of human motion [31]. Smooth priors have taken many forms across domains, such as movement primitives [32], smoothed- [11, 12] or coloured noise [4]. We use Gaussian processes [33] as action priors and show how they be scaled to nigh-dimensional action spaces through factorization of the covariance. Evaluating on simulated robotic systems, reproduce prior results policy search while transferring these ideas to MPC, matching prior performance with respect to sample efficiency while ensuring smooth actuation.
Contribution. First, we present perspective of episodic siterence-basede control based on Gibbs posteriors. Using this view, we then present novel Monte Carlo variants that incorporate the approx- imate inference error due to importance sampling, simplifying the hyperparameter while providing regularization. Thirdly, we demonstrate how richer Gaussian process priors can be combined with these regularized Gibbs posteriors for Monte Carlo MPC using online sequential inference, which achieves greater smoothness and sample efficiency than standard white noise priors. We highlight connections between this approach to MPC and effective prior approaches to episodic policy search.
2 Monte Carlo Methods for Optimal Control
This section outlines the problem setting and introduces variational optimization and posterior pol- icy iteration methods. We consider the standard (stochastic) optimal control setting in discrete- time, with states E Red,, actions a E IRd Optimization is framed as maximizing a reward r: Rdd x Rdd -> R under dynamics p(8 8;, and initial state distribution p(81).
max ar....a
E E at)
s.t. 8841 ~ p(: 8c, ar), 81 p(8))
This work focuses on the episodic setting, where optimization is performed after evaluating the current solution over a finite-time horizon T. We frequently use the episodic return R, where R(S, A)= Ei= r(8t, a:), using upper-case to denote sequences, e.g. A:= (a!...,,,a
2.1 Variational Optimization with Gibbs Posteriors
The optimization outlined above is amenable to gradient- solvers such as stochastic differential dynamic programming [34]. However, to aid optimization through exploration and regularization, we can consider optimizing a parametric belief over action sequences g E Q. The variational for- mulation (Equation 3) generalizes Bayes' rule beyond optimizing likelihoods and resembles many learning algorithms [35, 36). This work concerns optimizing an open-loop action sequence to max- imize an episodic return. Bayesian inference of an action sequence from data, known as input esti- mation, can be performed using message passing of the appropriate probabilistic graphical model, capturing the sequential structure of the problem and necessary priors [27]. If the measurement log-like liglliill is replaced with the control objective, this inference computation can be shown to have precise dualities with dynamic programming- objective provides a powerful suite of inference tools for efficient computation, it requires treating the control objective as a Markovian log-likelihood which not the case for episodic objectives. The Gibbs likelihood is general treatment of the abjective-as-likeliiie (Definition 1) [38, 39].
optimal control [37]. While this switch in

---

Definition 1. (Gibbs likelihoods and posteriors) For a loss f and prior p(z)) the Gibbs posterior 9a for parameter x is derived by constructing the Gibbs likelihood exp(-a fx)) from the loss,
1 Zo
(x(x)
f(z)
Za f(x(=) dx, 2 0.
(2)
This posterior minimizes the following objective
qa arg min
Q (==n(t))[(aa]] + [TT..lg())) p())]
This objective appears in PAC-Bayes methods [38], mirror descent methods [40] and Bayesian inference as the evidence lower bound objective when f(z) is a negative log-likelihood [39].
Augmenting the variational optimization objective with prior regularization (Equation 3), we obtain an expression of the optimal belief in the action sequence (Equation 2). The parameter has range of meanings, depending on context. In PAC-Bayes it is the dataset size, in mirror descent it is an update step size and in risk-sensitive control it is the sensitivity [41, 42]. Example in Appendix A examines a tractable linear-quadee mnear--uadrrtic-- Gaussian Gaussian example of this update, demonstrating its relation to Newton-like optimization and highlighting the effect a has the regularized update.
2.2 Posterior Policy Iteration
The optimal control problem (Equation 1) ambiguous regarding whether the action sequence or state-action trajectory is the optimization variable. Applying the Gibbs posterior to the optimal con- trol setting recovers Rawlik et al.'s posterior policy iteration [41], which can be implemented using the joint distribution policy. We consider the following joint state-action distribution, that fac- torizes in the following Markovian fashion p(S, AA = =)=((s)) ) II" P(8 8,, a,) p(a, s,). Pos- terior policy iteration updates the state-action distribution through the policy, constructing a Gibbs likelihood from the reward, as the dynamics and initial state distribution are constant.
Definition 2. (Posterior policy iterations (PPI) [43]) As the initial distribution and dynamics are shared by the prior and posterior joint state-action distribution, the joint Gibbs posterior (S, A) x exp(aR((S, p(S, A) can be alternatively expressed using the policy posterior update qa (A S) x exp(aR(SS, A)) p A S).
Using this update, the key decisions are choosing p(A S). a and the inference approximation. p and go are Gaussian, then PPI involves iterative refinement of the distribution. In the Monte Carlo setting, la takes the form of an importance -ppighteee-weighhed empirical distribution. To apply iteratively, p is updated using the M- M-projection, following the objective (Equation 3), i.c. a weighted maximum likelihood fit of the policy parameters [29]. This approach is a stochastic approximate expectation maxinization (SAFM) method [44] and described fully in Algorithm 1 in the Appendix. We argue key aspect of PPI methods is how to specify the inverse temperature a during optimization (Section 3), it has strong influence on the posterior, which is important when fitting rich distributions such as Gaussian processes (Section 4) from samples. Gaussian process action priors can be applied to several control settings, such as policy search and model predictive control (Section 6).
3 Posterior Policy Constraints for Monte Carlo Optimization
The Gibbs posterior in Definition has been adopted widely in control, albeit from a range of dif- ferent perspectives, such as Bayesian smoothing [23], solutions to the Feynman-Kan Kac equation [45], maximum entropy [26], mirror descent [46] and entropy-rreularrng
reinforcement learning [47]. An open question is how best to set for Monte Carlo optimization? Relative entropy policy search (Definition 3), provides a principled and effective means of deriving a for stochastic optimization, using the constrained optimization view of entropy- regulai-re optimal control.
Definition 3. (Episodic relative entropy policy search (eREPS) [29]) Maximize the expected return, subject hard KL. bound c on the policy update,
DK (AIS) l1 [(A <c The posterior policy takes the form go (A(S) x exp(aR) p(A(S), where a derived from Lagrange multiplier calculated by minimizing the empirical dual G(-) using N sumples,
max Eac -pt- (sr, a (:1s) -pp
[R(s), a.)
min, G(a) = = -+
+ 1 log A)) exp(aR(S A)) as : + log N EN exp(aRn).
3

---

While REPS is principled approach to stochastic optimization, we posit two weaknesses: The hard KL constraint is difficult to specify, as it depends on the optimization problem, distribution family and dimensionality. Secondly, the Monte Carlo approximation of the dual has no regularization and may poorly adhere to the KI. constraint without sufficient samples. Therefore, we desire an alterna- tive approach that resolves these two issues, capturing the Monte Carlo approximation error with a simpler hyperparameter. To tackle this problem, we interpret the REPS update as a posttrrr, where the temperature is calculated using the KL constraint. We make this interpretation concrete by reversing the objective and constraint, switching to an equality constraint for the expectation,
ming [go A S) 11 ((A S)] S.L. (a (-0-0((1)))) 41 -1>p()) [ r(8t, a+)] = R
This objective is a minimum relative entropy problem [48], which yields the same Gibbs posterior as eREPS (Lemma 1, Appendix A). With exact inference, a suitable prior and oracle knowledge of the maximum return, this program computes the optimal policy in a single step by setting lt to the optimal value. However, in this work, the expectation constraint requires nelf-normalized importance sampling (SNIS) on sampled returns R(n) using samples from the current policy prior,
1 (-)())) au (1)), pp)) E r(s,, az)l En q/p R()) En R(") exp(a R( R". Σ, exp(a R(n))
(n)
Rather than specifying R here, we identify that this estimator is fundamentally limited by inference accuracy. We capture this error by applying an Is-derived concentration inequality to this estimate (Theorem 1) [49]. This lower bound can be used an objective for optimizing a, balancing policy improvement with approximate inference accuracy,
Theorem 1. (Importance sampling estimator concentration inequality (Theorem 2, (491)) Let q and P be two probability densities such that q < and d2 || p) < +oc. Let σ .2..... ON i.i.d. random variables sampled from p and f + x RR be a bounded function (Ilffl so < +00). Then, for any O < ò S 1 and N >> O with probability at least 1 -
115 0z / (1 - 6)d2lg(2)) p(²)]
E [fx))] Z N Ei= (a))))))))))())))
(4)
5 N
The divergence term dzlgl1 p] is the exponentiated Rényi-2 divergence, exp Dzfall p). While this is tractable for the multivariate Gaussian, is otherwise not available in closed form. Fortunately, we can use the effective sample size (ESS) [50] as an approximation, as N, x N d2 p) [49, 51] (Lemma 2, see Section A of the Appendix). Combining Equation 4 with our constraint, instead of setting R², we maximize the IS lower bound Ri to form an objective for the inverse temperature a which incorporates the inference accuracy due to the sampling given inequality probability 1 1-6 8,
max ma, (a, 5) = E.../p [R] - En) En(6, = || - 5)
1
(5)
N₂
We refer to this approach as lower-bound policy search (LBPS). This objective combines the ex- pected performance of 9a. based on the iS estimate [=.//l-), with regularization En based on the return and inference accuracy. Treating p. N, ||/2]]>> as task-specific hyperparameterss the only al- gorithm hyperparameter ō E [0, 1) defines the probability of the bound. In practice, self-normalized importance sampling is used for PPI, as the normalizing constants of the Gibbs likelihoods are not available. While Metelli et al. also derive an SNIS lower bound [49], we found, as they did, that the lower bound with SNIS estimates work better in practice due to the conservatism of the SNIS bound. An interpretation of this approach is that the Rényi-2 regularization constrains the Gibbs posterior to be one that can be estimated from the finite samples, as the divergence is used in eval- uating IS sample complexity [52, 53]. Moreover, the role of the ESS for regularization is similar to the 'elite' samples in CEM. Connecting these two mechanisms as robust maximum estimators (Sec- tion A), we also propose effective sample size policy search (ESSPS), which optimizes to achieve desired ESS N Rényi-2 divergence bound, using the objective mina No -N More details regarding PPI (Section A) and temperature selection methods (Table 1) are in the Appendix.
This section introduces two methods, LBPS and ESSPS, for constraining the Gibbs posteriors for Monte Carlo optimization. These methods provide statistical regularization through soft and hard constraints involving the effective sample size, which avoids the pitfall of fitting high- siig--dmensiooall distributions to a few effective samples. A popular setting for these methods is MPC, which performs episodic optimization over short planning horizons while adapting each time step to the current state. Moreover, for optimal control, we also need to specify a suitable prior over action sequences. To apply PPI to the MPC setting, we must implement online optimization given this prior over actions.

---

Gaussian Noise
Smooth Noise (3=0.25)
Smooth Actions (3=0,8)
Coloured Noise (6=2.5)
SE Kernel (1=0.2
Periodic Kernel (1=0.2, T=0.5)
Figure 2: A practical aspect of Monte Carlo control methods for robotics is optimizing smooth ac- tion sequence. This example shows a non-smooth optimal sequence ---, which may undesirable, though optimal, to fit exactly. Prior methods struggle at providing both effective smooth solutions in the mean - and action samples -, as they ultimately fit the action distribution in an inde- pendent fashion. Using kernel-derived covariance function provides both. The line denotes the optimization horizon, beyond which is exploratory actions derived from both the posterior and prior.
4 Online Posterior Policy Iteration & Prior Design
In this section, we derive the online realization of posterior policy iteration that uses and maintains correlated action priors, computing the finite-horizon H future actions given a likelihood on a subset of actions from the past. R represents the return-based Gibbs likelihood term (Definition 1),
9a (a R1 = qq((a..--) R1:7) da1:-1 x p( (1:r) p((1:c-)) dai:t-1, (6)
where r S l H. As an analogy, this is equivalent to combining forecasting with state estimation, iee p( V1:)) for states x and measurements y. For correlated priors on the action space, this computation is tractable if working with Gaussian processes. In fact, a recurring aspect across several many posterior policy iteration-like approaches is the use of Gaussian process policies,
IL, E,), N() T E.))())
(Independent Gaussian noise, e.g. [18)),
(Bayesian linear regression, e.g. ProMP [32]);
(A|S)
W + K,s, E.,
(time-varying linear Gaussian e.g. [41, 54, 42)),
GP(N())) E())),
(non-parametric Gaussian process [55]).
Despite the simplicity of Gaussian action noise, for robotics, more sophisticated noise is often de- sired for safety and effective exploration [56. 29]. Prior work has proposed first-order smoothing [11, 12]. Using vi (n) ~O 1). B [0, 1] and Σ, = L.L.T actions are sampled using
a (n) However, in practice it is also implemented as a.") 3(1 + L,v, (n)) +1 (1-3)111))) 3) While this approach directly smooths the actuation, it also introduces a lag, which may deteriorate performance. Other approaches have used colored noise for sampling the noise n [4]. Contrast these approaches to Gibbs sampling a multivariate Gaussian joint distribution with 1-step cross-correlations (58),
p4 L,n(") (m) n. (m) _)( (1 3)m (m) or nhe (n) =B0 + v(1 B2) ni-1
(n)
which is a/j- 1 1
(n)
(n)
(1) where (1))) p4: Et,- E,, pc-1), and
-p1-1 - Σ, E E1,-1E, , - The differences are subtle, but important. The initial proposed sampling scheme essentially adds correlated noise to the mean for exploration, but does not con- sider the smoothness of the mean itself. The practical implementation incorporates the previous action, but through exponential smoothing, which introduces a fixed lag that potentially degrades the quality of the mean action sequence. Correct sampling of the joint distribution has neither of these issues and naturally extends to correlations over several time steps. We do this a gen- eral fashion by considering the (continuous time) Gaussian process (see Section G, Appendix), so p(a,) = N Ex,,t, = (P(p(t), E(t)) for a discrete- time sequence t = [t......t,) tyl. Propo- sition 1 shows how the time shift for MPC be implemented in general fashion when using GPS. Proposition Given a Gaussian process prior (P(((()), E({) and multivariate normal posterior NN Ee,d)(n) for t to tz. the posterior for to to t1 is expressed qa (a =
1 = Mtysa + 22 Etgita|R Etzzt, Eigtta Atiit Sigita -
ΣT
where Mt
11 (M
and At,it, - (E - De E
This update combines the new sequence prior from la to la and the previous likelihood used in the update for I to 12, obtained from the posterior and prior. Note, the cross-covariance Eigg is computed using the covariance function of the prior GP. The proof is in Appendix A.
'ee the source code for Nagabandi et al. [11] and MBRL-1ib [57].
5

---

For a stationary kernel, fixed planning horizon and fixed control frequency, the term ttitt is E 2 and is constant, so can be computed at initialization avoid repeated inversion. Figure 2 demon- strates how this update lets us combine our prior with previous posterior in a principled fashion. Moreover, its continuous-tin the time window, for planning at different timescales [30].
construction means that the time resolution can be updated, not just
Compared to the independence assumption, modeling correlations between actions introduces com- plexity. The full covariance over (flattened) time and action has a complexity R(T which is infeasible to work with. Assuming independence between actions, a GP per action has complexity of R(T requiring da GPs to be fit, which is not desirable for online methods such as MPC. To avoid the linear scaling w.r.t. da, we propose using the matrix Normal distribution (Definition 4) for scalability, as it is parameterized into single T and d, dimeensional covariances,
Definition 4. (Matrix Normal Distribution (MaVN) (591) For a random malrix X E RUXP, it fol- lows the distribution X (M,MM, E,E,, where M Rnxp, K E ST. and E S isand only if vec( -vee((((M))) Σ @ for Kronecker product & and (X -
Xx 0}.
Using the Kronecker-structure covariance provides a useful decomposition of the time-based co- variance K, that defines correlations between time steps, and an action covariance Σ that captures correlations between actions. Typically we assume actions are independent, but cross-correlations could be learned from experience for richer coordination. While this Kronecker structure does not fully capture the correlations between time and actions, the structure is very useful for MPC on robotic systems, where the actions space could be very high but the planning horizon is sufficiently small for covariance estimation using a reasonable number of Monte Carlo rollouts.
Feature Approximations. Despite the matrix Normal factorization, computing the correlations be- tween actions still requires a dense H x H covariance matrix K for planning horizon H. To sparsify this quantity, we consider kernel approximations, such as the canonical basis functions Σ,, k(,, x.,) and spectral approximations using random features Σn (-) [60], for a Bayesian linear model Φ, W. Focusing on the squared exponential (SE) kernel. this results in radial basis function (RBF) and random Fourier features (RFF) respectively. Interestingly, RBF features are closely related to probabilistic movement primitives, used extensively in policy search for robotics [32]. For one- dimensional inputs, RFFs are effectively approximated by applying Gauss-Hermice quadrature [61] to the random weights [62]. RBF features and RFFs approximate w.r.t. time and frequency respec- tively and could be combined [63]. Using these continuous-time features, the optimization is now abstracted from planning horizon and control frequency, providing much greater flexibility. Sec- ondly, due to the features, factorized weight covariance approximation does not sacrifice smooth- ness. Moreover, the moment updates described above are not needed, as only Φe is updated.
Related Work
Inference-based control. Posterior policy iteration was proposed by Rawlik al. [41] and cov- ers prior methods developed from Bayesian smoothing [23, 37], expectation maximization [22, 56),
entropy regularization [47, 9] and path integral [64] perspectives. For MPC specifically, the path integral-based MPPI was proposed [18]. with alternative formulations based on mirror descent [46] and variational inference [5. 65, 66]. Mukadam al. [25] models the optimal state-action distri- bution as a sparse Gaussian process and uses lincarization for approximate inference. The same approach is used for Gaussian process motion planning [30], which are also optimized using sam- pling [8]. Gaussian quadrature is also used for inference-basee
MPC [27]. Concurrent work uses the ESS for temperature adjusting heuristic for MPPI [15] and also combines policy search with MPC using PPI techniques [67]. See Section B for a more in- in-depth discussion on these related works.
Policy design and regularization. Smooth actuation is important in robot learning for safety and exploration, having been proposed for Monte Carlo MPC [11, 12, 4] and more broadly incorporated using augmented objectives, parameter sampling and policy design, e.g. [68, 69, 70].
Stochastic search. Probabilistic interpretations of black-box optimization algorithms are well es- tablished [71, 72, 731. however prior work did not connect the ESS and elite samples. CEM and extensions have also been adopted widely as a solver for MPC [3, 4, 6].
Gaussian processes for control. This work adopts GPs for correlated action priors. This is distinct from prior work which uses GPS to approximate dynamics or value functions, e.g. [74, 75, 76, 77].

---

CEM
LBPS
ESSPS
MPPI
PI
HumanoidStandup-y
door-v0
hammer-ve
-105
-104
3
Smoothness, Returnt
1
1,000
2
0.5
2,000
1
0
100
400
400
75
300
300
50
200
200
25
100
100
0
16
32
128 256 512
1024
16
32
64
128
256
512
1024
16
32
128 256 512 1024
Rollout samples
Rollout samples
Rollout samples
Figure 3: MPC return and smoothness with white noise priors. Displaying quartiles over 50 seeds.
These priors require a large number of samples for good performance, across methods and tasks.
6 Experimental Results
We assess the Gibbs posterior methods and policy design empirically across various settings. Black- box optimization (Section 6. considers standard benchmarks, while policy search (Section 6.2) optimizes action sequences for robotic task. For MPC (Section 6.3), evaluate online PPT ap- proaches with white noise and smooth priors on high-dimensional, contact-rich tasks. For the code, see github. com/
6.1 Black-box Optimization
To understand the behaviour of the proposed PPI variants, the performance of LBPS and ESSPS on range of standard black-box optimization functions over a range of hyperparameters, with cREPS and CEM as baselines, these methods, as each solver exhibits consistent ESS for a given hyperparameter value. However, the uniform weights used by CEM (Figure maintain entropy longer than ESSPS (Figure 8), which can lead to better optima, so the ESS is not sufficient to fully capture the behavior of these solvers.
shown in Appendix D.I. Figures 7 10 show that ESS a useful metric for
6.2 Policy Search
As LBPS and ESSPS are closely related to eREPS, we repeat the experiment from prior work perform- ing the 'ball a cup' task using policy search using a Barret WAM [14], which has been shown to transfer to the physical system [78, 14, 79], Matrix normal RBF and REF policies. From the kernel perspective, this feature approximation mo- tivated by the large (T ~ 1000) task horizon. The results in Appendix D.2 confirm that these solvers are all capable of solving the task, based on success rate, where RBF (Figure 11) and RFF features (Figure 12) perform equally well w.r.t. the convergence of the success rate for each approach.
benchmark task. Moreover, replace ProMPS with
6.3 Model Predictive Control with Oracle Dynamies
We evaluate online PPI across range of high-dimensional robotic control tasks in MuJoCo [80], including
in Gym [81] and door-v0, hammer-v0 from mj envs, using the Adroit hand (Figure 1) [82]. To measure smoothness, we adopt the FFT-based score Nf. ΣN azfi [68]. with sampling frequency fs and N resolvable frequencies, / with amplitudes We compute the Euclidean norm of the action sequence over time and apply the smoothness measure to this signal. For the evaluation, timestep. To assess the impact of approximate inference, we assess performance over an logarithmic range of sample rollouts, following prior work [4]. Details may be found in Appendix E.2.
2
focus a low computational budget, with or 2 iterations per
White noise priors. Figure shows MPC with white noise priors using LBPS and ESSPS, with MPPI, CEM and 2r baselines (see Table 1). While each solver performs comparably for 1024 rollouts, the low sample regime shows greater performance variance. While MPPI seems particularly effective, Figure 13 shows that its average ESS is particularly low, = 1 for many cases. Combined with the fixed variances, this suggests optimization is closer to greedy random search than importance sampling, The poor door-v0 performance of ESSPS is due to slow opening, rather than task failure.

---

iCEM (coloured noise)
LBPS (SE kernel)
ESSPS (SE kernel)
MPPI (smooth actions)
MPPI (smooth noise)
MPPI (SE kernel)
SiumanodSStandug
door-v0
hammer-vo
-10*
-104
3
Smoothness, Return
1
4,000
2
0.5
2,000
1
0
0
100
400
400
75
300
50
200
200
25
100
100
0
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
16
32
512 1024 Rollout samples
64
128
256
Rollout samples
Rollout samples
Displaying quartiles over 50 seeds. Compared to white noise priors, smooth action priors improve sample efficiency dramatically, but only PPI methods (LBPS, ESSPS) preserve this smoothness in the downstream control.
Figure 4: MPC return and smoothness with smooth action priors.
Policy design for smooth control. Figure 4 shows online PPI with action priors. LBPS, ESSPS and MPPI use the SE kernel, with iCEM [4] and MPPI with smooth actions and noise as a baseline. The smooth action distributions greatly improves performance across models, tasks and sample sizes, due to effective exploration. As desired, the richer SE kernel provides much greater smoothness, by up to factor of 2 compared to baselines, with limited impact to performance. It is unsurprising that the smoothness bias reduces performance optimal behavior is non-smooth, as illustrated in Figure 2. Appendix D.3.2 shows some of the action sequences from Figure 4, where see GP smoothness varies with increasing rollout samples and also results in significant actuator amplitude reduction.
Comparing kernel- and feature- policies. To assess feature approximations for smooth MPC, we replace the kernel with RBF and RFF features, while keeping the lengthscale fixed. These policies perform worse given fewer samples, but are comparable to the true kernel with sufficient samples (Figure 15). We attribute this to the compounding errors of kernel approximation and fewer effective samples. In contrast to the policy search task, RFFS appear superior to RBF features.
Learning priors from data. A benefit of using GP priors is the ability to optimize hyperparameters from expert demonstrations through the likelihood or a divergence. Moreover, the matrix normal decomposes temporal-
distribution is useful for analyzing high-dimensional action sequences, and action correlations into viewable covariance matrices. Section D.3.3 shows the matrix normal distributions of expert demonstrations of the tasks, obtained through human and RL experts. The results show that, surprisingly, the demonstrations are rougher than the smoothness achievable with MPC. We attribute this control artifacts from demonstration collection and the use of Gaussian noise by RI. agents. Applying the same methodology to the demonstrations of the smooth MPC agents proposed here extracts the expected action correlations across tasks. This analysis also raises the question of whether smoothness is an inductive bias we enforce for practicality. or phenom- ena we expect to arise from optimality. the latter, may be that the simulated environments or objectives considered are lacking components that encourage smoothness, such as energy efficiency.
7 Conclusion
We present broad perspective on episodic posterior policy iteration method for roboties and new methods for the Monte Carlo setting, based on regularizing the IS approximations. By considering vector-valued Gaussian processes for action priors, we have demonstrated how sample-efficient MPC can be performed as online inference and with greater control over actuator smoothness, connecting Monte Carlo MPC to prior work on policy search. This approach was validated on a set of high- dimensional MPC tasks closely matching baseline performance while achieving greater smoothness.
Limitations. Much of the prior work is motivated by simplicity, minimizing hyperparameter tuning and numerical procedures such as matrix inversion [64]. In contrast, the contributions of this work introduces complexity, i.e. online temperature optimization and the use of dense covariance matrices in order to perform more sophisticated approximate inference. While this additional complexity has an impact on execution time (Table 2, Appendix), we hope the sample-efficiency when combined with accelerations such as GPU-integration should produce real-time algorithms [13].
8

---

Acknowledgments
This work built on prior codebases developed by Johannes Silberbauer, Michael Lutter and Hany Abdulsamad. The large-scale experiments and ablations were conducted on the Lichtenberg high performance computer of the TU Darmstadt. The authors wish to thank Hany Abdulsamad, Boris Belousov, Michael Lutter, Fabio Muratore, Pascal Klink, Georgia Chalvatzaki, Kay Hansel, Oleg Arenz and the anonymous conference reviewers for helpful feedback on earlier drafts.
References
[1] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 2018.
[2] A. B. Owen. Monte Carlo theory, methods and examples. 2013.
[3] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
[4] C. Pinneri, S. Sawant, S. Blaes, J. Achterhold, J. Stueckler, M. Rolinek, and G. Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learn- ing, 2020.
[5] M. Okada and T. Taniguchi. Variational inference MPC for Bayesian model-based reinforce- ment learning. Conference on Robot Learning. 2019.
[6] M. Lutter, L. Hasenclever, A. Byravan, G. Dulac-Amold, P. Trochim, N. Heess, J. Merel, and Y. Tassa. Learning dynamics models for model predictive agents. arXiv preprint 1rXiv:2109.13311, 2021.
[7] F. Stulp, E. Theodorou, J. Buchli, and S. Schaal. Learning to grasp under uncertainty. In 2011 IEEE International Conference on Robotics and Automation, 2011.
[8] M. Kalakrishnan, S. Chitta, E. Theodorou, P. Pastor, and S. Schaal. STOMP: Stochastic tra- jectory optimization for motion planning. In 2011 IEEE International Conference on Robotics and Automation, 2011.
[9] C. Daniel, G. Neumann, O. Kroemer, J. Peters, al. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 2016.
[10] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. information-theorede model predictive control: Theory and applications to autonomous driving. IEEE Transactions on Robotics, 2018.
[11] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, 2019.
[12] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani. Data-efficient reinforce- ment learning for legged robots. In Conference on Robot Learning. 2019.
[13] M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots. STORM: An integrated framework for fast joint-space model predictive control for reactive manipulation. In Conference on Robot Learning, 2021.
[14) M. Lutter, J. Silberbauer, J. Watson, and J. Peters. Differentiable physics models for real-world offline model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, 2021.
[15] Carius, R. Ranftl, F. Farshidian, and M. Hutter. Constrained stochastic optimal control with learned importance sampling: A path integral approach. The International Journal of Robotics Research, 2022.
[16] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, Peters, and Schmidhuber. Natural evolu- tion strategies. Journal of Machine Learning Research, 2014.

---

[17] A. Abdolmaleki, R. Lioutikov, J. R. Peters, N. Lau, L. Pualo Reis, and G. Neumann. Model- based relative entropy stochastic search. In Advances Neural Information Processing Sys- lems, 2015.
[18] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 2017.
[19] R. Y. Rubinstein and D. P. Kroese. The Cross Entropy Method: Unified Approach To Com- binatorial Optimization, Monte Carlo Simulation. 2004.
[20] P. Dayan and G. E. Hinton. Using expectation iiciiion-maimmzzaiiinin for reinforcement learning, Neu- ral Computation, 1997.
[21] M. Toussaint and A. Storkey, Probabilistic inference for solving discrete and continuous state Markov decision processes. In International Conference of Machine Learning, 2006.
[22] J. Peters and S. Schaal. Reinforcement learning by reward weighte regression for operational space control. In International Conference on Machine Learning, 2007.
[23] M. Toussaint. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning, 2009.
[24] H. J. Kappen, V. Gómez, and M. Opper. Optimal control as a graphical model inference problem. In International Conference on Automated Planning and Scheduling, 2013.
[25] M. Mukadam, C. Cheng, X. Yan, and B. Boots. Approximately optimal continuous-time mo- tion planning and control via probabilistic inference. In IEEE International Conference on Robotics and Automation, 2017.
[26] S. Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. 2018.
arXiv preprint
[27] J. Watson, H. Abdulsamad, R. Findeisen, and J. Peters. Efficient stochastic optimal control through approximate Bayesian input inference. arXiv preprint arXiv:2105.07693, 2021.
[28] H. J. Kappen and H. C. Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 2016.
[29] M. P. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foun- dations and Trends in Robotics, 2013.
[30] M. Mukadam, J. Dong, X. Yan, F. Dellaert, and B. Boots. Continuous- time Gaussian process motion planning via probabilistic inference. The International Journal of Robotics Research, 2018.
[31] T. Flash and N. Hogan. The coordination of arm movements: An experimentally confirmed mathematical model. Journal of Neuroscience, 1985.
[32] A. Paraschos, C. Daniel, J. Peters, G. Neumann, et al. Probabilistic movement primitives. In Advances in Neural Information Processing Systems, 2013.
[33] C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[34] E. Theodorou, Y. Tassa, and E. Todorov. Stochastic differenniall dynamic programming. In American Control Conference, 2010.
[35] A. Zellner. Optimal information processing and Bayes's theorem. The American Statistician,
1988.
[36] M. E. Khan and H. Rue. The Bayesian learning rule. arXiv preprint arXiv:2107.04562 2021.
[37] J. Watson, H. Abdulsamad, and J. Peters. Stochastic optimal control as approximate input inference. In Conference on Robot Learning, 2019.
[38] B. Guedj. primer on PAC-Bayesian learning. In Proceedings of the second congress of the French Mathematical Society, 2019.
10

---

[39] P. Alquier. J. Ridgway, and N. Chopin. On the properties of variational approximations of Gibbs posteriors. Journal of Machine Learning Research, 2016.
[40] B. Dai, N. He, H. Dai, and L. Song. Provable bayesian inference via particle mirror descent. In International Conference on Artificial Intelligence and Statistics, 2016.
[41] K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Robotics: Science and Systems, 2012.
[42] Watson and J. Peters. Advancing trajectory optimization with approximate inference: Ex- ploration, covariance control and adaptive risk. In American Control Conference, 2021.
[43] K. C. Rawlik. On probabilistic inference approaches to stochastic optimal control. PhD thesis, The University of Edinburgh, 2013.
[44] C. Wirth, J. Fürnkranz, and G. Neumann. Model-free preference-based reinforcement learning.
In Conference on Artificial Intelligence, 2016.
[45] S. Satoh, H. J. Kappen, and M. Saeki. An iterative method for nonlinear stochastic optimal control based on path integrals. IEEE Transactions on Automatic Control, 2017.
[46] N. Wagener, C. an Cheng, J. Sacks, and B. Boots. An online learning approach to model predictive control. In Robotics: Science and Systems, 2019.
[47] J. Peters, K. Mülling, and Y. Altün. Relative entropy policy search. In Conference on Artificial Intelligence, 2010.
[48] J. Campenhout and T. Cover. Maximum entropy and conditional probability. IEEE Trans- actions on Information Theory, 1981.
[49] A. M. Metelli, M. Papini, N. Montali, and M. Restelli. Importance sampling techniques for policy optimization. Journal of Machine Learning Research, 2020.
[50] A. Kong. A note importance sampling using standardized weights. University of Chicago, Dept. of Statistics, Technical Report, 1992.
[51] C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. In Advances in Neural Information Processing systems, 2010.
[52] S. Agapiou, O. Papaspillipoollos, D. Sanz-Alonso, and A. M. Stuart. Importance sampling: Intrinsic dimension and computational cost. Statistical Science, 2017.
[53] J. Hermández-Gonzákee and J. Cerquides. A robust solution to variational importance sampling of minimum variance. Entropy, 2020.
[54] V. Gómez, H. J. Kappen, J. Peters, and G. Neumann. Policy search for path integral control. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2014.
[55] H. van Hoof, G. Neumann, and J. Peters. Nan-paramctric policy search with limited informa- tion loss. Journal of Machine Learning Research, 2017.
[56] J. Kober and J. Peters. Policy search for motor primitives in robotics. In Advances in Neural Information Processing Systems, 2009.
[57] L. Pineda, B. Amos, A. Zhang, N. O. Lambert, and R. Calandra. MBRL-Lib: A modular library for model-basee reinforcement learning. arXiv preprint arXiv:2104.10159 2021.
[58] A. Doucet. A note on efficient conditional simulation of Gaussian distributions. Departments of Computer Science and Statistics, University of British Columbia, 2010.
[59] A. P. Dawid. Some application. Biometrika, 1981.
distribution theory: notational considerations and a Bayesian
[60] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007.

---

[61] F. B. Hildebrand. Introduction to numerical analysis. Courier Corporation, 1987.
[62] M. Mutny and A. Krause. Efficient high-dimensional Bayesian optimization with additivity and quadrature Fourier features. In Advances in Neural Information Processing Systems, 2018.
[63] Wilson, Borovitskiy, A. Terenin, P. Mostowsky, and M. Deisenroth. Efficiently sampling functions from Gaussian process posteriors. In International Conference on Machine Learning, 2020.
[64] E. Theodorou, J. Buchli, and S. Schaal. A generalized path integral control approach to rein- forcement learning. The Journal of Machine Learning Research, 2010.
[65] A. Lambert, A. Fishman, D. Fox, B. Boots, and F. Ramos. Stein variational model predictive control. In Conference on Robot Learning, 2020.
[66] Z. Wang, O. So, J. Gibson, B. Vlahov, M. S. Gandhi, G.-H. Liu, and E. A. Theodorou. Varia- tional inference MPC using Tsallis divergence. In Robotics: Science and Systems, 2021.
[67] Y. Song and D. Scaramuzza. Policy search for model predictive control with application to agile drone flight. IEFE Transactions on Robotics, 2022.
[68] S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko. Regularizing action policies for smooth control with reinforcement learning. In IEEE International Conference on Robotics and Au- tomation, 2021.
[69] A. Raffin, J. Kober, and F. Stulp. Smooth exploration for robotic reinforcement learning. In Conference on Robot Learning, 2021.
[70] D. Korenkevych, R. Mahmood, G. Vasan, and J. Bergstra. Autoregressive policies for continuous control deep reinforcement learning. In International Joint Conference on Artificial Intelligence, 2019.
[71] Stulp and Sigaud. Path integral policy improvement with covariance matrix adaptation. In International Conference on Machine Learning, 2012.
[72] Y. Ollivier, L. Arnold, A. Auger, and N. Hansen. Information-geomeni optimization algo- rithms: A unifying picture via invariance principles. Journal of Muchine Learning Research, 2017.
[73] A. Abdolmaleki, B. Price, N. Lau, L. P. Reis, and G. Neumann. Deriving and improving CMA- ES with information geometric trust regions. In Proceedings of the Genetic and Evolutionary Computation Conference, 2017.
[74] F. Berkenkamp and A. P. Schoellig. Safe and robust learning control with Gaussian processes. In European Control Conference, 2015.
[75] J. Kocijan, R. Murray-Smith, C. Rasmussen, and A. Girard. Gaussian process model based predictive control. In American Control Conference, 2004.
[76] M. Deisenroth, C. Rasmussen, and J. Peters. Gaussian process dynamic programming. Neuro- computing, 2009.
[77] M. Maiworm, D. Limon, J. Maria Manzano, and R. Findeisen. Stability of Gaussian process learning based output feedback model predictive control. In IFAC Conference on Nonlinear Model Predictive Control, 2018
[78] P. Klink, H. Abdulsamad, B. Belousov, C. D D'Eramo, J. Peters, and J. Pajarinen. A probabilistic interpretation of self self-paced learning with applications to reinforcement learning. Journal of Machine Learning Research, 2021.
[79] F. Muratore, C. Eilers, M. Gienger, and J. Peters. Data-efficient domain randomization with Bayesian optimization. IEEE Robotics and Automation Letters, 2021.
[80] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physies engine for model-based control. In IEEE International Conference on Intelligent Robots and Systems, 2012.
12

---

[81] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym, 2016.
[82] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Robotics: Science and Systems, 2018.
[83] B. D. Anderson and J. B. Moore. Optimal filtering. 2012.
[84] T. D. Barfoot, C. H. Tong, and S. Särkkä. Batch continuous-time trajectory estimation as exactly sparse Gaussian process regression. In Robotics: Science and Systems, 2014.
[85] O. Cappé, S. J. Godsill, and Moulines. An overview of existing methods and recent advances in sequential Monte Carlo. Proceedings of the IEEE, 2007.
[86] K. Asadi and M. L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning. 2017.
[87] C. Chu, J. Blanchet, and P. Glynn. Probability functional descent: A unifying perspective GANs, variational inference, and reinforcement learning. In International Conference Machine Learning, 2019.
[88] A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 2003.
[89] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg. B. Boots, and E. A. Theodorou. tnformition-theeretiic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, 2017.
[90] Virtanen, R. Gommers, E. Oliphant, M. Haberland, Reddy, Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. Jarrod Millman, N. Mayorov, A. R. Nelson, E. Jones, R. Kern, E. Larson, C. Carey, i. Polat, Y. Feng, E. W. Moore, J. Vand erPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Hen- riksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, van Mulbregt, and S... Contribbbuu.. SciPy 1.0: Fundamental Algorithms for Scientific Comput- ing in Python. Nature Methods, 2020.
[91] P. Dutilleul. The MLE algorithm for the matrix normal distribution. Journal of Statistical Computation and Simulation, 1999.
[92] L. Peng, H. Qian, Shen, C. Zhang, and F. Li. Generative actor-critic; An off-policy algorithm using the push-forward model. arXiv preprint arXiv:2105.03733 2021.
[93] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.
[94] S. Särkkä and A. Solin. Applied Stochastic Differential Equations. Institute Mathematical Statistics Textbooks. Cambridge University Press, 2019.
13


Reviews:
[{"review_id": "xeQXm4YplcC", "reviewer": "Reviewer_9hau", "summary": "The paper proposes a novel way to perform inference-based control with Monte Carlo sampling methods in online sequential inference using Gaussian Process priors to promote smoothness in both action priors and samples. The novelty lies in how the action distribution is factorized and the introduction of a theoretically grounded approach to optimize the likelihood temperature.\n\nThe relationship of the proposed method and prior work, such as CEM and MPPI is well presented and explored. In fact, sections 2-4 describing the background and method are laid out very clearly and with a very solid theoretical setting. Prior work is reframed in the same statistical interpretation and provide an insightful interpretation of their connection to the current work. Additionally, the paper considers a extensive number of relevant previous works, close to a literature review on sampling-based control as inference.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written and has extensive background research. The content is delivered in a clear and logical progression and there are several simulated results. The contribution is also relevant and although both LBPS and ESSPS still have hyperparemeters to control how the temperature is optimized (respectively the probability of the bound and effective sample size) these offer a more intuitive selection. Lastly, the authors present a feasible implementation and address how the proposed method could be efficiently implemented for online use.\n\nOn the other hand, I feel the motivation for the desirability of smooth priors and actions could be stronger. At passages there seems to be a lack of distinction between correlation and smoothness, and while correlation is important for a coordinated exploration of the sampling space, it was not clear or well referenced when smooth controls are necessary. In fact, it is known that for minimum-time control problems where the Hamiltonian is linear w.r.t. the control variable, the optimal solution will be a bang-bang controller and many lower-level actuators already cope with abrupt changes in the control signal. Furthermore, on some of the simulation tasks, e.g. black-box optimization, there seems to be a gap in performance which is not attributable to feature approximations and it would be valuable to have a possible explanation from the authors. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I strongly recommend that the paper be accepted for oral presentation. Despite my criticism on the motivation above, the contributions are clear and relevant and the theoretical framework is very well presented. The method is validated through simulation results only, but is similar enough in nature as to accept that it would be extendable to hardware where the baselines have been applied before. This is further emphasized when considering the computational performance comparison listed on supplementary material.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I have the following minor corrections and/or recommendations:\n- In line 97 \"from\" should read \"form\";\n- In line 129 \"Reny\" should read \"Renyi\";\n- In the equation in line 154, I believe the first integral term should be the\n  joint distribution and not the marginal;\n- The claim in line 159 warrants a citation;\n- Reference [13] points to a paper published on CoRL 2022;\n- Please identify what the parameter `k` refers to on Figures 5 and 6;\n- On Figures 9 and 10 are rollouts referring to the cumulative sampled rollouts\n  or episodes?"}}, {"review_id": "ZJZiBgiA5l2", "reviewer": "Reviewer_vNLR", "summary": "The paper studies episodic posterior policy iteration method, specifically the questions of choosing the temperature hyperparameter of the Gibbs likelihood and inducing smoothness. The approach suggested by the authors exhibits better behavior in terms of smoothness and regret than the baselines considered.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Disclaimer**. I may lack a sufficient background in control, my primary expertise is in Gaussian processes. These are a part of the paper present even in the title, but it is not the central part. I apologize for any poor judgment that may follow in my review, I keep my confidence score very low to reflect this.\n\nHaving that said, I spent an honest 7 hours trying to comprehend the paper, without much success. The paper builds on a huge amount of prior work (there are almost a hundred references!) and does not seem to introduce the necessary background well. To be honest, on several occasions it seemed to me as if the paper was specifically composed by putting as much of the specialized terminology/methods or claims from the literature on an inch of paper as it is possible.\n\nLet me be more specific. First, at the end of page 2, the Gibbs posterior is introduced. It is immediately cast to an optimization problem. Then, on page 3, this optimization problem is replaced by another, now a constrained optimization. So far so good, I lose track after, at: “we reverse the objective and constraint and adopt an equality constraint”. Why do we do this? Then the argument proceeds by approximating some constraints (several times), replacing something by its lower bound in the concentration inequality, all this using multiple results from the literature. And in the end it is absolutely not clear what we have obtained. Are we now able to automatically tweak the temperature hyperparameter? Did we prove/infer something or was the argument just a heuristic motivation? In the former case, it obviously lacks rigor. In the latter case, it lacks motivation and explanations for each of the steps. If these would have taken up too much space, maybe it is worth extending the paper to the full-sized journal manuscript? Otherwise, maybe formulating some of the results as theorems may help, or explicitly stating the intermediate results and their meaning in the context of the whole paper.\n\nOn the other hand, I would like to mention that a software implementation is in the supplementary, which I think is a huge plus.\n\n**Small Comments**\n1. Line 12. “for to” -> “to”.\n2. Figure 1. There is no legend, it is not clear e.g. what a_0 is. Furthermore, this figure is first referred to from the main text on page 8. For a reader who only just read page 1, where the figure is located, it is not clear what the abbreviations mean.\n3. Equation (1). Is there a reason to write “a_1, a_2, … ” instead of “a_1, a_2, …, a_T”? The former notation is usually reserved for infinite sequences.\n4. Line 56. Maybe “setting, performs” -> “setting, i.e. performs”?\n5. Line 57. In “R(S, A)”, the “S” was not formally defined.\n6. Line 61. “Q” is not defined.\n7. Line 63. Probably “in an episodic return” -> “is an episodic return”.\n8. Line 121: “is an hyperparameter” -> “is a hyperparameter”.\n9. Equation in Lemma 2. \\mathbb{D}_2 seems to be undefined.\n10. Line 144. Maybe “posterior to one” -> “posterior to the one”?\n11. Line 242. Maybe “is an useful” -> “is a useful”?\n12. Figures 3 and 4. It would be good to explain what smoothness means in the caption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper was extremely hard to read for me and after spending 7 hours trying to understand it, I failed. It may very well be caused by a misalignment between my expertise and the subject of the paper, hence my low confidence. However, I cannot vote for acceptance of a paper I am sure I have not understood.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Clarity. See the “Strengths And Weaknesses” part of the review.", "reviewer_expertise": "1: The reviewer's evaluation is an educated guess"}, "extra_text": {}}, {"review_id": "LPaiLkH1FGE", "reviewer": "Reviewer_VNXb", "summary": "This paper proposed a novel method for smooth control in Model Predictive Control (MPC). It achieves better smoothness by incorporating Gaussian Process action priors in inference-based control. In order to reduce computation in GP, it factorized the covariance matrix by matrix Normal distribution [58]. Also, it proposed two new algorithms (LBPS and ESSPS) for optimizing the temperature in the Gibbs posterior, based on regularized Importance Sampling (IS) approximations. The effectiveness of the proposed algorithms as well as the smoothness are verified in simulation benchmark tasks.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Novel use of GP for action prior with efficient implementations for high dimensional control\n- Abundant ablation studies\n- The proposed methods were able to achieve a better smoothness empirically\n- Leveraged the IS lower bound [48] to form a new objective for temperature adaptation. The proposed perspective of inference-based control based on Gibbs posterior unified some previous methods\n\n### Weaknesses\n\n**Experiments** \n\nThe experiments are based on having access to the ground truth model but this setting is only vaguely mentioned in the last two lines of the paper… “vague” as in “future work will consider… additional consideration of dynamics approximation”. Please be explicit about the setting.\n\nLIne 271-272: SE leads to better smoothness “by a factor of 2 compared to baselines”, but it doesn’t seem to be the case for MPPI in Fig.4. Am i misinterpreting it?\n\nLine 275 discusses the results from Fig.13. But the figure does not show SE Kernel results despite having it in the legend.\n\n**Presentation**\n\nThe paper appears a bit unorganized. It covers a lot of technical background as well as the technical contribution but I feel that the contribution is a bit scattered throughout the text. As a result, the contribution of the paper is a bit obscured and takes a lot of time for the reader to parse.\n\n**Methods**\n\n- I missed it but I did not find the descriptions on how to choose N* in ESSPS.\n- Also, I could not fnid how the SE kernel parameters were chosen.\n\n**Minor**\n\n- Fig. 5,6,7,8: what do the parameters mean?\n\n- The adaptive temperature strategy in Sec. 3 appears to be a bit disconnected from the policy iteration algorihtm, without reading further into Appendix A and B. It’s partly from the switch of notations at the beginning of Sec 4 (using O to represent the Gibbs likelihood). It would be better to explicitly write it out. \n\n- Line 685: should -> showed?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper seems to contain decent contribution: inference-based control perspective based on Gibbs posteriors and new methods for temperature adaptation and smooth control. \nI think that the presentation could be improved though, as explained in the weakness section.\n\n======= \npost-rebuttal\n=======\nI've raised the score to strong accept to reflect what I mentioned earlier in the rebuttal response.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "Please make the setting regarding the use of GT model clear.\n\nPlease see the weaknesses section for other issues.\n\nQuestion: \nPerhaps i misunderstood it, but I thought that the GPMP [30] only used GP to model state sequence. But the paper suggested (below line 158) that it was used to model the policy. Is that right? \n"}}, {"review_id": "Ghk3Z0knkFU", "reviewer": "Reviewer_qNLg", "summary": "This paper considers Monte Carlo based optimal control through the lens of inference-based control whilst considering\nthe policy search, motion planning, and model predictive control (MPC) settings.\nThe paper has two main theoretical contributions.\nFirstly, they introduce an adaptive scheme for dynamically setting the likelihood temperature parameter $\\alpha$ \nin the Monte Carlo setting.\nTheir method is based upon optimising a lower bound on the expected return of the posterior based on the \nimportance sampled estimate using the prior.\nSecondly, they propose a method for smoothing action sequence samples by considering a continuous-time Gaussian process \nover the actions. \nIn contrast to the previous Monte Carlo approaches cited in the paper, their method produces smooth action trajectories\nby capturing correlations between the time steps.\nThey test their methods in multiple simulated environments.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths**\n\n1. The adaptive temperature scheme presented in this paper is well grounded theoretically whilst also performing well experiments.\n    - Their method can be used for MPC, motion planning and policy search.\n2. Their approach for modelling action sequences clearly results in smooth action sequences.\n    - First time I have seen RFF features used to learn a policy.\n3. Although I have less experience with Monte Carlo inference-based control, \n    the paper appears to cite the relevant literature well. \n4. I liked how the authors draw connections between methods that use the Gibbs posterior (Bayesian smoothing, maximum entropy, mirror descent and entropy-based RL).\n\n**Weaknesses**\n\n\n1. The motivation is not immediately clear. \n    - The authors state that adaptive temperature schemes are crucial for optimization and that in PPI it has a strong influence on the posterior.\n    However, they do not provide any references or intuition.\n    - I think this forms a large motivation for the paper.\n    - Interestingly, the motivation for the adaptive temperature scheme on their website is very intuitive and easy to follow.\n    - I think Section 2 should contain a figure similar to the first figure from the website as it helps provide intuition about how the temperature $\\alpha$ effects the posterior $q_{\\alpha}$.\n\n2. I found Sections 2 and 3 are hard to follow.\n    - I think they need more signposting to keep the story consistent throughout Sections 2 and 3.\n    - For example, Section 2 ends abruptly and I wasn't too sure about the main takeaways.\n        - Why not add something like this to the start of section 3.\n\n        > \"In this section we introduce our method for adaptively setting the likelihood parameter. It builds on the connection between  INSERT CONCISE DESCRIPTION OF SECTION 3\"\n\n   - Further to this, I wasn't too sure why we were jumping from Theorem to Lemma in Section 3.\n    I think the authors need to restructure this section or at least offer more signposting to help the reader through\n    a section so heavy on mathematics.\n    - I would be tempted to put Equation 5 at the start of section 3, then explain the meaning of the terms and \n    show where they come from / how they are derived.\n\n3. There are a few places where the authors offload work to the reader. \n    For example, I often had to refer to two or three figures, \n    often on different pages to understand a single statement made in the experiments section.\n    Further to this, I often had to figure out what (sub)figures to check.\n    Some examples:\n    - Section 6.2 the authors write,\n\n        > \"The results in Appendix D.2 confirm that these solvers are all capable of solving the task, while RBF and RFF features perform equally well\"\n\n        - What exactly should I be looking at? The top rows of Figure 5:8 showing the expected return converging? Assuming this is what I'm meant to be looking at, why not write something like:\n\n        > \"Figures 5:8 in Appendix D.2 show the expected return converging for all of these solvers (for at least some parameter settings), which confirms they are all capable of solving the task. Also note that RBF and RFF features perform equally well.\"\n\n    - Line 275: \"Figure 13 replaces the SE kernel with...\"\n        - To understand this paragraph I had to read Figure 13 in the appendix to understand that the first two sentences were referring to MPC.\n        - Why not write something like this:\n\n        > \"Figure 13 shows MPC results with finite feature approximation where the SE kernel has been replaced with RBF and RFF features.\"\n\n    - Section 6 first paragraph: \"Black box optimization compares...\" - I had no idea this was referring to Section 6.1 until my second read of the paper. Similarly for \"Policy search compares...\". Consider changing these to \"In Section 6.1 we compare\" etc.\n\n4. I found the equations for Lines 174:179 hard to follow. Why not tell the punch line first? Introduce the equation you want (equation 6) and then explain what the different terms are. In general, this writing style is much easier to follow. \n\n5. The approach for obtaining smooth controls presented in this paper is elegant from a probabilistic\n    modelling perspective, however, I am curious how it compares to encoding smoothness into the reward/cost function.\n    For example, adding a quadratic cost term on the controls is commonly used in control to encode the notion of minimum \n    effort/energy and is really simple to implement.\n    I am not suggesting the method presented in this paper is not a useful approach, I am more suggesting that the authors\n    should highlight how their method differs from (and hopefully improves on) adding an extra cost term.\n    The authors touch on this in lines 289/290 but I am interested if they have any further thoughts?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think this paper presents some nice work.\nTheir lower bound for optimizing the temperature seems to be grounded well from a theoretical perspective\nand the results suggest that it works well in simulation.\nWhilst I am less sure about the novelty of modelling the action sequence with continuous-time Gaussian processes,\ntheir method does achieve smooth control and I liked seeing results using GP feature approximations.\n\nAlthough the work seems strong, the paper suffers from two limitations.\nFirstly, I do not think the need for an adaptive temperature scheme in PPI is introduced very well.\nInterestingly, the motivation for the adaptive temperature scheme on their website is very intuitive and easy to follow.\nThe authors should consider putting some of this into the paper.\n\nSecondly, I found the paper difficult to read.\nThis may be due to the amount of work the authors are trying to fit in and the quite obvious lack of space.\nNevertheless, I think this needs to be addressed.\nIn particular, I think Section 2 ends abruptly as I wasn't too sure what I was meant to take away from Section 2.\nFurther to this, the start of Section 3 does not detail what the section is going to be about.\nAs a result, I ended up being lost whilst jumping from Theorem to Lemma.\nI think the authors need to restructure this section or at least offer more signposting to help the reader.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. Introduce the need for an adaptive temperature scheme more clearly.\n    - Perhaps use a figure similar to the ones used on the website?\n\n2. Figure 1 is never referenced. This figure is a nice motivation for smooth action sequences so use it!\n\n2. Add more signposting or restructure Sections 2 and 3 to make them easier to follow.\n    - There's a lot of maths here. Hold my hand and guide me through it...\n    - I am a big believer in \"telling the punch line first\" when it comes to academic writing. I'd encourage the authors to try and follow this approach here.\n\n3. When making statements about observations from figures (especially in the results section), walk me through what I should be looking at and make the statements so informative that I don't actually need to refer to the figure(s).\n\nMinor fixes:\n\n- Lemma 1: Does $\\lambda$ represent Lagrange multipliers? This should be explicitly stated.\n- Figure 1: Add x-axis ticks for reward. I had to use a ruler to compare iCEM and PPI.\n- Line 4: missing \"s\" from \"optimization\"\n- Line 12: remove \"for\"\n- Line 53: \"optimal setting\" should be \"optimal control setting\"?\n- Line 55: maximising the expected reward?\n- Line 63: Consider replacing \"concerns\" with \"considers\"\n- Line 63: \"open-look\" should be \"open-loop\"?\n- Line 63: \"f in an\" should be \"f is an\"?\n- Line 99: \"This approach is an\" should be \"This approach is a\"\n- Line 129: \"Reny-2\" should be \"Renyi-2\"?\n- Line 134: Add equation number for IS lower bound \n- Line 154: Introduce $\\mathcal{O}$ as a binary random variable $\\mathcal{O} \\in \\{0, 1\\}$ where $\\mathcal{O}_{t}=1$ indicates time step $t$ being optimal and that you denote $\\mathcal{O}_{1:t} = \\{\\mathcal{O}_i=1\\}_{i=1}^{t}$. Readers unfamiliar with inference-based control may be confused otherwise.\n- Equation below Line 154: time indices look wrong. If you want $q_{\\alpha}(\\alpha_{t:t+H} | ...)$ then shouldn't the integral be over $\\text{d}\\alpha_{1:t-1}$?\n- Line 155: should $h$ be $H$?\n- Line 208: \"approximate applying\" should be \"approximate by applying\"?\n- Line 218: \"popularized [18]\" should be \"popularized in [18]\"?\n- Line 219: Introduce acronym \"PIPC\"\n- Line 240-242: Sentence doesn't read well. Should it say something like: \"To understand the behaviour of the proposed posteriors as solvers, the performance results of LBPS and ESSPS on a range of standard black-box optimization functions over a range of hyperparameters, with e REPS and CEM as baselines, are shown in Appendix D.1.\"\n- Line 242: \"is an useful\" should be \"is a useful\"\n- Line 263: Introduce acronym \"$\\text{PI}^2$\"\n- Line 276: \"lengthscale.\" should be \"lengthscale fixed.\"?\n\n- Line 11, 64, 223, 228, 285: No comma before \"and\"\n"}}]

Author rebuttal:

Decision: Accept (oral)