Learning Sampling Distributions for Model Predictive Control
Authors: Jacob Sacks
Byron Boots
Venue: CoRL 2022 Poster (2022)
Keywords: Model Predictive Control
Normalizing Flows

Abstract: Sampling-based methods have become a cornerstone of contemporary approaches to Model Predictive Control (MPC), as they make no restrictions on the differentiability of the dynamics or cost function and are straightforward to parallelize. However, their efficacy is highly dependent on the quality of the sampling distribution itself, which is often assumed to be simple, like a Gaussian. This restriction can result in samples which are far from optimal, leading to poor performance. Recent work has explored improving the performance of MPC by sampling in a learned latent space of controls. However, these methods ultimately perform all MPC parameter updates and warm-starting between time steps in the control space. This requires us to rely on a number of heuristics for generating samples and updating the distribution and may lead to sub-optimal performance. Instead, we propose to carry out all operations in the latent space, allowing us to take full advantage of the learned distribution. Specifically, we frame the learning problem as bi-level optimization and show how to train the controller with backpropagation-through-time. By using a normalizing flow parameterization of the distribution, we can leverage its tractable density to avoid requiring differentiability of the dynamics and cost function. Finally, we evaluate the proposed approach on simulated robotics tasks and demonstrate its ability to surpass the performance of prior methods and scale better with a reduced number of samples.

Learning Sampling Distributions for Model Predictive Control
Jacob Sacks
Byron Boots University of Washington
University of Washington
bboots@cs washington edu
Abstract: Sampling-based methods have become a cornerstone of contemporary approaches to Model Predictive Control (MPC), as they make no restrictions on the differentiability of the dynamics or cost function and are straightforward to parallelize. However, their efficacy is highly dependent on the quality of the sam- pling distribution itself. which is often assumed to be simple. like Gaussian. This restriction can result in samples which are far from optimal, leading to poor perfor- mance. Recent work has explored improving the performance of MPC by sampling in a learned latent space of controls. However, these methods ultimately perform all MPC parameter updates and warm- warm-starting between time steps in the control space. This requires us to rely on number of heuristics for generating samples and updating the distribution and may lead to sub-optimal performance. Instead, we propose to carry out all operations in the latent space, allowing us to take full advantage of the learned distribution. Specifically, we frame the learning problem as bi-level optimization and show how to train the controller with backpropagation through-time time. By using normalizing flow parameterization of the distribution, we can leverage its tractable density to avoid requiring diffcrentiability of the dynamics and cost function. Finally, evaluate the proposed approach simulated robotics tasks and demonstrate its ability to surpass the performance of prior methods and scale better with reduced number of samples.
Keywords: Model Predictive Control, Normalizing Flows
Introduction
Sequential decision making under uncertainty is fundamental problem in machine learning and robotics. Recently, model predictive control (MPC) has emerged as a powerful paradigm to tackle such problems real-world robotic systems. In particular, MPC has been successfully applied to helicopter aerobatics [1]. robot manipulation [2, 3, 4], humanoid robot locomotion [5], robot- assisted dressing [6], and aggressive off-road driving [7. 8, 9]. Sampling-basee approaches MPC are becoming particularly popular due to their simplicity and ability to handle non-di non-differentiable dynamics and cost functions. These methods work by sampling controls from a policy distribution and rolling out an approximate system model using the sampled control sequences. They then use the resulting trajectories to compute an approximate gradient of the cost function and update the policy. Next, the controller samples an action from this distribution and applies it to the system. It repeats the process from the resulting next state, creating a feedback controller. Between time steps, it warm-starts the optimization process using modification of the solution at the previous time step.
An important design decision is the form of the sampling distribution, which is often simple, e.g. a Gaussian, such that we can efficiently sample and tractably update its parameters. However, this also has drawbacks: without much control over the distribution form, samples often lie in high-cost regions, hindering performance. This can be particularly problematic in complex environments with sparse costs or rewards, as a poorly parameterized distribution may hinder efficient exploration, leading the system into bad local optima. A side effect is that we often require many samples to accomplish the objective, increasing computational requirements. There have been extensions which target more complex distributions, such as Gaussian mixture models [10] and particle method based on Stein variational gradient descent (SVGD) [11]. However, there is large amount of structure in the environment that these methods fail to exploit. Instead, an alternative approach is to learn sampling distribution which can leverage the environmental structure to draw more optimal samples.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Prior work on learning MPC sampling distributions generally requires differentiability of the dynamics and cost function [12. 13]. Power and Berenson [14, 15] circumvent this issue by leveraging normalizing flows (NFs) [16, 17, 18], which have tractable log- fog-likelihood. This property allows them to learn flexible distributions by directly optimizing the MPC cost without requiring differentiability via the likelihood-ratio gradient. However, limitation of their approach is that all online updates to the distribution and warm-starting between time steps occur entirely in the control space, leaving the latent distribution fixed. This forces them to apply heuristics to generate samples by combining those from the learned distribution with Gaussian perturbations to the current control-space mean. These restrictions prevent us from fully taking advantage of the learned distribution and potentially throws away useful information. Additionally, their approach does not allow for the incorporation of control constraints directly into the sampling distribution, which is important for real- world robots.
Instead, we propose to alter the optimization machinery to operate entirely in the latent space. As the NF latent space follows simple distribution, it remains feasible to perform MPC updates in this learned space and update the latent distribution online. Specifically, during an episode, the parameters of the latent distribution are updated with MPC while those of the NF remain fixed. Then during training, after each episode, the parameters of the NF are updated. We can frame this setup as a bi-level optimization problem [19] and derive method for computing an approximate gradient through the latent MPC update. This involves treating MPC as a recurrent network, where the control distribution acts a form of memory, and unrolling the computation to train with backpropagation- through-time timetttmmeee (BPTT). However, it is no longer clear how to warm-start between time steps because there is no clear delincation of time in the latent space. Moreover, the usual method of warm-starting, which simply shifts the current plan forward in time, may be sub-optimal. Therefore, we propose to learn shift model, which performs all warm-starting operations in the latent space. Finally, we show how to alter the NF architecture to incorporate box constraints on the sampled controls.
Contributions:
In this work, we build upon recent efforts to learn sampling distributions for MPC with NEs by moving all online parameter updates and warm-start operations into the latent space. We accomplish this by framing the learning problem as bi-level optimization and derive an approximate gradient through the MPC update of the latent distribution to train the network with BPTT. Addition- ally. we show how to parameterize the flow architecture such that we can incorporate box constraints on the controls. Finally, we empirically evaluate our proposed approach on simulated navigation and manipulation tasks. We demonstrate its ability to improve performance over the baselines by taking full advantage of the learned latent space. We find that the performance of the controllers with our learned sampling distributions scales more gracefully with reduction in the number of samples.
2 Sampling-Based Model Predictive Control
We consider the problem of controlling a discrete-time stochastic dynamical system, which is in state Ie RVV at time step t. Upon the application of control u, e M, the system incurs the instantaneous cost c(Ie, u,) and transitions to xc+1 ~ f Mz). We wish to design a policy uN ~ x-1[2;) such that the system achieves good performance over T steps. Instead of finding a single, globally optimal policy, MPC re-optimizes local policy at each time step by predicting the system's 's behavior over a finite horizon H < T using approximate model f. Specifically, in sampling-based MPC, we sample control sequence i; ~ T.))), where û, A (Me, dc+1,"' (+)#-1),, and our policy is parameterized by / E O. We rollout our model starting at F4 using these sampled controls to get our predicted state sequence a, A Se41. ... ,d, with f. = F₁. The total trajectory cost is
H-1
Clăr, a.) = Σ
clerth, ūeth) + Cterm (f++H),
A==
where C-erm(-) is a terminal cost function. We then construct a statistic J(0; xz) defined cost C'.,, â.) and use the rollouts to solve 0, +- arg mingee ((0; After optimizing our parameters, we sample the control sequence ân ~ TT, (.), apply the first control to the real system (i.e. H4 = (4), and repeat the process. Because each parameter O₂ depends the current state, MPC effectively yields a state-feedback policy, even though the individual distributions give us an open-loop sequence.
In this paper, we consider popular sampling-based based MPC algorithm known as Model Predictive Path Integral (MPPI) control [7, 8J. MPPI optimizes the exponential utility or risk-seeking objective:
j(0; xx) = - log EE
n exp (- 1 C(a,, a.)]
(2)
2

---

where 3 >> 0 is a scaling parameter, known as the temperature. As we do not assume that the dynamics or cost function are differentiable, we compute the gradients via the likelihood ratio derivative:
E no..
$ (c(
Vo Pologmol To (ūs)
v((0:))))
(3)
B B
(()
In MPPI, the policy is assumed to be factorized Gaussian of the form
H --1
H-1
T aa)
II R=
To, (ūc+h)
= II (ū Petto Eeth). h==
(4)
Previous work by Wagener et al. [9] has shown that optimizing this objective with dynamic mirror descent (DMD) [20] and approximating with Monte Carlo estimates gives us the MPPI update rule:
Pe-h (1-7t)Acth + 7
W. ΣØ= e Jc a(")
(5)
where jeth is the current mean for each time step and 7/t the step size. Between time steps of DMD, we get ji++h from our previous solution f4++h by using shift model f +()4+a). This shift model aims to predict the optimal decision at the next time step given the previous solution. In the context of MPC, it allows us warm-start the optimization problem to speed up convergence, as we can only approximately solve the optimization problem due to real-time constraints.
3 Learning the Sampling Distribution of MPC
3.1 Representation of the Learned Distribution
Instead of using uninformed sampling distributions, learned distributions can potentially exploit structure in the environment draw samples which more likely be collision-free and close to optimal. However, such learned distributions must be sufficiently expressive order to better capture near-optimal, potentially multimodal, behavior. They must also be parameterized such that is tractable to sample from and update online. If the distribution has large number of parameters, the number of samples required to efficiently update them online may be computationally infeasible. And ideally, the form of our distribution would be such that we could find a closed-form update.
One path towards meeting these criteria is to maintain simple latent distribution from which can sample, and then learn a transformation of the samples which maps them to a more complex distribution. During training, learn the parameters of this transformation, which can be conditioned on problem-specific information, such as the starting and goal configurations of the robot and obstacle placements. However, when executing the policy during episode, the parameters of this learned transformation remain fixed, and instead, we update the parameters of the latent distribution. Concretely, we consider learning distribution TTo which defined implicitly follows:
2. ~ (-)p).))
i = hx(ecc
(6)
where 2, A (2,, R141.-... -1), is a context variable describing the relevant information of the environment, Po is the latent distribution with parameters 0, and h is the learned conditional transformation with parameters as vectors in RM MH If pe is a Gaussian factorized as in Equation (4) and we assume that h, is invertible, we prove Appendix A.6 that the corresponding DMD update to the latent mean is simply Equation (5), except that we replace the controls in the weighted sum with the latent samples.
Moving forward, we assume that both & and û, are stacked
3.2 Formulating the Learning Problem
Learning the distribution тe.i amounts to solving a bi-level optimization problem [19], in which one optimization problem is nested another. The lower-level optimization problem involves updating the latent distribution parameters at each time step, 0,, by minimizing the expected cost with DMD. The leper-level optimization problem consists of learning such that MPC performs well across number of different environments. To formalize this, first consider that have some distribution of
3

---

environments c -~CC C() over which we wish MPC to perform well. For each environment, our system has some conditional initial state distribution zo ~ p(-le). The objective we wish minimize is then
T-1
e(0, (;c) = -
Σ J( Ax
where = (fo, f1. ......)..))). and our cost statistic, J. now depends on À and c as well. This objective measures the expected performance of the intermediate plans produced by MPC along the I' steps of the episode. Our desired bi-level optimization problem can be formulated as:
min Be ((((())))))) s.t. (()) R arg min (c))
0
where = indicates that we approximate the solution of the optimization problem with an iterative algorithm that may also be parameterized by A, as the exact minimizer is not available in closed form. Moreover, the notation (A)) indicates the dependence of the lower-level solution on the upper-level parameters. In our case, we solve the lower-level problem with DMD, where we also parameterize the shift model, & (:) making it a learnable component and conditioned on
The normal shift model in MPC simply shifts the control sequence forward one time step and appends a zero or random control at the end. However, because we are performing this update in the latent space, there is no clear delineation between time steps of the latent controls, as they are coupled according to the learned transformation. Therefore, there is no way to easily perform the equivalent shift operation in the latent space. As such, we instead learn this shift model along with the transformation. Besides, the standard approach described above may not be optimal. By learning it, we may be able to further improve performance. This is especially true because the performance hinges greatly on the quality of the shift model since we only run one iteration of the DMD update.
3.3 Parameterizing with Normalizing Flows
In order to optimize the upper-level objective in Equation (8) with respect to X, we need to be able to compute the density T directly. Therefore, we choose to represent It with a normalizing flow (NF)
[16, 18, 17], which explicitly learns the density by defining an invertible transformation that maps latent variables to observed data. Generally, we compose a series of component flows together, i.c.
hy hAx hAx-1
hx, which define series of intermediate variables Do.... 9K-1, 9K. with yo = and ÿK- = ú. The log-likelihood of the composed flow is given by:
K
log T (f(c) = log pa (2)
-
log det
0ÿ:-1
In this work, we make use of the affine coupling layer proposed by Dinh et al. [18] part of the real non-volume- preeeneiinne (ReaINVP) flow. The core idea is to split the input â into two partitions â = (ū,, R, where 11 and 12 are partition of [1, M Hl). and apply
(10) where 8A and fx are the scale and translation terms, which are represented with arbitrary neural networks, and is the Hadamard product. This makes computing the log-determinant term in Equation (9) and inverting the flow fast and efficient. Now, in robotics, we often have lower and upper limits on the controls. These are usually enforced in sampling-based MPC by either clamping the control samples passing them through scaled sigmoid. However, instead of enforcing the constraints heuristically after sampling, we learn a constrained sampling distribution directly. Since the sigmoid function is invertible and has tractable log- Bog-determinant (shown in Appendix A.8), can simply append one after fxx in the NF and scale it by the control limits. This ensures that control constraints are satisfied by design and taken into account while learning the distribution.
01, û, 01,- = â, C cxp s ( + ca),,
3.4 Training the Sampling Distribution
Computing gradients through the upper-level objective is not straightforward, both the expectation and the inner terms of Equation (7) depend on X. Therefore, the state distribution depends on the NF and latent shift model. One way around this issue is to consider a modified objective at each batch d:
T
EO >; = EE
j( X; 2 c)
(11)

---

which fixes the outer expectation to be with respect to the current policy. Intuitively, this choice trains the NF to optimize the MPC cost function under the state distribution resulting from the current policy Ao, We then update the outer expectation distribution at each episode.
Now, we only have to focus on computing the gradient V (O,(O,,)), X; ær, c) for each time step, which can be computed similar to Equation (3) and approximated with Monte Carlo sampling:
N (.(( Σ
wi V log (2),
(1) (0)
(12)
where the weights wi are defined according to Equation (5). The log-likelihood is given by Equa- tion (9). the gradient of which involves computing the backwards pass through the network hx. However, we also have to consider the dependence of the latent distribution parameters (()) on X. Therefore, we must backpropagate through the MPC update:
E "E e hi
(x(A) = (1 (1--( + AAp, AP4:
(13)
IF
Ca c((()))))
where the previous shifted mean (A) is given by the learned latent shift model. Note that we have rewritten the expectations in terms of the control distribution, rather than the latent distribution. This is necessary in order to derive the following approximate gradient without requiring differentiability. To compute the gradient of Equation (13), we must approximate the gradient of Aj4; with respect to daue
X, which we can compute
M1 M2M2, where we define:
1 :c) ha ha(a ;) log 7 B(A)AA (a) (1/]
M
1=1
(14)
M₂ Σ
w :c, M3 = Σ wa V. log क TT(A) (c).
The derivation of this approximate gradient can be found in Appendix A.7. Note that computing the gradients V log T (ā, (a("(c) (c) will also require us to backpropagate through the shift model due to the dependence of on Therefore, even when the step size is set to one, i.e. 7t = 1, we introduce form of recurrence between time steps. Please see Appendix A. for additional visualizations of the computational graph generated by an episode and further descriptions of the overall algorithm.
Related Work
Multiple works have considered sampling distributions beyond simple Gaussians, such as Gaussian mixture models [10] and particle method based on Stein variational gradient descent (SVGD) [11]. Additionally, most implementations use heuristics to modify samples and squeeze out additional performance gains [4, 21]. In terms of learning the distribution, Amos and Yarats [12] learn a latent action space for their proposed differentiable cross-entrope method (CEM) controller. However, they require all components of the pipeline to be differentiable and do not consider learning a shift model. Agarwal et al. [13] learn normalizing flow in the latent space of variational autoencoder (VAB). Yet, they also require differentiability, use expert demonstrations to learn the latent space of the VAF, and have no means of warm-starting between time steps. Wang and Ba [22] propose to use learned feedback policy to wtrm-start MPC, but still rely on Gaussian perturbations to the proposed action sequence. The authors also explore performing online planning in the space of the network's parameters, which results in a massive action space that may be hard to scale.
Power and Berenson [14, 15] also train normalizing flow to use as the sampling distribution for MPC, but they do not learn a latent shift model and perform all operations in the control space. Moreover, they mix the latent samples with Gaussian perturbations to the current control-space mean, as they do not update the latent distribution directly. This prevents them from fully taking advantage of the learned distribution and throws away useful information which could potentially improve performance. In fact, we show in Appendix A.4 that the learned shift model contributes significantly to the performance gains. Additionally, a primary focus of their work is how to
5

---

MPPI
FlowMPPI
NFMPC
100
45
40
80
35
Success Rate
Tota Cost
20
20
16
10
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
Num. Samples
Num. Samples
Figure Success rate and cost distribution on the PNRANDDYN environment across different number of samples.
NFMPC
FlowMPPI
MPPI
Figure 2: Visualization of a trajectory and top samples from (top) NFMPC, (middle) FlowMPPI, and (bottom) MPPI on the PNRANDDYN task.
handle out-of- distr---iitiiitionnn (OOD) environments by learning posterior environment context variables. We could combine their approach with ours by conditioning the learned shift model on the inferred environment context for improved generalization. Finally, normalizing flows have also been used to improve exploration RL [23, 24, 25] by providing a more flexible, and potentially multimodal, distribution. They have also been employed in sampling-basee motion planning [26, 27] to provide good proposal configurations to speed up convergence.
Experimental Results
In all experiments, we denote proposed approach NFMPC, the baseline MPPI implementation as MPPI, and the method by Power and Berenson [14, 15] as FlowMPPL. Details about the hyper- parameters, implementation, tasks, and training can be found in Appendix A.2. We evaluate fixed set environments, which includes start states, goal locations, and obstacle placements, and run 32 rollouts for each sample amount. Our primary metrics for comparison defined as the percentage of times the task goal was achieved, and the average cost of trajectories which successfully completed the task. Additionally, the NF in the control pipeline. Therefore, we measured the average change in wall clock time across different amounts of samples for NEMPC and FlowMPPT in Appendix A.5.
the success rate,
cannot ignore the overhead introduced by
5.1 Planar Robot Navigation
We begin by applying NFMPC to planar robot navigation task in which a holonomic point-robot must reach goal position while avoiding eight dynamic obstacles, which move around randomly at

---

MPPI
FlowMPPI
NFMPC
107
80
60
Total Cost
50
Success
10a
211
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
Num. Samples
Num. Samples
Figure 3: Success rate and cost distribution on the FRANKAOBSTACCIES environment across different number of samples.
each time step (PNRANDDYN), The point-robot has double integrator dynamics with stochasticity on the acceleration commands to create a mismatch between the predictive model used by MPC and the true environment, Each obstacle's current position is perturbed with Gaussian noise and clipped be within map bounds, and the starting and goal locations of the robot are randomized each episode. The NF for both NFMPC and FlowMPPI conditioned on the obstacle locations, current state, and goal position. Note that we do not condition the shift model, as this consistently hurt performance.
We quantitatively compare all controllers in Figure 1. The trajectory cost box plots represent the median and quartiles of the distribution. We find that NFMPC consistently outperforms the both MPPI and FlowMPPI. While all controllers reach 100% success rate 1024 samples, NFMPC achieves a 20% and 8% lower median cost over MPPI and Flow FPoWMPPI, respectively. We also find that NFMPC scales more gracefully overall the number of samples reduced. For instance, able to withstand a 32x decrease in the number of samples (1024 to 32) while reducing success rate by only 3% and increasing median cost by 8%, nearly matching FlowMPPI 1024 samples. In comparison, MPPI reduces success rate by 6% and increases median cost by 43%. Similarly, NFMPC outperforms FlowMPPI all sample amounts in terms of median cost, although at 16 samples FlowMPPI achieves a slightly higher success rate than NFMPC. We visualize trajectories and top samples in Figure The green and red dots are the starting and goal locations, respectively, and the blue dot is the current position of the robot at the given time step. The thick black line is the resulting path taken by the controller, while the gray lines are the top samples generated at the current state. In this example, FlowMPPI nearly collides with an obstacle, while NFMPC and MPPI are able to safely reach the goal. FlowMPPiover commits to narrow corridor and unable to reroute in time to account for the new obstacle location. NFMPC takes a similar trajectory to FlowMPPI, however, it is able to pause until the obstacle moves out of the way to proceed towards the goal. We also provide additional experiments with static environments and comparing unconditional and conditional models in Appendix A.3.
5.2 Franka Panda Arm
Next, we apply NFMPC the FRANKA task, which involves controlling 7 degree-of-freedon (DOF) Franka Panda robot arm and steering it towards randomly placed target goal from a fixed starting pose while avoiding single pole obstacle. The NF for both NFMPC and FlowMPPI is conditioned on the obstacle locations, initial state, and goal position. It is also important to note that no controller achieves a 100% success rate, as not every randomly generated environment feasible. As shown in Figure 3, at 1024 samples, NEMPC achieves a success rate of 84%, while FlowMPPI and MPPI only succeed 81% and 78% of the time, respectively. Again, NFMPC scales better with reduced number of samples. With 64 x decrease in the number of samples (1024 to 16), NFMPC only drops success rate by 29%. Meanwhile, FlowMPPI decreases by nearly 70% to a success rate of 25%. However, both fair better than MPPI, which drops by 80% success rate of 16%.

---

NFMPC
FlowMPPI
MPPI
Figure 4: Visualization of a trajectory and top samples from (top) NFMPC, (middle) FlowMPPI, and (bottom) MPPI on the FRANKAOBSTACLES task.
These results support the hypothesis that learning to perform MPC updates in the latent space of the NF and training the controller as recurrent network improves performance.
To more clearly understand what NFMPC is doing differently, visualize the performance of all three controllers on a held-out validation environment in Figure 4. The green and red markers indicate the end-effector and goal positions, respectively, while the single pole the obstacle which must be avoided. Additionally, the green trajectories represent the top samples from the controller at each time step. While MPPI collides with the obstacle, both NFMPC and FlowMPPI learn take a different path which is collision-free. FloW generates better initial trajectories than NFMPC, which are more pointed towards the goal location and achieve a greater velocity. However, NFMPC is able to better adapt the sampling distribution throughout the episode and reach the goal more quickly. Finally, in Appendix A.4, we present results for the environment with no obstacles, comparison of conditional and unconditional models, and a breakdown of the individual cost terms for all models to gain insight into the learned sampling distributions.
also perform an ablation which removes learned shift model from NFMPC to illustrate that it is crucial component.
Limitations
major limitation of NFMPC is that the learned distribution and shift model are only valid for fixed horizon and control dimensionality. Therefore, these components cannot be directly transferred to new robots or for alternate horizons without being retrained. However, this could potentially be remedied by novel architectural innovations and training distributions across both environments and robots. Moreover, the learned distribution specific to the environmental distributions on which was trained. Therefore. it does not always perform as well when transferred to out-of-distribution environments. However, this is always going to be a challenge for any learning-based method and addressing it is open question for future research. Finally, both approach and FlowMPPI introduce an additional overhead to due to running the NF which cannot be ignored. This increase in wall clock time may be worth the additional performance gains. Additionally, we scale better with reduction in samples compared to all baselines. Therefore, we can potentially alleviate some of the introduced overhead by redueing the number of samples while still meeting the application demands.
7 Conclusion
We presented a method for learning MPC sampling distributions with normalizing flows (NFs) which moves all online parameter updates and warm-starting operations into the latent space. We show how to frame the problem as bi-level optimization and derive an approximate gradient through the MPC update to train the distributions. Additionally, we illustrate how incorporate control box constraints directly into the NF architecture. Through our empirical evaluations in both simulated navigation and manipulation problems, we demonstrate that our approach is able to surpass the performance of all baselines. Moreover, we find that controllers which move all operations into the latent space are often able to scale more gracefully with a reduction in the number of samples. These results indicate importance of leveraging the latent space in learned sampling distributions for MPC. Finally, because we learn all components the controller through episodic interactions with the environment, they can potentially be trained to account for the modeling errors in the MPC controller.

---

References
[1] P. Abbecl, A. Coates, and A. Y. Ng. Autonomous Helicopter Acrobatics through Apprenticeship Learning. The International Journal of Robotics Research (IJRR), (0(13:1608-1163) 2010.
[2] V. Kumar, E. Todorov, and S. Levine. Optimal Control with Learned Local Models: Application to Dexterous Manipulation. In JEEE International Conference on Robotics and Automation (ICRA), pages 378-3 378-383. IEEE, 2016.
[3] C. Finn and S. Levine. Deep Visual Foresight for Planning Robot Motion. In IEEE International Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[4] M. Bhardwaj. B. Sundaralingam, A. Mousavian, N. Ratliff, D. Fox, F. Ramos, and B. Boots. STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation, arXiv preprint arXiv:2104. 10di::2104.133500 13542, 2021.
[5] T. Erez, K. Lowrey, Y. Tassa, V. Kumar, S. Kolev, and E. Todorov. An integrated system for real-time Model Predictive Control of humanoid robots. In IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 292-299 IEEE, 2013.
[6] 7. Erickson, H. M. Clever, G. Turk, C. K. Liu, and C. C. Kemp. Deep Haptic Model Predictive Control for Rosot-Assisted Dressing. In IEEE International Conference on Robotics and Automation (ICRA), pages 4437-4444. IEEE, 2018
[7] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. In IEEE International Conference on Robotics and Automation (ICRA), pages 1433-1440. IEEE, 2016.
[8] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference Robotics and Automation (ICRA), pages 1714-1721. IEFF, 2017.
[9] N. Wagener, C.-A. Cheng, J. Sacks, and B. Boots. An Online Learning Approach Model Predictive Control. arXiv preprint arXiv: 290.208902.77. 2019.
[10] M. Okada and T. Taniguchi. Variational Inference MPC for Bayesian Model-based Reinforce- ment Learning. Conference on Robot Learning (CoRL), pages 258 272. PMLR, 2020.
[11] A. Lambert, A. Fishman, D. Fox, B. Boots, and F. Ramos. Stein Variational Model Predictive Control. arXiv preprint 20XXv:20/100764/// 2020.
[12] B. Amos and D. Yarats. The Differentiable Cross-Entropy Method. In International Conference on Machine Learning (ICML), pages 291 1302.. PMLR, 2020.
[13] S. Agarwal, H. Sikchi, C. Gulino, and E. Wilkinson. Imitative Planning using Conditional Normalizing Flow. arXiv preprint arXiv:2007.16162 2020.
[14] T. Power and D. Berenson. Variational Inference MPC for Robot Motion with Normalizing Flows. Advances in Neural Information Processing Systems (NeurIPS) Workshop on Robot Learning: Self-Supervised and Lifelong Learning, 2021.
[15] T. Power and D. Berenson. Variational Inference MPC using Normalizing Flows and Out-of- Distribution Projection. arXiv preprint arXiv:2205.04mm 044667 2022.
[16] L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear Independent Components Estimation. arXiv preprint arXiv: 1410.866. 2014.
[17] D. Rezende and S. Mohamed. Variational Inference with Normalizing Flows. In International Conference on Machine Learning (ICML), pages 1530-1538. PMLR, 2015.
[18] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. arXiv preprint 1605 08803, 2016.
[19] Bard, Practical Bilevel Optimization: Algorithms and Applications, volume 30. Springer
Science & Business Media, 2013.

---

[20] E. Hall and R. Willett. Dynamical Models and Tracking Regret in Online Convex Programming. In International Conference on Machine Learning (ICML), pages 579-587, PMLR, 2013.
[21] C. Pinneri, S. Sawant, S. Blaes, Achterhold, J. Stueckler, M. Rolinek, and G. Martius. Sample- efficient Cross-Entropy Method for Real-time Planning. arXiv preprint arXiv:2008.0633 06389, 2020.
[22] T. Wang and J. Ba. Exploring Model-Based Planning with Policy Networks. arXiv preprint 1906. 1906.06499,, 2019.
[23] Y. Tang and S. Agrawal. Boosting Trust Region Policy Optimization by Normalizing Flows Policy. arXiv preprint arXiv:/809.10326 2018.
[24] Y. Tang and S. Agrawal. Implicit Policy for Reinforcement Learning. arXiv preprint arXiv: 1806 06798, 2018.
[25] B. Mazoure, Doan, A. Durand, J. Pineau, and R. D. Hjelm. Leveraging exploration in off- off-policy algorithms via normalizing flows. In Conference on Robot Learning (CoRL), pages 430 444. PMLR, 2020.
[26] T. Lai and F. Ramos. Learning to Plan Optimally with Flow-based Motion Planner. arXiv preprint 200:20000.132:3 2020.
[27] Lai, W. Zhi, T. Hermans, and F. Ramos. Parallelised Diffcomorphic Sampling- based Motion Planning, arXiv preprint arXiv:2108.1177 2021.
[28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin,
N. Gimelshein, L. Antiga, et al. PyTTrcc: Imperative Style, High-Performane Deep Learning Library. Advances in Neural Information Processing Systems (NeurIPS), 32:8026- 8037, 2019.
[29] H. Halton. Algorithm 247: Radical-inverss quasi-random point sequence. Communications of the ACM, 7(12):701-702, 1964.
[30] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. arXiv preprint arXiv: 108. 10470, 2021.
[3]] Ba, J. R. Kiros, and G. Hinton, Layer Normalization. arXiv preprint arXiv:1607.06450. 2016.
[32] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. arXiv preprint 14Xi 6980, 2014.
10


Reviews:
[{"review_id": "xTi2W9H7Ean", "reviewer": "Reviewer_bXam", "summary": "This paper presents an approach to learning sampling distributions for MPPI. The authors leverage normalizing flows, and backpropagate through the MPC optimization procedure to train the flow model. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nOverall, I think the paper is a good one. The method is straightforward, easy to understand, and clearly yields strong practical benefits. \n\n## Weaknesses\n\nThere are a handful of small issues which, if addressed, would substantially strengthen the paper. \n\n- The biggest issue is lack of investigation of OoD tasks. The authors explicitly discuss this in their limitations section, but it really is extremely important. The paper needs to investigate performance on OoD tasks and should include more complex environments (possibly featuring multiple obstacles). In general, the authors should investigate leveraging more information in the normalizing flow. For example, scaling to alternative, more complex environments would likely be substantially improved by conditioning the NF on environmental information (which as far as I can tell is not currently done). This approach has been widely investigated in the literature on learning sampling distributions for sampling-based motion planning, and I would point the authors toward that area for inspiration (the authors should probably also discuss that literature in their related work section, as it is highly related). \n- The authors should report success/cost versus wall clock time in addition to versus number of samples. The performance for a small number of samples is obviously much better than standard MPPI, but understanding the time cost of the sampling (especially for NF models) is important. \n- It is unclear if the action sequence samples are decoded/passed through the NF as iid samples or jointly. If each set of action sequences is decoded independently, investigating a joint distribution over all samples is potentially very interesting as the model would be able to learn variance reduction strategies (eg antithetic sampling). \n- The authors propose to use NF models primarily due to the fact that they enable exact evaluation of likelihoods. Generally, with the world of generative models evolving as quickly as it currently is, I would try to minimize the explicit dependence on the NF models. In particular, the authors currently state a set of necessary features for the learning model in several spots across the paper. The paper would be improved by formally stating, in one spot, the necessary features for a generative models to be used in this work, and then discuss how NF models are one example of a model that meets the desiderata whereas eg VAEs are not. This hopefully will improve the applicability of this paper as generative models continue to progress. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors should extend their method to OoD tasks, as well as improve some elements of the writing. I am willing to improve my impact score from 3 to 4 if there is (substantial) progress made in the OoD task area.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As discussed above, the main issue to address is OoD tasks, as well as several more minor issues such as changing the framing around the generative model. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "KpdAgoOeQQw", "reviewer": "Reviewer_BrmL", "summary": "This work uses bi-level optimization and a normalizing flow parameterization of learned sampling distributions. This allows for learned sampling distributions for MPC without requiring differentiability of dynamics and cost functions. This approach allows for box constraints on the controls (ex joint limits).\n\nThe proposed method is compared to MPPI, FlowMPPI on a 2d navigation task, and a 7dof reaching task with a Frank robot.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe proposed approach appears to scale more gracefully to smaller numbers of samples, particularly for the harder Franka & Franka Obstacles tasks. \n\nWeakness:\nI still don't have a good intuition for the average trajectory cost in terms of either wall clock time or required number of operations for inference. This work demonstrates that their approach can operate with significantly fewer samples than MPPI, but is unclear what the advantage of less samples actually implies. How much faster or slower is it to run this NF based approach vs MPPI? Does the 16-sample NFMPC run any faster or require less compute than a 1024 sample MPPI? \n\nThere maybe clear benefits to the proposed approach here, that this manuscript would benefit from if articulated. Alternatively, if the proposed approach is actually slower or more computationally expensive even when using less samples, this work would be less compelling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written, figures are clean and the ideas are well presented.  It would be great if authors could provide some insight into the issues section to help understand the impact of this work.\n\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "\nMy main concern is that number of samples is an insufficient metric to compare this method to existing baselines. If your approach only requires 16 samples, while MPPI requires 1024, how much faster is your method or how much less computation is required?"}}, {"review_id": "FhZGdlv7R0w", "reviewer": "Reviewer_aRE3", "summary": "This paper proposes improves previous works on model-predictive control based on normalizing flow. The previous works learn the sampling distribution in the original control space. In contrast, this paper proposes to perform the learning of the sampling distribution in the latent space. As a result, the entire learning algorithm can be cast as a bi-level optimization problem which alternately optimize the sampling distribution and the normalizing flow which is responsible for mapping a sample in the latent space to the control space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed improvement of updating the latent sampling distribution is reasonable and solved in an intuitive manner.\n- The paper is in general well written, although I have trouble understanding some of the technical details such as Section 3.4, but it may be due to my lack of knowledge in the relevant fields such as normalizing flow.\n\nWeaknesses:\n- The experiments indeed show the improvements of the proposed method over the two baselines under comparison, however, I feel that some of the experimental results need more explanations. For example, in all experiments, it seems that the method of FlowMPPI usually performs worse than the simple baseline of MPPI especially with small sample sizes (e.g., see lines 246-248). Why is this the case? Next, as said in lines 251-252, the proposed NFMPC is able to discover more diverse paths, so what's the intuition behind this improved behavior of NFMPC? Is it because updating the sampling distribution in the latent space, as done by the proposed method, helps learn a better sampling distribution?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper makes a reasonable and useful contribution of sampling-based model predictive control.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the question under \"Weaknesses\" above.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)