Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Authors: Xuesu Xiao
Tingnan Zhang
Krzysztof Marcin Choromanski
Tsang-Wei Edward Lee
Anthony Francis
Jake Varley
Stephen Tu
Sumeet Singh
Peng Xu
Fei Xia
Sven Mikael Persson
Dmitry Kalashnikov
Leila Takayama
Roy Frostig
Jie Tan
Carolina Parada
Vikas Sindhwani
Venue: CoRL 2022 Poster (2022)
Keywords: Model Predictive Control
Transformers
Performers
Highly-Constrained Navigation
Social Navigation
Learning-based Control

Abstract: Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied public spaces. 
To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the robust handling of system constraints from Model Predictive Control (MPC). 
Our approach, called Performer-MPC, uses a learned cost function parameterized by vision context embeddings provided by Performers---a low-rank implicit-attention Transformer. We jointly train the cost function and construct the controller relying on it, effectively solving end-to-end the corresponding bi-level optimization problem. 
We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in different challenging real-world scenarios. 
Compared with a standard MPC policy, Performer-MPC achieves >40% better goal reached in cluttered environments and >65% better on social metrics when navigating around humans. 

Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation
Xuesu X Tingnan Zhang") Krzysztof Choromanski Edward Lee", Anthony Francis", Jake Varley", Stephen Tu², Sumeet Singh', Peng Xu Fei Xia Sven Mikael Persson", Dmitry Kalashnikov's Leila Takayama",
Roy Frostig', Jie Tan", Carolina Parada and Vikas Sindhwani "Equally Contributing Authors George Mason University Everyday Robots "Robotics @ Google "Hoku Labs
Despite decades of research, existing navigation systems still face real- world challenges when deployed in the wild, e-g., cluttered home environments or in human-occupied public spaces. To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the bust handling of system constraints from Model Predictive Control (MPC). Our approach, called Performer-MPC. uses a learned cost function parameterized by vision context embeddings provided by Performers low-rank implici--attentin Transformer. We jointly train the cost function and construct the controller relying it, effectively solving end-to-end the corresponding bi-level optimization prob- lem. We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in dif- ferent challenging real-world scenarios. Compared with a standard MPC policy, Performer-MPC: achieves -40 % better goal reached in cluttered environments and >65% % better on social metrics when navigating around humans.
Abstract:
Keywords: Model Predictive Control, Transformers, Performers, Highly-
Constrained Navigation, Social Navigation, Learning- Control
Figure 1: Left: Standard MPC efficiently cuts blind corners, forcing the human to back up: Right: Social Performer-MPC. avoids cutting blind corners, enabling safe navigation around humans.
1 Introduction and Related Work
Real-world robot deployment in human-centric environments, such as cluttered homes or crowded offices, remains an unsolved problem [1, 2]. These challenging situations require safe and efficient navigation through tight spaces, such as squeezing between coffee tables and couches, handling tight corners, doorways, untidy rooms, and on. An equally critical requirement is to navigate in manner that complies with unwritten social norms around humans.
Classical approaches using model-based control [3, 4, 5, 6] can already move robots from one point another safely and reliably. However, when deploying these systems in the complex real world [7, 8, 9], extensive engineering effort is required to construct world representations [10, 11, 12], model vehicle kinodynamics [13, 14], hand-craft cost functions [15, 16], fine-tune system parameters [17],
Project page: attps://perfornen http::/mwwmmmmm.tg........ github.io
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

and design backup planners to recover from stuck scenarios [18, 19, 20, 21]. While providing verifiable guarantees, these cascaded components need to be nand-engineered before deployment based on the roboticist's best expectations of what would be encountered in the real world, and become cumbersome when in-situ modifications are necessary to enable adaptive behaviors [17].
In contrast to these classical methods, machine learning enables robots to learn these behaviors directly from data [22]. End-to-end learning [23, 24] an appealing paradigm to reduce the engi- neering effort and cascading errors caused by separate components, but it usually requires extensive real-world training data or simulation with inevitably simplified human and environment represen- tations. Most importantly, it lacks safety, optimality, generalizability. interpretability. and explain- ability, which are crucial for real robots moving around humans [22. 25. 26]. Therefore, researchers have looked at individually learning global planners [27], local planners [28. 29, 30, 31, 32, 33], and other navigation components including cost representations [34, 35. 36, 37], kinodynamic mod- els [38, 391, and planner parameters [40, 41, 42, 43, 44, 45, 46] to enable both better navigation performance, and also off-road [47, 48] and social navigation [1, 49, 50].
Both classical and learning-based methods have their merits. Model Predictive Control (MPC) [3, 4, 5, 6] enables synthesis of real-time feedback controllers for robots operating in real-world environments that satisfy given safety constraints, optimality criteria, and kinodynamic models. To get the best of both worlds, we design a class of Learnable-MPP policies enabling robots to learn navigation behaviors in real-world use cases by combining the flexibility of learning from demon- strations with the optimality properties (e.g.. collision-free, shortest path) of MPC solutions. Our framework can also been seen as a class of Implicit Behavior Cloning policies [51] that are aware of real-worls robot-environment and robot-human interactions. Our contributions in this paper are three-fold:
We augment the cost function of MPC with learnable components parameterized by rich Transformer-basse latent embeddings of real-world context. Transformer architectures [52, 53, 54] have produced stunning advances in language modeling [55, 56, 57, 58, 59, 60], im- age generation [61, 62, 63, 64, 65, 66]. and multi- modal reasoning [67, 68, 69, 70, 71, 72]. This indisputable success comes at a computational price in proportion to the massive num- ber of parameters learned (c.g., 175 billion for GPT-3 [56]) as well as quadratic scaling in input sequence length of the core attention modules of these models. By generating context- dependent quadratic costs using Performers [73] low-rank linear-attention Transformer, we demonstrate how we can embed powerful pixel-to-pixel attention mechanisms in MPC while crucially retaining real-time solutions a CPU onboard mobile robot.
Using distributed bilevel optimization with implicit differentiation mechanisms, we train navigation policies on expert demonstrations to handle difficult navigation scenarios, with data augmentation strategies to mitigate well-known distribution shift issues that frequently plague behavioral cloning and other imitation learning approaches. We demonstrate that our Performer-MP
outperforms its counterparts in real-world chal- lenging navigation scenarios, including highly constrained and human-occupin environ- ments. Performer-MPC: learns to achieve >40 % better goal reached in highly constrained environments and >65 % better behavior captured by social metrics defined in the Ap- pendix when moving around humans in a social navigation pilot study.
2 Performer-MPC: Learnable MPC with Scalable Real-Time Attention
In order to respond to dynamic uncertainty in the environment, the principal challenges of synthesis of model predictive controllers [74, 75, 3] are: (i) to construct cost functions that remain suitable across a wide variety of robot-enviromment situations, and (ii) to generate reliable solutions to the underlying trajectory optimization problems in real-time. This work focuses on the first challenge above, in particular, by eschewing the classical approach of hand- -and-eengieerrdd cost functions, and adopting learning-basee
inverse optimal control [76, 77, 78, 79, 80, 81] framework, where the sensory/visual context is used to induce what MPC problem to solve in real-time in order to generate actions. We first provide some details on training and inference of learnable MPC framework, and then discuss the details of the learnable components; see Fig. 2 for an overview.
2.1 Learnable Model Predictive Control
Let Co denote the current "context", e.g., such as a list of tensors encoding for example RGB/D streams, force-torque readings, and proprioceptive states over short tine-w.ndow. Consider

---

Transigg
Learned Ceet
Expar
Query
X3
X
viku
Eribed
Gendhoieation
MPC
Ergkeeeed
Trejecbrrex
(Dsoupancy Gnd, etc.)
(Scllaale Aftertion)
Inference
Overview of the Performer-MPC. The final latent embedding of the patch highlighted in red is used to construct context-dependent learnable cost. The backpropagation (red arrows) is through the parameters of the Transformer (see Sec. 2.2 for more details regarding Performers).
Figure 2:
Learnable-MPC feedback policy implicitly defined by solving the following parametric optimal control problem at each time instant:
T-1
arg mi
Jclu, Σ
cxx, uz Co. 0) + cr (x1; C).
(la)
s.t.
x++1 141=f(x4, u,; Xo = 9Co, given.
(1b)
Denote the optimizer as u² (xo: 0). and the corresponding optimal state sequence as x" (Xo: Co, 0). Here, 0 are learnable parameters for stagewise and terminal cost neural networks CT the dynamics function, and current state estimator. While our framework generalizes to learning cost, dynamics, and state estimators simultaneously. in this paper we focus on the inverse optimal control setting: We study how the multi-layer self-attention cores Transformers may be embedded in the cost networks to handle sensor fusion while retaining real-time speed expectations of MPC. The dynamics function here corresponds to the differential drive dynamics of our robot.
While the MPC-structured policy can be trained using any flavor of Reinforcement Learning, real- world trial-and-error data is prohibitively expensive and general reward function that captures all intricacies of different real-world scenarios is difficult design. Therefore, we take an Imitation Learning approach where the robot has access to expert demonstrations. The MPC structure provides form of a strong inductive bias for Imitation Learning, and can lead to improved data efficiency, robustness, and generalization. Denote ú' (u/...., -/;-1) and x = (X). ..., x7:)) as the control and state sequence for the jth demonstration snippet, with associated sensor context Cj. which can be extracted from oflline planning human teleoperation. We optimize follows:
N
0r arg min
Σ Ji (0)la',
where Ji denotes total imitation loss that measures discrepancy between MPC-generated and expert state-control trajectories. We assume that J, also admits stagewise and terminal decomposition using loss functions/ and lr:
Th-1 [xi (0) ui (0), (0),
x u + Iu (x7,
x
(3a)
x fxx (0), of (0)),
(0) x
(3b)
Above, x"", T the MPC solution with cost parameters given context Cj and initial state xi.-
The training optimization problem (3) has embedded the MPC optimization, (1). Together, the two may be viewed where the higher-level searches for the best cost-network parameters via imitation loss minimiza- tion, while the lower-level synthesizes the optimal predictive control sequences given fixed 0. To use stochastic gradient descent for the higher-level problem, we need the gradient of Jr with respect to 0 evaluated at control sequence u" (0x) where 02 denotes the parameters during the current iterate k during training. This quantity decomposes as a vector-Jacobian product (VJP),
Training via Bilevel Optimization:
instance of bilevel optimization,
VoJi (u" (0 x*) = Vi,Ji (0t) x²) "pp" (0x).
The first term in the product on the right hand side is the gradient of the total imitation loss, which can be efficiently computed using the Adjoint method in Optimal Control [82] thanks to its stagewise
3

---

structure. The second term is the sensitivity of the MPC solution with respect to parameters, which may be efficiently computed using the Implicit Function Theorem (IFT), featured in several works exploring differentiable "optimization layers" [83, 84, 85, 86), see Appendix for details.
MPC Solver: We use second order Causs-Newton trajectory optimizer [87, 88] called Iterative LQR (iLQR) [89] with line searches inspired by Differential Dynamic Programming (DDP) [90, 87]. At each iteration, iLQR quadratizes the cost and linearizes the dynamics to compute the search direction by solving a time-varying LQR (TVLQR) [82] problem. Upon convergence, a single additional LQR solve suffices for computing (()"(() (4).
While one may use the first component of the optimal solution for problem (1), i.c., (xo; Co, 0). the policy map, we noted better performance by leveraging a secondary (non- learnable) MPC problem similar to (I), featuring a "tracking" objective w.r.t. the solution of the learnable MPC problem. The details of this "tracking MPC" problem are provided in the Appendix.
Policy:
2.2 Attentive Cost Functions for Learnable MPC
We adopt an inverse optimal control framework for Icarnable MPC whereby only the cost function is learnable. We structure this cost the sum of a user-engineered function and context-dependent quadratic, parameterized by an embedding matrix P and vector q (described in more detail below):
c(x, u, t; Co. u, +
u,
x 11
P"(Co,O)P(Co) 0)
+ aCoo, 0)
x 11
(5)
Here, Trefers to the hand-designed cost function (see Appendix), appended to a Transformer-backer cost model that attends to the current context Co to generate residual quadratic cost terms for MPC to optimize. This structure removes the computational cost of repeated quadratization of a large net- work in the iIQQR solver. Furthermore, since the residual cost is convex and well-conditioned, crude MPC solutions can generate reliable descent directions for the higher-level optimizer, even though applying IFT in gradient computation assumes the MPC solution is precisely a local minimum. We next describe the details of the embeddings P and q.
Generating Context-Dependent Transformer Embeddings: We outline general Transformer- based backend for learnabeeMMPCs which leads to Performer-MPCS. The backend maps the current contexts Co into a latent embedding which can be reshaped into the matrices P and q to support the quadratic parameterization of Eqn. 5. For concreteness, let Co be an image frame, i.e., the occupancy grid in the robot frame. As in Vision Transformer architectures [91, 92, 93], each frame is first independently pre-processed by convolution layer, and then flattened to sequence. Each element (token) of the sequence corresponds to a different patch of the original frame which is then enriched with positional encodings. The length L of this sequence is patch-size hyper-paramens The preprocessed input is then fed to regular attention and MLP layers. The final embedding of latent representation of the entire context Co parameterize the
one of the tokens is chosen learnable cost (e.g., via de- de-vectorization to and as in Eqn. 5). Even though we take the final embedding of single token, it contains signal from all the tokens since attention mixes information across tokens.
The attention used in regular Transformer architectures [52] linearly projects tokens' embeddings (via trainable transformations) into three matrices, Q, K, V _4xdd called queries, keys and values respectively. The output of the attention is then defined as:
Att(Q, K, V) = D-AAA A =
/Vd), D= diiggAA (I))
(6)
where A LR is called the attention matrix. In the above formula, exp(.) is applied element- wise, 1, is the all-ones vector of length L, and diag(-) is diagonal matrix with the input vector as the diagonal. Space and time complexity of computing Eqn. 6 are: 0 Ld) and O(L²d) respec- tively, because A has to be explicitly stored. Quadratic time and space complexity in the number of patches makes this approach prohibitive for real-time robotic navigation. To address this, we apply a class of low-rank implicit innar-attention Transformer architectures, called Performers [73].
Performers interpret attention matrices A R..II. kernel matrices, i.e., A(i,j) is defined A(i,j) K(q, ky), with q: /k, standing for the 2th/ /5th query/key rew-vector in Q/ K and kernel

---

a
C
Global Minimum
Goal
Local Minimum
b
P
Figure 3: Experiment Scenarios: (a) Learning avoid local minima during doorway traversal, (b) to maneuver through highly constrained spaces, and (c) to enable socially compliant behaviors for blind corner and (d) pedestrian obstruction interactions.
K Rd Rd
R defined for the (randomized) mapping o Rd R.m (for 0)
y) = Ex Φ(y)].
(7)
We call Φ(v) a (random) feature map for v E Rd For Q' K' E R. with rows given ((q.) and o(k;) respectively, Eqn. leads directly to the efficient attention mechanism of the form:
Att(Q, K, V) = (Q'(('))) V)),
D diag(Q Q
(8)
Here Att stands for the approximate attention and the parentheses indicate the order of computa- tions. Such a mechanism characterized by space complexity of ( T.d + md) and time complexity of O ([md) as opposed to O + L.d) and ((I2d) of the regular attention mechanism. Different mappings e give rise to different Performer variants. Two most popular ones [73] are:
(x) def 1
eepp Vm |
(exp(w] x).... ex.(,..x and,
@retu (x) def 1 (ReLLU())),..... ReLU(())) (with m = d),
Vm
for wor....... Wm ~ No.. I2) and we will refer to the corresponding Performers as Performer-Exy and Performer-Rel.U respectively. Mapping 6eep provides unbiased estimation of the softmax attention- kernel from Eqn. 5, but requires random projections, whereas Φrein defines weaker iitention-kerned but does not use random projections and leads the fastest Performer variant. In Section 3.2 and the Appendix, we provide comprehensive speed benchmark for Performer-MPC: variations, demonstrating Pixel-to-Pixel attention real-time speeds.
Experiments
Our Perf corfoormerMPPa is tested on differential-drive wheeled robot, which has 3D LiDAR in the front, and depth sensors mounted on head (see Fig. 1). It three-layer Performer-ReLU model with mlpdim 64 and one head. Our policies are trained four TPUs and then deployed on CPU onboard the robot. We use the differentiable Iterative LQR implementation of trajax [94] for MPC training and inference. Please refer to the Appendix for more details.
5

---

b
train
test
test 20
test
test
10
50000 100000 150000
200000 400000 500000
5000
10000
15000
25000 50000 75000 100000
Visualization of test data examples in the (a) doorway traversal, (b) highly con- strained obstacle course, (c) blind corner, and (d) pedestrian obstruction scenarios. PePformer-MPC trajectories aiming the goal are always closer to the expert demonstrations compared to the RMPC trajectories. Bottom: Train and test curves, where the vertical axis represents loss values (Hausdorff distance to the expert trajectories) and horizontal axis represents training steps.
Figure 4: Top:
Performer- is compared with two baselines, a regular MPC policy (RMPC) without the learned cost components, and an Explicit Policy (EP) that predicts a reference/goal state using the same Per- former architecture, but without being coupled to the MPC structure. The control action (i.e., final policy output) is implicitly defined via the solution of the "tracking MPC" problem (see previous section), where the reference trajectory generated by one of RMPC, EP, or MMC
We evaluate our method in four scenarios, one in simulation and three in the real world (Fig. 3). For each scenario, the learned policies (EP and Performer-MPC) ) are trained with demonstrations specifically collected for that scenario. To address the distribution shift issue, we not only collect positive examples where the robot is driving smoothly with the intended behavior, but also start the robot randomly selected "disadvantage locations" (e.g.. near-collision situations), and steer the robot to recover from them. For more data collection details please refer to the Appendix. We visualize the planning results of Performer-MPC (green) and RMPC (red) along with expert demonstrations (grey) in the top half of Fig. 4 and the train and test curves in the bottom half.
3.1 Experimental Results
Learning to Avoid Local Minima: We first evaluate our method a simulated doorway traversal scenario (Fig. 3a). 100 start and goal pairs are randomly sampled from opposing sides of the wall. A planner guided by a greedy cost function often leads the robot to a local minimum, i.c., getting stuck the closest point the goal on the other side of the wall. Although such problem can be mitigated by using global planner, we use this as test case to showcase the learning results. We generate 2000 expert demonstrations using solves for intermediate way points provided by a Dijkstra's global planner. Using these off-line demonstrations, Perf ormer-MPC learns cost landscape that steers the robot towards the doorway, even if it must veer away from the goal and travel further. Performer-Mee passes the doorway in 86 out of 100 trials while RMPC only passes 24 out of 100.
off-line iLQR planner [89], which iteratively
We next test our method in a challenging real-world scenario-a a cluttered home/office setting where the robot must perform sharp. near-collision maneu- vers (Fig. 3b). A global planner provides coarse way points for the robot to follow. Each policy is run times and we report Success Rate (SR) and average Completion Percentage (CP) with vari- ance (VAR) of the obstacle course that the robot is able to traverse without collisions or getting stuck outperforms both RMPC and in SR and CP.
Learning Highly- -oabhl--Contrraieed Maneuvers:
(Fig. 5). Performer-Pee
Learning to Anticipate Pedestrians at Blind Corners: Going beyond static obstacles, we apply our method to social robot navigation [1], where robots must respect unwritten social norms for which cost functions are hard to design. One such scenario is blind corner (Fig, 3c), where robots

---

Goal
SR
CP : VAR
69 13%
Start
EP
3/10
62 # 15%
9/10
6%
Figure 5: A 4.5 10 m² obstacle course with policy trajectories and failure locations indicated by crosses for RMPC, EP, and Pcrformer-MPC.
should avoid the inner side of hallway comer in case a human suddenly appears in this "blind spot". For blind corner, we collect 30 demonstrations with human driving the robot from randomly chosen location on one side the corner to the other side a socially compliant manner. After training, we evaluate each policy twenty times in the real world: ten times in the corner where the demonstrations were collected (seen), and ten times run, the robot and pedestrian human subject will approach the corner from opposite sides, while a third-party observer monitors from a distance. We use a social navigation evaluation protocol [95] in which pedestrians and observers rate the performance of the policy using standardized questionnaire scored with Likert scales [96] which we combine into a joint social navigation score (see Appendix for further details). Social navigation scores for blind corner scenarios are shown in Fig. 6a. RMPC has the least social compliance: its hand-crafted cost function efficiently cuts the corner, causing uncomfortable near-collisions (Fig. 1). EP performs slightly better than Performer-MPe in the seen environments, but does not generalize well to unseen scenarios, reaching the goal).
different corner (unseen). During each
and unseen
with worse social scores and 20% failure rate (e.g., safety stops
Learning Respect Comfort Distance When Obstructed by Pedestrians: Another common social navigation scenario is pedestrian obstruction, when human unexpectedly impedes the prescribed path of a robot (Fig. 3d). While statie obstacle avoidance is a largely solved problem, pedestrian obstruction is particularly challenging for MPC policies that are guided by way- points that were valid before the human entered the environment. A hand-crafted cost function may guide the robot too near to the human, causing uncomfortably close interactions or the robot getting stuck right in front of the human. We evaluate policy performance for pedestrian obstructin iincctiion using the social navigation evaluation protocol [95]. Again, Performer-MEB is the most socially compliant (Fig. 6b), and in a few cases even shows emergent maneuvers unseen in the dataset (i.e., passing the human the left if there is not enough space the right, while the demonstrations only include righ passing). In contrast, RMPC usually gets stuck front of the human due to local minimum close the goal behind the human. While EP does stay away from the human
(a) Blind Corner Social Navigation Score
(b) Fedestrian Obstruction Social Navigation Score
Seen Scenario Unseen Scenarlo
Seen Scenario Unseen Scenario
5
3
1
Regular MPC
Explicit Policy* Porformer-MPC
Regular MPC Explicit Policy" Performer-MPC
Figure 6: Social Navigation Results. (a) In the blind corner scenario, Performer-MPC achieves better social navigation score than RMPC and similar in the seen environment. the unseen environment Perf Pnfformer-MPCC is better than both baselines. EP fails to complete unseen 20% of the time (e.g., not reaching the goal or triggering a safety stop, denoted by *). (b) In the pedestriam obstruction scenario, both EP and CefformerMPP show similar social scores and are supe- rior to RMPC, but EP fails in seen or unseen conditions 5% of the time (again denoted by *). Performer-MPH thus shows better generalizability in both social score and navigation success.

---

subject in both seen and unseen, it struggles to reach the goal location and sometimes comes close colliding with nearby walls, leading to a 5% overall failure rate. Please refer the Appendix for detailed description of both social scenarios and statistical analysis of the results.
3.2 Speed Studies Over Various Performer Architectures
Below present speed ablation studies over various Performer architectures leading Performer- ReLU as a default choice. More detailed studies are given the Appendix. Tests presented here 100 100 images, with patch size 5 X 5 and two architecture sizes: (a) medium with I = 3 layers, = 1 head, and mlpdim = 64 (8.3M parameters) and (b) large with / = 6 layers, 1024 (24.6M parameters). We benchmark Performer-Exp (for the latter one varying the number of random projections rps used and either redrawing them not at each forward pass), measuring wall-clock time taken by the MPC. For the fastest Performer-Rel.. variant, we run additional studies (this time measuring CPU-time to distill time taken solely by the MPC from I/O time, etc.) for varying patch sizes, showing that we can reach Pixel-to-Pixel attention with near real-time speed (1 (11.3ms per MPC iteration). The results presented tables Tab. and 2.
= 3 heads, and mlpdim
Performer-ReLL. and
Table Speed ablation tests for different variants of Performers with 100 100 input images. The architecture deployed on the real robot is denoted in bold.
model size
Performer type
redraw
rps #
wall-clock time per MPC-iteration [ms]
medium
ReLU
N/A
N/A
8,3
medium
Exp
False
8
21.5
medium
Exp
False
16
21.6
medium
Exp
False
64
23.6
large
ReLU
N/A
N/A
13.8
large
Exp
False
8
72.9
large
Exp
False
16
66.9
large
Exp
False
64
83.0
Table 2:
Speed ablation over patch sizes for medium-sized Performer-Rell with 100 x 100 inputs.
patch sizes
x
x
5 5
10 10
number of params (M)
15.7
9.93
8.50
8.33
8.16
CPU-time per MPC-iteration (ms)
11.3
2.3
1.5
0.5
4 Limitations
Currently our Transformer-backenn uses spatial attention, but in principle it can leverage the tempo- ral axis. For example, in a face-to-face approach with walking human in hallway, motion history may shed light on how the human intends to move in the future, c.g., yielding left or right. promis- ing future research direction is to add history dependency that the robot sequentially reason about potential future interactions and conflicts. Furthermore, exploring richer modalities [97] than the occupancy grid (e.g., RGB images, human traces, language contexts [98, 72)), to enable robot- environment and robot-human interactions beyond simple geometry is another natural way to extend approach. Another limitation is while the quadratic cost assures global convexity and training stability, it limits the expressiveness and complexity of the cost function. Furthermore, the cost function is learned individually for each navigation scenario, but it is unclear how one stand-alone learned cost function can handle multiple scenarios. Also, our user study pilot questionnaire could be refined, and social evaluations could be expanded to broader set of scenarios.
5 Conclusions
We present this paper Performer-Ne MPC learnable MPC system utilizing scalable Transformers to learn rich context representations parameterizing trainable cost function. We show that Performer- MPCs can be used as robotic controllers for navigation where regular MPCs struggle, including learning to avoid local minima, to maneuver through highly constrained spaces, and adhere to unwritten social norms, while maintaining real-time speed, even for nearly Pixel-to-Pine Pixel attention.
challenging real-world environments

---

References
[1] C. Mavrogiannis, F. Baldini, A. Wang, Zhao, P. Trautman, A. Steinfeld, and Oh. Core challenges of social robot navigation: A survey, arXiv preprint arXiv:2103.05668 2021.
[2] X. Xiao, Z. Xu, Z. Wang, Y. Song, G. Warnell, P. Stone, T. Zhang, S. Ravi, G. Wang. H. Kar- nan, et al. Autonomous ground navigation in highly constrained spaces: Lessons learned from the barn challenge at iera 2022. arXiv preprint arXiv:2208. 10473, 2022.
[3] C. E. Garcia, D. M. Prett, and M. Morari. Model predictive control: Theory and practice survey, Automatica, 25(3):335- 348, 1989.
[4] Y. Wang and S. Boyd. Fast model predictive control using online optimization. IEEE Trans- actions on control systems technology, 18(2):267-278, 2009.
[5] S. Maniatopoulos, D. Panagou, and K. J. Kyriakopoulos. Model predictive control for the navigation of nonholonomic vehiele with field-of-view constraints. In 2013 American control conference, pages 3967-3972. IEEE, 2013.
[6] T. Fork, H. E. Tseng, and F. Borrelli. Models and predictive control for nonplanar vehicle navigation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 749-754, IEEE, 2021.
[7] Everyday Robots. Everyday robots. http:://eeeryyarroootss...... 2022.
[8] Amazon. Meet scout. meet-scout, 2022.
[9] Starship. Starship.
2022.
[10] A. Elfes. A tesselated probabilistic representation for spatial robot perception and navigation. In JPL, California Inst. of Tech., Proceedings of the NASA Conference on Space Telerobotics, Volume 2, 1989.
[11] A. Elfes. Using occupancy grids for mobile robot perception and navigation. Computer, 22 (6):46-57, 1989.
[12] Dellaert, M. Kaess, et al. Factor graphs for robot perception. Foundations and Trends@ in Robotics, 6(1-2):1-13, 2017.
[13] S. Rabiee and J. Biswas. A friction-based kinematic model for skid-steer wheeled mobile robots. In 2019 International Conference on Robotics and Automation (ICRA), pages 8563- 8569. IEEE, 2019.
[14] Y. Tian and N. Sarkar. Control of a mobile robot subject to wheel slip. Journal of Intelligent & Robotic Systems, 74(3):915-922 2014.
[15] D. Lu, D. Hershberger, and W. D. Smart. Layered costmaps for context-sensitive navigation. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 709- 715. IEEE, 2014.
[16] L. Jaillet, J. Cortés, and T. Siméon. Sampling- path planning configuration-space costmaps. IEEE Transactions on Robotics, 26(4))635-646, 2010.
[17] K. Zheng. Ros navigation tuning guide. In Robot Operating System (ROS), pages 197-226. Springer, 2021.
[18] OSRF. Ros wiki base. http:://lkk..ros.rrr//oooo.oo. 2018. Accessed: 2022- 06-12 12.
[19] M. McNaughton, C. Urmson, J. M. Dolan, and J.-W. Lee. Motion planning for autonomous driving with conformal spatiotemporal lattice. In 2011 IEEE International Conference Robotics and Automation, pages 4889-4895. IEEE, 2011.
[20] D. Fox, Burgard, and S. Thrun. The dynamic window approach to collision avoidance. IEEE Robotics & Automation Magazine, 4(1):23-33, 1997.

---

[21] S. Quinlan and O. Khatib. Elastic bands: Connecting path planning and control. In [1993] Pro- ceedings IEEE International Conference on Robotics and Automation, pages 802-807. IEEE, 1993.
[22] X. Xiao, B. Liu, G. Warnell, and P. Stone. Motion planning and control for mobile robot navigation using machine learning: a survey, Autonomous Robots, pages 1-29, 2022.
[23] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint 1604.00600.033 2016.
[24] M. Pfeiffer, M. Schaeuble, J. Nieto, R. Siegwart, and C. Cadena. From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots. In 2017 ieee international conference on robotics and automation (icra), pages 1527-1533. IEEE, 2017.
[25] Z. Xu, X. Xiao, G. Warnell, A. Nair, and Stone. Machine learning methods for local motion planning: A study of end-to-end vs. parameter learning. In 2021 IEEE International Sympo- sium on Safety, Security, and Rescue Robotics (SSRR), pages 217-222. IEEE, 2021.
[26] Z. Xu, Liu, X. Xiao, A. Nair, and Stone. Benchmarking reinforcement learning techniques for autonomous navigation. Technical report, The University of Texas at Austin, 2022.
[27] X. Yao, J. Zhang, and J. Oh. Following social groups: Socially compliant autonomous naviga- tion dense crowds. arXiv preprini arXiv: 19x:vv:91111.
2019.
[28] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-base -ased planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5113-5120 IEEE, 2018
[29] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2))2007-2014, 2019.
[30] X. Xiao, B. Liu, G. Warnell, and P. Stone. Toward agile maneuvers in highly constrained spaces: Learning from hallucination. IEEE Robotics and Automation Letters, 602):1503-1510, 2021.
[31] X. Xiao, B. Liu, and P. Stone. Agile robot navigation through hallucinated learning and sober deployment. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7316-7322. IEEE, 2021.
[32] Z. Wang. X. Xiao, A. J. Nettekoven, K. Umasankar, A. Singh, S. Bommakanti, U. Topcu, and Stone. From agile ground to aerial navigation: Learning from learned hallucination. In 2027 IEFF/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 148-153. IEEE, 2021.
[33] B. Liu, X. Xiao, and P. Stone. A lifelong learning approach to mobile robot navigation. JEEE Robotics and Automation Letters, 6(2): 10900100001110 1096, 2021.
[34] K. S. Sikand, S. Rabiee, A. Uccello, X. Xiao, G. Warnell, and J. Biswas. Visual representa- tion learning for preference-aware path planning. In 2022 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2022.
[35] Drews, G. Williams, B. Goldfain, E. A. Theodorou, and J. M. Rehg. Aggressive deep driving: Model predictive control with a enn cost model. arXiv preprint arXiv:1707.05300 2017.
[36] M. Wigness, G. Rogers, and L.. Navarro-Serment. Robot navigation from human demon- stration: Learning control behaviors. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1150-1157. IEEE, 2018
[37] B. Kim and Pineau. Socially adaptive path planning in human environments using inverse reinforcement learning. International Journal of Social Robotics, 8(1):51-66, 2016.
10

---

[38] H. Karnan, K. S. Sikand, P. Atreya, S. Rabiee, X. Xiao, G. Warnell, P. Stone, and J. Biswas. Vi-ikd: High-speed accurate off-road navigation using learned visual-inertial inverse kinody- namics. arXiv preprint arXiv:2203. 15983, 2022.
[39] X. Xiao, J. Biswas, and P. Stone. Learning inverse kinodynamics for accurate high-speed off- road navigation on unstructured terrain. IEEE Robotics and Automation Letters, 6(3):6054- 6060, 2021.
[40] D. Deso--F--eetño, E. Zulueta, U. Frmandez-Gamiz A. Saenz-Aguire, and R. Martinez. Predictive dynamic window approach development with artificial neural fuzzy inference im- provement. Electronics, 8(9):935, 2019.
[41] M. Bhardwaj. B. Boots, and M. Mukadam. Differentiable gaussian process motion planning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10598- 10604. IEEE, 2020.
[42] X. Xiao, Z. Wang, Z. Xu, B. Liu, G. Warnell, G. Dhamankar, A. Nair, and P. Stone. Appl: Adaptive planner parameter learning. Robotics and Autonomous Systems, 154:104132, 2022.
[43] X. Xiao, B. Liu, G. Warnell, J. Fink, and P. Stone. Appld: Adaptive planner parameter learning from demonstration. IEEE Robotics and Automation Letters, 5(3):4541-4547, 2020.
[44] Z. Wang, X. Xiao, B. Liu, G. Warnell, and P. Stone. Appli: Adaptive planner parameter learn- ing from interventions. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2021.
[45] Z. Wang, X. Xiao, G. Warnell, and Stone. Apple: Adaptive planner parameter learning from evaluative feedback. IEEE Robotics and Automation Letters, 6(4):7744-7749, 2021.
[46] Z. Xu, G. Dhamankar, A. Nair, X. Xiao, G. Warnell, B. Liu, Z. Wang, and P. Stone. Applr: Adaptive planner parameter learning from reinforcement. In 2021 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 6086-6092. IEEE, 2021.
[47] M. Bajracharya, A. Howard, L. H. Matthies, B. Tang, and M. Turmon. Autonomous off-road navigation with end-to-end learning for the lagr program. Journal of Field Robotics, 26(1): 3-25, 2009.
[48] J. A. Bagnell, D. Bradley, D. Silver, B. Sofman, and A. Stentz. Learning for autonomous navigation. IEEE Robotics & Automation Magazine, 17(2):74-84, 2010.
[49] R. Mirsky, X. Xiao, J. Hart, and P. Stone. Prevention and resolution of conflicts in social navigation-a a survey. arXiv preprint arXiv:2106 12113, 2021.
[50] V. Tolani, S. Bansal, A. Faust, and C. Tomlin. Visual navigation among humans with optimal control supervisor. IEEE Robotics and Automation Letters, 6(2):2288-223) 2021.
[51] Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference Robot Learning, pages 158-168. PMLR, 2022.
[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[53] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018
[54] T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers. 1rXiv:2106.04555 2021.
arXiv preprint
[55] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2:X1v:2204.02311, 2022.
11

---

[56] T. Brown, B. Mann, N. Ryder. M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[57] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, Fi- rat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Machine Learning, pages 5547-5569. PMLR, 2022.
Conference
[58] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
[59] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhu- moye, G. Zerveas, V. Korthikanti, et al. Using deepspeed and megatron to train megatron- turing nig 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11999 2022.
[60] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[61] B. Zhang, S. Gu, B. Zhang. J. Bao, D. Chen, E. Wen, Y. Wang, and B. Guo. Styleswin: Transformer-basee gan for high-resolution image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11304-11344 2022.
[62] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthe- sis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[63] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International conference on machine learning. pages 1691-1703, PMLR, 2020.
[64] L. Zhao, Z. Zhang, T. Chen, D. Metaxas, and H. Zhang. Improved transformer for high- resolution gans. Advances in Neural Information Processing Systems, 34:18367-18880, 2021.
[65] K. Lee, H. Chang, L. Jiang, H. Zhang, Z. Tu, and C. Liu. Vitgan: Training gans with vision transformers, arXiv preprint arXiv:2107.04588) 2021.
[66] Y. Jiang, S. Chang, and Z. Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34:14745-14758, 2021.
[67] J.BB. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Mil- lican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[68] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[69] J. Wang. Z. Yang. X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2410000000.. 2022.
[70] R. Hu and A. Singh. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439- 1449, 2021.
[71] V. Likhosherstov, A. Arnb, K. Choromanski, M. Lucic, Y. Tay, A. Weller, and M. De- hghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2/11. 122993,, 2021.
[72] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sind- hwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reason- ing with language. arXiv preprint 12XXi:2204400598 2022.
12

---

[73] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, Mohiuddin, L. Kaiser, D. B. Belanger, J. Colwell, and Weller. Rethinking attention with performers. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[74] F. Borrelli, A. Bemporad, and M. Morari. Predictive control for linear and hybrid systems. Cambridge University Press, 2017.
[75] B. Rawlings. Tutorial overview of model predictive control. IEEE control systems magazine, 20(3):38-53 (20::388522200 2000.
[76] S. Levine and V. Koltun. Continuous inverse optimal control with locally optimal examples. arXiv preprint 1rXiv:1206.4000 126..417,7 2012.
[77] K. Mombaur, A. Truong, and J.-P. Laumond. From human to humanoid locomotion in- verse optimal control approach. Autonomous robots, 28(3)369-383, 2010.
[78] S. Boyd, L. EI Ghaoui, E. Feron, and V. Balakrishnan. Linear matrix inequalities in system and control theory. SIAM, 1994.
[79] M. Palan, S. Barratt, A. McCauley, D. Sadigh, V. Sindhwani, and S. Boyd. Fitting linear control policy to demonstrations with a kalman constraint. In Learning for Dynamics and Control, pages 374-383. PMLR, 2020.
[80] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter. Differentiable mpe for end-to-end planning and control. In Advances in Neural Information Processing Systems, pages 8289- 8300, 2018.
[81] E. Pauwels, Henrion, and J.-B. Lasserre. Inverse optimal control with polynomial optimiza- tion. In 53rd IEEE Conference on Decision and Control, pages 5581-5586. IEEE, 2014.
[82] D. Bertsekas. Nonlinear programming. Journal of the Operational Research Sociery, 48(3): 334-334, 1997.
[83] Lorraine, Vicol, and D. Duvenaud. Optimizing millions hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552, 2020.
[84] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable convex optimization layers. In Advances in neural information processing systems, pages 9562-9574, 2019.
[85] A. Agrawal, S. Barratt, S. Boyd, and B. Stellato. Learning convex optimization control policies. arXiv preprint arXiv: 1912.0999,, 2019.
[86] M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-López, F. Pedregosa, and J.PP. Vert. Efficient and modular implicit differentiation. arXiv preprint 1rXiv:2105.1518. 2021.
[87] C. Dunn and D. P. Bertsekas. Efficient dynamic programming implementations of new- ton't 's method for unconstrained optimal control problems. Journal of Optimization Theory and Applications, 63(1):23-38, 1989.
[88] S. J. Wright. Solution of discrete-time time optimal control problems on parallel computers. Parallel Computing, 16(2-3):221-2377 1990.
[89] W. Li and E. Todorov. Iterative linear quadratic regulator design for nonlinear biological move- ment systems. In ICINCO (1), pages 222-229. Citeseer, 2004.
[90] D. Murray and S. Yakowitz. Differential dynamic programming and newton's method for discrete optimal control problems. Journal of Optimization Theory and Applications, 43(3): 395-414, 1984.
13

---

[91] A. Dosovitskiy. L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[92] P. Ramachandran, N. Parmar, A. Vaswani, 1. Bello, A. Levskaya, and J. Shlens. Stand-alone self-attention in vision models. Advances in Neural Information Processing Systems, 32, 2019.
[93] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, Uszkoreit, and L.. Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021.
[94] R. Frostig, V. Sindhwani, S. Singh, and S. Tu. trajax: differentiable optimal control on accel- erators, 2021. URL http://github. com/
[95] Pirk, Lee, X. Xiao, L.. Takayama, A, Francis, and A. Toshev. A protocol for validating social navigation policies, 2022.
[96] R. Likert. A technique for the measurement of attitudes. Archives of psychology, 1932.
[97] H. Karnan, A. Nair, X. Xiao, G. Wamell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters, 2022.
[98] M. Ahn, Brohan, N. Brown, Chebotar, O. Cortes, B. David, Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204 01691, 2022.
14


Reviews:
[{"review_id": "zV2slnbjCtV", "reviewer": "Reviewer_yaP4", "summary": "The paper proposes an approach that allows incorporation of complex rules (such as social norms, obstacle avoidance in cluttered scenarios) into the behavior generated by standard MPC policies. The main idea of the approach is based around transformer-generated cost functions trained through imitation learning (behavior cloning in this instance). The authors demonstrate effectiveness of their approach on a real robotics platform in multiple scenarios, including navigation in cluttered scenes and socially-acceptable navigation in human-occupied environments. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Praise:\n* The work tackles an important problem of incorporating complex behaviors, while retaining advantages of the existing control approaches: optimality, interpretability.\n* The work conducts interesting experiments on a real robotics platform and demonstrates some level of generalization on novel (albeit similar) cases.\n\n\nConcerns:\n* Literature overview is not complete: Authors cite a lot of papers for transformers in unrelated/weakly-related fields (which is definitely excessive), but the section on inverse-RL and cost learning is tiny. Even the relevant papers cited by the authors are just mentioned: there is no detailed comparison, i.e. how the current work advances already existing methods? It makes it harder to appreciate what has been done in the paper. Just to name a few other examples from the existing relevant body of literature that is worth including / comparing to:\n   * Sermanet et al. Time-Contrastive Networks: Self-Supervised Learning from Video.\n   * Hsu et al. Unsupervised Learning Via Meta-Learning\n   * Finn et al. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization.\n   * Bechtle et al. Meta-Learning via Learned Loss.\n\n   * Each task is learned separately, hence, it brings a question how to stitch them later. Although the authors explicitly mention this limitation it renders results quite a bit less impressive. Also, the experiments are conducted in somewhat similar environments and do not exhibit strong distributional shifts under proposed inputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work tackles an important problem and demonstrates interesting results. The paper is decently written and generally easy to follow. \nAmong the issues are i) somewhat weak comparison to the existing body of work and ii) overall results could demonstrate better generalization under multiple tasks and stronger distributional shifts. \nThe ideas and results are not revolutionary by any means, but I appreciate the amount of work put into producing this paper - it has decent quality and results look convincing, hence I am inclined to accept it.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See my CONCERNS section", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "dVLi_Xjx6mN", "reviewer": "Reviewer_UiTD", "summary": "This paper presents a novel way of augmenting the cost function of a Model Predictive Controller by allowing the cost function to be a combination of a fixed one and a learned one. The learned part of the cost function takes the most general form of non-linear least squares and learns the matrix coefficients directly using the embeddings of a vision transformer architecture. They use Performers instead of Transformers to speed up the training and forward pass. Using bi-level optimization, the gradients are propagated through the MPC optimizer.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths and learnings from this paper are:\n - The paper is very well written and very self-explanatory. The figures are very well chosen, and the theoretical part is rigorous and sound.\n - It’s the first approach (to my knowledge) that uses vision transformers in combination with an MPC in the loop to learn the cost function of said MPC. It can be an inspiration for future researchers, and it has the potential of being, in my opinion, a very good research direction.\n - It uses bi-level optimization to propagate the gradients through an optimizer. I believe that using this tool is a very elegant way of getting the best of both worlds, learning-based and model-based, to work together in what they respectively excel at.\n\nAnd weaknesses:\n - While the approach renders very good results and it’s elegant, it is compared with rather naive baselines. Furthermore, the existence of the MPC in the loop is not ablated in any way (since in all approaches there is a “Tracking MPC” problem that is solved at the end).\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this paper can have a major impact on the way that Model Predictive Control is used together with learning-based approaches. The tendency to use control in the last layer in robotics applications is gaining more and more popularity, and I believe that it can be a natural way of including model-based guarantees in an otherwise learning-based architecture.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": " - Have the authors tried not to use a hand-designed cost function? There is no mention as to why the authors need to use this hand-designed cost function. It would be worth commenting about this.\n - It would be very nice to have (or comment on) an extra comparison against end-to-end policies trained using imitation learning in order to understand better the benefits of using an MPC in the loop.\n - Throughout the paper, it’s mentioned that the “Tracking MPC” is used for all the approaches, and in the Appendix, this decision is argued by stating that the comparisons with the other baselines are fairer because all approaches use it. But then, the entire approach is doing planning and not control. The tackled problem is the generation of a trajectory that is then tracked with an MPC. I believe that this fact deserves more explanations.\n - “Real-Time Attention” is in the title of the manuscript, and the application of “pixel-to-pixel attention mechanisms in MPC” is one of your contributions. However, there are no ablations regarding the attention mask. How are the attention mechanisms helping here? It would be interesting to see what the agent pays attention to in different scenarios, e.g., when avoiding a human or when going through the door.\n - The approach assumes knowledge of the state of the robot, but this fact is only mentioned in the appendix (we get the state of the robot from a SLAM pipeline). It would be desired to highlight this fact earlier in the manuscript since it’s an important aspect of the approach.\n"}}, {"review_id": "RIhQYbz3_9y", "reviewer": "Reviewer_3ALJ", "summary": "This paper introduces a novel learnable MPC approach by enhancing the traditional MPC cost with learnable components. These learnable components use transformer-based embeddings of real-world context. The overall model is learned in imitation learning fashion by leveraging expert demonstrations. Various optimizations are considered to enable the model to run in real-time. Through experiments, the model is less likely to be stuck in local optimum and are more social when compared to the traditional MPC. The model is also less likely to run into obstacles when compared to pure expert demonstration models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is very clearly written. The authors have very cleanly formulated their approach and provide good reasoning describing every single step. I also concur that learnable-MPC combines the best of both worlds with the learning part being able to encode complex environmental and social context and the MPC part providing safety guarantees to some extent. The experiments are also carefully designed around 4 example scenarios and support the authors' claims.\n\nI still have a few questions about the experiments:\n- For learning highly constrained maneuvers, are the human expert demonstrations conducted in the same cluttered environment as in the evaluation? How well does Performer-MPC perform in unseen cluttered environments when compared to RMPC. (Or how does RMPC perform because I saw Performer-MPC generalizes in one instance shown in the video)\n- For blind corner and pedestrian obstruction scenarios, I do not see any statistical analysis. Are the comparisons statistically significant among the social score ratings of RMPC, EP and Performer-MPC? How many subjects are there? Are these the same as the pilot studies mentioned in the supplementary materials?\n\nI am also curious about how this method generalizes to more complex pedestrian scenarios, such as navigating through a crowd. How many demonstrations are needed to be successful in this? Crowd behaviors are highly variable and unconstrained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper's approach is novel and can have a very good impact on the research community. Its experiment designs are pretty thorough, but a few questions remain. My suspicions are RMPC might also work well in unseen cluttered environments. And Performer-MPC may not be statistically better than EP when it comes to social score ratings.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Copied from strengths and weaknesses:\n- For learning highly constrained maneuvers, are the human expert demonstrations conducted in the same cluttered environment as in the evaluation? How well does Performer-MPC perform in unseen cluttered environments when compared to RMPC. (Or how does RMPC perform because I saw Performer-MPC generalizes in one instance shown in the video)\n- For blind corner and pedestrian obstruction scenarios, I do not see any statistical analysis. Are the comparisons statistically significant among the social score ratings of RMPC, EP and Performer-MPC? How many subjects are there? Are these the same as the pilot studies mentioned in the supplementary materials?\n\nThe reference section may need some organizing: For example, I saw a double reference [1] and [48]."}}, {"review_id": "Q3fk0pPnXHL", "reviewer": "Reviewer_Xcqt", "summary": "This paper presents a novel learning-based MPC approach where the cost function is learned using the expressiveness of deep neural networks such as transformers. These extract the cost directly from context information and trained using imitation learning while the MPC is solved using classic optimization techniques. The proposed solution is compared with two main baselines. The first one includes is a classic MPC whereas the second one is an explicit policy method that directly generates control actions without using the MPC structure.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the approach is interesting. However, there are several aspects that are not fully convincing. First, it is not clear why the proposed representation as input to generate the final latent embedding is selected. How other representations affect the learning module. This is not clear. Second, the current setup is tested and analyzed in separate scenarios and every time needs to be separately retrained. This is a huge limitation of this approach and it is not clear if the architecture is able to learn different scenarios at the same time. The network is tested except for minor variation in the same scenario or task it was trained for. Third, it is not clear by using a Performer type of architecture rather than the original transformer if there is any loss in performance or in the ability to learn the cost function. It would have been interesting to have some study about that at least showing what the different architectures learn in terms of costs if not possible to deploy all of them directly in the MPC for computational reasons. This can also be extended considering different architectures beyond transformers. Finally, it is not clear what is the model used in the current robot and at which level of abstraction the control commands are sent and how some constraints like pedestrian detections are incorporated in the regular MPC approach. An appropriate constraint should not let the robot get stuck in front of the human. The proposed comparison is probably not fair in this case.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The approach is interesting, but the there are several limitations that have been highlighted that prevent its ability to be fully utilized in real-world settings.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "As explained in the other section there are several issues that have been summarized.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)