Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
Authors: Atanas Mirchev
Baris Kayalibay
Patrick van der Smagt
Justin Bayer
Venue: ICLR 2021 Poster (2021)
Keywords: Generative models
Bayesian inference
Variational inference
SLAM
Deep learning

Abstract: We solve the problem of 6-DoF localisation and 3D dense reconstruction in spatial environments as approximate Bayesian inference in a deep state-space model. Our approach leverages both learning and domain knowledge from multiple-view geometry and rigid-body dynamics. This results in an expressive predictive model of the world, often missing in current state-of-the-art visual SLAM solutions. The combination of variational inference, neural networks and a differentiable raycaster ensures that our model is amenable to end-to-end gradient-based optimisation. We evaluate our approach on realistic unmanned aerial vehicle flight data, nearing the performance of state-of-the-art visual-inertial odometry systems. We demonstrate the applicability of the model to generative prediction and planning.

Published as a conference paper at ICLR 2021
VARIATIONAL STATE-SPACE MODELS FOR LOCALISATION AND DENSE 3D MAPPING IN 6 DoF
Atanas Mirchev
Baris Kayalibay
Patrick van der Smagt
Justin Bayer
Machine Learning Research Lab, Volkswagen Group, Munich, Germany (atanas mi rchev, bkayal bay, bayer @ rgmax.
ABSTRACT
We solve the problem of 6-DoF localisation and 3D dense reconstruction in spatial environments as approximate Bayesian inference in deep statesppce model. Our approach leverages both learning and domain knowledge from multiple-view geometry and rigid-body dynamics. This results an expressive predictive model of the world, often missing in current state-of-the-art visual SLAM solutions. The combination of variational inference, neural networks and a differentiable raycaster ensures that our model is amenable to end-to-end gradient- paadee optimisation. We evaluate our approach on realistic unmanned aerial vehicle flight data, nearing the performance of state-of-the-ar visual-inertial odometry systems. We demonstrate the applicability of the model to generative prediction and planning.
INTRODUCTION
We address the problem of learning representations of spatial environments, perceived through RGB-D and inertial sensors, such as in mobile robots, vehicles or drones. Deep sequential generative models appealing, wide range of inference techniques such as state estimation, system identification, uncertainty quantification and prediction is offered under the same framework (Curi et al., 2020; Karl al., 2017a; Chung al., 2015). They can serve so-called world models or environment simulators (Chiappa et al., 2017; Ha & Schmidhuber, 2018 1), which have shown impressive performance on variety of simulated control tasks due to their predictive capability. Nonetheless, learning such models from realistic spatial data and dynamics has not been demonstrated. Existing spatial generative representations are limited to simulated 2D and 2.5D environments (Fraccaro et al., 2018).
On the other hand, the state estimation problem in spatial Swvironmentt-SSSAM a variety real-world settings, including cases with real-time constraints and on embedded hardware (Cadena et al., 2016; Engel et al., 2018; Qin et al., 2018; Mur-Arta & Tardós, 2017). While modern visual SLAM systems provide high inference accuracy, they lack predictive distribution, which is a prerequisite for downstream perception-cone control loops.
been solved in
Our approach scales the above deep sequential generative models to real-world spatial environments. To that end, integrate assumptions from multiple-vin view geometry and rigid- dynamics com- monly used modern SI.AM systems. With that, our model maintains the favourable properties of generative modelling and enables prediction. We the recently published approach of Mirchev et al. (2019) as a starting point, in which variational state-space model, called DVBF-LM, is extended with spatial map and attention mechanism. Our contributions are as follows:
We use multiple-view geometry to formulate and integrate differentiable raycaster, an attention model and a volumetric map.
We show how to integrate rigid-body dynamics into the learning of the model.
We demonstrate the successful use of variational inference for solving direct dense SLAM for the first time, obtaining performance close to that of state-of-the-art localisation methods.
We demonstrate strong predictive performance using the learned model, by generating spatially-come consistet real-world drone-fligh data enriched with realistic visuals.
We demonstrate the model's applicability to downstream control tasks by estimating the cost-to-go for collision scenario.

---

Published as a conference paper at ICLR 2021
slar, mus spoed
Prolicaa
colorr
depth
depth
MOCAP
proposed
[m]
Figure I: Illustration of the proposed quadcopter localisation and dense mapping. Left: top-down view of the localisaton estimate. Right: generative depth and colour reconstructions for one time step.
The contributions allow the reformulated model to tackle realistic RGB-D scenarios with 6 DoF.
2 RELATED WORK
Generative models for spatial environments GTM-SM (Fraccaro et al., 2018) focuses on long- term predictions with a non-metric deterministic external memory. Chaplot al. (2018) formulate an end-to-end learning model for active global localisation, filtering with likelihood update predicted by a neural network. The agent can turn in four directions and move on a plane, perceiving images of the environment. VAST (Corneil et al., 2018) assumes discrete state space for generative model applied to the 2.5D Vizdoom environment. Whittington et al. (2018) model agents moving on a grid and place cells. Other works propose end-to-end
2D grid with latent neurologically-nnspo learnable generative scene models (Eslami et al., 2018; Engeleke et al., 2020), without considering the agent dynamics. Like in the above, put major emphasis the generative predictive distribution of our model. With it, the agent can imagine the consequences of its future actions, a prerequisite for data-efficient model-based control (Chua et al., 2018; Hafner et al., 201 20;9;;;; Becker-Ehnne et al., 2020). However, the aforementioned deep generative spatial models have only been applied on simulated 2D, 2.5D (movement restricted to plane) and very simplified 3D environments.
A major challenge when scaling to the real world is to ensure that the learned components, and in turn the generative predictions, generalise observed but yet unvisited places. Gregor et (2019) highlight another problem, that of long-term consistency when predicting ahead, and address it by learning with overshooting. In contrast, our method resolves these issues by injecting a sufficient amount of domain knowledge, without limiting the flexibility w. t. learning. To this end, we begin by sharing the probabilistic factorisation of DVBF-LM (Mirchev al., 2019), a deep generative model that addresses the tasks of localisation, mapping, navigation and exploration in 2D. We then redefine the map, the attention, the states, the generation of observations and the overall inference, allowing for real-worle 3D modelling and priming method for data-efficient online inference in the future. We discuss why these changes are necessary more thoroughly in appendix A.
Combining learning and spatial domain knowledge Fully-learned spatial models with an explicit memory component have been studied by Parisotto & Salakhutdinov (2018); Zhang et al. (2017); Oh et al. (2016). Further relying on geometric knowledge, Tang & Tan (2019) propose learning through the whole bundle adjustment optimisation, formulated on CNN feature maps of the observed images. Czarnowski et al. (2020) define a SLAM system based on learned latent feature codes of depth images, a continuation of the works by Zhi al. (2019); Bloesch et al. (2018). Factor- graph maximum posteriori optimisation is then conducted, substituting the observations for their respective low-dimensional codes, leading to point estimates of the individual geometry N keyframes and the agent poses over time. Wei et al. (2020) maintain cost volumes (Newcombe et al., 2011 1) for discretised poses and depth, and let a 3D CNN learn how to predict the correct geometry and pose estimates from them. Depth cost volumes are also used by Zhou et al. (2018) in learning predict depth and odometry with neural networks. In the work by Yang et al. (2020), networks that predict odometry and depth are combined with DSO, leading SLAAM system that utilises learning to its
2

---

Published as a conference paper at ICLR 2021
advantage. Jatavallabhula et al. (2019) investigate differentiable SLAM, treating odometry estimation and mapping separately. The considered rendering gradients in that method are from the fused map to the observations, which is the opposite of the gradient paths used for learning in work.
As in our approach, the geometric assumptions in the majority of these works allow the systems to generalise more easily to unseen cases and real-world data. What distinguishes explicit generative model, able to predict the agent movement and observations in the future. Addi- tionally, our approach is probabbioobbi,,,,,,, maintaining complete distributions over the variables of interest, whereas the aforementioned approaches are not. Our method is also end-to-end differentiable and can be implemented in auto-diff frameworks, welcoming learned components. We are bridging the gap between probabilistic generative models, learning and spatial domain knowledge.
method is an
Depth estimation and differentiable rendering Recent promising approaches combine learning and non non-paramttric categorical distributions for depth estimation (Laidlow et al., 2020; Liu et al., 2019), fusing likelihood terms into a consistent depth estimate. Such depth estimation is compatible with system and can be used formulate priors, but for now we rely as a first step (Hirschmuller, 2007). Inferring whole scenes parameterised with neural networks by backpropagation through realistic differentiable rendering has also become a prevalent direction of research (Bi et al., 2020; Mildenhall et al., 2020; Sitzmann et al., 2020). In our method the occupancy and colour map are inferred in similar way, but the raycasting scheme we follow is simple, meant only to illustrate the framework as a whole. We note that the current inference times of c.g. Bi al. (2020) amount to days (see appendix of that work), which is hard scale to online inference. Extending our approach with more advanced rendering method is the subject of future work.
traditional method
Bayesian SLAM inference To keep exposition brief, we refer to (Cadena et al., 2016) for an overview of modern SLAM inference and focus only on approaches that have applied Baly-BBaysiinn methods to SLAM. The inference in this work can be categorised as probabilistic SLAM, other prominent examples of which are FastSLAM (Montemerlo al., 2002) and RBPF SLAM (Grisetti et al., 2005). What distinguishes our method is the application variational inference with SGVB (Kingma& & Welling, 2014). Our model does not restrict the used distributions and allows any differentiable functional form, which enables us to use neural networks. The contribution by Murphy (Murphy, 1999) one of the first infer global map with Bayesian methods. Bayesian Hilbert maps (Senanayake & Ramos, 2017) focus on a fully Bayesian treatment of Hilbert maps for long-term mapping dynamic environments. Stochastic variational inference is used to infer agent poses from observed 2D image features in (Jiang et al., 2017; Jiang et al., 2019). DVBF-LM (Mirchev et al., 2019) uses Bayes by Backprop (Blundell et al., 2015) for the inference of the global map variable.
3 METHOD
Background We adhere the graphical model of DVBF-LM (Mirchev et al., 2019), but introduce novel design choices for every model component and implement the overall inference differently, allow for real-world 3D modelling. In the following, we will first describe the assumed factorisation and then explain the introduced modifications. The assumed joint distribution of all variables is:
p(XIT, ZL.T MI.T M UITT-1) T
T-1
= =p(M)p(zi II p(x: mm)p(m;) 24, M) II p(zc-1 ze, 4:),
(1)
where X1:7 are observations, 21.T agent states, mi.7 map charts and u1:r- conditional inputs (controls). The factorisation defines traditional state-space model extended with global map variable M. For a single step I, an observation X; is generated from a map chart m,-the relevant extract from the global M around the current agent pose (cf. fig. 2a). Chart extraction is given by p(m; M)) which can be seen as an attention mechanism. In this graphical model, SLAM is equivalent to inference of the agent states Zi.7 and the map M. For the remainder of this work, we assume all observations x, E R xxxx are RGB-D images. Next, we will describe the functional forms of the map M, the attention pl (m, 24, M), the emission p(x; m.) and the states 21.7.
3

---

Published as a conference paper at ICLR 2021
1.0
oscuppanc values
XBI
0.
observed
04
Ag
M
M
0.
6.25
6.50
7.00
1.75
depth values
(b)
Figure 2: (a) One time step of the proposed probabilistic graphical model. (b) Linear interpolation during ray casting for a single ray the emission model. dg is the depth corresponding to the first ray value that exceeds The output depth is formed by linearly interpolating between dę-1 and dy based on the occupancy values o o.J.k-1 and o.d.k.
Geometric map The map random variable M = (M _Mao)) consists of two components.
M E xixxx a spatially arranged 3D grid of scalar values that represent occupancy. M represents the parameters of assigns an RGB colour value to each point in space. In this work, the network weights are deterministic and point-estimated via maximum likelihood, the fully-Bayesian treatment of the colour map is left for future work. The prior and approximate posterior distributions over the occupancy map are:
neural network fM R - [0, 255]s. The network
p( = II II 1,, (M( = II N GJ.k
G5.k Pig.es R...).
i.J.k
Here and for the rest of this work 9s will denote variational approximate posterior distribution, with all its optimisable parameters summarised in Φ. We assume p(Moss) and qo (Mccc) factorise over grid cells. The variational parameters pi.jkk Oij.,k optimised with Bayes by Backprop (Blundell et al., 2015).
Attention In the proposed model, the composition of the attention p(m; Z₁, M) and the emission understanding of
p(x; m,) implements volumetric raycasting. We engineer them based geometry to ensure generalisation across unseen environments. The attention p(m; 2t, M) forms latent charts m,, which correspond to extracts from the map M around 2,- We identify m, with the part of the map contained the frustum of the current camera view. To attend to that region, first the intrinsic camera matrix K (assumed be known) and the agent pose Z; are used to cast ray for any pixel [i in the reconstructed observation. The ray is then discretised equidistantly along the depth dimension into r-many points, resulting into a collection of 3D world coordinates P. E I xxxxxxxx3 Depth candidate values d E (ke] are associated with each point along ray, where is a resolution hyperparameter. The latent chart m, = (Or, C.) factorises into an occupancy chart O E IR wxxxxr and a colour chart C₁ E Ruxxxrx3 Let pt E R be 3D point in the spanned camera frustum. To form the occupancy chart o,, cells from the map M around p are combined with weighted kernel o.tk Ei.., M,,,, (1,A (nisk). Note that here 1, h, indices of the occupancy map voxels. We choose trilinear interpolation kemnel for merging only eight map cells per point. This makes the attention fast and differentiable w.r.t 2,. The colour chart c, = f (pa) is formed by applying f the colour neural network, point-wise to each 3D point. In this work. we keep the chart m, deterministic. The full attention procedure can be described as:
p(m; Zt. M) = II 6(m 5k SA(M, (ik)) Bisk = T(z,) -i. 1. i5k
Here T(z;) SE(3) denotes the rigid camera transformation defined by the current agent state z, and j, l index the points lying inside the attended camera frustum.
4

---

Published as a conference paper at ICLR 2021
Emission through ray casting The emission model factorises over the observed pixels:
p mi)
I mc), p(x m,)
p(d? of Or, Ce)-
operates on the extracted chart m, (or: ct). Here x E IR+ denotes an RGB-D pixel value, i.c. for each pixel [7, j] we reconstruct a depth de and a colour value c." The mean of the depth value dy is formed by a function SE: fE(O,) = min k S.L. of jkk
> 7.
kEir]
SE traces the ray for pixel [ searching for the minimum depth d = ck for which the occupancy value of exceeds a threshold 5 (a hyperparameter). Since the above min operation is not differen- tiable os, we linearly interpolate between the depth value for the first ray hit and its predecessor to form the mean of the emitted depth (cf. fig. 2b):
(o.) + (1 - (fEE de
a
J. -J.A-T' 02
The mean of the emitted colour fe, eith directly corresponds the k-th element of the attended colour values, where & is the index of the first hit from raycasting above. A heteroscedastic Laplace distribution is assumed for both the emitted depth and colour values:
p(x² m.) Laplace (x (vY, (²))) diag(o
Agent states All agent states are represented as vectors 2, = (Ae, We 2,tt) E R.d. X E R the agent location in space. W, E H is the agent orientation, represented
quaternion. 2, E Rd.-7 remainder. Depending on the used transition model, z,"t can be Å, alone or it can contain an abstract latent portion not explicitly matching physical quantities. The approximate posterior variational family over the agent states factorises over time:
(o(Z)T) = II go(2)) II N((z nº (i.dag(oo))))
Here μ Rdd and of E R& are free variables for each latent state and are optimised with SGVB (Kingma & Welling, 2014). Notably, the above factorisation over states bears similarity to pose-graph optimisation. One can see the individual terms qo (z.) as graph nodes, and the loss terms induced by the transition and emission in the objective presented next as the edge constraints.
Overall objective The elements described so far, together with the transition p(Zr+1 Ze; Up) discussed in the next section, form the probabilistic graphical model in eq. (1). The assumed variational approximate posterior is
9 (o(Z):))g = p(Zi,T,M XI.T u1
For the optimisation objective 1999), given as
the negative evidence lower bound (ELBO) (Jordan al.,
T
Lelbo = -Eg
log p(x, m:)
T
+ (L( ((M)) + Eg
KL (Zs)
p(z, 21-1, u.11)
(2)
We employ the approximate particle optimisation scheme from (Mirchev et al., 2019) deal with long data sequences. The only optimised parameters are Φ, containing the parameters of the map and the agent states.
Making image reconstruction tractable Using the full observations during inference is not fea- sible, raycasting for all pixels is too computationally demanding. To ensure tractability of the inference method we therefore use reconstruction sampling (Dauphin et al., 201 1), emitting random part of x; at time, by randomly selecting c-many pixel coordinates jim for every gradient step. Here is a constant much smaller than the image size wh, speeding up gradient updates by a few orders of magnitude. Note that this results in unbiased, faster and more memory-effictee Monte Carlo approximation of the original objective, avoiding loss of information due to subsampling or sparse feature selection.
0 b5 set 0 for < 1 and fÆ(O:) re if no value exceeds along the ray.
5

---

Published as a conference paper at ICLR 2021
4 LEARNING RIGID-BODY DYNAMICS
The introduced model factorisation includes a transition p(z(+1 Zt, uz). which allows the natural inclusion of agent movement priors. This is reflected in the corresponding KL terms in eq. (2). Note that using variational inference lets us integrate any differentiable transition model as-is, without additional linearisation. the following, assume the agent has an inertial measurement unit (IMU) providing readings X (linear acceleration) and ".tt (angular velocity) over time, which we choose to treat as conditional inputs u, = =>; imu (
Engineering rigid-body dynamics the absence of learning, one can use an engineered transition prior that integrates the IMU sensor readings over time. The latent state Zz (A,, W., A,) then contains the location, orientation and linear velocity of the agent every time step. The transition is defined as:
p z Zz, u.) = (Z++1 fr uz), diag(or))))
The state update fr implements standard rigid-body dynamics using Euler integration (see ap- pendix D.3). This engineered model will serve counterpart for the learned transition model presented next.
Learning a dynamics model Engineered models of the agent movement are often imperfect or not available. We therefore provide a method for learning a probabiiitiiiiiccccccc transition model from streams of prerecorded controls and agent pose observations, which we can then seamlessly include as prior in the full model. We do not learn the transition with per-step, fully-supervised regression. Instead formulate generative sequence model for T time steps. This allows aleatoric uncertainty in the observed agent states from the uncertainty in the transition itself. We follow the literature on variational state-space models (Fraccaro al., 2016; Karl et al., 2017a). We assume we have sequence of locations ΛET and orientations W1:T as observations. and sequence of IMU readings, as well as per-rotor revolutions per minute (RPM) and pulse-width modulation rpm
separate the
ima (PWM) signals, as conditional inputs U1:7-1 (A generative state-space model:
imnu
pwm 1) We define the
T-1 II
p( (AAI: WI.T Z1:T U1:T-1) = 5(2, (p() (A wr 21)
pe, (2141 2t, ur u:)p(()+1, 2.41 2441).
The objective is to learn generative transition parameters Or, such that the marginal likeli- hood of observed agent poses pe, (ALr,, ΦIr u1:r-1) is maximised. The latent state is Z: (A,, w., År, ), identifying its first three components with location, orientation and lin- ear velocity. The remainder z!"t acts as an abstract state part. Its role to absorb any quantities that might affect the transition, for example higher moments of the dynamics sensor biases accumulated over previous time steps. The transition is implemented as a residual neural network on top of Euler integration:
pe, (Z++1
Z,, ue) = -N(z++1 M4+1s diag(or(+1)2)
Jr (2; uz) MLP, u:), a = =MLP, (2,, uz),
P11
0
where fo is the engineered Euler integration from the previous section and the abstract remainder of the latent state formed entirely by the network (MLP). This strong inductive bias shapes the transition to resemble regular integration in the beginning of training, exploiting engineering knowledge, while still allowing the MI.P to eventually take over and correct biases as necessary.
The emission isolates the location and orientation from the latent state its mean:
م( wr 2 = . NV{ de (Ar, w.). )
The inference over the latent states uses Gaussian fusion as per (Karl et al., 2017b) and the necessary inverse emission is given by bidirectional RNN that looks into all observations and conditions:
((z,) du., Wirr u1-r-1) N(2 RNN(A1 Wrrr urrr-1)

---

Published as a conference paper at ICLR 2021
We minimise the negative ELBO w.r.t 0r, omitting the conditions in q for brevity:
[(Or) Bg
lo wr 20) + B.g (L((((Z)) || Por 2 (b_1))
EXPERIMENTS
The experiments are designed to validate three model aspects the usefulness of the reconstructed 3D world maps, multi-step prediction given future controls and the localisation quality.
For evaluation, we use the Blackbird data set (Antonini al., 2020). It consists of over ten hours of real quadcopter flight data. The ground truth poses are recorded by motion capture (MOCAP) system. For each trajectory. Blackbird contains realistic simulated stereo images. We obtain depth from these images using OpenCV's Semi-Global Block Matching (SGBM) (Hirschmuller, 2007) and treat the left RGB camera image and the estimated depth as an observation X,. We evaluate our method the test trajectories used in (Nisar et al., 2019). Other trajectories with overlap are used for training the learned dynamics model and model selection. All model hyperparameters are fixed to the same values for all evaluations. More details can be found in appendices C and D.
5.1 DENSE GEOMETRIC MAPPING
A fused dense map, obtained as an approximate variational posterior qa(M), allows us to simulate (emit) the environment from any novel view point. We take the NYC subway station Blackbird environment as an example, in which the test set trajectories take place. Figure 3a shows image reconstructions, generated along an example trajectory segment. The model can successfully generate both colour and depth based on q (M). Note that the true observations are not needed for this, as all of the information is recorded M through gradient descent. Even though we use reconstruction sampling during training, on average all image pixels contribute to learning the map. This leads to dense predictions of the agent's sensors. The inferred map correctly filters out wrong observations, as can be seen in the top-down point-cloud comparison in fig. 4d, noting the subway station columns.
5.2 USING MAPS FOR DOWNSTREAM TASKS
Besides parameterising predictive emissions, a map can be used to define downstream navigation and exploration tasks. Since the map is modelled as a random variable, the approximate posterior (M) also gives us an uncertainty estimate, in this work only applying to M Figure 4a shows horizontal slice from the occupancy grid, along which the map uncertainty is evaluated. The uncertainty is low inside the subway station, and high on the outside where the agent has not visited. Meaningful map uncertainty is useful for cooommttionthhorede exploration (Mirchev et al., 2019).
The occupancy map can also be used to construct navigation plans, by computing a collision cost-to- go J(z)) = Eu (u:rr.):r>p()) [> e( U: 2:)] (Bertsekas, 2005). For example, the cost c(u, z) can be the occupancy value in the map at 2,, defining collision cost, To simulate control policy, use an empirical distribution of randomly picked 40-step sequences of IMU readings u1.7 from the test data. Then we generatively sample future states Z1:T for these controls and evaluate the respective costs. Figure 4c shows (zz evaluated for all states along the considered 2D slice of the map. The cost-to-go is high near the walls and columns, and gradually drops off in the free space.
5.3 GENERATING FUTURE PREDICTIONS
The proposed model can predict the agent movement and observations for a sequence of future conditional inputs, i.e. p(Z2T, X1.7 u1:7-1, 21). Such predictions are crucial for model-based control, and typically not readily available from modern visual SLAM systems. Figure 3b shows predictions of the full spatial model for 200 steps in the future, comparing both the engineered and learned transition priors. Long-term prediction is significantly better with the learned transition, indicating that it corrects biases present in the agent sensors. Conversely, with the engineered transition localisation drift is higher and wrong map regions queried for generation, translating into wrong depth and colour predictions. We refer to the supplementary vidco for more examples using the predictive model. Evaluated on 200 test set trajectories with 100 steps, the learned transition
7

---

Published as a conference paper at ICLR 2021
colour
depth
enn..
(a)
(b)
Figure 3: (a) Emissions at different trajectory points, sampled at one second interval. Top to bottom:
predicted colour, observed colour, predicted depth, observed depth. (b) Generative predictions using the engineered transition vs. the learned transition in the complete model. Left: top-down view of 200-step location predictions. Right: predicted colour and depth for the same step for both models.
ICA
Tim
-50
-15
[m)
[m]
(a)
(b)
(c)
(d)
Figure 4: An illustration of reconstructed dense map for the NYC subway station environment. Note that columns from the subway are captured in the reconstructed map. The figures show horizontal map slice. (a) Occupancy map uncertainty (white means low uncertainty). (b) Occupancy map mean (white means occupied). (c) Collision cost-to-go (red means high cost). (d) Generated point cloud (black, for inferred agent poses) vs. data point cloud (red, for ground truth MOCAP poses).
leads to better predictions than its engineered counterpart, with an average translational RMSE of 1.156 and rotational RMSE of 0.096, compared to 768 and 0.111 for the engineered model. Similarly, the pixel-wise oggliikelhood of the observation predictions - -1.23 when the learned model is used, and 1..85 for the engineered model, averaged 1000 images. This evaluation clearly illustrates the positive effect of the learned transition on the model's predictive performance.
5.4 AGENT I.CAALISATINN
Finally, we evaluate localisation performance during SLAM inference. Figure shows estimates for the fastest trajectories in the test (up 4m/s). We refer to appendix F and the supplementary video for more inference examples in different environments. Table summarises the average absolute RMSE for all test trajectories, evaluated following Zhang Scaramuzza (2018). Here we compare to the results reported by Nisar et al. (2019), including VIMO (Nisar et al., 2019) and VINS-MONO (Qin et al., 2018)-two state-of-the art visual odometry methods. In this case both systems are carefully tuned to the environment. While the RMSE of our method is not as low as that of the baselines, we note that these absolute values. The online translational error of our method does not exceed 0.4m, which is practical considering the average 232m trajectory length. Our method also succeeds on the fastest star test trajectory, for which both baselines have been reported to fail without specific retuning. We note that the two systems in real-time, whereas currently method does not-we will tackle real-time inference in our future work (see appendix E).
We also compare to the system benchmarks provided by Antonini et al. (2020), including results for VINS-MONO, VINS-Fusion (Qin et al., 2019) and ORE-SLAMP (Mur-Artal & Tardós, 2017). We the same star and picasso trajectories reported in table 1, and refer Antonini et al. (2020) for the exact evaluation assumptions. We also explicitly restate note made by the authors: the benchmarked systems are not carefully tuned to the Blackbird environment and their loop-closure
8

---

Published as a conference paper at ICLR 2021
OaS SLAM
GCD SLAM
Vini prrg
(m)
[m]
Sbitnc mosdled
dirzace traeetdddddd (xs)
(a)
(b)
(c)
Figure 5: (a,b) Top-down view of localisation for star, forward yaw, max speed 4.0m// and picasso, constant yaw, max speed 4.0m/s. s. (c) Localisation with our method compared to the benchmark results reported by Antonini et al. (2020) for picasso, constant yaw and star, forward yaw, 1 to 4m
Table I:
Absolute localisation RMSE for each test trajectory. See appendix B for a discussion of the two noticeable translation RMSE outliers with the learned transition.
Translational RMSE [m]
Rotational RMSE [rad]
proposed (eng. trans.)
proposed (leam. trans)
VIMO
VINS
proposed
proposed
VIMO
VINS
(oug. trans.)
(lemm trass.)
picasso
m//a
0.139
0.143
0.055
0.097
0.053
0.052
0.013
0.011
0.136
0.131
0.040
0.043
0.069
0.064
0.007
0.008
m//s
0.120
0.122
0.043
0.045
0.073
0.070
0.005
0.005
4 m/s
0.174
0.368
0.049
0.056
0.124
0.149
0,009
0.011
star
m//s
0.137
0.133
0.088
0.102
0.057
0.056
0.008
0.008
m/s
0.163
0.626
0.082
0.133
0.061
0.157
0,010
0.011
m/s 0,281
0.187
0.183
0.235
0.080
0.059
0.015
0.016
m/s 0.156
0.160
0.065
0.059
modules are disabled, which might not be reflective of their best possible performance. Therefore, these baselines represent the performance an off-the-shelf visual odometry system without changing its hyperparameters. Figure 5c summarises the comparison, reporting error statistics for segments of different length (x-axis) divided by the distance travelled. Localisation with our method is robust and drift does not compound, which we attribute to the global map variable M serving an anchor.
Overall, localisation is successful for test trajectories and its accuracy is practical and close to that of state-of-the systems, while all merits of deep probabilistic generative modelling retained.
CONCLUSION
This work is the first to show that learning a dense 3D map and 6-DoF localisation can be accomplished in deep generative probabilistic framework using variational inference. The proposed spatial model features an expressive predictive distribution suitable for downstream control tasks, it is fully-differentiable and can be optimised end-to-end with SGD. We further propose probabilistic method for learning agent dynamics from prerecorded data, which significantly boosts predictive performance when incorporated in the full model. The proposed framework was used to model quadcopter flight data, exhibiting performance close to that of state-of-the- of-tth-art visual SLAM systems and bearing promise for real-world applications. In the future, we will address the current model's speed limitations and move towards downstream applications based on the learned representation.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th (USENIX) Symposium on Operating Systems Design and Implementation ((OSDI) 16), pp. 265-2833 2016.

---

Published as a conference paper at ICLR 2021
Amado Antonini, Winter Guerra, Varun Murali, Thomas Sayre-McCord, and Sertac Karaman. The blackbird uav dataset. The International Journal of Robotics Research, 000:077836492200083331, 2020. doi: 10.1177/027835442200333
Philip Becker-Ehmele Maximilian Karl, Jan Peters, and Patrick van der Smagt. Learning to fly via deep model-based reinforcement learning, 2020.
Dimitri P. Bertsekas. Dynamic Programming and Optimal Control, volume I. Athena Scientific, Belmont, MA, USA, 3rd edition, 2005.
S. Bi, Zexiang Xu, Pratul P. Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milovs Havsan, Yannick Hold-Geoffroy, D. Kriegman, and R. Ramamoorthi. Neural reflectance fields for appearance acquisition. ArXiv,
2020.
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J. Davison.
Codeslam learning a compact, optimisable representation for dense visual slam. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2560-2568, 2018.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. CoRR, Abs/15055004244 2015.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian D. Reid, and John J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans. Robotics, 1309-13333 2016.
Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov. Active neural localization. In International Conference Learning Representations, 2018. URL https: http///ppneereeee revi ew. net/forund ddryy-G_6666
Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openreview.nee net fateee//ooommmmdeeeeeeee 6xvglx.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4764-4765 2018.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2980-2983 2015.
Dane Corneil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement learning with variational state tabulation. volume 80 of Proceedings of Machine Learning Research, pp. 1049-1058, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Sebastian Curi, Silvan Melchior, Felix Berkenkamp, and Andreas Krause. Structured variational inference in partially observable unstablegaussian process state space models. volume 120 of Proceedings of Machine Learning Research, pp. 147-157, The Cloud, 10-11 Jun 2020. PMLR.
J Czarnowski, T Laidlow, R Clark, and AJ Davison. Deepfactors: Real-time probabilistic dense monocular slam. IEEE Robotics and Automation Letters, 5:721-728, 2020. doi: 10. 10.1109//ra.020. 2965415. URL http:h/tt// /tpp/xxxxoooo. wereii.orgg111 10 1109 2020.2965421211
Yann N. Dauphin, Xavier Glorot, and Yoshua Bengio. Large-scale learning of embeddings with
reconstruction sampling. In Proceedings of the 28th International Conference on Machine Learning, ICML, 201 1, Bellevue, Washington, USA, June 28 July 2, 2011, pp. 945-952, 2011.
J. Engel, Koltun, and D. Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, March 2018.
10

---

Published as a conference paper at ICLR 2021
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 36046394)1120442221 2018. doi: 10.1126/sciencs aar6 170. URL https: / htt///sciiee science. sciencemag wwapz////o////t/toeeeee /36
Marco Fraccaro, Søren Kaae Sønderby, UIrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. pp. 2199-220122 2207, 2016.
Marco Fraccaro, Danilo Jimenez Rezende, Yori Zwols, Alexander Pritzel, M. Ali Eslami, and Fabio Viola. Generative temporal models with spatial memory for partially observed environments. CoRR, 0tas/1804.094/0 2018.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping belief states with generative environment models for rl. In Advances in Neural Information Processing Systems, pp. 13475- 13487, 2019.
Giorgio Grisetti, Cyrill Stachniss, and Wolfram Burgard. Improving grid-based slam with rao- blackwellized particle filters by adaptive proposals and selective resampling. Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 2432-2437, 2005.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2450-2462. Curran Associates, Inc., 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint A9Xiv: 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019b.
Heiko Hirschmuller. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on pattern analysis and machine intelligence, 30(2):328-341, 2007.
Krishna Murthy Jatavallabhula, Ganesh Iyer, and Liam Paull, gradslam: Dense slam meets automatic differentiation, 2019.
X. Jiang, M. Hoy, H. Yu, and J. Dauwels. Linear-complexity stochastic variational bayes inference for slam. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1-6, 2017.
Xiaoyue Jiang, Hang Yu, Michael Hoy, and Justin Dauwels. Robust limear-complexity approach to full slam problems: Stochastic variational bayes inference. 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), pp. 1-5. IEEE, 2019.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233) 1999.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In 5th International Conference on Learning Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017a.
Maximilian Karl, Maximilian Soelch, Philip Becker-Ehmed Djalel Benbouzid, Patrick van der Smagt, and Justin Bayer. Unsupervised real-time control through variational empowerment. arXiv preprint arXiv: / 1710.05001, 2017b.
11

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abss/441..6980 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference Learning Representations (ICLR), 2014.
T. Laidlow, J. Czarnowski, A. Nicastro, R. Clark, and S. Leutenegger. Towards the probabilistic fusion of learned priors into standard pipelines for 3d reconstruction. 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 7373-7379, 2020. doi: 10.1 N..109//CRA4094
2020.9197001.
Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa Narasimhan, and Jan Kautz. Neural rgb (r) d sensing: Depth and uncertainty from video camera. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10986-10995, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Atanas Mirchev, Baris Kayalibay, Maximilian Soelch, Patrick van der Smagt, and Justin Bayer. Approximate bayesian inference in spatial environments. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgau, Germany, June 2019.
Michael Montemerlo, Sehastian Thrun, Daphne Koller, and Ben Weghreit. Fastslam: factored solution to the simultaneous localization and mapping problem. In AAAI/IAAI, 2002.
Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 3355):225-1222,,22 2017.
Kevin P. Murphy. Bayesian map learning in dynamic environments. Advances in Neural Informa- fion Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 December 4, 1999], pp. 1015-1021, 1999.
Richard A. Newcombe, Steven Lovegrove, and Andrew J. Davison. Dtam: Dense tracking and mapping in real-time. 2011 International Conference on Computer Vision, pp. 1320-2327 2011.
Barza Nisar, Philipp Foehn, Davide Falanga, and Davide Scaramuzza. Vimo: Simultaneous visual
inertial model-based odometry and force estimation. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgau, Germany, June 2019.
Junhyuk Oh, Valliappa Chockalingam, Satinder P. Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pp. 2790-2799, 2016. wtt.:/ooo///ooeeeeeeeeeoooooee/e http:/poooeeeinngeeemee http///appss/////eeeee 8 oh1 6 html.
URL http::ttp/// / jmlr.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement
learning. In International Conference on Learning Representations, 2018. URL https://
openreview / forum?id BidrummmddB
Tong Peiliang Li, and Shaojie Shen. Vins-mono: robust and versatile monocular risual-inertins state estimator. IEEE Transactions on Robotics,
2018.
Tong Qin, Jie Pan, Shaozu Cao, and Shaojie Shen. A general optimization-based framework for local odometry estimation with multiple sensors. arXiv preprint arXiv:1901.036:: 2019.
Ransalu Senanayake and Fabio Ramos. Bayesian hilbert maps for dynamic continuous occupancy mapping. In Proceedings of the 1st Anmual Conference on Robot Learning, volume 78 of Proceed- ings of Machine Learning Research, pp. 458-471, PMLR, 13-15 Nov 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Chengzhou Tang and Ping Tan. BA-net: Dense bundle adjustment networks. In International Conference on Learning Representations, 2019. URL https: http////pee///eeeee//e/ee net/ forum? id=B1 gabhRex..
12

---

Published as a conference paper at ICLR 2021
Dustin Tran, Matthew D. Hoffman, Dave Moore, Christopher Suter, Srinivas Vasudevan, Alexey Radul, Matthew Johnson, and Rif A. Saurous. Simple, distributed, and accelerated probabilistic programming. In Neural Information Processing Systems, 2018.
Xingkui Wei, Yinda Zhang, Zhuwen Li, Yanwei Fu, and Xiangyang Xue. Deepsfm: Structure from motion via deep bundle adjustment. In ECCV, 2020.
James Whittington, Timothy Muller, Shirely Mark, Caswell Barry, and Tim Behrens. Generalisation of structural knowledge in the hippocampal-entorhing system. In Advances in neural information processing systems, pp. 8484-8495, 2018.
Nan Yang, Lukas von Stumberg, Rui Wang, and Daniel Cremers. D3VO:
Deep depth, deep pose and deep uncertainty for monocular visual odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural slam: Learning to explore with external memory. arXiv preprint arXiv:1706.09522 2017.
Zichao Zhang and Davide Scaramuzza. A tutorial on quantitative trajectory evaluation for visual(- inertial) odometry. In IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS), 2018.
Shuaifeng Zhi, Michael Bloesch, Stefan Leutenegger, and Andrew Davison. Scenecode: Monocular dense semantic reconstruction using leamned encoded scene representations. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp- 11776-11785, 2019.
H. Zhou, B. Ummenhofer, and T. Brox. Deeptam: Deep tracking and mapping. In European Conference on Computer Vision (ECCV), 2018.
13

---

Published as a conference paper at ICLR 2021
A USE OF DOMAIN KNOWLEDGE WHEN DESIGNING LEARNED SPATIAL MODELS
X3
8F
8x
1/
21
(a) When the attention the map is too narrow, emit- ting from observed but yet unvisited places suffers from non-local generalisation issues in an online setting.
(b) A 2D illustration attending to the whole field of view of the agent. Note that in our method the occu- pancy map is a 3D volume, and the attended camera frustum resembles pyramid.
Figure 6: Simplified 2D examples of the conceptual difference the chosen attention can have on generating observations.
What distinguishes our method from DVBF-LM (Mirchev et al., 2019) and other previous 2D and 2.5D deep generative sequence models is that introduce geometric inductive biases (multiple-view geometry, rigid-body dynamics) in a deep generative sequence model (inferred via the ELBO).. We hope the following overview DVBF-LM's methodology will exemplify why such assumptions could be necessary and motivate our design choices.
DVBF-LM models the world 2D chessboard where the content of each grid cell describes 360 horizontal panoramic view around the z-axis, centered at the agent 2D location. Observation predictions are cropped from that view, relative to the way the agent is facing in 2D (and transformed by an MLP). This already consitutes a set of basic geometric assumptions, but they can be limiting, as explained below. First all, the agent movement is restricted to a plane, with rotations around one axis (no 6-DoF modelling). The main problem, however, is that the map is updated very locally, only at the 2D location of the agent, because the attention of DVBF-LM only considers four map cells directly underneath. For example, if the agent is standing 5m in front of wall and facing it, DVBF-LM only stores this information at that location in the map. If the agent moves 2m closer to the wall, it is now accessing different cells, whose content is completely arbitrary (see fig. 6a). These cells will have to learn the presence of the wall again. The map is thus unnecessarily redundant and the agent would have visit virtually all map cells infill everything. This means observation predictions from cells which have not been visited before will not be meaningful, even when they were already in the field of view (FoV) of the agent. Localisation also suffers, as the map more casily allowed to establish multiple modes for the pose posterior because of the redundancy (same content can be stored different places by mistake), further complicating the perceptual aliasing problem of spatial environments.
By contrast, our method we use raycasting and attend the whole camera frustum, accessing all map content in the field of view of the agent, as shown in fig. 6b. Note that the depiction is intentionally simplified to 2D for the sake clarity, while the actual attention operates 3D. In the above example, the wall and the empty space in front of the agent would be captured as soon as the agent sees the wall for the first time, and can be predicted from all regions in the FoV. This constitutes strong geometric inductive bias that will generalise across different environments.
our approach
In general, the lack of geometric assumptions in previous deep spatial generative models (c.g. the ones described in the first paragraph of section 2) attractive, as it lessens the need for domain knowledge, instead attempting to learn the whole system end-to-end. However, learning everything
14

---

Published as a conference paper at ICLR 2021
becomes problematic (not just in terms of runtime) when the inference needs to happen on- on-the-fly, fly and data is scarce, as is the case for autonomous agents. When the agent cannot afford to exhaustively observe the whole environment, insufficient data for learning from scratch can lead to problems with consistency in the map, consistency in the long-term predictions and ambiguous agent localisation. One option is address this through learning certain components in advance (e.g. the dynamics model, see section 4). Once image data is involved, however, the generalisation of learned components is much harder ensure because the curse of dimensionality. Observations can vary greatly from one scene to the next, and free 6-DoF movement of the agent exacerbates the problem further. Instead of collecting data exhaustively, in an attempt to pretrain network to learn how to render, we rely on the geometric knowledge that is already available to us to design map, attention and the emission in section Such assumptions are also at the core of many of the models discussed in the second paragraph in the related work (section 2). In contrast to these models, however, in our work we weave the aforementioned inductive biases into the deep generative framework, which is tubly-probabilisticc, implies end-to-end differentiability and cases the introduction of new learned components when needed. Therefore, our goal is to maintain an expressive predictive distribution p(Z2:T: X1:7 U1:7-1: 21) that lets predict the future, precisely aligning with the performed inference based on past data at the same time. We stress that maintaining full probabilistic predictive model is needed quantify uncertainty when planning for future actions of the agent.
do with
15

---

Published as a conference paper at ICLR 2021
B CASE STUDY: DRONE LANDING
In terms of the integrated transition p(Z++1 Zz, uz), we find that the learned transition performs on par with its engineered counterpart when used for inference in the full model. However, when max speed 4.0
inspecting the absolute RMSE in table 1, there appear to be two outliers-picaseee m/s and star, max speed 2.0 m/s, when using the learned transition. closer examination showed that this is entirely due to large localisation error during landing at the end of the trajectories. In both cases the agent fails to land correctly or falls over, leading to out-of-distribtiinn conditions u, from the IMU sensor (cf. fig. 7), for which the learned model does not generalise. Such behaviour is not unexpected when it comes to neural networks. Localisation beforehand is stable along the whole trajectory, the landing tracking failure happens only during the last 2 seconds of movement.
Angular velocity (x. y.
Linear acceleration (x,)zz)
2
IMU value
IMU value
0
-2
0
50
100
150
50
100
150
time [s]
time [s]
distance error [m]
Linear accelerations in
5.0
flight
landing
density
2.5
0.0
50
100
150
1.5
-1.0
-0.5
0.0
0.5
1.0
time [s]
value
Figure 7: Illustration of the discussed failure landing of the quadcopter the end of the test trajectories, taking slar, max speed 2.0 m/s as an example. The problematic segment of the trajectory marked with red vertical band. Note the outlier controls at the end. Top left: angular velocity IMU readings. Top right: linear acceleration IMU readings. Bottom left: absolute Euclidean distance error w.r.t. ground truth MOCAP locations. Note how the error large only at the end the trajectory, and directly coincides with the outlier controls. Bottom right: comparison of the distribution of linear accelerations in y during flight (red) vs. during landing (gray). The controls during landing are out of distribution for the learned transition model.
16

---

Published as a conference paper at ICLR 2021
C DATA DETAILS AND OVERALL SETUP
For all presented experiments we used the Blackbird data set (Antonini et al., 2020), which can be found here: https: http::///ttthae com / mm meit-ffsttin tt-fast/llaabbird-eeeeeee we describe the exact pre-processing of the data.
In the following
C.I DATA PARTITIONING AND USAGE
The data was partitioned into a training, a validation and test set. The trajectory shapes (c.g. star, picasso. patrick, etc.) in every split were different. This was done to prevent accidental overfitting of the learned transition model any particular trajectory shape. particular, the test set contains the trajectories star, forward yaw and picasso, constant yaw, traversed at different speeds, everything else used for training and validation. This the same test setup as the one used by Nisar et al. (2019) (considered baseline). Appendix G lists the exact trajectories used for each split, along with the number steps in each trajectory after pre-processing (which includes subsampling).
The training set used only to train the learned transition model pe, (Z++1 Zz, ut), following the method described in section 4. The learned transition model is trained separately before is used as a prior in the full spatial model. Pretraining the model multiple trajectories beforehand, opposed to training it from seratch during SLAM inference, ensures that the transition really captures the agent dynamics and does not overfit the currently explored environment. The training trajectories were further randomly rotated around the axis and linearly translated in space, to promote generalisation.
The validation set was used for checkpointing and selecting the best weights 0, of the neural network in the learned transition model, based on the ELBO defined in section 4. The validation set was also used for hyperparameter selection for the engineered transition model, the learned transition model and the full spatial model. The best hyperparameters for all models (cf. appendix D) were selected with random search. You can find details for the search ranges in appendix
The test set was used for evaluation only-all all results reported in the experimental section of the paper were done on the test data. The full spatial model was tested with the forward yaw star trajectories, speeds 1.0 m/s to 4.0 m/s and with the constant yaw picasso trajectories, speeds 1.0 m/s to 4.0 m/s, to match the evaluation by Nisar et al. (2019). These trajectories take place the NYC subway station environment.
C.2 DATA PREPROCESSING
Fach trajectory contains IMU, RPM and PWM readings, MOCAP ground truth pose observations, as well as simulated grayscale images from foriard-facing stereo pair and downward-facing camera.
We pre- processed every trajectory in the following way:
Ground truth MOCAP state readings were extracted from the provided ROS bags.
Downward- facing images were ignored.
The remaining data was subsampled to 10 Hz, using nearest neighbour subsampling based on the provided time stamps.
Depth was then precomputed from the left and right images using OpenCV's SGBM (Hirschmuller, 2007).
For every time step, the right colour image was then ignored, while the left image and the depth estimate were together treated as an RGB-D observation x, Rwxhx4
The provided IMU readings are measured in the coordinate frame the IMU sensor. Therefore, they had to be rotated to the body frame of the agent, using the respective quaternion provided with the data set: = [0.707, 0.005, - 0.004, 0.706)
The intrinsic parameters of the camera, specifying the intrinsic camera matrix K, and the stereo baseline were fixed to the values provided with the data set:
Stereo baseline: 0.1m.
Image size: 1024 x 768.
17

---

Published as a conference paper at ICLR 2021
Focal length, 665.1.
Focal length, y: 665.1.
Principal point offset, x: 511.5.
Principal point offset, y: 383.5.
Depth was computed based on the disparity values produced by SGBM. We used the following parameters, keeping the default values for everything else:
Block size:
Min. disparity:
Max. disparity: 256.
We did not filter the images and the produced depth values in any way and treated them as direct observations, to limit the pre-processing steps as much possible and rely the defined spatial model instead. Note that due to the simplicity of the method used for depth estimation, the depth observations contain significant amounts of noise (e.g. fig. 4d). The pixel values for the color images were normalized to in the range [0, 1].
D MODEL DETAILS
This section lists all model and optimisation hyperparameters used for generating the results reported in the paper.
D.1 FULL SPATIAL MODEL
The full spatial model uses either the learned or the engincered transition, see appendix D.2 and appendix D.3 for their respective hyperparameters.
D.1.1 OPTIMISATION
After subsampling, the trajectories in the Blackbird dataset can still contain thousands of time steps. To deal with this, during inference the proposed model follows the approximate particle optimisation scheme introduced by Mirchev al. (2019), using chunks time steps for every gradient update, using 50 approximating state particles and refreshing the respective particles after every gradient step.
Adam (Kingma Ba, 2014) is used to optimise the parameters for the approximate posterior distribution U (z : MM.. Table 2b lists the used optimiser hyperparameters. Note that due to the employed attention p(m, 2., .M), due to the reconstruction sampling in the emission and due to the approximate optimisation scheme mentioned above (which uses only chunk of the trajectory at a time), only a part of the parameters for the occupancy map and the full trajectory of agent states is used for gradient step. In other words, gradient updates happen locally in the spatially arranged occupancy map parameters, based on the currently selected local chunk of the agent trajectory. Because of this, momentum is disabled in Adam for both the occupancy map and the agent states, to avoid accidental drift in regions currently not covered by the optimisation.
For every variable in question, the same optimiser is used optimise the mean and scale of the assumed distribution (where applicable). For any scale distribution parameters (e.g. standard deviations for the assumed Gaussian distributions), the optimisation is performed in log-space to satisfy the positive value constraint.
D.1.2 INTTIALISATION OF NEW POSES
While performing SLAM, data is added incrementally to the spatial model, to emulate online usage of the method. new time step f is explored every 500 gradient steps, effectively adding a new observation x, and conditional input u, the data. Whenever parameters of the corresponding state z, are unlocked for optimisation and initialized according to table 2a.
time step is added, the
18

---

Published as a conference paper at ICLR 2021
Table Model details.
(a) Hyperparameters of the individual spatial model components.
Component
Parameter
Value
init. value for M4 mean of por (z: M4 u.)
init. value for oz
0.01
00
grid size
200 x 200 x 200
init. value for p.s.k init. value for Jss.k
-0.5
0.1
M
# hidden layers # hidden units activation residual connections
256
softsign true
p(m; 2z, M)
(ray resolution) max depth (ke) # reconstr. pixels
0.1m
20m
200
(b) Optimisation hyperparameters for the full spatial model.
(c) Hyperparameters of the learned transition model.
Component
Parameter
Value
Variables
Parameter
Value
pe, (Ze+1 Ze, Uz)
hidden layers
5
71:7
Adam, learning rate
0.001
# hidden units
64
Adam, 3
0.0
activation
relu
Adam, 32
0.999
residual connections
true
M dre
Adam, learning rate
0.05
size of x rest
8
Adam, $1 0.0
Adam, 32
0.999
((Zi RUZT Wirr upr))
RNN type
LSTM
# RNN units
64
M co!
Adam, learning rate
0.001
Adam, 31 0.9
Adam, $2
0.999
D.1.3 CHOICE OF APPROXIMATE POSTERIOR OVER STATES
In (Mirchev et al., 2019) a bootstrap particle filter is used to implement the variational posterior over agent states. The increased model complexity due to the presented raycasting model makes the application of particle filters with sufficiently many particles costly. In section 3 we therefore introduce an approximate posterior over the agent states do (Z1:r), with free state parameters Φ. that factorises over time. The chosen variational states approximation represents a shift in perspective from filtering towards a method more similar pose-graph optimisation.
D.1.4 HANDLING ORIENTATIONS
The portions Wi of the sampled latent states z, are identified with quaternions and are explicitly normalised to unit quaternions after gradient updates before they are used in the of the model. This means that the assumed Gaussian distribution is not directly expressed on the manifold 8O(3). but we found that this parameterisation works well enough.
D.1.5 COMPONENT PARAMETERS
The rest of the hyperparameters for the full spatial model, including initial values for the optimised variational parameters, specific to the individual model components and are listed in table 2a.
The scale og for the heteroscedastic Laplace emission p(x; mm) is learned with gradient descent (applied in log-space). The colour part of o is forced to be 10 times smaller than that for the depth part, to account for the different scales the observed values (colour range is [0, 1]. depth range is [0, 20).
19

---

Published as a conference paper at ICLR 2021
D.2 LEARNED TRANSITION
The learned transition is obtained by training the generative parameters Or in the context of the model defined in section 4. When reconstructing orientations in that model, we do not reconstruct quaternions directly because of the ambiguity q -q Instead we reconstruct the rotation matrix corresponding to the observed orientation Wt, i.c. the mean of the emission p(w; Ze) is rotation matrix constructed from the quaternion w, that's 's part of the latent state 2,. When used in the full spatial model, the weights of the transition neural network 0r are fixed their pretrained values. This is done to avoid accidental overfitting of the transition parameters to the current trajectory and current environment for which SLAM performed.
The selected hyperparameters of the learned transition are listed in table 2c,
D.3 ENGINEERED TRANSITION
The mean of the engineered transition prior p(z++1 fr, which implements the rigid body dynamics
Ze. us) from section is formed by a function
Ae+1
A + ArAe
(r(2,,u))
Wi+1 A++1
We 4 R(ws, )ist Al imm L + R(w:) (At)²
Here ① denotes standard quaternion integration. The standard deviation dT of p(z Zt. u,) (a Gaussian) is hyperparameter, estimated via search on the validation Its diagonal entries 0.01 for the location state dimensions, 0.001 for the orientation state dimensions and 0.001 for the velocity state dimensions. The gravitational force g is subtracted from the IMU readings when applying the Euler integration given by fr. The time delta for the integration is set to At = 0.1, to match the 10Hz data subsampling.
E HYPERPARAMETER SEARCH AND EXECUTION DETAILS
All models are implemented in python using TensorFlow (Abadi et al., 2016), making use of automatic differentiation to optimise model parameters. This allows for the easy integration of neural networks and makes end-to-end optimisation straightforward. In TensorFlow 1.15, one gradient step of the model takes 0.0607s average, without XLA compilation. In TensorFlow 2.3 gradient step of the model takes 0.0073s on average, with XIA compilation enabled. The inference experiments the paper currently assume 500 iterations per added data point (once every 0.1s for 10Hz stream), which amounts to 36.5s of inference runtime per Is of real-time movement in the newer TensorFlow version. As already mentioned the main text, we have not yet optimised the model for real-time inference. and we will address this in our future work. The large speed-up achieved by simply enabling XLA and switching to a higher version of TensorFlow indicates that a lot of the computations in the model can be improved further (e.g. by moving to a C++ runtime or writing dedicated CUDA kernels). We also anticipate that better initialisation and careful tuning the model hyperparameters will let system reach interactive operation rates.
All probabilistic modelling aspects were implemented using Edward (Tran et al., 2018). Hyperpa- rameter search (HPS) experiments were executed in a cluster with 8 Tesla V100 GPUs and 40 Intel Xeon E5-2698 CPU cores. Because of the large amount of Blackbird data (4.7 TB) and the current model's 'ss speed limitations, the search was performed for subset of all possible hyperparameters. The performed model selection is therefore not exhaustive, possibly leaving potential for improvement. In the following list the considered hyperparameter search ranges and the number of trials for each model.
E.1 HPS: FULL SPATIAL MODEL
The hyperparameters for the full spatial model were selected based on localisation RMSE for 500-step segments of trajectories in the validation set. A total of 200 experiments conducted, randomly picking a trajectory segment on which SLAM inference is performed. The considered parameter ranges listed in table 3a.
20

---

Published as a conference paper at ICLR 2021
Table 3: HPS details.
(a) HPS ranges for the full spatial model.
(b) HPS ranges for the learned transition model.
Parameter
Range
Parameter
Range
reconstr. pixel count
[100, 200, 500]
learning rate
[0.0001, 0.0005, 0.001, 0.005
(M
(CC) init. value stddev
00,
2
[8, 16
P
MM poc) stddev
1.0, 10.0
# hidden units
[32, 64, 128, 256
M
1000 grid side
[50, 100, 200]
# layers
[2, 4,5)
M acc learning rate
[0.01, 0.05, 0.001]
activation
[softsign, relu]
M col learning rate
00.01, 0.05, 0.001
# RNN units
32, 64, 128, 256]
ZTT learning rate
[0.01, 0.05, 0.001]
(c) HPS ranges for the engineered transition model.
Parameter
Range
À stddev
0.0001, 0.001, 0.01
w stddev
0.0001, 0.001, 0.01
Á stddev
0..0001, 0.000, 0.01
E.2 HPS: LEARNED TRANSITION MODEL
The learned transition hyperparameters were selected based on the validation set ELBO value from model discussed in section 4. A total of 400 experiments were conducted, using all of the blackbird training and validation data. The considered parameter ranges are listed table 3b.
E.3
HPS: ENGINEERED TRANSITION MODEL
The engineered transition hyperparameters were selected in the same HPS for the full model discussed above, based on localisation RMSE for 500-step segments of trajectories in the validation set. The considered parameter ranges are listed in table 3c.
21

---

Published as a conference paper at ICLR 2021
F
FURTHER INFERENCE EXAMPLES
Predicted colour
Predicted colour
clover, max speed 3.0 mas
potrick, mas speed 3.0 más
peopact
Observed colour
Observed colcur
[oo)
Prodicted depth
Prodicted depth
SGBM depth
SGBM depch
[m]
sphins, max speed
max speed
MOCAP
MOCAP
Observed
Predicted depth
[n]]
Predicted depth
SGBM depth
SGBM depth
X [m]
mm
Figure 8: Examples of inference in the full spatial model, expressed localisation and mapping for segments of the following trajectories: clover, forward yaw, max speed 3.0 m/s, patrick, forward yaw, max speed 3.0 m/s, sphinx, forward yaw, max speed 3.0 m/s and sid, forward yaw, max speed 3.0 m/s. Left every subfigure: top-down view of the localisaton estimate. Right on every subfigure: depth and colour reconstructions from the generative model for one time step.
22

---

Published as a conference paper at ICLR 2021
G
DATA SET SPLITS
Training set
Test set
trajectory
# stops
duration [s]
trajectory
# steps
duration [s]
3dFigure8, constant yaw. 1.0 m/s
2087
208.7 7
star, constant yaw. 1.0 m/s
2680
268.0
3dFigure,, constant yaw. m/s
2130
213.0
star, constant yaw.
263.5
3dFiguref, constant yaw. 3.0 m/s
2091
209.1
star, constani yaw. 3.0 m/s star, constant yow.
2640
264.0
MFiguree, constant yaw, 3dFigure8, constant yaw, 5.0 m/s ampersand. constant yaw. m/s
213.6
266.7
2240
224.0
star, constant 5.0 m/s
2611
261.1
2086
208.6
star, forward yaw. 1.0 m//
1491
149.1
ampersand. constant yaw. 2.0 m/s
2061
206.1
forward yaw. 2.0 m/s star, forward yaw. 3.0 m/s star, forward yaw. star, forwand yaw. 5.0 m/s picasso, constant yaw, m/s picasso, constant yaw. 2.0 m/s
1503
150.3
amporsand. constant yaw, m/s
833
83.3
1523
152.3
clover, constant yaw, mus clover, constant yaw. 2.0 m/s clover, constant yaw, m/s clover, constant 4.0 m/s clover. constant yaw, 5.0 mus dicee yaw. m/s dice, constant yaw, 3.0 m/s constant yaw. figureR, constant yaw. 1.0 m/s figure8, constant yaw, mv's
257.5
2605
260.5
1108
110.8
204.0
2631
263.1
2071
207.1
2607
260.7
picasso, constant yaw. 3.0 m/s
2072
2U7.2
2605
260.5
picuso, constant yaw. 4.0 m/s
2109
210.9
833
83.3
picasso, constant yaw. 5.0 m/s
2626
262.6
261.4
picasso,
yaw.
1513
151.3
picasso, forward yow. 3.0 más
2083
208.3
2052
205.2
picasso, forward yaw, mls
2057
205.7
figur88, constant 5.0 m/ss mouse, constani yaw, 1.0 m/s
1999
199.9
picasso, forward 5,0 m/s
891
89.1
2641
264.1
comsiant yaw, 2.0 mouse, constant yaw, m/s mouse, consiant yaw, mouse, constant yaw. 5.0 m/s oval. constant yaw, 2.0 m/s oval. constant yaw. m/s oval, cunstamt yaw, 4.0 m/s thrice,
2674
267.4
Validation
833
83.3
trajectory
# steps
duration [s]
262.0
2655
265.5
sid, constant yaw. 1.0 m/s
2688
268.8
2033
203.3
sid, constant yaw, 2.0 m/s
2677
267.7
833
83,3
sid,
yaw. 3.0 m/s
833
83.3
2035
203.5
sid, constant yaw, 4.0 m/s
2668
266.8
yaw, 1.0
2726
272.6
sid, constant yaw. 5.0
2661
266. 1
thrice, constant yaw. 2.0 m/s constant yaw. m/s thrice, constant yaw, 4.0 m/s thrice, constant yaw. 5.0 m/s
2658
265.8
sid, forward yaw. 1.0 m/s sid, forward yaw, m/s sid, forward
897
89.7
265.6
110.9
2656
265.6
m/s
833
83.3
2621
262.1
sid, forward yaw. ml's sid, forward yaw. m/s sphinx, constant yaw; m/s
1109
110.9
tiltedThriee. constant tiltedThrice, constant yaw, 2.0 m/s
m/s
2624
262.4
1110
111.0
2624
262.4
2612
261.2
riltedThrice, constant yaw, m/s
833
83.3
sphinx, constant yaw. m/s
2601
260.1
titteThriic,, constant yau, 4.0 m/s
2602
260.2
sphinx, constant 3.0 m/s
833
83.3
riltedThrice, constant yaw, winter, constant yaw. m/s winter, constant yaw, mas winler, constunt
257.4
sphinx, constant yaw, m/s sphinx, forward yaw. 1.0 m/s sphinx, forward yaw, 2.0 m/s sphinx, forward yaw. 3.0 m/s sphinx, forward yaw. 4.0 m/s
2560
256.0
2625
262.5
1088
108.8
2570
257.0
1087
108.7
m/s
833
83.3
833
83.3
winter, constant yaw. 4.0 m/s winter, constant yaw, m/s
2569
256.9
1086
108.6
2620
262.0
bentDice, constant yaw. bentDice, constant 2.0 m/s
262.4
ampersand. forward yaw. 1.0 ma's
1100
110.0
2698
269.8
ampersand, forward yaw. m/ss
beatDice, constant yaw. m/s
417
41.7
clover, forward yaw. 1.0 m/s
1088
108.8
bentDice, comstant yaw, m/s
2632
263.2
clover. Jorward yaw, 2.0 m/s
1088
108.8
beniDice, forwand yaw. 1.0 m/s
1058
108.8
clover, forward yaw. 3.0 m/s
833
83,3
beniDice, forward yaw, 2.0 m/s
1088
8
clover, forward yaw, 4.0 m/s clover, forward yaw.
1101
110.1
bentDice, forward yaw, 3.0 m/s
833
83.3
1091
109.1
patrick, constant yaw. patrick, constant 2.0 m/s
dice, forward yaw. 1.0 m/s
1096
109.6
2584
258.4
dice, forward yaw. 2.0 m/s
109.9
patrick, constant yaw. m/s
417
41.7
dice, forward yaw. 3.0 m//s mouse, forward yaw. 1.0 mls forward yaw, m/x
833
83.3
patrick, constant yaw. m/s
2611
261.1
1110
111.0
patrick, constant yaw, 5.0 m/s
2580
258.0
1107
1 10.7
patrick, forward yaw. m/ss
108.9
mouse, forward yaw, 3.0 m/s
833
83.3
patrick, forward yaw. 2.0 ma's
1089
108.9
mouse, forward yaw, 4.0 m/s mouse, forward yaw. 5.0 mls oval. forward yaw, m/s oval, Forward yaw. 2.0 m/s oval, forward yaw, 3.0 m/s
110.8
patrick, forward yaw. más
1107
1 110.7
patrick, forward yaw. 4.0 mas
1091
109.1
1086
108.6 6
892
89.2
833
83.3
oval, forward yaw, 4.0 m/s
1086
108.6
thrice, forward yaw. 1.0 m/s yaw. thrice, forward yaw. m/s thrice, forward yaw, m/s Uhrice, forward yaw. 5.0 m/s
1088
108.8
108.8
833
83.3
1088
108.8
1088
108.8
filtedThrice, forward yaw, 1.0 m/s
898
89,8
tiltedhhrice, forward yaw, 2.0 m/s
1088
108.8
ultedThrice,, forward yaw. 3.0 m/s
833
83.3
tiltedTarice. forward yaw. tiltedThrice, forward yaw. 5.0 m/s
108.8
1087
108.7
winter, forward yaw, 2.0 m/s
1089
108.9
winter, forward yaw. 3.0 m/s
833
83,3
winter, forward yaw, 4.0 m/s
1091
109.1
23


Reviews:
[{"review_id": "xBMl4Vm985", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper addresses the problem of dense RGB-D SLAM. The key idea is to formulate the problem as a deep state-space model and infer the state of the latent variables (i.e. pose and geometry) using variational inference. The experiments demonstrate that the method performs well in a challenging quadcopter dataset. However, the advantages of the approach are not demonstrated. \n\nStrengths\n- The proposed approach is a very elegant and principled formulation of the dense SLAM problem. The probabilistic graphical model considers the observations, dynamics and latent states of the agent (i.e. the pose and dense map). As far as I am aware, not many deep slam approaches consider all these factors in the estimation process. The proposed model is therefore a nice unification of neural networks, dynamics and multi-view geometry.\n\n- The method has a number of interesting capabilities such as the ability to predict the future trajectory of the camera. The demonstration of the usefulness of the inferred map in downstream tasks such as navigation is also very interesting.\n\nWeaknesses\n- The related work is severely lacking. Specifically, the paper seems to marginalise most of the recent methods that try to combine deep learning with dense SLAM. The authors summarize the whole collection of such approaches with a single sentence: “depth or semantic representations with existing SLAM methods has also become a prevalent direction of research”. This isn’t a fair representation of existing works and doesn’t really do much to position the authors’ novel contribution with respect to the existing literature. The authors need to discuss and contrast their approach to works such as BA-Net, DeepFactors , Neural RGB->D, DeepSFM etc. Currently, the related work is too focussed on traditional SLAM approaches (like VinsMono) and 2D/2.5D methods like VAST/DVBF-LM.\n\n- The proposed method requires RGB-D as input which is obtained from SGM in the paper. How sensitive is the approach to noise in the depth input, and can the approach work with just RGB input? The reason I consider this a weakness is that many existing RGB-D SLAM approaches (even traditional ones like ORB-SLAM2) have very high accuracy and in some sense can be considered a “solved problem”. This is not really the case for dense monocular SLAM which is still a very challenging problem.\n\n- Furthermore, the advantages of the proposed approach are not described in the paper. In the evaluation the improvement over ORB-SLAM2 in terms of accuracy doesn’t appear to be very significant. Is it simply an improvement in accuracy or are there other advantages as well? \n\n- The experiments and comparisons are also rather limited. The authors have only evaluated on a single quadcopter dataset which doesn’t really show the generalizability of the trained model. ORB-SLAM2 and VINS-MONO don’t have any trained components and are therefore completely general (apart from the need for tuning some settings). How does the proposed approach perform when trained on Blackbird and evaluated on EuRoC, for example? \n\n- The runtime of SLAM approaches is quite important, especially for application on platforms such as quadcopters. It would be good if the authors could include some indication of the runtime and whether the inference can be done in realtime.\n\nMinor points\n\nIt would be good to bold the best performing methods in the table as this makes it easier to see the relative performance of the methods.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "scwBHMqKfi-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThe paper proposes a framework built on DVBF-LM, extended to dense 3D mapping. Overall I find the work somewhat incremental over DVBF-LM and that the methodology lacks clarity. \n\n## Strengths\n\n - The work builds on a fundamentally new and interesting line of generative variational approaches to SLAM\n - It also builds on recent work in differentiable SLAM systems\n - It shows results on datasets taken from an actual quadrotor.\n - Results seem promising (but comparisons to some other related approaches might be missing)\n \n\n## Weaknesses\n\n - I found the methodology extremely difficult to follow. There are many variables used, and some choices are simply not explained or not clear. This becomes very laborious. One such example is that the variable $o$ is used to represent occupancy as well as the agent location in space. In all these variables, I missed a definition for $q_\\phi$. Part of this is also that non-standard terms and notation are used. As far as I know, \"map charts\" is not a common term. Neither is emission model (should be measurement model). Although I do see that these notations are borrowed from DVBF-LM.\n\n - I feel there are some related works that are missing from the discussion. The approach in [1] formulates the fully differentiable dense SLAM problem (including ray casting) on real world datasets, [2] is a recent novel 3D rendering approach and also does differentiable ray casting,  [3] demonstrates an end to end approach to learning measurement likelihood models with an RL active localization framework, and [4] which is also uses a clever combination of deep learning and multi-view geometry to produce dense 3D maps. There are others. In general, differentiable ray casting is not new, and the authors should make a more concerted effort to cite this work and place their approach within this context.  \n\n[1] ∇ SLAM: Dense SLAM meets Automatic Differentiation\nKM Jatavallabhula, G Iyer, L Paull\n2020 IEEE International Conference on Robotics and Automation (ICRA), 2130-2137\n\n[2] Neural Reflectance Fields for Appearance Acquisition\nS Bi, Z Xu, P Srinivasan, B Mildenhall, K Sunkavalli, M Hašan, ...\narXiv preprint arXiv:2008.03824\n\n[3] Active Neural Localization\nDS Chaplot, E Parisotto, R Salakhutdinov\n6th International Conference on Learning Representations (ICLR-18)\n\n[4] Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction\nT Laidlow, J Czarnowski, A Nicastro, R Clark, S Leutenegger\n2020 IEEE International Conference on Robotics and Automation (ICRA), 7373-7379\n\n[5] Neural scene representation and rendering\nSMA Eslami, DJ Rezende, F Besse, F Viola, AS Morcos, M Garnelo, ...\nScience 360 (6394), 1204-1210\n\n - It would nice to know in the related work section what exactly it is about previous methods that makes them not applicable to \"real-world 3D modelling\". In particular, the very related DVBF-LM which method builds upon. \n\n - The parts of the \"Method\" section which are taken from DVBF-LM should be moved to a section called \"Background.\" That would enable the reader to more clearly differentiate what is new here. \n\n## Other Comments/Questions\n\n - You state that \" $M^col$ represents the parameters of a feed-forward neural network $f_{M^col}: R3 \\mapsto [0, 255]^3$. The network assigns an RGB colour value to each point in space.\" What colours are assigned to free space? What about viewpoint dependent illumination?", "extra_scores": {}, "extra_text": {}}, {"review_id": "o204HUSetg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel learning-based visual-inertial odometry algorithm. The algorithm simultaneously reconstructs the world map as well as the states of the agent from the stereo RGBD sensors.  The world is modeled as an occupancy grid with color. A graphical model with attention mechanism and ray casting is used to model how the world and the agent state renders the RGBD sensor data. ELBO is used to optimize the model. The technical details look sound to me.\n\nThe learning part of the paper is to use neural networks to model the dynamics: the distribution of the next agent state given the current state and the IMU is modeled as a normal distribution whose mean and standard deviation are determined via a neural network that takes the current state and the IMU as the input. The neural network is trained on pre-recorded agent states and IMU data captured in the MOCAP. Note that the neural network builds upon the Euler integration based IMU model; it models the residual between the true agent next state and the prediction using the Euler-integration based IMU model. \n\nThe paper performs extensive experiments on a dataset that includes agents of different velocities and different types of trajectories. Qualitatively the trajectories generated by the proposed method look great in the figures. Quantitatively, the average localization error is around 0.2% of the trajectory length, demonstrating the practicality of the method. \n\nHowever, compared with the SOTA methods, the proposed method is slower and at the same time has higher localization errors. In addition, the learning-based dynamics model does not outperform the Euler integration based IMU model, despite the former builds upon the latter. The paper also reports two outlier data points in the experiments where the learning-based dynamics model leads to significantly higher localization errors. In the appendix, the authors analyze these two data points and identify the cause: the landing of the agents in these two trials fail and lead the model to learn ungeneralizable dynamics.\n\nThe presentation of the paper is great. The technical parts of the paper are written concisely and precisely with clear motivation and intuition. I find the paper easy to understand. \n\nTo summarize:\n\nPros:\n1. Great presentation\n2. Novel formulation and learning-based dynamics model\n3. Extensive experiments\n\nCons:\n1. The proposed method is behind the state-of-the-art in both speed and localization accuracy\n2. The learning-based dynamics model proposed by the model does not outperform the closed-form Euler integration based one", "extra_scores": {}, "extra_text": {}}, {"review_id": "BpGKLJN-Xj-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a Deep Variational Bayes Filter (DBVF) for Deep-Learning based SLAM in 3D environments. It builds upon similar work for 2D environments in [Mirchev et. al. 19], and learns a full 3D RGBD occupancy map and a sequence of 6 DoF poses (localization) using raw stereoscopic camera data. Differentiable ray-casting and an attention model is described to access the learnt global map to give a local map and an expected observation - using an emission model from the current pose and local map. A transition model describing the evolution of the dynamics of the agent is also learnt. A variational approximation of the actual posterior (of the sequence of poses and the map, given the sequence of observations) is learnt by optimizing the standard ELBO equation from Variational Bayes. Such deep generative models, once learnt (in an unsupervised way) for an environment, allows one to hallucinate a sequence of poses and observations, given the learnt map and control inputs. This allows downstream robotic control tasks like environment exploration and path planning to be integrated into the model. Experiments on a simulated dataset with a flying drone in a subway and living room environments demonstrate good SLAM performance (that approach traditional methods): bird's eye view projections of the 6 DoF poses and the emitted maps closely match the ground truth poses and the occupancy grid. This is a well-written paper that represents a good step up from [Mirchev et. al. 19] to formulate a DVBF with realistic RGBD data streams. The authors mention that the computational times for this method is still far from conventional SLAM techniques - an actual quantification of the time taken during inference would be useful.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for SLAM like dense 3D mapping (colored occupancy grid) based on differentiable rendering with a possibility to provide a probabilistic generative predictive distribution, evaluated on UAVs.

Initially this paper has a wide spread of reviews, with ratings between 4 and 9. Reviewers appreciated the elegant and principled formulation and the interest of the predictive distribution. On the downside, several issues were raised on the incremental nature wrt to DVBF-LM; presentation and writing being very dense and difficult to follow; positioning wrt to prior art; performance with respect to known visual SLAM SOTA baselines; limited evaluations.

The authors provided responses to many of this questions and also updates to the paper, which convinced several reviewers, who unanimously recommended acceptance after discussion.

The AC concurs.
Decision: Accept (Poster)