PRISM: Probabilistic Real-Time Inference in Spatial World Models
Authors: Atanas Mirchev
Baris Kayalibay
Ahmed Agha
Patrick van der Smagt
Daniel Cremers
Justin Bayer
Venue: CoRL 2022 Oral (2022)
Keywords: generative model
SLAM
Bayes filter
uncertainty
differentiable rendering

Abstract: We introduce PRISM, a method for real-time filtering in a probabilistic generative model of agent motion and visual perception. Previous approaches either lack uncertainty estimates for the map and agent state, do not run in real-time, do not have a dense scene representation or do not model agent dynamics. Our solution reconciles all of these aspects. We start from a predefined state-space model which combines differentiable rendering and 6-DoF dynamics. Probabilistic inference in this model amounts to simultaneous localisation and mapping (SLAM) and is intractable. We use a series of approximations to Bayesian inference to arrive at probabilistic map and state estimates. We take advantage of well-established methods and closed-form updates, preserving accuracy and enabling real-time capability. The proposed solution runs at 10Hz real-time and is similarly accurate to state-of-the-art SLAM in small to medium-sized indoor environments, with high-speed UAV and handheld camera agents (Blackbird, EuRoC and TUM-RGBD).


PRISM: Probabilistic Real-Time Inference in Spatial World Models
Atanas Mirchev', Baris Kayalibay', Ahmed Agha', Patrick van der Smagt', Daniel ; Bayer'
Machine Learning Research Lab, Volkswagen Group, Technical University of Munich atanas www.heerarggaa.......mm. ai
Abstract: We introduce PRISM, method for real-time filtering in probabilistic generative model of agent motion and visual perception. Previous approaches either lack uncertainty estimates for the map and agent state, do not run in real-time, do not have a dense scene representation or do not model agent dynamies. Our solution reconciles all of these aspects. We start from a predefined state-space model which combines differentiable rendering and 6-DoF dynamics. Probabilistic inference in this model amounts simultancous localisation and mapping (SLAM) and is intractable. We use series of approximations to Bayesian inference to arrive at probabilistic map and state estimates. We take advantage of well-established methods and closed-form updates, preserving accuracy and enabling real-time capability. The proposed solution runs at 10Hz real-time and is similarly accurate to state-of-the-aa SLAM in small to medium-sine sizzuu sized indoor environments, with high- speed UAV and handheld camera agents (Blackbird, EuRoC and TUM-RGBD).
Keywords: generative model, SL.AM, Bayes filter, uncertainty, diff. rendering
1 Introduction
Moving agents perceive streams of information, typically mix of RGB images, depth and inertial measurements, Probabilistic generative models [1] are principled way to formalise the synthesis of this data, and from these models inference can be derived through Bayes' rule. We focus on exactly such inference and target the agent states and the scene map, a problem known as simultancous localisation and mapping (SLAM). We treat model, such that the combination is useful for model-based control: the posterior inference serves as state estimator and the predictive state-space model as simulator with which to plan ahead [2].
posterior approximation for given state-space
To pave the way towards decision making, we believe an inference method should have:
a compatible predictive model for both RGB-D images and 6-DoF dynamics;
principled state and map uncertainty;
real-time performance on commodity hardware;
state-of-the-ard localisation accuracy.
We motivate these requirements further in appendix J. Prominent methods like LSD-SLAM [3], ORB-SLAM [4], DSO [5] have propelled visual SLAM forward, with heavy focus on large-scale localisation. The core of modern large-scale SLAM maximum a-posteriori (MAP) smoothing in a probabilistic factor graph [6, 7]. At present this demands sparsity assumptions for computational feasibility, which obstructs the tight integration of dense maps and rendering. Nonetheless, for smaller scenes the recent popularity of neural models (c.g. NERF [8]) has sparked interest in inference through renderer (e.g. [9, 10, 11]), but dynamics modelling and uncertainty have remained out of scope. Conversely, classical filtering comes with dynamics and uncertainty in real-time (e.g. [12, 13, 14]).
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

inferred
predicted
ground truth inf. map cloud
2
Figure 1: Inference is tailored to the depicted predictive model. Predicting future rollouts, shown, required for optimal control. Ground-trud trajectory in black, inferred trajectory from past data in blue. In orange, see uncertainty envelopes for the predicted future states. On the right, predicted and ground-truth future images. Visualised in 2D for clarity, our method operates in 3D.
but over time has given way to large-scale smoothing [6] and to our knowledge has not been well explored for the integration of dense differentiable rendering and dynamics on moderate scale.
Overall, we find there is a need for cohesive inference solution that satisfies our requirements. We thus contribute by meeting all the above goals, emphasising the link to predictive model (fig. 1).
start from generative model Mirchev et [15], who combine differentiable rendering and agent dynamics in probabilistic framework. The authors considered stochastic variational inference for this model, applying off-line with runtime orders of magnitude too long for on-line use. derive
We pursue alternative route for real-time inference: from the generative assumptions approximations to the true marginal filters over the last state and map [16]. By focusing recursive filtering updates, we identify where established probabilistic inference and computer vision techniques can be used, putting emphasis on fast closed-form updates. We find this divide-and-come -onide- strategy a good compromise for achieving the aforementioned objectives under computational constraints.
We evaluate the proposed solution on two unmanned aerial vehicle (UAV) data [17, 18) and TUM-RGBD [19]. Our method PRISM runs 10 Hz real-time with similar localisation accuracy to state-of th-the-rrt SLAM in moderately-sine sized indoor environments. It provides uncertainty estimates and features predictive distribution that can both render images and forecast the agent's 's movement.
2 Related Work
Generative models Generative state-space models simulate the formation of observed data over time in a Markov chain [1, 12, 20, 21, 22, 23], serving as world models [24, 25]. With their agent dynamics and tute-to--bbeervtton emission models we can imagine future rollouts for planning [2. 26, 27, 28, 29, 30, 31, 32, 33J. We abide by this framework and design a posterior inference for a spatial state-space model, to enable on-line control. Among such models (e.g. [34, 35, 36, 37, 38, 39, 40]), tailor our inference to the model of Mirchev al. [15]. It scales to 3D with rendering and 6-DoF dynamics. We contribute real-time inference that fits its probabilistic formulation.
SLAM through image synthesis The assumed generative model renders RGB-D images, which is related to SLAM through full-image synthesis. Traditional methods feature varied maps, from volumetric to surfels (e.g. [41, 42, 43, 44, 45, 46, 47, 48, 49]), and commonly estimate new camera poses by aligning new observations to a rendered image with variants of point-to-platt to-plane ICP with photometric consistency [50, 51, 52, 53, 44]. We extend this optimisation with dynamics in our approximate state filter [54]. A recent trend is use implicit scene representations like NERF (e.g. [8, 55, 56]) with high rendering fidelity. Gradient-based pose inference through NERF-like rendering has received attention [57, 58], with iMAP [11] and NICE-SLAM [9] being two real-time

---

solutions. The mapping runtime of such methods is weighed down by optimisation through the renderer. Rendering can be sped up by decomposing parameters over space, e.g. by using voxels or primitives [59, 60, 61, 62, 63], but how to update neural maps in closed form remains unclear. Therefore, we rely on vanilla voxel grid maps [15, 64), as their probabilistic treatment and closed-form updates are straightforward, leaving implicit representations for future work. We note that none of the aforementioned methods incorporate dynamics and uncertainty, which distinguishes approach.
Probabilistic SLAM inference SLAM filters are thoroughly explored for flat 2D modelling [65, 12, 66, 13, 67, 68, 69]. but have been superseded by MAP smoothing in modern visual SLAM (e.g. [3, 4, 70, 71, 72, 73)). primarily due to scalability concerns [6, 74]. However, as of now smoothing not computationally feasible without sparsity assumptions. We therefore reexamine filtering for differentiable rendering, as we aim to obtain dense map posterior with uncertainty in real-time (see appendix J for further motivation). Filters may benefit from the dense modelling of observations [74], which aligns with our objective, and we will demonstrate they can be feasible solution for moderately sized indoor environments. For the states, we use a Laplace approximation [75] and velocity updates similar to those in extended Kalman filters [12]. For the map, occupancy grids are a common probabilistic choice [64, 66, 76] and closed-form mapping has been used in that context [69]. To enable rendering we provide a similar derivation, but for a signed distance function (SDF), which is related. Probabilistic SDF mapping dates back Curless and Levoy [41], and SDF updates have well-known probabilístic interpretation [77, 78). We use these approximations to arrive at holistic probabilistic solution that scales dense 3D modelling in real-time.
3 Overview
We approach on-line SLAM inference with two aims in mind. First, we want to harmonise our map and state estimation with predictive model. Second, we want to quantify uncertainty: estimates and predictions should account for modelling inaccuracies as well as measurement and process noise. Both are important for autonomous decision making. To achieve this, we derive a Bayesian posterior in the probabilistic model of Mirchev et al. [15], to ensure that inference matches the forward model. Before we delve into our proposed solution, we present a practical summary. At every time step:
1. we point-estimate the agent's 's pose using gradient descent, involving geometry and dynamics. 2. we extend the pose with a Gaussian covariance matrix through a Laplace approximation.
3. with the pose, we estimate the agent's 's current velocity in closed form.
4. with the pose and the current observation, we update the map in closed form.
We use well-established methods for the above. In 1. we combine assumed density filtering [79], point-to-plane ICP [50] and photometric alignment [51, 52]. In 2. we use a Laplace approximation [75, 80]. In 3. we use linear-Gaussian updates, akin to Kalman filters [12]. In 4. we first derive generic closed-form map updates, which boil down to SDF updates [41] for our generative assumptions.
We contribute by deriving holistic Bayesian inference from the generative model started with. In doing so, identify where traditional techniques are applicable to make practical algorithm.
4 Methods
In the following we will denote generative distributions, true posterior distributions and conditionals with p(-). Respectively, approximate distributions will be denoted with q(-). Approximation steps will be indicated by s in equations. We use (-) to subsume estimated distribution parameters into Φb subscript ' indicates that variable or distribution different at every time step.
4.1 Background
We start with overview of the generative model of Mirchev et al. [15] from which we will derive the inference. We assume sequence of RGB-D CGB-D&o observations X1.7 and sequence of agent states

---

Z1:T driven by controls -1:7-1 form a Markovian state-space model, Each observation is constructed from a respective state with rendering emission model p(x; M,Zz), where M is a global latent random variable for a dense map. A transition model p(zz Zz-1, 14-1) accounts for the agent dynamics, where u, are known acceleration controls. Assuming 2 given, the joint distribution is:
p(M, 22:T, X1:T U1:T-1,21) = p(M)p((x M,,
II p(z: Zt-1, 4x-1)p(xx Ze M).
t=2
The map is a 3D voxel grid of occupancy and color-each cell contains four values. The emission is fully-lifferentiable and performs volumetric raymarching, searching for a unique hit position at a surface along each ray [81]. The transition performs Euler integration, using the acceleration controls and maintained velocity from the latent state. Appendix B and the original paper have the details.
4.2 Posterior Choice
First we need to choose which posterior to approximate. For example, Mirchev et al. [15] approx- imate the full posterior over the map and all states p(M, Z2:T XI:T, u Zi) with variational inference [82]. While generic, this approach is slowed down by rendering at every optimisation step [54), and the inevitable stochastic optimisation demands multiple steps until convergence. addition, estimating the posterior over all states scatters the optimisation budget across the whole trajectory.
To enable real-time inference we target alternative posterior, the filter p(M, 2t X1:t, ; ) as the last state belief is enough for planning ahead [2]. Since filters can be updated recursively [16, 80), we can use closed- form updates for fast inference. Still, maintaining the joint distribution is too costly because of the large dense 3D map M. Instead, we approximate the two marginal filters:
(M (M H₂) p(M X1:6. -1:1-1.71)
q (z)))= plz H₂) = p(zz X1:,, U1.1-1,Z1)
where H, = X1 11 2 More details about this modelling choice can be found in appendix A. We draw attention to the shorthand notation p( Hi), which will appear again in the following.
4.3 Approximate Filtering
For both marginal filters, we will arrive at adequate approximations by reusing the following equation:
/ p(zz 1 u, (1 2, H(-1)dz:-1, (I)
p(M ze H₂) x p(x+ Ze. M)
This is a classic recursive expression of the Bayes filter [16]. Starting from each true marginal posterior, we will first expand the joint, then use eq. (1) and apply set of approximations. Next we will discuss our final result, we defer the detailed derivation of both filters to appendices C and E.
4.3.1 Marginal Map Filter
We begin with the map approximation, starting from the true marginal Bayes filter:
/ p(M H.)
p(M| H1)
u: x / p(xe ze, Mc f plz, 11 1)(M 2:-1 He-l) dz, dz:
2 p(xe 22, M) x (M)
(2)
{M X,, 2x) (M) =: qi (M).
(3)
Equations (2) and (3) hide a few approximations detailed in appendix C. The resulting solution takes a nominal state sample 2,, with which a map update qM Xx, 2,) is applied to the previous map belief a We set 2, to the mean of the current state belief qf (z.). Accepting some bias, we do this for speed as it is our best guess for 2, without extra computation." Intuitively, the map update
E.g. the size full-covariance Gaussian representations [12] or carrying multiple maps parallel for a Rao-Blackwellised particle filter [13, 14] hecome prohibitive. Appendix F discusses this approximation further.

---

qMl X,, 2,) populates the map such that the observation can be reconstructed. Our derivation of the updates is similar to the one by Grisetti et al. [69] for 2D occupancy maps, but now applied to 3D.
The above approximation generic, agnostic to the specific map and rendering assumptions. In prac- tice, we need closed-form map update qMM Xx, 2,) that is faithful to the emission p(x; 22, M). In this work, follow Mirchev et al. [15] and use Gaussian map that factorises over voxels:
qe (M) = II [[
M Mijka diag( diaag((oi))) ( (ookk.)) )²).
59k
Here the indices ijk run over voxels 3D grid. For this specific representation and the assumed surface-basel rendering, we identify that the map update gM x 2t) can be implemented as probabilistic signed distance function (SDF) update [41]. We provide the technical details in appendix D. SDF updates for voxel maps are traditional concept in computer vision, and prior work has considered their probabilistic interpretation before [77. 78]. We contribute by identifying the place of such updates in probabilistic filter that follows the generative model of [15]. A detailed discussion of how the above relates to classical SDF update equations can be found in appendix D.
The above approximations are motivated by the real-time constraint. For example, one could optimise eq. (2) directly with gradient descent through the renderer, but evaluating the emission is expensive and hinders accurate convergence are desirable, as optimisation would then be stochastic and gradients noisy [83]. In contrast, the derived one-shot map updates are meant to have cost similar to emitting just once, while capturing uncertainty as well. We show some of the differences between the two approaches in section 5.3.
budget. This is particularly true when uncertainty estimates
4.3.2 Marginal State Filter
Similarly, for the state filter we start from the true marginal and arrive at approximations via eq. (I):
p(z= Hi)
p(M He) dM
/ p(xe ze, M) / plae ze-1, Ur Ur--;; ()M 21-1 H+-1) dM
x
p Xt zpose M) (z
11-1, Hz 1) (z) 120000 uc-1, Hl-11
(4)
~ o8 (z/000) x (z()) 20000 u H,-1) =: 9: (z:).
(5)
We detail all the approximations that lead to eq. (4) in appendix E. In eq. (4) we have three terms: an image reconstruction likelihood, a Gaussian pose prior and linear Gaussian velocity conditional given a pose. The latter two we obtain analytically with a linear approximation of the transition model and the previous Gaussian belief q.- (zz-1) 1) (c.f. appendix E). First, using the first two terms of eq. (4) we define a maximum a-posteriori (MAP) objective for pose optimisation:
arg max log p Xx M, 2. pose) puse
log (z) pz.e
pose
u:-1, Hc-1).
Here, M is a nominal map sample set to the mean of the previous map belief oe (M). The term log (z) pose u, H1-1) is an approximate dynamics prior over the current pose, it makes the pose respect the transition model. The term log |,, pose) represents reconstructing the current observation, optimising it for the current pose will align the observation to the map. However, evaluating this rendering term in every gradient step is inefficient. Because this, we replace it with the cradiction-to-observaae
objective used by Kayalibay et [54], Nießner et al. [45], Newcombe et al. [84]. We refer to [54] for further motivation and we list the technical details in appendix E.
The above optimisation gives us a MAP pose estimate, which we denote with fu Next, we apply a Laplace approximation [75] around it to obtain a full covariance matrix Σ which captures the curvature of the objective. This leaves us with a full Gaussian belief over the current pose:
@ (2) Nx22 pose
1 pope DP06e)
AAppendix F discusses this approximation further.
5

---

state map strueture
(b) Blackbird map uncertainty.
inferred
interred
map structure view directions
(a) Example mapping and localisation
(c) EuRoC map uncertainty.
Figure 2: (a) 3D reconstruction, example emission and inferred trajectory for EuRoC/V 102 and TUM-RGBD fr3/office. (b) Blackbird experiment. Top-down map uncertainty on the left, black is uncertain, orange is precise. Precision highest in triangle around the center, which the camera frustum where the agent remains sitting platform for a long time, the orange triangle amidst the map point cloud on the right. (c) Analogous EuRoC experiment. Map uncertainty is high outside of the room, at the center and behind the structures the left due occlusion. The uncertainty in the center is high because the agent primarily looks outwards (view directions in the right image).
Finally,
can combine this Gaussian with the Gaussian velocity conditional U. H,-1) (the third term in eq. (4)) into full-state belief in closed form:
puse
(2,) = N(z: p4, Σ,) = NZ p D "(" vel D2z. + c,, Ere))
This is approximate, we do for speed and find does not harm localisation in practice. Appendix describes how the linear Gaussian terms come to in more detail.
Experiments
Originally we out with a few goals: the inference method should be faithful the generative assumptions, it should quantify uncertainty and it should in real-time. What follows is empirical analysis of these aspects. We evaluate on the EuRoC [17], Blackbird [18] and TUM-RGBD [19] data sets. The agent in the former two is an unmanned aerial vehicle (UAV), with speed of up to 4 m/s. For Blackbird, we use Semi-Globa Block Matching (SGBM) for stereo depth estimation [85]. For EuRoC, we use the ground-tree truth Leica MS50 depth readings provided by [10]. We pretend the IMU readings from these data sets are our control inputs. For TUM-RGBD we not feed any controls and assume a constant-velocity transition. All experimental details are in appendix G.
5.1 Inference Through Probabilistic Generative Model
First we look into the synergy between the inference and the generative assumptions. In fig. 2a see mapping and localisation examples. The inferred scenes are consistent, with no dramatic offisess in geometry. More importantly, rendering from the inferred map using the emission p(x, 2, M) works as expected (see middle row), indicating that map updates are consistent with the generative assumptions. This evident from the accuracy of the inferred state trajectories as well (last row), as the pose optimisation objective from section 4.3.2 uses rendered images every filtering step. A potential discrepancy between the inference and the generative assumptions would lead to errors that would accumulate over time, which is not the case.

---

procisson of &
obs. coler
dens (x.) de:
precesion of &q qumeea frustum
obs. color
de (xye) de.
camera frustum
55
673
50
abs. depth
(2.)) de
45
abs. depih
d (el de
40
35
(a) location uncertainty
(b) qr (yaw) orientation uncertainty
Figure 3: Inferred state uncertainty. Inferred trajectories are colored by precision (inverse uncertainty) of certain state dimension, followed by observations, followed by columns of the tracking Jacobian for that same state dimension. (a) Here the precision in 2 (vertical movement) is high (yellow), because the z-orthogonal floor produces consistent Jacobian (bottom right). (b) Here the precision in qN orientation (yaw, azimuth) is low (violet), as there are no orthogonal surfaces (i.e. facing sideways). Note the low Jacobian magnitude of the horizontal floor this time (bottom right).
Map uncertainty The inferred map uncertainty is determined by the map updates. We show its interpretable effects in figs. 2b and 2c for two examples, one from Blackbird and another from EuRoC. Our map updates are akin to traditional SDF updates and the main factor that decides whether map region is certain how often was observed. Regions that were occluded by objects, are behind walls or were rarely view remain uncertain, e.g. as seen fig. 2c. In contrast, the agent spends lot of time looking certain map region, the uncertainty there decreases, seen in fig. 2b.
State uncertainty In fig. 3 we analyse state uncertainty by looking at the variance for individual dimensions. We notice that state uncertainty changes along the trajectory. Uncertainty is determined by what the agent currently sees, based on the geometric relationship between the agent movement and the observed scene (c.g. fig. 3a and fig. 3b). This effect can be explained we examine the Laplace approximation used estimate pose covariances. At any given time step, we set the covariance to Here H is the Hessian of the tracking objective at the mean pose
Dp x - x - -2JJ" estimate and is the Jacobian. The Jacobian connects the pose to all image pixel errors. The more consistent Jacobian entries are for given pose dimension, the smaller the variance for that dimension will be. We refer to appendix H for more details about the map and state uncertainty quality.
5.2 Localisation Accuracy
We compare PRISM's localisation stute-of-the-aee methods in moderately-sizer indoor environ- ments. We consider both baselines with dense maps (TANDEM [10], VSSM-LM [15], iMAP [11], NICE-SLAM [9], VIO [86]) and sparse methods without rendering (ORB-SLAM2 [4], VINS [71], VIMO [70]). The results are in table 1. For the considered trajectories accuracy comparable to the baselines, with differences
few centimeters. At the same time, our inference boasts a predictive state-space model with both rendering and dynamics as well uncertainty estimates, which is not common in the dense visual SLAM literature. Finally, in fig. 4 we see example inferred agent velocities, noting the uncertainty bands. This is possible because model the agent dynamics.
Our localisation accuracy on Blackbird is better than the off-line variational inference results of VSSM-LM presented by Mirchev et al. [15], and at the same time our solution runs in real-time and also captures uncertainty. This shows the advantages the proposed wiiide-and-conquer filtering.
5.3 Approximations for Runtime Improvement
All of our approximations motivated by the real-time siletimme constraint, dictating the need for closed-for map updates, a Laplace approximation, lincarisation assumptions and surrogate pose optimisation objective. Figure 5 shows runtime breakdown for different image resolutions, measured on an
'Last 10 skipped, the drone hits the ground during landing.
7

---

Table 1: Localisation absolute error RMSE meters on
EuRoC [17], Blackbird [18] and TUM-RGBD [19].
inferred
true
Trajectory
Ours
Code
TANDEM
ORB
-0.25
VIO
SLAM2
-0.25
BuRoC/V101
0.041 (#: 0.002)
0.05
0.09
0.031
EuRoC/V100 102
0.035 (#: 0.002)
0.07
0.17
0.02
-0.25
400 time step
EuRoC/V103
0.042 (# 0.002)
0.07
0.048
EaRoC/V201
0.037
(+: 0.001)
0.10
0.09
0.037
Figure 4: Inferred zyz-vclocity.
EuRoC/V202
0.035
(# 0.003)
0.06
0.12
0.035
EuRoC/V203
0.275
LXBS equitions spdite
runtime [ms]]
Trajectory
Ours
VSSM
LM
VIMO
VINS
1.0
pose optin.. rendering
Laplace appes.
100
picasso. m/s 0.064 (#: 0.003)
0.139
0.055
0.097
50
picasso, m/s 0.053 (= 0.003)
0.136
0.040
0.043
picasso, m/s 0.061 (# 0.003)
0.120
0.043
0.045
520
picasso, 4 m/s
0.079 (= 0.005y*
0.174
0.049
0.056
image resolution
star; m/s
0.089 (#: 0.007y*
0.137
0.088
0.102
Figure 5: Runtime breakdown.
star, 2 m/s
0.111 (= 0.009)
0.163
0.082
0.133
star, 3 m/s
0.115 (= 0.012)
0.281
0.183
0.235
Vi-bused (SGD)
smallest en. level
star, 4 m/s
0.153 (= 0.015)*
0.156
X
20c oEt
1.0
SDF level
X
SDF update
0.6
NICE SLAM
ORB
Trajectory
Ours
iMAP
0.3
SLAM2
frl/desk
0.053 (# 0.003)
0.049
0.027
0.016
10'
10'
10'
fr2/xyz
0.029 (#:: 0.001)
0.02
0.018
0.04
runtime [ms]
fr3/office
0.083 (#: 0.001)
0.058
0.03
0.01
Figure 6: Mapping comparison.
NVIDIA 1080 Ti GPU and an Intel(R) Xeon(R) W-2123 CPU at 3.6 GHz. The heaviest operations are rendering and the gradient- baseed pose optimisation. Based on movement speed, rendering can happen periodically, whenever leaves us with total runtime of 10 Hz to 15 Hz, updating the map and state at every data step. In fig. 6 we also compare closed-form map updates to map inference via gradient-descent (e.g. in [15, 8, 11, 9I). While gradient-desee descent is more accurate on bigger budget. it is much more expensive. For example, to match the accuracy of the closed-form updates, which take less than 10 ms, one would need ca. 250 ms optimisation, which impractical. These runtimes are for voxel grid that significantly faster than neural representations [54], which would only exacerbate the problem.
anchor image prediction for pose optimisation is needed. This
Limitations and Conclusion
SDF voxel grids allow for closed-form updates, but their memory footprint limits the maximum resolution and scene size. Voxel hashing [45] or octrees [87] can directly replace them for memory efficiency. Neural maps and dynamically changing maps have remained out of our scope. Their probabilistic formulation and closed-form updates require further investigation. Our map factorises voxels with no inter-region correlation, which could also be improved. PRISM provides interpretable uncertainty in real-time, but estimation is approximate. Obtaining perfectly calibrated uncertainty budget remains an open question (see appendix H). While filtering works for our generative assumptions indoors, filters cannot revisit past errors and can drift in large scenes with high levels exploration [6]. We leave acaee inference considerations for future work.
We have introduced PRISM, method for probabilistic filtering predefined spatial state-space model. Our solution runs in real-time, provides state and map uncertainty, and infers dense map and a 6-DoF state trajectory with velocities. It is comparably accurate to state-of-the-art SLAM in indoor environments. To the best of our knowledge this is the first real-time sully-probabilistic solution for SLAM that combines differentiable rendering and agent dynamics. We validated our method on three challenging data sets, featuring unmanned aerial vehicles and a handheld camera. The results are promising, establishing PRISM as a viable state estimator for downstream model-ban based control.

---

Acknowledgments
We thank our reviewers for the thoughtful discussion, helped better position our contribution.
References
[1] D. Koller and N. Friedman. Probabilistic Graphical Models Principles and Techniques. MIT Press, 2009.
[2] D. P. Bertsekas. Dynamic programming and optimal control, 3rd Edition. Athena Scientific,
2005. ISBN 1886529264. URL http://wwww.orlcatt.rrrr///o////
[3] J. Engel, T. Schöps, and D. Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference computer vision, pages 834-849. Springer, 2014.
[4] R. Mur-Artal and Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 1355):1255-1262 2017.
[5] J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, Mar. 2018.
[6] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics, 32(6):1309-1332 2016.
[7] R. Kschischang, B. Frey, and H.-A. Loeliger. Factor graphs and the sum- sum- sum-product algorithm. IEEE Transactions information theory, (7(2):498-511 2001.
[8] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[9] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[10] L. Koestler, N. Yang, N. Zeller, and D. Cremers. Tandem: Tracking and dense mapping in real-time using deep multi-view stereo. In Conference Robot Learning (CoRL), 2021.
[11] E. Sucar, Liu, Ortiz, and A. J. Davison. imap: Implicit mapping and positioning in real-time, 2021.
[12] R. E. Kalman et al. A new approach to linear filtering and prediction problems [i]. Journal of basic Engineering, 82(1):35 45, 1960.
[13] M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit. Fastslam: A factored solution to the simultaneous localization and mapping problem. In AAAI/IAAI, 2002.
[14] G. Grisetti, C. Stachniss, and W. Burgard. Improved techniques for grid mapping with rao- blackwellized particle filters. IEEE transactions on Robotics, 23(1):34-46, 2007.
[15] A. Mirchev, B. Kayalibay, P. van der Smagt, and J. Bayer. Variational state-space models for localisation and dense 3d mapping in dof. In International Conference on Learning Representations, 2021. URL
[16] S. Särkkä. Bayesian Filtering and Smoothing. Number 3. Cambridge University Press, USA, 2013. ISBN 1107619289.
[17] M. Burri, J. Nikolic, Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart. The euroc miero aerial vehicle datasets. The International Journal of Robotics Research, 35 117 2016.

---

[18] A. Antonini, W. Guerra, V. Murali, T. Sayre-McCord, and S. Karaman. The blackbird uav dataset. The International Journal of Robotics Research, 001:0778564221908333,, 2020. doi: 2001100117700883492000333
[19] Sturm, N. Engelhard, F. Endres, W. Burgard. and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. In Proc. of the International Conference on Intelligent Robot Systems (IROS), Oct. 2012.
[20] L. Rabiner. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2)257-286, 1989. doi: 10.1 10995. 18626.
[21] M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning state space models from raw data. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[22] R. Krishnan, Shalit, and D. Sontag. Deep kalman filters. arXiv preprint 15.00115 2015.
[23] Chung, Kastner, L.. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015, URL https://prooeddings www.niinnnnnnnnnissneeee. cc cc/paaprr/211//t1le Paper. pdf.
b6 1G116.99000333222221112220003113
[24] S. Chiappa, S. Racanière, D. Wierstra, and S. Mohamed. Recurrent environment simulators. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:// openreview net foorumidd=Bis:ai
[25] D. Ha and J. Schmidhuber, Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2450-2462. Curran Associates, Inc., 2018.
[26] M. Deisenroth and C. E. Rasmussen, Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-II), pages 465-472. Citeseer, 2011.
[27] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages 2555 2565. PMLR, 2019.
[28] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020. URL www.:e/ooorrrdddd110000..
[29] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-755 2020.
[30] Beeker-Ehmck, M. Karl, J. Peters, and P. van der Smagt. Learning to fly via deep model-based reinforcement learning, 2020.
[31] M. Karl, M. Soelch, P. Becker-Ehmek, D. Benbouzid, van der Smagt, and J. Bayer. Unsuper- vised real-time control through variational empowerment. arXiv preprint 17X1v:1710.05101, 2017.
[32] D. Corneil, W. Gerstner, and J. Brea. Efficient model-based deep reinforcement learning with variational state tabulation. volume 80 of Proceedings of Machine Learning Research, pages 1059-1058, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
10

---

[33] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4754 4763, 2018.
[34] M. Fraccaro, D. J. Rezende, Y. Zwols, A. Pritzel, S. M. A. Eslami, and F. Viola. Generative tem- poral models with spatial memory for partially observed environments. CoRR, hbbs/180.099400 2018.
[35] S. M. A. Eslami, D. Jimenez Rezende, Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruder- man, A. A. Rusu, L. Danihelka, K. Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, H. King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukcuoglu, and D. Hassabis. Neural scene representation and rendering. Science, 12046394)110041D1 1210, 2018. doi: 10.1 www.iiiennnnccreo...e. 70. URL https: http:p/:/sceenncscciiieseee sciencemag..... crrg//ontet//////e 360/6394/124
[36] K. Gregor, D. J. Rezende, F. Besse, Y. Wu, H. Merzic, and A. van den Oord. Shaping belief states with generative environment models for rl. In Advances in Neural Information Processing Systems, pages 13475-13487, 2019.
[37] A. Mirchev, B. Kayalibay, M. Soelch, P. van der Smagt, and J. Bayer. Approximate bayesian inference in spatial environments. In Proceedings of Robotics: Science and Systems, Freiburgim- Breisgau, Germany, June 2019.
[38] S. Gupta, V. Tolani, J. Davidson, S. Levine, Sukthankar, and Malik. Cognitive mapping and planning for visual navigation, 2019.
[39] E. Parisotto and R. Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In International Conference on Learning Representations, 2018. URL https: //openreview net/forumiid-Bt922
[40] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. In International Conference on Learning Representations (ICLR), 2020.
[41] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996.
[42] C. Carr, K. Beatson, B. Cherrie, T. J. Mitchell, W. R. Fright, B. C. McCallum, and T. R. Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 67-76, 2001.
[43] R. A. Newcombe, S. Lovegrove, and A. J. Davison. Dtam: real-time. 2011 International Conference on Computer Vision, pages 2320-22- 2007,222,220 2011.
Dense tracking and mapping in
[44] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. Davison, P. Kohi, J. Shotton, Hodges, and A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pages 127-136. IEEE, 2011.
[45] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 32(6):1-11, 2013.
[46] M. Keller, D. Lefloch, M. Lambers, S. Izadi, T. Weyrich, and A. Kolb. Real-time 3d recon- struction in dynamic scenes using point- based fusion. In 2013 International Conference on 3D Vision 3DV 2013, pages 1-8, 2013. 10. 10i:10.1109/3DDV 10933DV.2013.9.
[47] T. Whelan, S. Leutenegger, R. F. Salas-Moreno. B. Glocker, and A. J. Davison. Elasticfusion: Dense slam without pose graph. In Robotics: Science and Systems, 2015.

---

[48] Y.-P. Cao, L. P. Kobbelt, and S. Hu. Real-time high-accuracy three-dimensional reconstruction with consumer rgb-d cameras. ACM Transactions on Graphics (TOG), 37:1 16, 2018.
[49] Y. Xu, L. Nan, L. Zhou, J. Wang. and C. C. Wang. Hrbf-fusion: Accurate 3d reconstruction from rgb-d data using on-the-fly implicits. ACM Transactions Graphics (TOG), 11(3):1-19, 2022.
[50] Y. Chen and G. Medioni. Object modelling by registration of multiple range images. Image and vision computing, 10(3):145-155) 155, 1992.
[51] F. Steinbrücker, J. Sturm, and D. Cremers. Real-time visual odometry from dense rgb-d images. In 2011 IEEE international conference computer vision workshops (ICCV Workshops), pages 719-722. IEEE, 2011.
[52] C. Audras, A. Comport, M. Meilland, and Rives. Real-time dense appearance-basee slam for rgb-d sensors. In Australasian Conf. on Robotics and Automation, volume 2, pages 2-2, 2011.
[53] C. Kerl, J. Sturm, and D. Cremers. Dense visual slam for rgb-d cameras. In 2013 IEEE/RS/ International Conference on Intelligent Robots and Systems, pages 2100-2106. IEEE, 2013.
[54] B. Kayalibay, Mirchev, van der Smagt, and J. Bayer. Tracking and planning with spatial world models. arXiv preprint arXiv:2201.10335, 2022.
[55] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[56] Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam. isdf: Real-time neural signed distance fields for robot perception. arXiv preprint arXiv:220.002296 2022.
[57] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. Inerf: Inverting neural radiance fields for pose estimation, 2021.
[58] Z. Wang. S. Wu, W. Xie, M. Chen, and V. A. Prisacariu. Nerf-: Neural radiance fields without known camera parameters, 2021.
[59] C. Reiser, Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands tiny mips, 2021.
[60] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmano, and Y. Sheikh. Neural volumes. ACM Transactions on Graphics, 38(4):1-14, Jul 2019. ISSN 1557-7368. doi: 145/33344663 3323020. URL http: //t://dxx.oo http:://xxxddx..o//// org 1011/111333330 03014///33003463333000...
[61] L. Liu, Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. NeurIPS, 2020.
[62] S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph., 40(4), jul 2021. ISSN 0730-0301. doi: 10:10.1145// 040.1444//45062235599666 URL https://doo http:////oooooo.o..////o 1145//4506666. 3459863.
[63] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 10:11:102 July 2022. doi: 10. 1111/352822233333112, URL httpps:///ii... org/10..1110
[64] H. Moravec and A. Elfes. High resolution maps from wide angle sonar. In Proceedings. 1985 IEEE International Conference on Robotics and Automation, volume 2, pages 116-121. IEEE, 1985.
12

---

[65] S. Thrun, W. Burgard, and D. Fox. Probabilistie Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, 2005. ISBN 0262201623.
[66] K. P. Murphy. Bayesian map learning in dynamic environments. In Advances in Neural Information Processing Systems 12, INIPS Conference, Denver, Colorado, USA, November 29 December 1999]. pages 1015-1021, 1999.
[67] D. Hahnel, W. Burgard, D. Fox, and S. Thrun. An efficient fastslam algorithm for generating maps of large-scale eyclic environments from raw laser range measurements, In Proceedings 2003 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453), volume 1, pages 206-211 vo 2003. doi: $oi:10.111911 (0..110//RRSS0000 200.0..000..120663
[68] M. Montemerlo, S. Thrun, D. Koller, B. Wegbreit, et al. Fastslam 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. I/CAI, volume 3, pages 1151-1156, 2003.
[69] G. Grisetti, C. Stachniss, and W. Burgard. Improving grid- grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pages 2432-2437, 2005.
[70] B. Nisar, P. Foehn, D. Falanga, and D. Scaramuzza. Vimo: Simultancous visual inertial model-based odometry and force estimation. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgauu Germany, June 2019.
[71] T. Qin, Li, and S. Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4):1004 1010000000,,,,000000000000000000 2018.
[72] A. Rosinol, T. Sattler, M. Pollefeys, and L. Carlone. Incremental visual-inertial 3d mesh generation with structural regularities. In 2019 International Conference on Robotics and Automation (ICRA), 2019.
[73] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an open-source library for real-time metric-semantic localization and mapping. In IEEE Inil. Conf. on Robotics and Automation (ICRA), 2020. URL httpp::/giiaaii
cwm/MIT SPARS/Kimessss
[74] H. Strasdat, J. M. Montiel, and A. J. Davison. Visual slam: why filter? Image and Vision Computing, 30(2):65-77, 2012.
[75] P. S. Laplace. Memoir on the probability of the causes of events. Statistical Science, 1(3): 364-378, 1986. ISSN 08834237. URL http: http://wwww..t..roo/o http//sttbee/224/////////
[76] R. Senamayake and Ramos. Bayesian hilbert maps for dynamic continuous occupancy mapping. In Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 458-471. PMLR, 13-15 Nov 2017.
[77] C. Heránndez, G. Vogiatzis, and R. Cipolla. Probabilistic visibility for multi-view stereo. In 2007 IEEE Conference Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007.
[78] W. Dong, Q. Wang, X. Wang, and H. Zha. Psdf fusion: Probabilistic signed distance function for on-the-fly 3d data fusion and scene reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 701-717, 2018.
[79] M. Opper and O. Winther. bayesian approach to on-line learning. 1999.
[80] C. M. Bishop. Patlern recognition and machine learning. springer, 2006.
[81] S. Parker, P. Shirley, Y. Livnat. C. Hansen, and P.-P. Sloan. Interactive ray tracing for isosurface rendering. In Proceedings Visualization '98 (Cat. No. 98CB36276), pages 233-238. IEEE, 1998.
13

---

[82] D. P. Kingma and M. Welling. Auto-eng variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL
[83] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-87) 2017.
[84] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pages 127-136, 2011.
[85] H. Hirschmuller. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on pattern analysis and machine intelligence, 30(2):328-341, 2007.
[86] X. Zuo, N. Merrill, W. Li, Y. Liu, M. Pollefeys, and G. Huang. Codevio: Visual-inertial odometry with learned optimizable dense depth. 2021 IEEE International Conference on Roboties and Automation (ICRA), pages 14382-14385 IEEE, 2021.
[87] F. Steinbrucker, C. Kerl, and D. Cremers. Large-scale multi-resolutie surface reconstruction from rgb-d sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 3264-3271, 2013.
[88] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G, Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Milng and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http::tttttttttee
com/ google/jax.
[89] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/ wtbb//412.0e 141.6690.. 2014.
[90] I.. Jospin, Laga, F. Boussaid, W. Buntine, and Bennamoun. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17 (2):29 48, 2022.
[91] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
14


Reviews:
[{"review_id": "yCZNtdmH9S0", "reviewer": "Reviewer_ZggF", "summary": "This paper describes a real-time filtering-based approach to simultaneous localization and mapping.  Notable features include an environmental model that captures dense geometry and appearance (using a volumetric grid representation), a differentiable rendering-based measurement model, and a 6DOF (constant-velocity) agent dynamics model that permits prediction of the agent’s future states, which is useful for downstream planning and control tasks (such as flight control of UAVs).  The proposed method is evaluated on 3 benchmark datasets involving AUV and handheld cameras (EuRoC, Blackbird, and TUM-RGBD), where it achieves real-time (10 Hz) performance and localization accuracy comparable with current state of the art SLAM solutions.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "One of the paper’s most prominent strengths is its careful and principled development of the proposed method.  At the beginning, the authors set themselves the task of devising a SLAM method that meets the following conditions:\n\n\n* Its environmental model includes dense geometry and appearance information\n* It provides a 6DOF model of the agent’s dynamics (which is useful for downstream prediction, planning, and control tasks)\n* It provides explicit and principled measures of uncertainty for its state and map estimates\n* It achieves real-time performance on commodity hardware\n\nGiven these desiderata, the paper then proceeds step-by-step, starting from the general (abstract) probabilistic formulation of the SLAM problem, to propose (i) specific representations for the map and state that satisfy the above conditions, (ii) suitable measurement and dynamics models (built on these representations), and (iii) approximate Bayesian updates for the map and state that permit the overall inference to be realized as a real-time filtering-based SLAM system.\n\nI especially appreciated the paper’s careful attention to detail when developing the proposed method.  The appendices in the supplementary material provide detailed derivations of each of the main results presented in the paper, including careful signposting of what approximations are being made where, and the rationale for doing so (with respect to the goals outlined above).  As a result the technical approach is well-founded and exceptionally clearly explained, and the experimental results demonstrate that the method achieves the stated objectives, specifically real-time operation and localization accuracy comparable with state of the art alternatives (at least for the three test cases considered in the experimental section – more on this below).\n\nIn terms of weaknesses:\n\n**The paper significantly mischaracterizes prior work on SLAM, including current state of the art alternatives.**  The paper repeatedly stresses that one of its central aims is to devise an inference method that provides principled uncertainty estimates for the state and the map, claiming this as one of its top-line contributions.  In so doing, it repeatedly attempts to draw a contrast between what it calls “classical probabilistic methods” (which it appears – based on the references it associates with this phrase – to identify with either *EKF-* or *particle filtering*-based approaches specifically) and “modern visual SLAM”.  For example, in its discussion of prior work, the paper asserts that “modern visual SLAM has drifted away from the probabilistic paradigm”.\n\nThis assertion is emphatically false.  It is true that modern visual SLAM methods are not based on *filtering* techniques, but this is because over the last ~10-15 years it has become well-understood that optimization-based *full smoothing* methods provide superior accuracy (because they do not suffer from particle depletion in high-dimensional estimation problems as particle filters do, nor irreversible linearization errors as EKFs do) and frequently faster computation (because they can take advantage of the sparsity present in the full smoothing optimization problem, sparsity which is *not* present in the mean-covariance state representation used in standard extended Kalman filters [and which this paper identifies as a key computational bottleneck, motivating the use of the independent marginal posterior filters for the state and map proposed in this work]).\n\nFrom a probabilistic standpoint, these graphical full smoothing methods perform maximum a posteriori (MAP) inference over the  joint distribution for the SLAM problem shown at the bottom of page 3: that is, they emphatically *are* probabilistically-grounded approaches.  Indeed, these approaches are typically developed by positing a probabilistic generative model for the data (just as this paper does), and then deriving from this the corresponding form of the MAP estimator.  Moreover, having calculated a MAP estimate, it is possible (and in fact common practice) to extract estimates of the posterior uncertainty using *exactly the same* Laplace approximation proposed in this work – except in the case of smoothing approaches, this is an approximation to the *full posterior* distribution, and therefore provides access to posterior covariances among *arbitrary subsets* of variables (something that the method proposed in this work actually cannot do, because it maintains separate marginal filters for the map and state).\n\n**Possible additional limitations of the approach and experiments not addressed:**  One limitation of the proposed approach is that its map representation is based on a voxel grid, which limits the scale of the environments that can be mapped at a given resolution (due to memory consumption).  As a result, the datasets that are used for the experimental evaluation cover a fairly small area by the standards of modern SLAM systems (roughly the size of a large single room – for example, the Blackbird dataset is an environment of size 11 x 11 x 5 meters).\n\nThis has important ramifications for the claim that PRISM achieves a “localization accuracy comparable to current state of the art SLAM systems.”  It is well-known that filtering approaches (like PRISM) have difficulty correcting for significant drift (in both the map and state estimates) after closing large loops (basically because doing this precisely would entail propagating corrections backwards through previous state estimates that have already been marginalized out of the filter).  In contrast, full smoothing approaches preserve (and can therefore *correct*, via nonlinear optimization) a joint estimate for the map and *all* current and previous states, and so tend to be much better at closing large loops.\n\nGiven these considerations, it is not clear to me whether the experiments reported in Section 5 are actually sufficient to reveal significant differences in localization performance that *may in fact exist* between PRISM and current state of the art alternatives – in particular smoothing-based approaches like ORB-SLAM – as these significant differences may only become apparent when closing loops over much larger spatial scales.\n\nPlease see the following references for a more in-depth discussion of all of the above points:\n\n* Dellaert, Frank, and Michael Kaess. \"Factor graphs for robot perception.\" Foundations and Trends in Robotics 6.1-2 (2017): 1-139.\n\n* Kaess, Michael, and Frank Dellaert. \"Covariance recovery from a square root information matrix for data association.\" Robotics and autonomous systems 57.12 (2009): 1198-1210.\n\nFinally, while not (strictly speaking) a “weakness”, I think **the paper could be greatly strengthened by including a more pointed discussion of PRISM’s potential *impact***.  \n\nAs the paper itself makes clear, most of the features present in PRISM are afforded by alternative techniques: graphical smoothing methods are built on probabilistic models, provide posterior uncertainty estimates, and can easily incorporate 6DOF agent dynamics [simply by specifying the appropriate conditional distributions p(x_{t+1} | x_t, u_t)].  Moreover, multiple works already combine graphical smoothing approaches with (joint) estimation of dense geometry and appearance models (see for example the recent work on Kimera, which is also missing as a reference from the current draft).  In my view the differentiable rendering measurement model proposed in this work may actually be its most distinguishing feature, but it’s not immediately clear to me what the practical advantages of this would be versus e.g. the mesh-based models used in Kimera.\n\n\nI understand that one of the technical goals of this project was to produce a method that combines all of the aforementioned features into a single estimation procedure, but what is *not* obvious to me is: what is the actual *impact* of having done this?  That is: what *specifically* does (or might) PRISM enable me to do that I could not have achieved by some other means?  [For example, I could easily imagine combining ORB-SLAM for visual-inertial state estimation (with agent dynamics) with some other system for dense mapping to obtain a (more loosely-coupled but) seemingly largely equivalent system using off-the-shelf components.  Why should I prefer PRISM to this alternative?  In particular, is there some *specific* application PRISM enables that I would not be able to tackle with this (straw-man) alternative?]\n\n\n* Rosinol, Antoni, et al. \"Kimera: From SLAM to spatial perception with 3D dynamic scene graphs.\" The International Journal of Robotics Research 40.12-14 (2021): 1510-1546.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": " While the paper’s technical development is carefully executed and explained exceptionally clearly, it also mischaracterizes the capabilities of prior state of the art SLAM methods, and therefore the positioning of this work with respect to prior art.  It also lacks experimental evidence necessary to validate one of the central claimed contributions (that PRISM’s localization accuracy is comparable to existing state of the art SLAM methods).\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* The characterization of prior work on SLAM should be clarified, and the positioning of this work with respect to prior art should be updated accordingly.\n\n* The limitations section should explicitly address the well-known limitations of filtering-based SLAM methods versus full smoothing approaches, and their ramifications for circumstances under which PRISM would provide an acceptable solution (i.e. these limitations would not *necessarily* be a significant issue over small spatial scales, as the experiments in the current draft show).\n\n* To support a claim that PRISM “achieves localization accuracy comparable with state of the art SLAM solutions”, it would be necessary to experimentally evaluate PRISM’s localization performance after closing loops on a much larger spatial scale.  Ideally, the experimental results would be expanded along these lines, but failing that, the paper should discuss the limitations of the current small-scale test cases in measuring this (potential) discrepancy.\n"}}, {"review_id": "YZ4CJz6LUwL", "reviewer": "Reviewer_uCb5", "summary": "This paper proposes computationally tractable methods and approximations to allow for inference of generative 3D spatial models in real-time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper demonstrates a clear method for tractable inference of generative spatial models.  The design choices and approximations that are made in order to achieve real-time performance are well-motivated and provide interesting discussions on context with previous work.  Overall, the paper was an enjoyable read, and presents an interesting view on the current state of real-time world models.\n\nWeaknesses:\n\nThere is no quantitative evaluation of the uncertainties.  For example, there should be a comparison with the offline method [12], which is the most related baseline.  Section 5.2 and Figure 3 make sense intuitively, but do not demonstrate whether these are meaningful.\n\nThe parameters are not unified across all maps while for baselines this may be the case.  A fair comparison would be to use a single set of parameters across environments, or at least show this along with the current evaluation.\n\nFor pose estimation, the scales of Laplace also vary across datasets.  Why can we trust these units in general? Depth and RGB losses have very different units, and photometric error uncertainties do not necessarily convey meaningful uncertainty on pose depending on units.  \n\nAdditional comparisons with [12], or an equivalent offline version of the proposed work should be shown.  [12] learns agent dynamics.  Is the filter version still compatible?  Ultimately, is this method useful for learning models beyond just single-session agent pose and map states?\n\nWhat are the benefits of the generative model vs. traditional SLAM representations? For a dense TSDF map and a motion model, future states could be predicted in a similar manner, and the Laplace approximation could be used to estimate pose uncertainty, while the TSDF already has weights which are essentially map uncertainties.\n\nOther comments:\n\nWhat is currently limiting map accuracy? It would be good to perform comparisons with higher image resolution and higher map resolution even if no longer real-time so that one could see whether design choices or the approximation gaps are the limiting factor.\n\nAdditional description on the implication of marginal filters would be beneficial.  How is this not assuming independence?  What do we lose by not maintaining the joint distribution?  Why is the mean-field approximation worse when it explicitly represents the joint?\n\nWhy is pose optimization first-order? Could this not be done using a second-order optimizer (such as Gauss-Newton)?\n\nWhy are the Laplace approximations noisy and require smoothing? \n\nLast line of page 20 has a typo, “exponential moving avarge”.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is clear, well-motivated, and demonstrates interesting methods to achieve tractable inference of 3D spatial models.  In particular, the discussions on where approximations are made and how it draws on previous work is a valuable contribution.  As mentioned above, there are some additional comparisons and improvements that should be made to justify the method's claims, such as the uncertainty quality and comparisons to non-filtering inference methods.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "D5kWJ3afKAc", "reviewer": "Reviewer_6pXK", "summary": "This paper presents a filtering approach for real-time 3D mapping and localization with a dense geometric map representation. The approach leverages a simple and classical heuristic, which is to decouple the map and robot state updates entirely. This enables fast, closed-form updates. The system is demonstrated on several benchmark datasets and has competitive estimation accuracy with state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The split filter idea is an interesting one, and it is worth noting this is actually a *classical* idea in robot perception, with a rich history dating back to early SLAM literature. However, it also has some known issues (see Weaknesses).\n\n- The approach is _fast_, operating in real-time with realistic hardware considerations, while maintaining a dense geometric map representation, which is the primary motivation.\n\n- The experimental evaluation is fairly thorough.\n\n- The empirical results on some benchmark SLAM sequences are good and competitive with state-of-the-art approaches.\n\n# Weaknesses\n(Note I am combining here weaknesses with clarifications / issues)\n\n- Clarification: The map cells $\\mathcal{M}_{ijk}$ have a Gaussian distribution with diagonal covariance, but what is the data in each cell that we are assuming has Gaussian distribution? What is its dimensionality? This should be made precise.\n- The language throughout could be made significantly clearer and the language could be simplified. For example, in the abstract: \"We start from a predefined world model which combines differentiable rendering and 6-DoF dynamics\" - what is meant by \"world model\" here? Likewise, in the introduction, we have: \"Our motivation for obtaining the inference procedure as the inversion of a well-defined forward model is that their combination lends itself to model-based control, where the former serves as a state estimator and the latter as a simulator with which to plan ahead\" which I found difficult to parse (what is the \"former\" and \"latter\" referring to here?). I would suggest taking a full pass through the paper and asking for each sentence \"How can this be made simpler?\"\n- The marginal updates appear to be discarding significant uncertainty information. Essentially, the current presentation is directly combining \"mapping while assuming known poses\" and \"localization assuming a known map.\" The consequence of this is that in scenarios where significantly more robot state/map uncertainty has accumulated, measurements from uncertain states will have equal influence to those from certain states on the quality of the map (and vice versa).\n- Because of this issue with uncertainties, an interesting demonstration would be to expand the spatial and temporal scale of experiments (e.g. demonstrate on a larger-scale dataset with more \"exploration\"). Since this approach doesn't have an obvious \"global\" loop closure component, I could see this being challenging, and it would be interesting to examine at what point the performance of the estimator degrades.\n- Not a weakness, but in light of the above considerations, it may be that this method is best suited to persistent operation is a relatively well-constrained environment where uncertainty accumulation can be limited by the spatial extent of navigation. Alternatively, one could imagine (in a less compute-constrained setting) using this method as a sort of low-computational-cost \"odometry\" method in conjunction with a SLAM system and maintaining correlations between the reference frames for _submaps_ built using the proposed mapping method (e.g. in a pose-graph optimization type approach). This could help scale this approach to larger environments.\n- Minor comment: it seems common in learning and inference literature to use $z$ to denote hidden states of a Markov model (i.e. the robot poses in a SLAM context) and $x$ to denote measurements, which is done here. It is more common in robotics literature to use $z$ to denote *measurements* and $x$ to denote the robot poses. There is no ambiguity in the current presentation, as these terms are clearly defined, but adopting the latter notation convention may make this work easier to parse at a glance for robotics researchers.\n\n## Details regarding uncertainty\n\nIn the map update step (eq (2)-(3)), we have the approximation (dropping the $\\phi$ superscript and bold font):\n\n$p(\\mathcal{M} \\mid H_t) \\approx p(x_t \\mid \\hat{z_t}, \\mathcal{M}) \\times q_{t-1}(\\mathcal{M}) \\triangleq q_{t}(\\mathcal{M})$.\n\nSince $p(x_t \\mid \\hat{z_t}, \\mathcal{M})$ (likewise the inverse model $q(\\mathcal{M} \\mid x_t, \\hat{z_t})$ that is actually used) is based on a point estimate, none of the uncertainty from the belief in the state $z_t$ is reflected in the updated map distribution $q_{t}(\\mathcal{M})$. Following this reasoning recursively (until we arrive at the initial map believe $p(\\mathcal{M})$, it's easy to see that uncertainty in $z_{1:t}$ is not reflected in the map uncertainty. Crucially, this is _not_ true of the actual marginal filter. Because of this point estimate approach, the update (in the aligned equations leading up to eq (2)) is perhaps not best thought of as approximating the marginal, but rather as _conditioning_ on the state $\\hat{z_t}$. This means the mapping procedure effectively assumes \"known\" poses (where those poses are derived from the state filter). The state filter, however, does the same conditioning trick (conditioning on the current estimate of the map at time $t$); therefore it is likewise discarding uncertainty in the map. The practical motivation for this approach is understandable: the actual marginal filters for the states/map (or the joint filter) would require maintaining correlations between robot states and map elements, and with a dense map representation this is too computationally expensive for real-time operation. However, the practical _consequence_ of this for a SLAM system is that in longer-term mapping and localization scenarios (particularly during exploration, where drift is a significant issue), the quality of the map and localization gets worse and worse *and* it is difficult to correct later after e.g. a loop closure.\n\nIt's worth noting that early work on SLAM focused on \"decoupling\" landmark and robot state uncertainty for the same computational reasons observed here. \nSee Durrant-Whyte and Bailey's \"Simultaneous Localization and Mapping (SLAM) Part 1\" for a contemporaneous review. The following quote from their paper is relevant:\n\"Most importantly, it was recognized that the correlations between landmarks, that most researchers had tried to minimize, were actually the critical part of the problem and that, on the contrary, the more these correlations grew, the better the solution.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is good overall and I believe the proposed systems seems interesting and useful. The experimental demonstration is convincing. Some minor aspects of the presentation could be improved (see Issues), and in particular the scenarios under which the approximations are reasonable (and their effect on uncertainty estimates) should be clarified.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "See Weaknesses for more detailed comments above; main points reproduced below:\n\n- Clarification: The map cells $\\mathcal{M}_{ijk}$ have a Gaussian distribution with diagonal covariance, but what is the data in each cell that we are assuming has Gaussian distribution? What is its dimensionality? This should be made precise.\n- The language throughout could be made significantly clearer and the language could be simplified. For example, in the abstract: \"We start from a predefined world model which combines differentiable rendering and 6-DoF dynamics\" - what is meant by \"world model\" here? Likewise, in the introduction, we have: \"Our motivation for obtaining the inference procedure as the inversion of a well-defined forward model is that their combination lends itself to model-based control, where the former serves as a state estimator and the latter as a simulator with which to plan ahead\" which I found difficult to parse (what is the \"former\" and \"latter\" referring to here?). I would suggest taking a full pass through the paper and asking for each sentence \"How can this be made simpler?\"\n- The marginal updates appear to be discarding significant uncertainty information. Essentially, the current presentation is directly combining \"mapping while assuming known poses\" and \"localization assuming a known map.\" The consequence of this is that in scenarios where significantly more robot state/map uncertainty has accumulated, measurements from uncertain states will have equal influence to those from certain states on the quality of the map (and vice versa).\n- Because of this issue with uncertainties, an interesting demonstration would be to expand the spatial and temporal scale of experiments (e.g. demonstrate on a larger-scale dataset with more \"exploration\"). Since this approach doesn't have an obvious \"global\" loop closure component, I could see this being challenging, and it would be interesting to examine at what point the performance of the estimator degrades.\n- Not a weakness, but in light of the above considerations, it may be that this method is best suited to persistent operation is a relatively well-constrained environment where uncertainty accumulation can be limited by the spatial extent of navigation. Alternatively, one could imagine (in a less compute-constrained setting) using this method as a sort of low-computational-cost \"odometry\" method in conjunction with a SLAM system and maintaining correlations between the reference frames for _submaps_ built using the proposed mapping method (e.g. in a pose-graph optimization type approach). This could help scale this approach to larger environments.\n- Minor comment: it seems common in learning and inference literature to use $z$ to denote hidden states of a Markov model (i.e. the robot poses in a SLAM context) and $x$ to denote measurements, which is done here. It is more common in robotics literature to use $z$ to denote *measurements* and $x$ to denote the robot poses. There is no ambiguity in the current presentation, as these terms are clearly defined, but adopting the latter notation convention may make this work easier to parse at a glance for robotics researchers."}}]

Author rebuttal:

Decision: Accept (oral)