Learning to be Smooth: An End-to-End Differentiable Particle Smoother
Authors: Ali Younis
Erik B. Sudderth
Venue: NeurIPS 2024 poster (2024)
Keywords: particle
filter
smoother
non-parametric
generative
discriminative
multi-modal
belief propagation
monte carlo
sequential
kernel density estimation
mixture
importance sampling
localization
autonomous driving
smoothing
filtering
Primary area: probabilistic_methods
TL;DR: An end-to-end differentiable framework for training discriminative particle smoothing algorithms, with state-of-the-art performance at city-scale localization from imagery.

Abstract: For challenging state estimation problems arising in domains like vision and robotics, particle-based representations attractively enable temporal reasoning about multiple posterior modes.  Particle smoothers offer the potential for more accurate offline data analysis by propagating information both forward and backward in time, but have classically required human-engineered dynamics and observation models.  Extending recent advances in discriminative training of particle filters, we develop a framework for low-variance propagation of gradients across long time sequences when training particle smoothers.  Our "two-filter" smoother integrates particle streams that are propagated forward and backward in time, while incorporating stratification and importance weights in the resampling step to provide low-variance gradient estimates for neural network dynamics and observation models.  The resulting mixture density particle smoother is substantially more accurate than state-of-the-art particle filters, as well as search-based baselines, for city-scale global vehicle localization from real-world videos and maps.

Learning to be Smooth: An End-to-End Differentiable Particle Smoother
Ali Younis and Erik Sudderth Department of Computer Science, University of California, Irvine, CA 92617 ayounis@uci. edu, sudderth@uci.ees
Abstract
For challenging state estimation problems arising in domains like vision and robotics, particle-based representations attractively enable temporal reasoning about multiple posterior modes. Particle smoothers offer the potential for more ac- curate offline data analysis by propagating information both forward and backward in time, but have classically required human- human-engineen engineered dynamics and observation models. Extending recent advances in discriminative training of particle filters, we develop framework for low-variance propagation of gradients across long time sequences when training particle smoothers. Our "two-filter" smoother integrates particle streams that propagated forward and backward in time, while incor- porating stratification and importance weights in the resampling step to provide low-variance gradient estimates for neural network dynamics and observation mod- els. The resulting mixture density particle smoother is substantially more accurate than state-of oftte-of-the-rtt artt particle filters, well as sarch-based baselines, for city-scale global vehicle localization from real-world videos and maps.
Introduction
Global localization of the state of a moving vehicle using a city-scale map is challenging due the large area, as well as the inherent ambiguity in urban landscapes, where many street intersections and buildings appear similar. Recent work on global localization [1-11] has typically localized each time point independently during training, sometimes followed by temporal post-processing, often with demanding requirements like near-exact external estimation of relative vehicle poses [11].
For a broader range state estimation problems in fields like vision and robotics, a number of methods for end-to-end particle filter (PF) training have been proposed [8, 12-16]. Learnable PFs are suitable for global localization because they can represent multi-modal posterior densities, propagate uncertainty time, and learn models of real vehicle dynamics and complex sensors directly from data. However, most learnable PF methods have only been applied to simulated environments [12-14], with only a few preliminary applications real-world data [8, 15].
Particle filters [17-21] only use past observations to estimate the current state. For offline inference from complete time series, more powerful particle smoothing (PS) algorithms [22-27] may in principle perform better by integrating past and future data. But to our knowledge, recent advances in end-to-end differentiable training of PFs have not been generalized to the complex PS scenario, requiring error-prone human engincering of PS dynamics and observation models. Classical work on generative parameter estimation via PS [28] is limited parametric models with few parameters. In contrast, we develop differentiable PS that scale to complex models defined by deep neural networks.
After introducing differentiable particle filters (Sec. 2) and classical particle smoothers (Sec. 3), develop our differentiable, discriminative Mixture Density Particle Smoother (MDPS, see Fig. 1) in Sec. 4. Thorough experiments in Sec. 5 then highlight the advantages of our MDPS over differential PFs on a synthetic bearings-only tracking task, and also show substantial advantages over search-based and retrieval-baut based baselines for challenging real-world ccty- global localization problems.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Timn StApE
Obearvation Encodtar
Map
OoseraatonE:
Fores Se
Edtrector
Feature
Extractor
Maseturaraact
Modal
Pertitta-
Projection
ja Lasiioaioo
Compus Mege de
Figure I: Left: Our MDPS method showing the forward and backward particle filters, which are integrated (via learned neural networks, indicated trapezoids) produce smoothed mixture posterior. Right: Feature encoders and measurement model used for global localization. First-person camera views are encoded into Birds-Eye- Eye-View View (BEV) feature map by extracting features before applying a geometric projection [11]. Map features are extracted via feed-forward encoder, and un no-nommalized particle weights are computed as an inner product between BEV features and features of local map extracted from the global map at the particle location.
2 Differentiable Particle Filters
Particle filters iteratively estimate the posterior distribution p(ux) Vy.r, d1:e) over the state at discrete time t given a sequence of observations Y/ and, optionally, actions dz PFs use a collection of N samples, or particles, It
(c) = {x; (1)
..... It. (N)) with weights w/ (wi (1) ..... W;
(N) to flexibly
capture multiple posterior modes nonparametrically. Classic PFs are derived from Markov generative model, leading to an intuitive recursive algorithm that alternates between proposing new particle
locations and updating particle weights. End-to-end training requires gradients for each PF step.
Partiele Proposals. At each iteration, new particles xt are proposed by applying a model of the state transition dynamics individually each particle 2²!²) p(z!") (8) |r|" (2
at), conditioned actions a, if available. Of note, only simulation of the dynamics model is required; explicit density evaluation is unnecessary. Using reparameterization [29-31], the dynamics model can be defined feed-forward
neural network f-) that transforms random (Gaussian) noise to produce new particles:
(i)
f(( (.))
(9) (i)
Mi
(),
Measurement Updates and Discriminative Training. Proposed particles are importance-weighted by the likelihood function, w x (x())= incorporate the latest observation yr into the
posterior. The updated weights are then normalized such that Σ w!"" 1. However, for complex observations like images or LiDAR, learning accurate generative models p(y((x:) is extremely challenging. Recent work [8, 12-15] has instead learned discriminative PFs parameterized by differentiable (typically, deep neural network) measurement models:
w (6)
((x)))) 1.) (9)
(1)
Here, [(x;; Mr) scores particles to minimize loss, such as negative-log lokeliiel or squared-error, the prediction of true target states H, that are observed during training.
2.1 Particle Resampling
The stochastic nature of PF dynamics causes some particles to drift towards states with low pos- terior probability. These low-weight particles do not usefully track the true system state, wasting computational resources and reducing the expressiveness of the overall approximate posterior.
Particle resampling offers a remedy by drawing a new uniformly weighted particle set a.) from (+) with each particle duplicated (or not) proportional to its current weight w( The simplest
It

---

multinomial resampling strategy [23, 32, 33] chooses particles independently with replacement:
x (1)
j j~ Cat(wul ...., wi 1
To maintain an unbiased posterior, resampled particles have weight vi!" N. Multinomial resam- pling may be implemented [32] by drawing a continuous Unif(O, 1) variable for each particle, and transforming these draws by the inverse cumulative distribution function (CDF) of particle weights.
Stratified resampling [23, 32, 33] reduces the variance of conventional multinomial resampling, by first partitioning the interval (0, 1] into N sub-intervals (0, & ul. n. 1]. One uniform variable then sampled within each sub-interval, before transforming these draws by the inverse CDF of particle weights. Our differentiable PS incorporate stratified resampling to reduce variance with negligible computational overhead, making training more robustly avoid local optima (see Fig. 2).
While other methods like residual resampling [34, 32, 35] have been proposed in the PF literature, this partially-deeeministic approach is less robust than stratified resampling in our experiments (see Fig. 2), and also much slower because residual resampling cannot be parallelized across particles.
For mixture density PS, particles resampled from continuous Gaussian mixture, in which all components share a common standard deviation 3. This resampling can equivalently be expressed as d #4 (i) + 3T) (d) where 1/2 (6) (O. I) and (9) is generated via discrete sampling of the mixture component means. We incorporate stratified resampling this step to boost performance.
2.2 Differentiable Approximations of Discrete Resampling
For discriminative PFs to effectively learn to propagate state estimates over time, gradients are needed for all steps of the PF. While differentiable dynamics and measurement models are easily constructed via standard neural-network architectures, discrete particle resampling is not differentiable.
Fruncated-Gradiee Particle Filters (TG-PF) [15]. the first so-called "differentiable" particle filter, actually treated the resampling step non-differentiable and simply truncated gradients zero resampling, preventing buck-propagation through time (BPTT) [36]. Due to this weakness, dynamics models were assumed known rather than learned, and measurement models were learned from biased gradients that fail to propagate information over time, reducing accuracy [14].
Soft Resampling Particle Filters (SR-PF) [13] utilize differentiable resampling procedure that sets particle resampling weights to be a mixture of the true weights and a discrete uniform distribution:
( st
j
Cat(u)
(1)
(N)),
(+) (1
A)w! (
N
Gradients are then propagated via the resampled particle weights defined as:
w
Vaul = V
w
(5)
(1 A) A/N
(1
A)w!
+ A/N
This simple approach resamples low-weight particles more frequently, degrading performance. The gradients of Eq. (5) also have substantial bias, because they incorrectly assume model perturbations only influence the particle weights in (5), and not the discrete particle resampling in (4).
Relaxations of Discrete Resampling, While discrete particle resampling could potentially be replaced by continuous particle interpolation with samples from Gunbbei-Sottma Concrete distribution [37, 38). no work has successfully applied such relaxations to PFs, and experiments in Younis and Sudderth [14] show very poor performance for this baseline. Alternatively, entropy- regularized optimal transport particle filters (OT-PF) [12] replace discrete resampling with an entropy-regularizes optimal transport problem, that minimizes Wasserstein metric to determine probabilistic mapping between the weighted pre-resampling particles and uniformly weighted restplesa particle. OT-PF performance sensitive to a non-learned entropy regularization hyperparameter, and the biased gradients induced by this regularization may substantially reduce performance [14]. Furthermore, "fast" Sinkhorn algorithms [39] for entropy-regularized OT still scale quadratically with the number of particles, and in practice are orders of magnitude slower than competing resampling strategies. This makes OT-PF training prohibitively slow on the challenging city-scale localization tasks considered in this paper, do not compare to it.
3

---

TD-F
AEDIT
POTH
FTBS
Figure 2: Box plots showing median (red line). quartiles (blue box), and range (whiskers) 11 training runs for Bearings-One Only tracking (Sec. 5.1). We boost the robustness the performing MDPF [14], which previously used multinomial resampling, by incorporating variance-redie reducedd stratified resampling; residual resampling is both slower and less effective. Stratified resampling provides larger advantages for the less- sophisticated TG-PF [15] and SR-PF [13] gradient estimators, but these baselines remain inferior to MDPF. Our MDPS substantially improves all PFs by incorporating both past and future observations when computing posteriors. Classic FFBS particle smoothers [24, 25] poor performance, even when provided the true likelihoods (rather than a learned approximation), showing the effectiveness of our end-to-end learning of particle proposals and weights, Forward PFs are initialized with noisy samples of the true state, while HOPF-Backkard (the backwards-ting PF component MDPS) is initialized by sampling uniformly from the state space.
2.3 Mixture Density Particle Filters
Mixture Density Particle Filters (MDPF) [14] are a differentiable variant of regularized PFs [40, 41]. MDPF estimates a continuous kernel state density [42] by convolving particles with a continuous, and differentiable, kernel function K (such as a Gaussian) with bandwidth hyperparameter B::
N
m(x;
(1) W2
.) =
Σ wi K(x, 21 (1)
(6)
Particles are then resampled a m(xx of via discrete resampling. Unbiased and low-variance Importance Weighted Sample Gradient (IWSG) [14] estimates may then be constructed by viewing the particle proposal g(z) =m(z Φ))
.3 from this continuous mixture instead
be fixed to the mixture model parameters Φo {z(²) ( [} the current training iteration. Gradients then account for parameter perturbations not by perturbing particle locations as in standard reparameterization [29-31], J, but by perturbing particle importance weights away from uniform:
(b)) m22 ④)) @0
V (1) Vom(2(0) ()) @))o==oo
m(((() @b)
m(2() Φ0)
With this approach, the bandwidth parameter 3 may also be tuned for end-to-end prediction of state distributions, avoiding the need for classic handwidth-selectin heuristics [42-44]. An "adaptive" variant of MDPF [14] incorporates two bandwidths, one for particle resampling (to propagate information over time) and a second for estimation of state posteriors (to compute the loss). Our MDPS also incorporates separate bandwidths for resampling and state estimation, detailed below.
From Filtering to Smoothing
Particle smoothers extend PFs estimate the state posteriors (((r))1irr) given full T-step sequence of observations. (To simplify equations, we do not explicitly condition on actions Q1.7 in the following two sections.) Particle smoothers continue approximate posteriors via collection of particles (I:N) with associated weights (1:N) where we use bi-directional overhead arrows to denote smoothed posteriors. Classical particle smoothing algorithms, which are non-differentiable and typically assume human-engineered dynamics and likelihoods, fall into two broad categories.
Backward Smoothing (FFBS) algorithms [24, 25] compute p(z, 181.r) by
factoring into forward filtering and backward smoothing components:
p(z ((r.)) f p( )
pl p(
(p IT.. dr
p(Iz|B1:d)
(8)
p Ire (p(p 1111e)
forward filtering
backward smouthing
natural algorithm emerges from Eq. (8), where conventional PF first approximates p(z; y1s)) for
all times via particles (1:N) with weights () A backward smoother then recursively reweights

---

the "forward" particles to account for future data, but does not change particle locations:
p P
w x wi
1+1
(9)
k= p
Because FFBS sets 20 2 it is only effective when filtered state posteriors p(z//11d substantially overlap with smoothed posteriors p(zr/yi:T) [25]; performance deteriorates when future data is highly informative. FFBS also requires explicit evaluation, not just simulation, of the state transition dynamics p( 12)), which is not tractable for dynamics parameterized as in Eq. (1).
Two Filter Smoothing (TFS) algorithms [22, 24, 25] instead express the smoothed posterior as normalized product of distinct forward-time and backward-time filters:
p( (1rr) p W1: ((ta))(((++arrr)
p(z, ((aaly)a))(((e11:) (m)).
(10)
p
Here p( may be approximated by standard PF, and p(U(+1:2/24) is the so-called backward information filter [25, 24] defined as
p(uer(())) - p(ye+ ur (p((++ (20 (
(11)
Because p(yerrlat) is likelihood function rather than probability density in æy, and it possible that P(BET Jay )dz, = oo. This is not an issue when ((Verraa (xz) is computed analytically as in Kalman smoothers for Gaussian models [45], but particle- base methods can only hope to approximate finite measures. Bresler [22] addresses this issue via an auxiliary probability measure 2t (zz):
P(yer (xt) x p(ZIlVeT) (x(x))
T
p(Zur Mer) x (=(x))(())
II
(12)
am t-1
From Eqs. (10,12), the smoothed posterior is reweighted product of forward and backward filters:
forward fitering backward filtering
p(x+(11rr)) pa W1a) p(3eele411r))
(13)
(xx))
This suggest an algorithm where two PFs are run on the sequence independently, one forward and one backward in time, to compute forward particles {>(1:N) /t (I:)) and backward particles ( (1:0)), Because continuously sampled forward and backward particle sets will not exactly align, classic TFS integrate these two filters by rewriting Eq. (13) follows:
p(ya (z ((xx///(41.)) ((xa((t-1)((1; 1affee--11---- dx
(14)
(x:)
This yields a particle re- re-weighting approach where backward filter particles (1:.)) using the forward filter particle set, to produce the final smoothed particle weights (>(1:N),
(1:N) are re-weighted
N
x
W
Σ
( =1
(15)
7e
1=1
which similar to FFBS, makes performance heavily dependant on significant overlap support between p(z, (1 and p( Like FFBS, TFS also restrictively requires evaluation (not just simulation) of the state transition dynamics.
Conventional TFS set (1:N)
+ (1:N)
4 Mixture Density Particle Smoothers
Our novel Mixture Density Particle Smoother (MDPS, Fig. 1) can be seen differentiable TFS, where the forward and backward filters of Eq. (13) are defined MDPFs (Sec. 2.3). Using discrim- inative differentiable particle filters (MDPFs) within the TFS frameworks, and replacing Eq. (14) with an importance-weighted integration of forward and backward particles, enables an effective and end-to-end differentiable particle smoother. We begin by rewriting Eq. (13)
p( (1rr)) x p(ye (2, (x W )p(x (e+1:T)
(16)
(x))
5

---

Edtiovad cacer
Recall
Tep Nada:
buand cedinn sho clllee
"Cheuling"
Posttion Emor [m]
Angle Error C7
Doss Brank Hoael
Figure 3: Position and error recall using the MGL [1L] dataset. Recall is computed with the top posterior mode
well as with the best of the top-3 posterior modes, extracted via non-maximal suppression. As expected, Retrieval [3] methods do poorly due to their lack of discrimination power between neighboring map patches. Dense scarch [1 does better by using fine map details during localization, but it requires a ground truth hint ("Cheating" with GT, which artificially improves performance) to work well at city-scale environments. Retrieval (PF) [8] uses unleamed state dynamics, which proves useful, but still suffers from the poor discriminative ability of retrieval. In contrast, MDPF [14] uses end-lo-end learned dynamics and measurement models, allowing for good performance but suffering from only using past information when estimating posterior densities. Our MDPS is able to learn similar strong dynamics and measurement models as MDPF, and also incorporates future well past information to achieve
accurate posterior density and thus higher recall.
where the forward and backward filters longer condition the current observation. This allows for functionally identical MDPFs to be used for both directions, simplifying implementation. MDPFs parameterize state posteriors continuous kernel density mixtures:
N
p(=xly)=-1) = Σ wi -(a- - 21; 3),
p(ZalUeTET) = Σ
( K( 10 (3) (17)
i=1
f=1
Unlike discrete probability measures, these continuous mixture distributions be combined via direct multiplication to give a smoothed posterior mixture containing N2 components, one for each pair of forward and backward particles. For this product integration of forward and backward filters, the normalizing constants for all pairs of kernels must be explicitly computed to correctly account for the degree which hypotheses represented by forward and backward partieles normalization constants are tractable for some simple kernels including Gaussians [46], but more complex for the other cases such as von Mises kernels of orientation angles [47, 48].
consistent, These
Direct mixture multiplication climinates the need to evaluate the dynamics model, as in classic TFS, but introduces significant overhead due to the quadratic scaling of the number of mixture components. To address this issue, our MDPS uses importance sampling where the smoothed posterior is defined by M 2 particles drawn from mixture of the filter posteriors:
((xa)) = p p (e41))))))
1 ....... .M.
(18)
By construction, this proposal will include regions of the state space that lie within the support of either p(xx(11:t-1) or p(z:)
p(z:)Bt-1::)) (8t-1:)), improving robustness. Our experiments set M 2N, drawing N particles from each of the filtered and smoothed posteriors. Given true dynamics and likelihood models, importance sampling may correct for the fact that smoothed particles are drawn from mixture rather than a product of filtered densities, as well as incorporate the local observation:
plyel 3 )p( (0 11. ( (1+11T)
M
(19)
The (T))
To more easily train a discriminative PS, rather than estimating each term in Eq. (19) separately, we directly parameterize their product via feed-forwan neural network I(-):
1( (Z (1)
Ue. p( VI: 1).-1), (++1:))
w(() (i)
1.
(20)
(7
The posterior weight network Z(.) scores particles based on agreement with yr, as well as consistency with the forward and backward filters, and implicitly accounts for the auxiliary distribution (-))). To allow state prediction and compute the training loss, the smoothed posterior is approximated as:
N
p(zelbirr) m(x.1
3
v 3),
(21)
where 3 is learned, dimension-specific bandwidth parameter.

---

Thee Thestop Thmesee
Threstmy
Tacatop
The shep
Theesip
POIT werr--taacttereee MOve
Trun State
Top 3 Modes
Figure 4: Example trajectories from the MGL dataset with observations shown in the top row. We show the current true state and state history (black arrow and black line), the estimated posterior density of the current state (red cloud, with darker being higher probability) and the top 3 extracted modes (blue arrows) for the MDPS as well its forward and backward MDPFs. Due ambiguity at early time-steps, MDPF [14] is unable resolve the correct intersection, and instead places probability mass multiple intersections. By fusing both forward and backward filters, our MDPS resolves this ambiguity with probability mass focused on the correct intersection. Furthermore, MDPS provides tighter posterior density than either MDPF-Forward or MDPF-Backwand
Training Loss and Gradient Computation. We discriminatively train our MDPS by minimizing the negative log-likelihood of the true state sequence:
C = - Σ log log(m(a)
T (1) 8 F)).
(22)
tET
During training, the IWSG estimator of Eq. (7) provides unbiased estimates of the gradients of the forward and backward resampling steps. We may similarly estimate gradients of the mixture resampling (18) that produces smoothed particles, enabling the first end-to-end differentiable PS:
V 7 & Voll Br. p( (()0) Via (ia-1),P2 P( (B++1)))
(23)
Training Details. Because the smoother weights of Eq. (20) cannot be effectively learned when filter parameters are random, train MDPS via three-stage procedure. In stage 1, the forward and backward PFs are trained separately (sharing only parameters for the encoders, see Fig. 1) to individually predict the state. In stage 2, the PFs frozen and the particle smoother measurement model I(-) of Eq. (20) trained. In stage 3, all models are unfrozen and trained jointly to minimize the loss in the MDPS output state posterior predictions of the true states. The forward MDPF,
backward MDPF, and MDPS posterior each have separate kernel bandwidths (3. 3. 3 that are jointly learned with the dynamics and measurement models. We randomly resample stochastic subset of the training sequences for each step, and adapt learning rates via the Adam [49] optimizer.
Computational Requirements. At training time, to allow unbiased gradient propagation, MDPS computes importance weights for each particle during resampling. For particles and T time-steps, this requires O2T operations. At inference time, importance weighting is not needed as the particle weights can simply be set as uniform, and resampling only requires (T operations. All phases of our MDPS scale linearly with N test time, in contrast with other differentiable relaxations such as OT-PF [12], which requires (TN22) operations for both training and inference.
5 Experiments
We evaluate our MDPS on synthetic bearings-only tracking task [14], as well as on real-world city-scale global localization. For all tasks, we estimate the MDPF/MDPS posterior distributions of a 3D (translation and angle) state (a, y, 0), using Gaussian kernels for the position dimensions, and von Mises kernels for the angular dimensions of the state posterior mixtures.

---

True State at Time
True State at Time t+1
Figure 5: Learned dynamics from the forward filter of MDPS trained the MGL dataset. Density cloud illustrates density of particles after applying dynamics while marginalizing actions. MDPS clearly learns informative, non-lincar dynamics models which aid in state posterior estimation.
5.1 Bearings Only Tracking Task
To allow comparison to prior discriminative PFs, the same bearings-oni tracking task as [14], where the 3D state of a variable- -velocity synthetic vehicle is tracked via noisy bearings from a fixed- position radar. 85% of observations are the true bearing plus von Mises noise, while 15% uniform outliers. Train and evaluation sequences have length find truncated BPTT [50] is not necessary if bandwidths are initialized appropriately. Filtering particles are initialized as the true state with added Gaussian noise, while MDPF-Backwar (and the MDPS backwards filter) are initialized with uniformly sampled particles to mimic datasets where often only the starting state is known. More details can be found in the Appendix.
50. Unlike Younis and Sudderth [14],
We compare our MDPS methods to several existing differentiable particle filter baselines, but no differentiable particle smoother baseline exists. Instead, we implement the classic FFBS [24, 25] algorithm (Sec. 3), which assumes known dynamics and measurement models. Since FFBS is not differentiable, we learn the dynamics model using the dataset true state pairs {zr-1, 2z} outside the FFBS algorithm. In order simulate from and evaluate the state transition dynamics, as needed by the FFBS, we parameterize the dynamics model to output a mean and use a fixed bandwidth parameter (tuned on validation data) to propose new particles. We also use the true observation likelihood the measurement model, instead of a learned approximation; this boosts FFBS performance.
Results. In Fig. 2 we show statistics of performance over 11 training and evaluation runs for each method. We compare to TG-PF [15], SR-PF [13], the classical FFBS [24, 25], and MDPF [14]. Interestingly, MDPF outperforms SR-PF and TG-PF even when the initial particle set is drawn uniformly from the state space as in MDPF-Backwarr
By incorporating more temporal data, MDPS substantially outperforms MDPF. Even when unfairly provided the true observation likelihood, FFBS performs poorly since particles are simply re-weighted (not moved) by the backward smoother. This inflexibility, and lack of end-to-end learning, makes FFBS less robust inaccuracies the forward particle filter.
We are the first to compare resampling variants in the context of modern discriminative PFs. Stratified resampling substantially improves TG-PF and SR-PF performance, but only modestly improves the worst- porss-performing MDPF runs. This may be because even with basic multinomial resampling, the voriinear MDPF gradients dramatically outperform all TG-PF and SR-PF variants. Residual resampling performs worse than stratified resampling, and is also much slower since it cannot be easily parallelized on GPUs, so do not consider it for other datasets.
5.2 City Scale Global Localization Task
Our global localization task is adopted from Sarlin et al. [11], where we wish to estimate the 3D state (position and heading) of a subject (person/bike/vehiile) as moves through real-world city-scale first-person images, actions are noisy odometry, and a 2D planimetric map is provided to help localize globally. We use the Mapillary Geo-Localization [11] and KITTI [51] datasets compare MDPS method to MDPF [14] well as other methods specifically designed for the global localization task, which are not sequential Monte Carlo methods.
environments. Observations are
Our global localization task is distinct from local localization systems, which aim to track subject positions relative to their starting position, instead of in relation to the global map origin. Visual SLAM systems [21] almost exclusively solve the local localization task, using the starting position as the origin of their estimated map. a map is provided, then just the localization part of Visual SLAM can be run, but detailed visual or 3D maps of the environment are needed. These have prohibitive memory requirements at city-scales, and need constant updating as the visual appearance of the environment changes (c.g., with the weather/seasons) [11]. Hybrid place recognition with localization
8

---

also requires detailed visual or 3D maps [52]. In our experiments, we instead seek to use planimetric maps for global localization, which are compact and robust to environment changes. It is not obvious how to apply SI SLAM//ybrrid place recognition systems to this type of map.
Retrieval methods [1-7] rely on latent vector similarity where map patches and the observation encoded into a common latent state space before doing a vector similarity search. These methods are trained using a contrastive loss [2, 53] that forces the observation encoding to be similar to its corresponding map patch encoding, while being dissimilar to other patch and observation encodings. Accuracy depends on map patch extraction density and patch similarity; similar patches are mapped to near-identical encodings, performance suffers. Zhou et al. [8] extend this framework using non- differentiable PF, where a retrieval- based measurement model is trained outside the PF framework, and non-lcarned dynamics are fixed to actions with added Gaussian noise.
Refinement methods [9, 10] refine initial position estimate via expensive optimization, by maximizing the alignment of features extracted from the observation and map. Due to the extreme non-linearity of this objective, refinement methods require an accurate initial estimate converge to the correct solution, if at all. This prevents their use for city-scale global localization.
Dense Search [1 extracts birds-eye-view (BEV) features from the observed images via geometric projection (see Fig. 1), before applying a dense search to align BEV features with extracted map features. Heuristic alignment probabilities may be produced by tracking alignment values during search, and applying a softmax operator. Higher discretization density boosts accuracy, but requires significantly more memory and compute. Temporal information can be used by warping probability volumes onto the current time-step, but this requires near-exact relative poses which Sarlin et al. [11] determine via expensive, black-box Visual-Inertial SLAM system [21].
5.3 Mapillary Geo-Localization (MGL) Dataset
In the Mapillary Geo-Localization (MGL) [11] dataset, images sequences are captured from handheld or vehicle-mounted cameras as a subject (person/bike/car) roams around various European and U.S. cities. Observations are set as 90º Field-of-Viee images in various viewing directions, with actions being noisy odometry. A planimetric map of the environment is also provided via the OpenStreetMap platform [54] at 0.5 meter/pixel resolution. We generate custom training, validation, and test splits to create longer sequences with T = 100 steps. For particle-based methods, we use stratified resampling and set the initial particle sets to be the true state with added noise. More details are in the Appendix.
Implementation Details. For MDPF and our MDPS, we set the dynamics model to multi-layer perceptron (MLP) network. The measurement model incorporates BEV features [11] and map features as illustrated in Fig. 1. The smoother measurement model incorporates additional inputs via an extra MLP. Memory constraints prevent Dense Search in city-scale environments, so we consider two methods to limit the search space. A sliding window limits the search space to 256m x 256m area that is recursively re-centered around the best position estimate at t - 1. propagated to f using a, We can also limit the search space to a 256m x 256m area containing the true state, though this artificially increases performance. We similarly limit the search space for Retrieval, which performs poorly in large environments; Zhou et al. [8] even limit the vector search to known road networks.
Results. We compare our MDPS to MDPF [14], Dense Search [11], Retrieval [3] implemented as detailed by [11], and Retrieval (PF) [8], reporting the position and rotation recall in Fig. 3. Due to ambiguity in large city environments (e.g., intersections can look very similar), estimated state posteriors can be multi-modal (see Fig. 4), and thus simply reporting accuracy using the highest probability mode does not fully characterize performance. We thus also extract the top-three modes using non-maximal suppression (see Appendix), and report accuracy of the best mode. Interestingly, MDPF and MDPS give dramatic improvements over baselines engineered specifically for this task, highlighting the usefulness of end-to-end training. MDPS outperforms MDPF by using the full sequence of data to reduce mode variance, and discard incorrect modes as illustrated in Fig. 4.
Informative dynamics models boost performance, as demonstrated by the MDPS and MDPF results in Fig. 3. We visualize the learned dynamics of the MDPS forward filter in Fig. 5. Good dynamics models keep particles densely concentrated in high-probability regions, while also including diversity to account for sometimes-noisy actions. This enables learning of more discriminative measurement models, since training encourages the weights model to disambiguate nearby particles.

---

Modia
Figure 6: Left: Lateral and longitudinal errors are in the vehicle frame of reference. Right: Position recall versus error for the KITTI [51] dataset. Recall is computed with the top posterior mode as well as with the best of the top-3 posterior modes. Longitudinal localization performance is poor for all methods due to lack of visual features. MDPF [14] offers dramatic improvements for lateral error
Retrieval [3], Retrieval (PF) [8] and Dense Search [11] baselines, even when these baselines are constrained to operate around the ground truth state ("Cheating" with GT). For longitudinal recall, methods using "Cheating" with GT have good performance because they are artificially constrained to be near the true state, and thus have significantly less position ambiguity along the roadway. MDPS offers further improvements over MDPF it maintains more diverse set of posterior modes, instead of prematurely collapsing to incorrect modes.
Computational Requirements. While MDPS is more accurate than other methods, it is also more efficient than dense search. Because dense search must try many options to find the best alignment of the BEV and extracted map features, it has complexity O (KK22H3) for K search rotations, and search locations defined on a grid of width W and height H. (For notational simplicity, we assume the BEV features and the map features have the same width and height.) This complexity can be reduced to O(K H log(W H)) by using the Fast Fourier Transform. In contrast, MDPS has complexity O(NW H), where N K log( W ), it only compares the BEV and map features at the particle locations. This allows MDPS to better scale to large- scale environments.
5.4 KITTI Dataset
We also evaluate MDPS method for the global localization task using the KITTI [51] dataset, where observations are forward-facing camera images from moving vehicle. We augment this datatset with noisy odometry computed from the ground truth states and use the default Train, Testl, and Test2 splits for training, validation, and evaluation respectively. Like MGL, a planimetric map of the environment is provided via the OpenStreetMap platform [54] 0.5 meter/pixel resolution. Due to the small size of the KITTI dataset, we tre-train all methods using MGL before refining on KITTI, using the same network architectures as was used for the MGL dataset. See Appendix for details.
Results. Due to the aeward-facing camera, the observation images lack visual features for useful localizing along the roadway, therefore we decouple the position error into lateral and longitudinal errors when reporting recalls in Fig. 6. Understandably, all methods have larger longitudinal error than lateral error. Interestingly, MDPF and MDPS offer similar Top 3 mode performance for small lateral errors (under meters) while significantly outperforming all other methods. When the lateral error is greater than 2 meters, MDPS sees performance gain as it maintains a more diverse set of posterior modes, whereas MDPF prematurely collapses the posterior density to incorrect modes.
5.5 Limitations
Like all particle-based methods, MDPS suffers from the curse of dimensionality [55] where particle sparsity increases as the dimension the state space increases, reducing expressiveness of state posteriors. More effective use of particles via smarter dynamics and measurement models, as enabled by end-to-end MDPS training, can reduce but not eliminate these challenges.
6
Discussion
We have developed a dilly-differentiable, two-filter particle smoother (MDPS) that robustly learns more accurate models than classic particle smoothers, whose components are often determined via heuristics. MDPS successfully incorporates temporal information from the past as well as the future to produce more accurate state posteriors than state-of-the-an differentiable particle filters. When applied to city-scale global localization from real imagery, our learned MDPS dramatically improves on search and retrieval-basee baselines that were specifically engineered for the localization task.
10

---

Acknowledgments
This research supported in part by NSF Robust Intelligence Award No. IIS-1816365 and ONR Award No. N00014-23-1-212..
References
[1] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where looking at? joint location and orientation estimation by cross-view matching. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[2] Sixing Hu, Mengdan Feng, Rang M. H. Nguyen, and Gim Hee Lee. Cym-net: Cross-view matching network for image-basee based ground-to-aerio geo-localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[3] Noe Samano, Mengjie Zhou, and Andrew Calway. You Are Here: Geolocation by Embedding Maps and Images. In European Conference on Computer Vision (ECCV), 2020.
[4] Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross- view image geo-localization beyond one-to-one retrieval. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3640-3649, 2021.
[5] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for image based cross-view geo-localization. In Advances Neural Information Processing Systems (Neurips), 2019.
[6] Yujiao Shi, Xin Yu, Liu Liu, Tong Zhang, and Hongdong Li. Optimal feature transport for cross-view image geo-localization. In Association for the Advancement of Artificial Intelligence (AAAT), 2020.
[7] Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. In European Conference on Computer Vision (ECCV), 2022.
[8] Mengjie Zhou, Xieyuanli Chen, Noe Samano, Cyrill Stachniss, and Andrew Calway. Efficient localisation using images and openstreetmaps. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
[9] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] Paul-Edouard Sarlin, Ajaykumar Unagar, Mâns Larsson, Hugo Germain, Carl Toft, Victor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[11] Paul-Edouard Sarlin, Daniel DeTone, Tsun- Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulo, Richard Newcombe, Peter Kontschieder, and Vasileios Bal- ntas, OrienterNet: Visual Localization in 2D Public Maps with Neural Matching. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[12] Adrien Corenflos, James Thortoon, George Deligiannidis, and Arnad Doucet. Differentiable particle filtering via entropy-regularizer optimal transport. International Conference on Machine Learning (ICML), 2021.
[13] Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter networks with application to visual localization. Conference Robot Learning (CORL), 2018.
[14] Ali Younis and Erik Sudderth. Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes. Neural Information Processing Systems (NeurIPS), 2023.

---

[15] Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End- to-end learning with algorithmic priors. Proceedings of Robotics: Science and Systems (RSS), 2018.
[16] Adam Scibior, Vaden Masrani, and Frank Wood. Differentiable particle filtering without modi- rying the forward pass. International Conference on Probabilistic Programming (PROBPROG), 2021.
[17] Neil J Gordon, David Salmond, and Adrian FM Smith. Novel approach to nonlinear/non- gaussian bayesian state estimation. IEE proceedings F-Radar and Signal Processing, 1993.
[18] K. Kanazawa, D. Koller, and S. Russell. Stochastic simulation algorithms for dynamic proba- bilistic networks. In UAI 11, pages 346-351. Morgan Kaufmann, 1995.
[19] A. Doucet. N. de Freitas, and N. Gordon, editors. Sequential Monte Carlo Methods in Practice. Opringer-Verlag, New York, 2001.
[20] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Proc., 10(2):774-188, February 2002.
[21] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, 2005. ISBN 0262201623.
[22] Yoram Bresler. Two-filter formulae for discrete-time non-linear bayesian smoothing. Interna- tional Journal of Control, 1986.
[23] Genshiro Kitagawa. Monte carlo filter and smoother for non gaussian nonlinear state space models, Journal of Computational and Graphical Statistics, 1996.
[24] Arnaud Doucet, Adam M Johansen, et al. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering. 2009.
[25] Mike Klaas, Mark Briers, Nando de Freitas, A. Doucet, Simon Maskell, and Dustin Lang. Fast particle smoothing: if had a million particles. International Conference on Machine Learning (ICML), 2006.
[26] Mark Briers, Arnau Doucet, and Simon Maskell. Smoothing algorithms for state-space models. Annals of the Institute of Statistical Mathematics, 2010.
[27] Herbert E Rauch, F Tung, and Charlotte T Striebel. Maximum likelihood estimates of linear dynamic systems. American Institute of Aeronautics and Astronautics journal, 1965.
[28] Nikolas Kantas, Arnau Doucet, Sumeetpal S Singh, Jan Maciejowski, and Nicolas Chopin. On particle methods for parameter estimation in state-space models. Statistical Science, 2015.
[29] Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In International Conference on Machine Learning (ICML), 2014.
[30] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. International Confer- ence on Learning Representations (ICLR), 2014.
[31] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), 2014.
[32] Randal Douc and Olivier Cappé. Comparison of resampling schemes for particle filtering. In ISPA 2005. Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, 2005., 2005.
[33] Tiancheng Li, Miodrag Bolic, and Petar M. Djuric. Resampling methods for particle filtering: Classification, implementation, and strategies. IEEE Signal Processing Magazine, 2015.
[34] Jun Liu and Rong Chen. Sequential monte carlo methods for dynamic systems. Journal of the American statistical association, 1998.
12

---

[35] Darrell Whitley. A genetic algorithm tutorial. Statistics and Computing, 1994.
[36] Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 1990.
[37] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztioo with gumbel-softmax. International Conference on Learning Representations (ICLR), 2016.
[38] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. International Conference on Learning Representations
(ICLR), 2016.
[39] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 2013.
[40] N. Oudjane and C. Musso. Progressive correction for regularized particle filters. International Conference on Information Fusion, 2000.
[41] C. Musso, N. Oudjane, and F. Le Gland. Improving regularized particle filters. In Sequential Monte Carlo Methods in Practice, 2001.
[42] B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, 1986.
[43] M Chris Jones, James S Marron, and Simon J Sheather. A brief survey of bandwidth selection for density estimation. Journal of the American Statistical Association, 1996.
[44] Adrian W Bowman. An alternative method of cross-validation for the smoothing of density estimates. Biometrika, 1984.
[45] B.D.O. Anderson and J.B. Moore. Optimal Filtering. Prentice-Hall, 1979.
[46] Alexander Ihler, Erik Sudderth, William Freeman, and Alan Willsky. Efficient multiscale sampling from products of gaussian mixtures. In Advances in Neural Information Processing Systems, 2003.
[47] K. V. Mardia and P. J. Zemroch. Algorithm 86: The von mises distribution function. Journal of the Royal Statistical Sociery. Series C (Applied Statistics), 1975.
[48] Geoffrey W. Hill. Algorithm 518: Incomplete bessel function i0. the von mises distribution [s14]. ACM Trans. Math. Softw., 1977.
[49] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[50] Herbert Jaeger. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the "echo state network" approach. RMDD-Forschungszennrir Informattionstechnik Bonn, 2002.
[51] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Rexearch (IJRR), 2013.
[52] Yuwei Wang, Yuanying Qiu, Peitao Cheng, and Junyu Zhang. Hybrid cnn-transformer features for visual place recognition. IEEE Transactions on Circuits and Systems for Video Technology. 2023.
[53] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
[54] OpenStreetMap contributors. Planet dump retrieved from https://planet.smmorrg https:
//www. www.opppnstreetmapp.. org, 2017.
[55] R. Bellman, R.E. Bellman, and Rand Corporation. Dynamic Programming. Rand Corporation research study. Princeton University Press, 1957.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
13

---

[57] Kaiming He. X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[58] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. IEEE International Conference on Computer Vision (ICCV), 2015.
14

---

A Additional Experiment Results
In this section we give additional experiment results for the global localization task on the MGL [11]
and KITTI [51] datasets.
A.1 MGL Dataset Additional Results
Tep
Figure 7: Additional example trajectories from the MGL dataset with observations shown in the top row. We show the current true state and state history (black arrow and black line), the estimated posterior density of the current state (red cloud, with darker being higher probability) and the the top 3 extracted modes (blue arrows) for each method. By using the full sequence of observations and actions when computing state posteriors, MDPS able to estimate a more accurate and tighter posterior than the forward backward MDPFs.
State
Figure 8: Additional example trajectories from the MGL dataset with observations shown in the top row. We show the current true state and state history (black arrow and black line). the estimated posterior density of the current state (red cloud, with darker being higher probability) and the the top extracted modes (blue arrows) for each method. By using the full sequence of observations and actions when computing state posteriors, MDPS able to estimate a more accurate and tighter posterior than the forward or backward MDPFs.
15

---

A.2 Kitti Dataset Additional Results
Figure 9: Additional example trajectories from the KITTI dataset with observations shown in the top row, We and state history (black arrow and black line), the estimated posterior density of the
show the current true state current state (red cloud, with darker being higher probability) and the the top 3 extracted modes (blue arrows) for each method.
4T
Figure 10: Additional example trajectories from the KITTI dataset with observations shown the top row. We show the current true state and state history (black arrow and black line), the estimated posterior density of current state (red cloud, with darker being higher probability) and the the top 3 extracted modes (blue arrows) for each method.
Additional example trajectories from the KITTI dataset with observations shown the top We show the current true state and state history (black arrow and black line). the estimated posterior density of current state (red cloud, with darker being higher probability) and the the top 3 extracted modes (blue arrows) for each method.
Figure 11:
16

---

B Pseudocode
Mixture Density Particle Filtering:
Given observations /t.r and actions 01:r with N being the number of particles
1. Initialize particle set {x1] (t) (3) using initial known state or via some other method,
Normalize weights such that Ei=1 w1 =1
2. For f 2 T and I,..,,
(a) Resample particle from mixture distribution
(t) m( W 31,
N
(b) Apply noisy system dynamics to particle:
x ( f(=(") az m)),
77~ ((0, 1)
(c) Compute particle weight (normalized such that Σ wi ( 1)
w (6) w. 1(=//) ;1)
(:) 3. Output: (s WiT, 11:77 (8}
Figure 12: The Mixture Density Particle Filter
Mixture Density Particle Smoothing:
Given observations y1.r and actions 01rr with being the number of particles
1. Compute forward filter particle using Mixture Density Particle Filtering with yırr and 01.r:
(
20 1:7:
1.7 3
I:T
Compute backward filter particle sets using Mixture Density Particle Filtering with yru and оT:1 (with time reversed);
E E. 1:T
w
33
3. For = 1,.., T and 1...,,N N
(a) Define:
(1) z
1:T W 1:7 3)+ 3 mm(
1
I i:T 1:7. 3)
to
((x) 2 m x
(b) Sample particles:
4 ~ q(x)
(c) Compute weights
{(,; (+)
1: w 1: 3) ) m 10 TO 3))
m
q
4. Output: (x
(1) 1:7
3
Figure 13: The Mixture Density Particle Smoother
17

---

Interestingly, performance C Number of Particles Ablation Study
Rocalll (96)
06
local Real
frod
Racal
lacal
3B'
10
30'
Number of Particles
Number of Particles
Figure 14: Recall of position and angle for MDPS using the MGL dataset [11] with varying numbers of particles inference time. Here specity the number of particles N used for the forward and backward filters of MDPS. The final MDPS posterior density is defined by 2N particles described platcaus quickly as we increase the number particles implying MDPS's ability to use particles smartly and efficiently, allowing for fewer particles to be used, lowering the computation and memory requirements neceded for effective models.
key hyper-parate prper-parameter for particle filters and smoothers is the number particles to use at inference time. Using more particles increases performance shown in fig. 14 but performance plateau. As seen in fig. 4, effective models tend to concentrate particles densely in likely regions of the state space. By using more particles, the particle density of these likely regions is increased but with diminishing returns, Each additional particle within the dense regions will vary only slightly from its neighbors, minimally adding to the particle diversity. Further using more particles increases computation and memory requirements making smarter models which are more particle efficient, such our MDPS, attractive for real world deployment
quickly
18

---

D Additional Experiment Details
D.1 Bearings Only Tracking Task
The Bearings Only Tracking Task adopted from Younis and Sudderth [14] aims to track the state of a vehicle it navigates a simple environment. No actions are provided for this task and the observations are given as noisy bearings to a radar station:
Uniform( -T, π) - (1 a)) VonMises(()(zz))
where (x,, is the true bearing with 0.15 and K 50. The velocity of the vehicle varies over time, changing randomly when selecting a target new way point with Im/s or 2m/s being equally likely. During training, ground truth states are provided every 4 time-steps however dense true states are given at evaluation time. All sequences are of length T = 50 and we use 5000, 1000 and 5000 sequences for training, validation and evaluation respectively.
For methods we use 50 particle during training and evaluation. For forward-in-time running PF methods, we initialize the particle set as the true state with small Gaussian noise (o 0.01) on the the initial particles we add Von Mises noise
x-y components of the state. For the angle components (with concentration k 100) to the true state. For backward-in in-timm running PF methods set the initial particle set as random samples drawn uniformly from the state space.
Learning rates are varied throughout the training stages ranging from 0.001 to 0.000001 though we find that all methods are robust learning rate selection when using the Adam [49] optimizer, with sensible learning rate effecting convergence speed but not performance.
For SR-PF [13] we À 0.1.
D.1.1 Model Architectures
All methods (baselines and ours) for the Bearings Only Tracking Task use the same dynamics and measurement model architectures which are described below.
Dynamics Models. We parameterize the dynamics model residual neural network as shown in figs. 15 and 16. To maintain position in-variance, mask out the position components of particles when input into the dynamics model. We also transform the angle component the state into vector representation T(0) (O), cos( before applying dynamics. Afterwards we transform the angle component the state back into an angle representation T(u, v) atan2 v) =
Input: Particles
Input: Handom Noren
Concatenate
Fully Connected
RoLU
Fully Connected (5)
USiggmold"2) 1)* Scale
Output: Particies
Figure 15: Dynamics model used for the Bearings Only Tracking Task. The output scaling scales the position components of the residual to be within [-5, 5] and 2, for the positional and angle (in vector representation) components respectively.
Measurement Models. Figure 17 shows the foed-forward neural network architecture for the measurement models used for the Bearings Only Tracking Task.
MDPS Forward Backward Combination. For the Bearings Only Tracking Task, the MDPS smoothed measurement model is very similar to the measurement model using for MDPF but with additional inputs. The MDPS smoothed measurement model network architecture is shown in fig, 18.
19

---

Inoub Partidie
Concarenatee
Fully Connected (8)
Hoon Noiss
RELU
Soals
Figure 16: Dynamics model used for FFBS in the Bearings Only Tracking Task. The output scaling scales the position components of the residual to be within [-5, and -2, for the positional and angle (in vector representation) components respectively. The dynamics model outputs a mean. Using this mean along with a hand tuned standard deviation allows for simulation from the dynamics model as well as evaluating state transition probabilities as required for FFBS. Of note: the angle dimension of the state is approximated by Normal distribution with bound variance avoid issues with angular discontinuities. The hand-tuned standard deviations values used are [1.0, 1.0, 1.25).
Isss Parteles
Figut Oiverwniorr
Bell
RUUU
Pully Cornrected
Partiele filter measurement model used for the Bearings Only Tracking Task. The output scaling scales the weights to within 10.00001, 1]
Figure 17:
D.2 Global Localization Task with Mapillary Geo-Location Dataset and KITTI Datasets
In this section we give more information about the experiments conducted with the Mapillary Geo-Location Dataset (MGL.) 1] and KITTI [51] datasets.
For all particle filter methods (including ones internal to MDPS) we use 250 particle during training and evaluation and initialize the filters using 1000 particles. For PF and smoother methods, initialize the particle set as the true state with Gaussian noise (a = 50 meters) on the x-y components of the state. For the angle components of the initial particles we add Von Mises noise to the true state.
Initial learning rates are varied throughout the training stages ranging from 0.01 to 0.000001 though find that all methods robust to learning rate selection when using the Adam [49] optimizer, with sensible learning rates effecting convergence speed. For comparison methods, we use the learning rates specified by the method authors select them via brief hyper-parament search if they not stated.
D.2.1 Mapillary Geo-Location Dataset Additional Details
In the MGL dataset, observations are images captured by various types of handheld or vehicle mounted cameras. Some cameras capture 360 images which require additional processing before being used as observations. These 360" images are cropped to 90 Field-of-View in random viewing directions, with the same viewing direction being used for the whole observation sequence. All images are then gravity aligned to produce the observation sequence. A planimetric map of the
20

---

Igg Shandien
Fab Coweriade
Faul
Filk Coner
Figure 18: Measurement model used for the Bearings Only Tracking Task when computing the smoothed particle weights for MDPS. The output scaling scales the weights to be within [0.00001,
environment is also provided via the OpenStreetMap platform [54] 0.5 meter/pixel resolution, and all observation images are publicly available under CC-BY-SA license via the Mapillary platform. All KITTI data published under the CC-BY-NC-S. licence.
Unfortunately the creators of the MGL. dataset trained their methods on single observations from the dataset and did not sequences during training [11]. As such, observations from sequences are scattered amongst the training and testing splits, preventing effecting training of methods that require longer uninterrupted sequence data such as MDPF and MDPS. We therefore create custom train, validation and evaluation splits of the MGL dataset in order to accommodate longer sequences for training and evaluation.
Due to data integrity and corruption issues we exclude all sequences from the "Vilnius" " portion the dataset.
D.2.2 MDPF/MDPS Model Architectures
Dynamics Models. The network architecture of the dynamics model used for the MGL and KITTI is shown in fig. 19 and is similar to that used in the Bearings Only Tracking task, using the same angle to vector particle transformation. Note for this dynamics model we do not mask out any component of the state.
Input: Particles
Input: Random Noise
Concatenate
Fully Connected (8)
Relli
Fully Cornected (8)
(Sigmaad"2)-1>> Scale
Output: Particles
Figure 19:
Network architecture for the dynamics model used for the Bearings Only Tracking Task. The output scaling scales the position components of the residual be within [-128, 128] and -2, for the positional and angle (in vector representation) components respectively.
Measurement Models. The measurement model used for MDPF and MDPS uses the Birds-Eye- View (BEV) feature encoder and the map encoder from the official Dense Search implementation released by Sarlin al. [11]. As shown in fig. 1, Birds-Eye-VVies (BEV) feature map is estimated from the observation using geometric projection [11] where columns of the observation image are considered polar rays with features binned into course depth planes projected away from the camera focal plane. This gives a top-down representation of the local area in polar coordinates (bearing and
21

---

course distance of an image feature from the camera center). The polar representation of the scene then sampled into top-down Cartesian coordinates to yield the final BEV feature map. We refer the reader to Sarlin et al. [11] and the appendix for more details. To compute particle weights, we compute the alignment, via a dot-product, between the BEV feature map and a local region from the neural map, cropped and rotated at the current particles location. For I(-) from eqn. 20 we use fully-connected layers to produce the final smoothed particle weights from the BEV-map dot-product alignment and the forward and backwart filter posterior densities. The map feature encoder is U-Net based architecture with VGG- 19 [56] backbone and the BEV feature encoder is based on multi- head U-Net with ResNet- 101 [57] backbone as well
differentiable but un-learned geometric projection. We refer the reader to Sarlin et al. [11] for more details about the encoders. To derive the un-normalized particle log- weights, the BEV encoding is compared. via dot-product, to local map patch extracted at a specific particle to compute an alignment value. This is akin to the dense search described in Sarlin et al. [11] but at only a single location determined by the particle.
MDPS Forward Backward Combination. The MDPS smoother measurement model differs from the measurement models of MDPF as it requires additional inputs as described in eqn 20. We implement this model as a 4-layered, 64-wide fully-connected feed-forward neural network with PReLU [58] activation's 's shown in fig. 20. Importantly all computed un-normalized weights are bound to be within [0.0001, 1] using a Sigmoid function with an offset. Input into this network is the GEV-map feature alignment computed in the same way the MDPF measurement model well as the posterior probability values from the forward and backward filters for the current smoothed particle.
Input: BEV-Mop Alighment
Input: Forward Posterior
Inout: Backward Postarior
Concatonate
Fully Connected (84)
PReLU
Fully Conmected (64)
Sigmoid Scale
Dutput: Waights
Figure 20: Network architecture for the MDPS smoothed measurement model used for the MGL and KITTI datasets.
D.2.3 MDPS Training Procedure Details
As stated in sec. 4, effective training of MDPS requires training in stages. Importantly due to VRAM constraints, we are unable to train large map and observation encoders on long sequences. Therefore we train on short sequences before freezing the encoders, training the rest of the models on longer sequences.
Training procedure for MDPS the MGL. dataset:
1. Train forward and backward MDPFs individually via independent loss functions, sharing map and observation encoders, on short sequences from the dataset. Here we hold the output posterior bandwidths fixed to prevent converging to poor local optima where the bandwidth is widened while the dynamics and measurement models are not informative.
2. Freeze all MDPF models, unfreeze the MDPF output posterior bandwidths and train long sequences.
3. Freeze all MDPF models (including bandwidths) and train the MDPS measurement model and output posterior bandwidth on long sequences.
4. Unfreeze all models except the map and BEV encoders and train MDPS on long sequences.
Due to the small size of the KITTI dataset along with pre-training using the MGL dataset, a special constrained training procedure is required to prevent immediate over-fitting to the training split.
22

---

Instead of jointly refining all components simultaneously, we fine-tuning the forward and backward MDPFs before freezing those models and fine-tuning the MDPS smoothed measurement model:
1. Train the forward and backward MDPFs individually via independent loss functions, sharing map and observation encoders, on short sequences from the dataset. We hold the output posterior bandwidths fixed.
2. Freeze BEV and map encoders, training the forward and backward MDPFs individually via independent loss functions on long sequences.
3. Freeze all MDPF models, unfreeze the MDPF output posterior bandwidths and train on long sequences.
4. Freeze all MDPF models (including bandwidths) and train the MDPS smoothed measure- ment model and output posterior bandwidth on long sequences.
5. Freeze all models except for the MDPS output posterior bandwidth and train on long sequences.
D.2.4 Baseline Implementation Details
The Retrieval [3] baseline as described by Samano et al. [3] encodes individual patches from the environment global map into the a latent space. This is inefficient if the map patches are densely sampled and is prohibitive to run for large environments. Instead a dense feature map can be predicted from the global map in one forward CNN pass to generate dense map encoding for map patches roughly sized according to the CNN receptive field [1.]. We adopt this dense encoding approach for Retrieval, implementing the Retrieval method specified by Sarlin al. [11].
Retrieval.
Dense Search. For Dense Search we use the official implementation released by Sarlin et al. [11] which differs from the text description present in the official paper. In the paper description a location
prior is computed from the provided map which estimates regions of the state space that are likely to be occupied (e.g. the prior says that rivers and inside buildings are unlikely to be occupied by car). The prior is then multiplied with the observation likelihood (probability volume computed via dense search) to produce the final state posterior. In the official implementation this prior is disabled making the state posterior simply the observation likelihood. Further we use VGG-19 [56] as our map encoder backbone.
Retrieval (PF). Zhou al. [8] embeds standard Retrieval methods within a nondiifferentiable particle filter where the dynamics are set as Gaussian Noise:
(4 + ac,7)
(24)
In our experiments we set 7 as 2.5m for the x-y state position components and 15 for the angular state components, chosen via brief hyper-parameter search. The measurement model is defined as
W
exp
(25)
20
where d.0 is the alignment of the observation latent encoding with the map patch encoding at the current particles location z .computed via the Retrieval method. After a brief search we set 2 in our experiments.
Applying baselines to city-scale environments. Due memory constraints, dense search over the whole map is not possible (approx. 809 GB is needed for T 100 length sequence at 0.5m per pixel resolution). We therefore offer 2 methods for applying this dense search city scales. Ground Truth (GT) Cheat Method: using the ground truth state, we extract a small region from the map in which we do dense search. This greatly reduces the search space, saving memory but also greatly (and artificially) improves performance. Sliding Window Method: At = 1 small region extracted around the true state is densely searched. At subsequent stees the best alignment from the dense search of the previous time-step is propagated using and used as the center of a new small region which is then searched. Retrieval methods tend to fail when applied to large environments with Zhou et al. [8] even limiting the search to patches on known road networks. To address this, we limit the search space of Retrieval like in Dense Search using the GT Cheat and Sliding Window techniques, considering map patches within small regions.
23

---

D.2.5 Top 3 Mode Finding via Non-Maximal Suppression
Due to multi- modality of the posterior estimate, simply extracting the top mode to evaluate errors is not a good gauge of performance. Instead we extract the top-3 modes from the posterior density for evaluation. This can be easily achieved via a non-maximal suppression scheme where modes are extracted before particles around those modes are deleted. Specifically after extracting the top mode, the distance of all particle to that top mode is calculated. Particle within some threshold of the top mode are deleted from the particle set and the weights of all remaining particles are re-normalized to admit valid probability density after deletion. The next top mode is then extracted and the deletion process repeated until a total of 3 modes are extracted. In our implementation, we delete particles that are within 5m and 30 of the top mode.
For methods that admit discrete probability volume (such as Dense Search and Retrieval), we suppress the values of all probability cells within 5m and 30 of the top mode during the deletion step. Extracting the top mode can be simply achieved by finding the maximum value within the discrete probability volume.
D.3 Compute Resources
We give an approximation for compute resources needed to run our experiments in tables 1, 2 and 3. Since our experiments are necked by GPU resources and require only minimal CPU and memory needs, we report the GPU needs and GPU runtime for each experiment. compute resources stated in this section, we used additional resources over the course of our project when developing our methods, though the amount of resources used is difficult to quantify and thus we do not report here.
addition the
For evaluation, all methods can comfortably run in under 2 hours for the full evaluation split of MGL (using a NVIDIA A6000 GPU) and in under hour for Bearings Only (using a NVIDIA RTX 3090 GPU) and KITTI (using a NVIDIA A6000 GPU).
Table I:
Computation needs for training Bearing Only Tracking Task. Of note: none of the methods require using the whole GPU and thus we usually train 2-3 methods per GPU simultaneously. The numbers reported assume training each method 11 times sequentially without running in parallel.
Experiment
GPU
GPU Runtime
TG-PF (Multinomial)
Ix NVIDIA RTX 3090
25 hrs
TG-PF (Stratified)
1x NVIDIA RTX 3090
~55 hrs
SR-PF (Multinomial)
1x NVIDIA RTX 3090
~21 hrs
SR-PF (Stratified)
1x NVIDIA RTX 3090
~21 hrs
MDPF (Multinomial)
1x NVIDIA RTX 3090
~00 hrs
MDPF (Stratified)
1x NVIDIA RTX 3090
88 hrs
MDPF-Backward
1x NVIDIA RTX 3090
80 hrs
MDPS
1x NVIDIA RTX 3090
160 hrs
Table 2: Computation needs for global localization on the MGL dataset. Retrieval (PF) requires no training since all trained models taken from the Retrieval baseline. Similarly MDPF requires training as it is trained within MDPS.
Experiment
GPU
GPU Runtime
Retrieval
1x NVIDIA A6000
12 hrs
Retrieval (PF)
NA (No Training)
Dense Search
4x NVIDIA A6000
48 hrs
MDPF
NA (No Training)
MDPS
3x NVIDIA A6000
72 hrs
24

---

Table 3: Computation needs for global localization on the KITTI dataset. Retrieval (PF) requires no training since all trained models are taken from the Retrieval baseline. For MDPF we report additional resources used during refinement on the KITTI dataset.
Experiment
GPU
GPU Runtime
Retrieval
1x NVIDIA A6000
~ hrs
Retrieval (PF)
NA (No Training)
Dense Search
4x NVIDIA A6000
18 hrs
MDPF
3x NVIDIA A6000
~ hrs
MDPS
3x NVIDIA A6000
hrs
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes the abstract and introduction state the claims making in the main paper text. We backup these claims with experimental results in sec. 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in the main paper text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
25

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We dont have any theoretical results but instead conduct experiments on existing datasets.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all important information for reproducing our results in the main text (secs. and 5) and the appendix (supplement).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
26

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code will be released GitHub if the paper is accepted after some code cleanup.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: wwtggguddess///e www.eoeSaaiiiiiiiooooooooooaa /otea//guudeee/eeeeeaaaaieeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Important details needed to understand the results are given in the main paper (sec. 5) with additional detail given in the appendix (supplement) for more curious readers.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27

---

Answer: [Yes]
Justification: Where possible we run experiments multiple times to generate error bars and other statisties about our methods. Some experiments require significant amounts of compute and thus we are unable to generate error bars for all experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This information is provided in the appendix (supplement). Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : htt////eerrpppeeeeee wwwturps.ccppuuuaee
Answer: [Yes]
Justification:
collected in outdoor public environments and thus contain images of people in public. The dataset owners have taken efforts to blur faces of all people and other identifiable information. Our work primarily focuses on state estimation systems for robotics and thus is low risk in terms of safety, ethics and fairness.
The real world datasets
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
28

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA].
Justification: Our work primarily focuses state estimation systems for robotics. As such there is little risk for abuse of our methods without substantial innovation by a bad actor and thus the broader negative impacts of work is very limited.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., depllymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our work focuses on general state prediction models that can be applied to range of tasks though only use synthetic or real world roboties datasets limiting the ability of our models to be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
29

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit all works we reference or use as well as state the copyright (when available) datasets in our experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
paperswithcode.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not create any new data assets. The final trained models are large and we do not have the hosting capability to make these models freely available online, instead we will release all code and instructions such that these models can be trained.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: Our work does not involve crowdsourcing or human subjects but rather focuses on roboties tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
30

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA].
Justification: Our work does not involve crowdsourcing human subjects but rather focuses on robotics tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "hfian1SLZQ", "reviewer": "Reviewer_tGtz", "summary": "The paper proposes a learnable differentiable particle smoother system that extends an existing differentiable particle filter to a smoother. The method utilizes two independent particle filters for the two smoothing directions with importance sampling to address computation scaling issues.", "questions": "- With regards to the sampling, in the paper M is selected to be equal between the two filter directions. What's the impact of selecting M and it's distribution onto the two distributions?\n- In the description of baseline methods it is mentioned that refinement methods require an \"accurate initial estimate\". How good does this have to be in practice and/or in the specific experiments shown in the paper?", "limitations": "The limitation section is very basic and really is about the limitation of the family of methods in general rather than the proposed method. Expanding on that and providing information about the proposed method would be expected.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Proposed smoothing approach outperforms filtering setting\n- Combination of sensible components into the proposed filtering setup", "weaknesses": "While the proposed method appears to work and improves over the baseline particle filter variant it is unclear how well it performs in contrast to other smoothers which would pose a fair comparison. Additionally, the particular task of large scale global localization is something that robotics has been investigating for a long time and as such one would expect to see comparisons to the typical way this is solved, such as full visual SLAM or hybrid place recognition with localization. As the KITTI dataset is used it is also disappointing that no comparison to the benchmark results of the dataset are presented. Overall it is unclear how well the proposed method works when compared to known approaches deployed in practice.\n\nThe power of a learnable filter/smoother is that it can learn models where it is hard or impossible to define them manually. However, the chosen application has very well defined and workable models making the motivation less convincing. As such it would be highly interesting to see the method be evaluated on scenarios where no models exist or even exist and are compared with. In line with that it would also be interesting to know how feasible it is to run the proposed method on higher dimensional data.\n\nThe method requires a significant amount of time to train and run which makes it questionable whether it is even practical for real applications. Another issue arises from this is that different methods were provided with different amount of training time. While it is valid to see how well methods work when given the same parameters (particles) it is also important to assess how good other methods would perform if they had access to the same amount of training time which for the MDPF vs MPDS is still twice as much.\n\nThe paper mentions that given true dynamics and likelihood models the sampling will compensate for the mismatch between samples and density mixture. Is that statement meant to indicate that these ground truth models are required, or that in an ideal case this would technically happen? If the first meaning is intended, where do these models come from? If the second meaning is intended, how close does the proposed method get to this and is there a way to assess that?\n\nIt appears that the absolute error of the proposed method and all others is exceedingly large with being many meters off to achieve any reasonable amount of recall. This is where reproducing numbers from the KITTI benchmark is crucial to get an idea of the hardness of the problem.\n\nFigure 5 is hard to interpret, especially since there are now comparisons to other methods provided. While one can see that the prediction somewhat maps to parts of the future state it is very noisy and unclear how good it actually is.\n\nGiven the massive training times of the method, it would be important to provide insights into the scalability both in terms of number of particles and dimensionality of the estimation problem. Another question that arises in that regard is how the proposed method compares to methods based around Stein variational inference [1] which is capable of handling high-dimensional problems very efficiently.\n\nOverall, while the proposed method performs better than other filtering methods it is unclear how practical the method is given the required compute time and limitation in the dimensionality of problems. This is further exarberated by the fact that there is no comparison to typical solutions of the chosen problem setups and a lack of particle smoother baselines.\n\n[1] Liu, Qiang, and Dilin Wang. \"Stein variational gradient descent: A general purpose bayesian inference algorithm.\" *Advances in neural information processing systems* 29 (2016).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback.  Please see our global response for discussion of training time and paper figure formatting.\n\nWith regard to comparison with existing works, we would like to make the distinction between local localization and global localization.  Local localization estimates the state in relation to the starting point. Global localization requires estimating the state in relation to a global map origin. Visual SLAM systems almost exclusively solve the local localization task, using the starting position as the origin of their estimated map. If a map is provided, then just the localization part of Visual SLAM can be run, but detailed visual or 3D maps of the environment are needed.  These have prohibitive memory requirements at city scales and need constant updating as the visual appearance of the environment changes (i.e. with the weather/seasons) [11].  Hybrid place recognition with localization also requires detailed visual or 3D maps [Wang et al., 2023]. Instead we seek to use planimetric maps for global localization, which are compact and robust to environment changes.  It is not obvious how to apply SLAM/Hybrid place recognition systems to this type of map.\n\nWe use the KITTI dataset in a way that makes comparison against the KITTI leaderboard impossible. The leaderboard ranks methods for local localization with no prior map provided. The methods seek to only accurately estimate relative position without regard to global position. In contrast, our method solves the global localization task using a map where the starting state itself is uncertain. Comparison of our method to the leaderboard would not be appropriate, as we are completing a different task. We instead compare against state-of-the-art prior methods for global localization (see Figs. 3 and 6) to illustrate task difficulty.  Compared to the prior state-of-the-art [11], we have both higher accuracy and lower localization cost at test-time.\n\nFigure 5 highlights the effectiveness of the learned complex, nonlinear dynamics model at propagating particles given noisy actions. Particles are spread to ensure some overlap with the true state, and the measurement model then concentrates probability mass over the true state using the latest observation.  Other methods do not have dynamics models and thus similar figures cannot be made.\n\nIn appendix B, we show some scalability results for the number of particles used during inference. Interestingly, models can be trained with few particles to limit computation requirements during training, but then many particles can be used during inference to increase accuracy. In this work, we focus on algorithmic innovations over prior differentiable particle filters/smoothers, and apply our MDPS to 3D problems.  However, in future work we plan to explore scalability to higher-dimensional problems.\n\nStein variational inference iteratively transforms particles from a starting distribution to samples from a known target distribution using the score function, and can be thought of as a Monte Carlo method that avoids computing normalizing constants. Although this method has been applied to state estimation [Maken et al. 2022, Fan et al. 2021], these methods assume the prior and dynamics models are fixed and known, and thus could not be applied to our discriminative training scenario where models must be learned from data. \n\n**To answer your questions:**\n\n- In section 4 we say *“Given true dynamics and likelihood models, importance sampling may correct for the fact that smoothed particles are drawn from a mixture rather than a product of filtered densities...”*. We are saying that we wish for samples from the product of densities (hard to sample from) but draw samples from a mixture of densities (easy to sample from) and use importance sampling [21] to compensate for this mismatch by weighting the drawn samples.  If the true dynamics and likelihood models are known then importance sampling would produce weighted samples from the true posterior distribution, but since these models are learned the weighted samples are from an approximation of the true posterior distribution. There is no easy way to assess the accuracy of the learned posteriors, since the true distributions are unknown.\n\n- See appendix B for experiments varying the number of particles $M$.  Sufficient number of particles are needed to accurately represent the forward and backward filter distributions.  $M$ that is too small can negatively impact MDPS by limiting the representational power of the two filters, and thus increasing variance when estimating gradients. Increasing $M$ can only help performance by increasing representation power.  We empirically set $M$ to be equal for both filters in order to simplify implementation, however tuning $M$ separately for each direction could potentially provide computational savings.\n\n- Refinement methods require an accurate initial estimate.  [11] sets this to be within a 64x64 meter box centered around the true position and within 45 degrees of the true bearing in order for the method to converge. In [9] this box is decreased to 40x40 meters, and the authors state that if the initial state is far from the true state, extracted features from the image and the map will be too dissimilar and the refinement method is often trapped in poor local optima. In contrast, we initialize our MDPS method with large uncertainty (particles randomly within 150 meters from the true state).\n\n**Additional References:**\n\nY. Wang, Y. Qiu, P. Cheng and J. Zhang, \"Hybrid CNN-Transformer Features for Visual Place Recognition,\" in IEEE Transactions on Circuits and Systems for Video Technology, 2023\n\nMaken, Fahira Afzal, Fabio Ramos, and Lionel Ott. \"Stein particle filter for nonlinear, non-gaussian state estimation.\" IEEE Robotics and Automation Letters, 2022\n\nFan, Jiaojiao, Amirhossein Taghvaei, and Yongxin Chen. \"Stein particle filtering.\" arXiv preprint arXiv:2106.10568 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFfYaBnA0M", "reviewer": "Reviewer_qoBM", "summary": "The author's propose the first differentiable particle smoother, building on the MDPF of Younis and Sudderth which replaces the multinomial resampling step with a sampling from a KDE mixture. Their smoother is a two filter smoother where both the forwards and backwards filters are MDPFs. Instead of using the forwards filter to re-weight the particles from the backwards filter, as is conventional, particles are drawn directly from the mixture of the forwards and backwards posteriors. The author's test their smoother, first a toy bearings only tracking example. Then the a more complicated visual localisation task as well and the KITTI dataset. The smoother out performs the state-of-the-art filtering and search-based methods tested.", "questions": "None.", "limitations": "It is mentioned in the limitations that the curse of dimensionality may be lifted somewhat with smarter proposal. There is no evidence that this algorithm nor the MDPF is adept at learning a non-bootstrap proposal.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. A differentiable particle smoother is the obvious next step to differentiable filtering, especially given much previous work trains their filters in an offline, batched way on with access to ground truth latent state in training. I.e. on smoothing problems. This is the first paper to successfully construct a differentiable smoother capable of learning the complete model.\n2. The experiment results are very strong and give good credibility to the algorithm. The used scenario is considerably more complex than those often used to test algorithms in a similar class.\n3. Using the direct product of the mixtures simplifies the calculations greatly.\n4. The paper is well organised and easy to follow.", "weaknesses": "1. This method and it's predecessor the MDPF appear to enjoy less theoretical support than other differentiable particle algorithms such as \\'{S}cibior and Wood's stop gradient resampling or Thornton, Corenflos et al.'s OT resampler. For example neither the gradient estimates nor state estimates appear to be consistent, or at least there is no proof they are.\n2. Unlike the MDPF, this smoother requires the pointwise evaluation of the posterior density. This differs from the usual two filter smoother, I am unconvinced that the KDE approximation is accurate enough for this purpose. Some more analysis could be used to justify this approach.\n3. This method has potential impact outside of just visual localisation, instead of several visual localisation experiments an example of a different problem could have boosted the apparent impact.\n4. The text on the figures is far too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your praise and feedback. We are grateful for the careful review of our work and appreciate your highlights on the novel nature of our differentiable particle smoother, and the strength of our experimental results compared to scenarios considered by related work.\n\n**To address some of your concerns please see below:**\n\n- We agree that KDE density approximations may not always be accurate, but established theory [41] tells us that KDEs consistently approximate any smooth density with a sufficient number of samples, and our experiments suggest they can provide accurate approximations with practical numbers of particles.\n\n- We too agree that our method has impacts outside of just visual localization and plan to apply our techniques to other tasks in future work.  But we would like to emphasize that given the complexity of the real-world datasets considered in our experiments, achieving these advances to the state-of-the-art in visual localization already required substantial work.\n\n- Please see our global response for more information on figures and formatting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zoN2QIIWde", "reviewer": "Reviewer_fbEx", "summary": "My understanding is that this work seeks to learn a state-space model by differentiating through a regularised version of the sequential Monte Carlo (SMC) approximation of the the (generalised) two-filter smoother.", "questions": "Can you explain how the proposed method achieves differentiable stratified resampling?", "limitations": "I appreciate the authors' candour that the proposed method will suffer from a curse of dimension.", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "I am not very familiar with the applications used as numerical examples but they seem to be reasonably close to real-world scenarios.", "weaknesses": "**Lack of clarity & rigour**\n\nI do not believe that the presentation of this work is adequate for NeurIPS.  The manuscript is much too informal and therefore very difficult to follow and verify. It reads like a mid-1990s engineering conference paper. For example, \n\n* The work should really include some pseudo-code. Without this, it is very hard to even understand what the proposed algorithm exactly is. It is not even entirely clear to me whether the proposed method seeks to learn only the measurement model or also the system equation. The short paragraph on \"Training details\" at the end of Section 4 is also highly insufficient.\n* Throughout Section 3 and 4, the same symbols seem to be used for model densities and their (e.g. Monte Carlo or other) approximations\n* Lines 70/71 claim: \"Our differentiable PS incorporate stratified resampling to reduce variance with negligible computational overhead, making training more robustly avoid local optima\". I don't find evidence for this and it is not clear to me how the authors would achieve differentiable stratified resampling. \n* Some notation is not or insufficiently clearly defined, e.g. $\\hat{w}^{(i)}$ and its gradient in Equation 7.\n* Line 108 mentions \"unbiased\" gradient estimates but I cannot see how this would be true for any of the proposed methodology.\n\n**Missing baselines**\n\nThere are well-known SMC based, recursive parameter-estimation methods against which the method should really be compared, e.g.:\n\nKantas, N., Doucet, A., Singh, S. S., Maciejowski, J., & Chopin, N. (2015). On particle methods for parameter estimation in state-space models.\n\n**Insufficient motivation**\n\nIt is not clear to me what the motivation for the proposed approach is, except that no one has previously tried to differentiate through the two-filter smoothing approximation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback.  \n\n**To address your concerns:**\n\n- Our MDPS method learns both the measurement model and the dynamics model via end-to-end training.  The “Training Details” paragraph at the end of section 4 was intended to highlight the 3-stage training approach. Additional training details for reproducibility can be found in the appendix. \n\n- We apologize that aspects of the notation were confusing, we will add more clarifications.  With regard to $\\hat{w}^{(i)}$ in Eq. (7), this refers to the unnormalized weight assigned to the resampled particle $\\hat{x}^{(i)}$.  It has weight 1 for the current model parameters, but weights must be adjusted as dynamics and observation parameters change during training, as captured by the gradient equation.\n\n- Line 108 mentions how unbiased gradient estimates may be produced via the Importance Weighted Sample Gradients (IWSG) estimator proposed in [14].  IWSG computes gradients based on importance sampling [21]. Briefly, importance sampling itself produces unbiased weighted samples from some distribution of interest, and therefore differentiating through importance sampling yields unbiased gradients as well.  Experiments in [14] show that this approach has dramatically reduced variance compared to reparameterization-based gradient estimators.\n\n\n- Thank you for pointing out the review of classic work by Kantas et al. (2015), we will add citation and discussion.  These parameter estimation techniques make different assumptions than our work:  they learn generative rather than discriminative particle filters, and assume that parametric dynamics and likelihood models with only a few unknown parameters are available.  (Experiments in Kantas have only 3 unknown parameters, and we are not aware of any classic work with more than a few dozen learned parameters.)  These methods would not practically scale to datasets with large numbers of sequences as we consider, because particle state estimates must be produced for every time step at every iteration.  Moreover, the neural network parameterizations of our dynamics and measurement models have over 500 million parameters, millions of times larger than work cited by Kantas.  The computational cost of classic particle learning algorithms scales poorly with the number of parameters, and it would be practically impossible to apply these methods to our models (without substantial innovations).  Indeed, these limitations motivated our research on the MDPS method. \n\n\n**To answer your specific questions:**\n\n- Stratified resampling [31,32] is a method of drawing i.i.d. samples from some target distribution while reducing the variance associated with those samples.  These samples can be thought of as uniformly weighted samples from the target distribution (i.e. where all the weights are equal). To compute gradients for these samples we use Importance Weighted Sample Gradients (IWSG) [1] which can be applied to any sampling method that produces i.i.d. samples. IWSG propagates gradients via adjusting the weights of the drawn samples (which are initially uniform), yielding a differentiable stratified resampling technique that has the lower sampling variance properties of stratified resampling and the unbiased gradients of IWSG.\n\n**Additional Reference:**\n\nKantas, N., Doucet, A., Singh, S. S., Maciejowski, J., & Chopin, N. (2015). On particle methods for parameter estimation in state-space models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Ry5qeNS12", "reviewer": "Reviewer_56kY", "summary": "This paper proposes an end-to-end differentiable particle smoother and presents its application to global localization of a subject moving through real-world city-scale environments. Although there exists prior work in differentiable particle *filtering*, such as Mixture Density Particle Filters (MDPF), *smoothing* yields a more accurate estimate for offline data processing as it incorporates all the observations taken along the full trajectory. This paper revisits the classic formulation of Two Filter Smoothing (TFS) and leverages its principle to fuse two MDPFs, one running forward in time and one backward, into the novel Mixture Density Particle Smoother (MDPS). MDPS achieves higher log-likelihood and better tracking accuracy compared to MDPF and other prior work.", "questions": "[EDIT] All major concerns have been resolved after the rebuttal.\n\n* For MDPS, did you ever try initializing the estimates from some random states rather than the noised version of the true states? I wonder whether smoothing can actually correct erroneous initial estimates. If true, that is another advantage compared to MDPF, since we cannot hope for filtering to fix errors in initial estimates at all.\n\n* MDPS requires more GPU runtime to train than MDPF by design, as presented in Tables 1, 2, and 3. Would MDPF still perform worse than MDPS even when you let MDPF train longer to match the GPU runtime of MDPS? In other words, would the loss and the accuracy of MDPF not improve further with additional training epochs?", "limitations": "* As a fundamental limitation of the proposed approach, training requires three separate stages and thus takes more GPU runtime. I would appreciate it if the authors could comment on the tradeoff between better performance and more demand for compute when compared to MDPF.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* To my knowledge, this paper proposes the first framework for end-to-end differentiable particle smoothing. Smoothing is a powerful tool for offline state estimation and expected to achieve better accuracy than filtering. The proposed MDPS is a valuable framework for offline state estimation that does not require hand-designed state transition or observation models.\n\n* The technical approach is a sound combination of MDPF and TFS. Although I have some remaining questions about the underlying math (which is discussed below), the authors seem to have successfully designed their differentiable smoothing framework by leveraging the principled TFS formulation.\n\n* Empirical results show a clear advantage of the smoothing framework compared to its filtering counterpart as well as other prior work based on non-probabilistic schemes (e.g. refinement).", "weaknesses": "[EDIT] All major concerns have been resolved after the rebuttal.\n\n* Some of the math presented in Section 3 for TFS seems to be incorrect. Specifically, I do not think equation (14) holds; it seems to be missing $p(y_t \\mid x_t)$ from its numerator. Similarly, the resulting equation (15) for the particle weights does make sense either; multiplying a weight by a weighted sum of estimated states would not result in a weight. It would instead result in a new state. Please check (14) and (15) again and correct any errors. Also, please clarify how (14) and (15) relate to the practical implementation of MDPS presented in Section 4. In its current form of the paper, (14) and (15) seem unnecessary for MDPS because we instead have the correct equation (16) to begin with.\n\n* Many figures should be updated for better readability when printed on paper. Figures 1, 4, 7 - 11 have map patches and RGB observations that are too small. I suggest that the authors increase the font size as well and make each component in the figures bigger. The same goes for Figures 3 and 6, where the legends are too small to associate different curves with different methods.\nMore details on MDPF would be appreciated at the end of Section 2 for self-completeness of the information. Specifically, I did not fully follow what the “adaptive” variant means and why two bandwidths was necessary.\n\n* More details on MDPF would be appreciated at the end of Section 2 for clarity. Specifically, I did not fully follow what the “adaptive” variant means and why two bandwidths were necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and constructive feedback.  \n\nPlease see our global response to all reviewers for discussion of training time and paper figure formatting.\n\nWe agree that adding some discussion of accuracy vs compute demands of MDPF and MDPS would be useful, and will do this. In general MDPS will have higher compute demands than MDPF due to running two particle filters, but both methods scale similarly with the number of particles, and our experiments show that the constant-factor cost increase of MDPS may substantially boost accuracy. Note that appendix B discusses accuracy vs. compute demands for MDPS with respect to the number of particles used.\n\nWe would like to thank you for taking the time to carefully review our math. Indeed there are factors missing from equations (14) and (15), below are the corrected versions (with the corrections highlighted in red):\n\n**Equation (14):** \n\n$p(x_t|y_{1:T}) \\propto \\frac{\\tilde{p}(x_t| y_{t+1:T}) \\color{red} p(y_t|x_t) \\color{black} \\int p(x_t|x_{t-1}) p(x_{t-1} | y_{1:t-1}) dx_{t-1}} {\\gamma_t(x_t)}$\n\n**Equation (15):**\n\n$\\overleftrightarrow{w}_{t}^{(i)} \\propto \\overleftarrow{w}\\_{t}^{(i)} \\sum\\_{j=1}^N  \\color{red} \\overrightarrow{w}\\_{t-1}^{(j)} \\color{black}  \\frac{p(\\overleftarrow{x}\\_{t}^{(i)} | \\overrightarrow{x}\\_{t-1}^{(j)})}{\\gamma_t(\\overleftarrow{x}\\_{t}^{(i)})}$\n\nEquations (14) and (15) are included in the paper to contrast our MDPS formulation with the two-filter formulation employed by most prior work. You are correct that they are not directly used in the derivation of MDPS in Sec. 4, and thus these missing terms do not impact the correctness of our results.\n\nWe agree that the details on MDPF in Sec. 2 are brief, and that expanding this section could be beneficial.  In short, the adaptive MDPF as described in [14] is simply a particle filter that decouples the distribution used for resampling and the distribution used as the posterior estimate. The rationale is that the distribution used for resampling particles may not necessarily be appropriate for estimating the state posterior, since the latter may concentrate probability mass more tightly when observations are informative. Adaptive MDPF decouples the resampling and posterior bandwidths and learns each separately, allowing the resampling and posterior distributions to be separately tuned for each application. \n\n**To answer your specific questions:**\n\n- With regard to MDPS initialization, in the Mapillary and Kitti experiments we initialize MDPS with particles within approximately 150 meters of the true state with the localization area only being 614x614 meters (i.e. particles are drawn randomly from about ~19% of the localization area). This initialization results in erroneous initial estimates for MDPF but interestingly we find that MDPS is able to correct these erroneous initial estimates by using information from the backward filter. Given this, we believe that MDPS could correct initial states when initialized with totally random particles. \n\n- Please see our global response to all reviewers for discussion of training time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you all for your feedback and helpful suggestions.  We want to address a few topics that were referenced by multiple reviewers.

First, we would like to emphasize that our work is the first and only generalization of state-of-the-art differentiable particle filters to the more challenging particle smoothing scenario.  It is for this reason that most of our experimental comparisons are not to other particle smoothers, but to state-of-the-art methods for the global visual localization task, where we find substantial advantages in both speed and accuracy.

We agree that some formatting aspects of the paper can be improved. Unfortunately due to space constraints, the size of some figure elements and fonts was limited. In future revisions, we will be sure to make changes to the figures to increase the size of renderings and fonts for better readability (moving some details to appendices if necessary).

With regard to training time, we train all methods to convergence. During training, we monitor the training loss, and decay the learning rate each time a training loss plateau is detected. Training is terminated once the training loss plateaus and the learning rate is sufficiently small that decreasing it has no effect on training.  Finally, we use validation data to select the best model to use for evaluation (to prevent overfitting).  We find that given this training methodology, additional training time for the various methods would not yield better performance as all methods were trained to convergence (i.e. given as much training time as needed to produce the best model). The improved performance of MDPS over MDPF is not due to additional training time, but rather due to better dynamics and measurement models being learned as well as using the backward filter to propagate information from future observations (i.e., using the full sequence to estimate the state posterior distribution). Once trained, MDPF is very efficient during inference as shown in [14]. Since our MDPS simply runs two MDPFs and an additional combination step, it is also efficient during inference, making real-world use more practical than other methods such as dense search [11].

Meta-review: This paper presents a novel end-to-end differentiable particle smoother called Mixture Density Particle Smoother (MDPS) for state estimation problems in robotics and computer vision. The key contributions are:

1. Extending recent advances in differentiable particle filters to the more complex particle smoothing scenario.
2. Developing a gradient propagation framework across long time sequences for training particle smoothers.
3. Integrating forward and backward particle streams with stratified resampling and importance weighting.
4. Demonstrating improvements over state-of-the-art particle filters and search-based baselines on real-world, city-scale global vehicle localization tasks, albeit under the assumption that the localization is over a planimetric global map.

The reviewers were split in their assessments, with two recommending acceptance and two recommending rejection. The positive reviews highlighted the novelty of the approach, the strong experimental results and specifically the performance gains over rfiltering-only approaches, and the potential impact on real-world applications. The negative reviews raised concerns about the clarity of the mathematical presentation and the technique/claims, and questions about the practical applicability given the computational demands.

The authors' rebuttals addressed many of these concerns, providing additional details on the mathematical foundations, clarifying the novelty of their approach, and explaining the experimental setup. They also acknowledged limitations and areas for future work. While there were specific concerns regarding insufficient baseline comparisons, the authors state their approach operates under significantly more challenging conditions (e.g. estimation over a much larger parameter counts). There has been a concern regarding the definition of stratified resampling, which despite author responses, does not seem to have been fully resolved. 

Despite the split opinions, I recommend accepting this paper with the condition that the authors address the clarity and rigor concerns in a revised version. The reasons for this recommendation are:

1. This is the first work to successfully develop an end-to-end differentiable particle smoother, which is a non trivial contribution to the field, whereas differentiable particle filters do exist.
2. The method demonstrates substantial improvements over baselines on tasks that are moderately challenging. 
3. The approach has the potential to address important problems in robotics and computer vision with potential real-world applications. 
4. The authors have actively participated in the discussion to help alleviate reviewer concerns, and have shown willingness to address reviewer concerns and improve the paper.

To address the remaining concerns, the authors should:
1. Improve the clarity and rigor of the mathematical presentation, particularly around gradient estimations, and the definition of stratified resampling in the context of the this paper. 
2. Add more detailed comparisons to classical particle smoothing methods, even if only in synthetic scenarios.
3. Provide a more in-depth discussion of the computational requirements and potential limitations in high-dimensional settings, while also highlighting the fact that inference is more efficient. 
4. Consider adding pseudocode for the algorithm to improve reproducibility.

In conclusion, while there are areas for improvement, the novel contributions and strong empirical results make this paper a valuable addition to the NeurIPS proceedings. The authors should use the revision period to address the noted concerns and strengthen the paper further.
Decision: Accept (poster)