Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes
Authors: Ali Younis
Erik B. Sudderth
Venue: NeurIPS 2023 poster (2023)
Keywords: particle
filter
mixture
belief propagation
nonparametric
deep learning
generative
discriminative
graphical model
multiple modes
mutli-modal

Abstract: Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods.  Such generative models may be inaccurate or unavailable for high-dimensional observations like images.  We instead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders.   While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities.  Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness.  On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, will also showing much greater stability across multiple training runs.

Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes
Ali Younis and Erik Sudderth {ayoounis, sudderth) edu Department of Computer Science, University of California, Irvine
Abstract
Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking prob- lems with known dynamics and observation likelihoods. Such generative models may be inaccurate or unavailable for high sigh--imensional observations like images. We instead leverage training data to discriminatively learn particle- representa- tions of uncertainty in latent object states, conditioned arbitrary observations via deep neural network encoders. While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, biased learning by truncat- ing gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities. Our theory and experiments expose dramatic failures of existing reparameteriizationbaann estimators for mixture gradients, issue address via importance-samplien gradient estimator. Unlike standard recurrent neural networks, our mixture den- sity particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness. On range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, while also showing much greater stability across multiple training runs.
Introduction
A particle filter (PF) [1-4] uses weighted samples to provide a flexible, nonparametric representation of uncertainty in continuous latent states. Classical PFs are generative models that typically require human experts to specify the latent state dynamics and measurement models. With the growing popularity deep learning and abundance of time-series data, recent work has instead sought to discriminatively learn PF variants that (unlike conventional recurrent neural networks) capture uncertainty in latent states [5-8]. These approaches are especially promising for high-dimensional observations like images, where learning accurate generative models is extremely challenging.
The key challenge in learning discriminative PFs is the non-differentiable particle resampling step, which is key to robustly maintaining diverse particle representations, but inhibits end-to-end learning by preventing gradient propagation. Prior discriminative PF' have typically used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients. These methods severely compromise the effectiveness PF training; prior work has often assumed exact dynamical models known, used very large numbers of particles for test data, due to these limitations.
After reviewing prior work on generative (Sec. 2) and discriminative (Sec. 3) training of PFs. Sec. demonstrates dramatic failures of popular reparameterization-aasss gradient estimators for mixture models. This motivates our importance weighted samples gradient estimator, which provides the foundation for the mixture density particle filter of Sec. 5. Experiments in Sec. 6 show substantial advances over prior discriminative PF for several challenging tracking and visual localization tasks.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X₁
X
a2
1++1
a2
Y/
Figure I: Sequential state estimation may be formulated via either generative (left) or discriminative (right) graphical models, In either case, dynamies of states 2i are influenced by actions de, and estimated via data y/-
2 Sequential State Estimation via Generative Models
Algorithms for sequential state estimation compute or approximate the posterior distribution of the system state Tt. given a sequence of observations yt and (optionally) input actions a, at discrete times & 1, ...... Sequential state estimation is classically formulated as inference in generative model like the hidden Markov model (HMM) or state space model of Fig. 1, with dynamics defined by state transition probabilities p(z; Øt-1, at), and observations generated via likelihoods p(3 22).
Given known Markov prior on latent state sequences, and a corresponding observation sequence, filtered state posterior p(æ; Vt: - W1) can in principle be computed via Bayesian inference. When the means of state dynamics and observations likelihoods are linear functions, and noise Gaussian, exact filtered state posteriors Gaussian and may be efficiently computed via the Kalman filter (KF) [9-11]. The non-Gaussian state posteriors of general state space models may be approxi- mated via Gaussians [12], but estimating the posterior mean and covariance is in general challenging, and cannot faithfully capture the multimodal posteriors produced by missing or ambiguous data.
2.1 Particle Filters
To flexibly approximate general state posteriors, particle filters (PF) [1-4] represent possible states nonparametrically via a collection of weighted samples or particles. The classical PF parameterizes
the state posterior at time by a set of N particles 2 = {x(] , 21 (NV) with associated weights
wi( (w:[' Wi }. PFs recursively update the state posterior given the latest observation yr }. and action Gg and may be flexibly applied to broad range of models; they only assume that the state dynamics may be simulated, and the observation likelihood may be evaluated.
Particle Proposal. To produce particle locations x, at time model of the state transition
dynamics is applied individually to each particle _ conditioned on external actions if available:
p( Ft-1 11) 4:).
Measurement Update. After a new particle set x, has been proposed, particle weights must be updated to account for the latest observation Yr via the known likelihood function:
w/ x p(ye 1 2
The updated weights are then normalized so that E wi = 1. This weight update motivated by importance sampling principles, because (asymptotically, ,....,,)))
N - oo) it provides unbiased approximation of the true state posterior p(#:/ yts ye,..., 91) ) xx p(y. xa)p((;
.............
Particle Resampling. Due to the stochastic nature of PFs, over time particles will slowly diverge to regions with low posterior probability, and will thus be given little weight in the measurement step. all but few particles have negligible weight, the diversity and effective representational power of the particle set diminished, resulting poor estimates the state posterior.
To address this, particle resampling is used to maintain diversity of the particle set over time, and may be performed at every iteration or more selectively when the "effective" sample size becomes too small [13]. During resampling, a new uniformly-weighe
particle set at(²) is constructed by discrete resampling with replacement, Each resampled particle copy of some existing particle, where copies are sampled with probability proportional to the particle weights:
=
j j~ Cat(w)) (1) ..... ....,,(!") (N))
(3)
After resampling, assigning uniform weights w:(') - maintains an unbiased approximate state posterior. This discrete resampling oon-differentiable, preventing gradient estimation via standard reparameterization [14-16], and inhibiting end-to-end learning of dynamics and measurement models.

---

2.2 Regularized Particle Filters
The regularized PF [17, 18] acknowledges that a small set of discrete samples will never exactly align
with the true continuous state, and thus estimates a continuous state density m (#t convolving particles with continuous kernel function K with bandwidth 3:
I x w/ 3) by
ml(a:
(3) (:) W 3) Σ un! K(I; 2 10 1=1
(4)
The kernel function could be a Gaussian, in which case 3 is a (dimension-specifhe standard deviation. The extensive literature kernel density estimation (KDE) [19] provides theoretical guidance on the choice of kernel. Quadratic Epanechnikov kernels take K(u; 3) = (1 (u/B)²) ) if lu/8| S 1, K K(u; () = 0 otherwise, and asymptotically minimize the mean- mean-squared-ens squared-error
the estimation of the
underlying continuous density. A number of methods have been proposed for selecting the smoothing bandwidth [19-21], but they often have asymptotic justifications, and can be unreliable for small N.
Attractively, regularized PFs resample particles #1 ~ m(x; xt Wi .3) from the continuous mixture density rather than the discrete particle set. By resampling from continuous distribution, regularized PFs ensure that no duplicates exist in the resampled particle set, increasing diversity while still concentrating particles in regions of the state space with high posterior probability. Our proposed mixture density PF generalizes regularized PFs, using bandwidths 3 that are tuned jointly with discriminative models of the state dynamics and observation likelihoods.
3 Conditional State Estimation via Discriminative Particle Filters
While classical inference algorithms like KFs and PFs are effective in many domains, they require faithful generative models. human expert must typically design most aspects of the state dynamics and observation likelihoods, and algorithms that use PFs to learn generative models often struggle to scale beyond low-dimensional, parametric models [22]. For complex systems with sigh-dimensional observations like images, learning generative models of the observation likelihoods is often impracti- cal, and arguably more challenging than estimating latent states given an observed data sequence. We instead learn discriminative models of the distribution of the state conditioned observations (see Fig. 1), and replace manually engineered generative models with deep neural networks learned from training sequences with (potentially sparse) latent state observations.
Discriminative Kalman Filters. Discriminative modeling has been used to learn conditional variants of the KF [23, 24] where the posterior is parameterized by a Gaussian state space model, and state transition and observation emission models are produced from data by trained neural networks. Like conditional random field (CRF) [25] models for discrete data, discriminative KFs do not learn likelihoods of observations, but instead condition on them. Discriminative KFs desirably learn a posterior covariance, and thus do not simply output state prediction like conventional recurrent neural networks. But, they are limited to unimodal, Gaussian approximations of state uncertainty.
Discriminative Particle Filters. It is attractive to integrate similar discriminative learning principles with PFs, but the technical challenges are substantially greater due to the nonparametric particle representation of posterior uncertainty, and the need for particle resampling to maintain diversity.
Like in generative PFs, discriminative PFs update the particle locations using dynamics model as in Eq. (1). Unlike classic PFs, the particle weights are not updated using a generative likelihood function. Instead, discriminative PFs compute new weights w.() using measurement model function f(2,; ye) trained to properly account for uncertainty in the discriminative particle posterior:
w (1) x E(x ; ye (e (8)
(5)
Here E(r, 31)) is a (differentiable) function optimized to improve the accuracy of the discriminative particle posterior, rather than generative likelihood.
Creating a learnable discriminative PF requires parameterizing the dynamics and measurement models as differentiable functions, such deep neural networks. This is straightforward for the measurement model (xx yt), which may be defined via any feed-forward neural network architecture like those typically used for classification (see Fig. 2). For the dynamics model, the neural network does not simply need score particles; it must be used for stochastic simulation. Using reparameterization [14- 16], dynamics simulation is decomposed as sampling from standard Gaussian distribution, and then
3

---

Mosture Deretty Particla Pileer
Adaptiie- Mexture Denetyy Particle Piltr
Desammic Model
Meassrement Model
Vil 2
Reaxpping
Trandkera
Pertefr Excader
Arliee Exeeder
Pariid Bacader
Oturrvation Excador
Laureed
HenrannDer Madel
Sandwidd
Hetwork
Hetward
hetwook)
Trensdom
Figure 2: Left: Our MDPF method showing the various cub-oompnennts. Middle: Our A-MDPF method with decoupled measurement models and bandwidths, Right: Dynamies and measurement model structures used in MDPF and A-MDPF. The dynamics and measurement models composed of several neural networks well as some fixed transforms, which convert the angular state dimensions of particles into a vector representation.
transforming that sample using a learned neural network, possibly conditioned on action a,:
xx( = f(n (6) {1) (.))
? (N, I).
(6)
The dynamics model f a₂) is a feed-forward neural network that deterministically processes noise 7), conditioned on 24-1 and a, to implement the dynamical sampling of Eq. (1). The neural network f may be flexibly parameterized because discriminative particle filters only require simulation of the dynamical model, not explicit evaluation of the implied conditional state density.
3.1 Prior Work on Discriminative Particle Filters
Specialized, heuristic variants of discriminative PFs have been previously used for tasks like robot localization [26] and multi-object tracking [27]. Principled end-to-end learning of a discriminative PF requires propagating gradients through the PF algorithm, including the discrete resampling step. Zhu et al. [28] explore replacing the PF resampling step with learned (deterministic) particle transform, but find that exploding gradient magnitudes prohibit end-to-end training.
Srancated-Gradiee Particle Filter (TG-PF). The first so-called "differentiable" particle filter [5] actually treated the particle resampling step as a non-differentiable discrete resampling operation, and simply truncated all gradients to zero when backpropagating through this resampling. Though simple, this approach leads to biased gradients, and often produces ineffective models because back- propagation through time (BPTT (29]) is not possible. Perhaps due these limitations, experiments in Jonschkowski et al. [5] assumed a simplified training scenario where the ground-trute dynamics are known, and only the measurement model must be learned.
Discrete Importance Sampling Particle Filter (DIS-PF). Šcibior et al. [6] propose a gradient estimator for generative PFs that may also be adapted discriminative PFs. They develop their estimator by invoking prior work on score-function gradient estimators [30], but we provide simpler derivation via importance sampling principles. Instead of discretely resampling particles as in Eq. (3),
a separate set of weights (c) is defined and used to generate samples:
(7) To account for discrepancies between these resampling weights v(t) and true weights w(,, the resampled particle weights wi(') are defined via importance sampling. The choice of (c) critically impacts the effectiveness of the resampling step. To maintain the standard PF update of Eq. (3), we
(6) It (1)
Cat(!!") (1) ..... .....,((()) (()))
(4) set u = = yielding the following resampled particle weights and associated gradients:
VaN (3)
W
wi
1,
Voul
(8)
=w
Intuitively, particle locations are resampled according to the current weights in the forward pass of DIS-PF. Then when computing gradients, perturbations of the observation and dynamics models are accounted for by changes in the associated particle weights.
A drawback of this discrete importance sampling is known as the ancestor problem: since the resampled particle set is constructed from a subset of the pre-resampled particles, no direct gradients exist for particles that were not chosen during resampling, they are only indirectly influenced by the weight normalization step. This increases the DIS gradient estimator variance, slowing training.
Soft Resampling Particle Filter (SR-PF). Karkus et al. [7] propose a soft-resampling (SR) mixing of the particle weights with discrete uniform distribution before resampling: (i) (1- --))w + +/NN.

---

Viewing particle locations as fixed, SR-PF evaluates the resampled particle weights and gradients as
W (
V Voll V (1
W
(9)
(1 1) A/N
X) 1 + A/N
While the SR-PF weight update is differentiable, it does not avoid the ancestor problem. By re- sampling low-weight particles more frequently, SR-PF will degrade overall PF performance when N is not large. More subtly, the gradient of Eq. (9) has (potentially substantial) bias: it assumes that perturbations of model parameters influence particle weights, but not the outcome of discrete resampling (7). Indeed in some experiments in [7], smoothing is actually disabled by setting À 0.
Concrete Particle Filter (C-PF). The Gumbel-softmas or Concrete distribution [2] approxi- mates discrete sampling by interpolation with continuous distribution, enabling reparameterized gradient estimation. Each resampled partiele is a convex combination of the weighted particle set:
N
cxp((lg((w)) Gtz)/A)
O33 E exp (log((lo((v)) (*) + Gaa/Aa
Gi ~ Gumbel.
(10)
j=1
Gradients of (10) are biased with respect to discrete resampling due to the non-learned "temperature" hyperparameter À > O. When À is large, the highly-biased relaxation will interpolate between modes, and produce many low-probability particles. Bias decreases as X + O, but in this limit the variance is huge. Even with careful tuning of X, Concrete relaxations are most effective for small N. Maddison et al. [32] focus on models with binary latent variables, and find that performance degrades even for N 8, far smaller than the N needed for practical PFs. While we provide C-PF as a baseline, we are unaware of prior work successfully incorporating Concrete relaxations in PFs.
Optimal Transport Particle Filter (OT-PF). OT-PF [8] modifies PFs by replacing stochastic resampling with the optimization-basee solution of an
optimal transport (OT)
problem. OT seeks probabilistic correspondence between particles x; with weights (:) as in (5), and a corresponding set of uniformly weighted particles, by minimizing a Wasserstein metrie W2
min a
11
Alog
S.L.
04j
1 N
0:j (9) (11)
N-1 (3)
Entropy regularization is required for differentiability, and OT-PF accuracy is sensitive to the non- leamned hyperparameter A > O. OT-PF uses this entropy regularregul mapping to interpolate between particles: a! = Σj= Nā. This assignment approximates the results of true stochastic resampling in the limit as N oo, but lacks accuracy guarantees for moderate N.
OT-PF solves a regularized OT problem which relaxes discrete resampling, producing biased gradients that are reminiscent (but distinct) from C-PF. Minimization of Eq. (11) via the Sinkhorn algorithm [33] requires (N2) operations. This OT problem must be solved at each step of both training and test, and in practice, OT-PF is substantially slower than all competing discriminative PFs (see Fig. 10).
Stable Gradient Estimation for Mixture Model Resampling
Training our discriminative mixture density particle filter (MDPF) requires unbiased and low- -wrrinreece estimates of gradients of samples from continuous mixtures. Mixture sampling classically decom- posed into discrete (selecting a mixture component) and continuous (sampling from that component) steps. The Gumbel-softmax or Concrete distribution [31, 32] provides a reparameterizable relaxation of discrete resampling, but may have substantial bias. Some work has also applied OT methods for gradient estimation in restricted families of Gaussian mixtures [35, 36). We instead develop importance-sampling estimators that are unbiased, computationally efficient, and low variance.
Implicit Reparameterization Gradients. Direct reparameterizztioon of samples, as used for the latent Gaussian distributions in variational autoencoders [14-16], cannot be easily applied to mixture models. Reparameterized sampling requires an invertible standardization function (2) = c that transforms sample 2, drawn from distribution parameterized by to auxiliary variable independent of 6. For mixture distributions, the inverse of Sz)2)) cannot be easily computed.
The implicit reparumeterization gradients (IRG) estimator [37, 38] avoids explicit inversion of the standardization function Sz).). Using implicit differentiation, reparameterization can be achieved so long computable (analytically numerically) and invertible S (2) exists, without the need to
5

---

A
Aen BLBS
0.0
Ao -3755
Figure 3: Left: Example changes in samples from ID mixture model with two Epanechnikov [34] components. Under IRG, changes in the mixture are shown by dramatic sample shifting, where some samples must change modes to account for the changes in the relative weights of each mixture mode. Explicitly plotting the particle transformation induced by IRG reveals a discontinuity (bottom). In contrast, IWSG smoothly reweights samples. Right: Example changes in samples from a 2D mixture of two Gaussians. IRG again induces large shifts as particles change modes, as demonstrated by the vector field (bottom).
explicitly compute its inverse. IRG gradients are expressed via the Jacobian of (z)):
V = (V=ss(() &
(12)
For univariate distributions, the cumulative distribution function (CDF) M²(2) is a valid standardiza- tion function. For higher dimensions, the multivariate distributional transform [39] is used:
(z)
| 22))..... M4
MZ | (2..)...--1)).
(13)
While IRG may be applied to any distribution with continuous CDF, and has been explicitly suggested (without experimental validation) for mixtures [37], we show that may have enormous variance.
Importance Weighted Sample Gradient Estimator. We propose a novel alternative method for computing gradients of samples from continuous mixture distribution. Our importance weighted sample gradient (IWSG) estimator employs importance sampling for unbiased gradient estimation. IWSG is related to the DIS-PF gradient estimator for discrete resampling [6]. but instead consider continuous mixture distributions with arbitrary component distributions. This resolves the ances-
try problem present in Šcibior et al. [6], since particle locations could be sampled from multiple overlapping mixture components, whose parameters will all have non-zero gradients.
Formally, we wish to draw samples (0)) ~ m(z Φ) from mixture with parameters Φ. Instead of sampling from m(z Φ) directly, IWSG samples from some proposal distribution 2 4(2). Importance weights (+) and associated gradients, for these samples then equal
( (² ())
V ())(2( (
(ow
(14)
g(((1)
(((())
Setting the proposal distribution q(2) m(= do) to be mixture model with the "current" parameters Φo at which gradients are being evaluated, the IWSG gradient estimator (14) becomes
(2)
②)) à=
Vaw(( vom(z()) Φ)) 0))o=oo
(15)
m (2(1))) @o)
m(2(0) | (0)
While current samples are given weight one because they are exactly sampled from a mixture with the current parameters Φu, gradients account for how importance weights will change mixture parameters & deviate from Φ0. Note that gradients are not taken with respect to the proposal q(2) = m(2 do) in the denominator of w()), since sample locations z()) are not altered by gradient updates; changes in the associated importance weights are sufficient for unbiased gradient estimation.
Instability Implicit Reparameterization Gradients. Reparameterization methods capture changes to a given distribution p(z[Ø), and thus to its parameters ø, by shifting the samples drawn from that distribution. When p(=(Ø) is unimodal, this induces a smooth shift the samples. When p(z(Ø) contains multiple non-overlapping modes, these shifts are longer smooth samples

---

IRGPY
Trancaled Grddient
Nature
Truth Learond Biziovade of
Leamed Belisato OTM
Gradieres Bctinado S
Leameel Vetiesato of
Gradivids Bctesato far
Gradieat Bidisade for
Thiniing Iteration
Trasing Btnation
Traniig Itrrtiion
Thaining Ituration
Traniing Paration
Traising Hteration
Figure 4: A simple temporal prediction problem where IRG has highly unstable gradient estimates. We use N= = 25 particles when training IRG-PF, IWSG-PF, and truncated gradients (TG-PF). We show the learned values for parameters B, w2, C1 and their gradients during training. IRG produces unstable gradients which prevent parameters from converging all, but IWSG allows for smooth convergence of all parameters. IWSG faster than biased TG, and nearly as effective as (expensive, and for general models intractable) mixture KF.
jump from one mode another; see Fig. 3. For mixtures with finitely supported kernels (like the Epaaechnikov) and non-overlapping modes, discontinuities exist in (c). and IRG estimates may have infinite variance. These discontinuities correspond to samples shifting from one mode to another, and the resulting non-smooth sample perturbations will destabilize backpropagation algorithms for gradient estimation. While the inverse CDF is always continuous for Gaussian mixtures, it may still have near-infinite slope when modes are widely separated, and induce similar IRG instabilities.
In contrast, our IWSG estimator reweights particles instead of shifting them. This reweighting is smooth and does not suffer from any discontinuities. To illustrate this, we create simple discriminative PF with linear-Gaussine Gaussian dynamics (16) and a bimodal observation likelihood (17):
p 1r-1; 0x) (1r AAx-1 + Ba,, o²),
(16)
ply x4) w, Nyr Ciz, + + 7²) + wyNorr( C27, + 72)
(17)
Exact posterior inference for this model is possible via mixture KF [11, 40] which represents the state posterior as Gaussian mixture. The number of mixture components grows exponentially with time, motivating PFs for long-term tracking, but remains tractable over a few time steps.
Using a fixed dataset and gradient descent, we train discriminative PFs using the IRG estimator, our IWSG estimator, well as with biased gradients that are truncated at each resampling step [5]. We initialize with perturbed parameter values and aim to learn the values of all dynamics and likelihood parameters, except for and 7 which are fixed to their true values. See Appendix for details.
Fig. 4 shows the estimates and gradients for a subset of the learned parameters, and several gradient estimators. IRG's unstable gradients prevent convergence of all parameters. For training (generative) marginal PFs [41] which approximate marginals with mixtures, Lai et al. [42] found that IRG gradient variance was large that biased estimators converged faster, but provided no detailed analysis. In contrast, IWSG converges smoothly for all parameters, and more rapidly than truncated gradients.
5 Mixture Density Particle Filters
Our mixture density particle filter (MDPF) is discriminative model inspired by regularized PFs, where resampling uses KDEs centered on the current particles. We apply IWSG to this mixture resampling, achieving unbiased and low-variance gradient estimates for our fully-differenniable PF.
MDPF does not incorporate human-specified dynamics or measurement models; some prior work assumed known dynamics to simplify learning [5, 7]. These models are parameterized as deep neural networks (NNs), and trained via stochastic gradient descent to minimize the negative-log- rogative-log-likelihooo likelihood of (potentially sparsely labeled) states I,, given observations Yt and (optionally) actions az As shown in Fig. 2, dynamics and measurement models are flexible composed from smaller NNs. Our MDPF places no constraints on the functional form of NNs, allowing for modern architectures such as CNNs [43] and Spatial Transformers [44] to be used. See Appendix for implementation details.
The MDPF uses KDE mixture distributions as in Eq. (4) to define state posteriors used to evaluate the training loss, as well as resampling. A bandwidth parameter /3 is thus required for each state dimension. Rather than setting / via the classic heuristics discussed in Sec. 2.2, we make 3 learnable parameter, optimizing it using end-to-end learning along with the dynamies and measurement models, This contrasts with prior work including SR-PF [7] and C-PF and OT-PF [8], which all include relaxation hyperparameters that must be tuned via multiple (potentially expensive) training trials.
We also extend the MDPF by decoupling the mixtures used for particle resampling and posterior state estimation. Using two separate measurement models, compute two sets of particle weights w!!")
7

---

Basting Ouy
chr
100re
Deepeied Modro
Donpesind Haze
F.
Figure 5: Box plots showing median (red line), inter- qutr--uartile (colored box) and range (whiskers) over several training runs on the Bearings-Only (11 runs) and Deepmind-Maze (5 runs) tracking tasks. MDPF and A-MDPF consistently perform well on both the NIL and RMSE metries. Our IWSG estimator is critical: IRG-MDPF performs very poorly due to unstable gradients, TG-MDPF is inconsistent and sometimes becomes trapped in local optima, and baselines with biased gradient estimators typically have inferior performance. LSTM achieves low RMSE in the Deepmind-Man Maze task however we find it simply propaapts using the noisy actions. DIS-PF performs well for Maze but with larger variability which can sometimes lead to worse performance than our more stable A-MDPE. Note that IRG-MDPF does not support Epanechnikov kernels, which induce discontinuous CDFs.
and w one for resampling and the other for posterior state estimation; see Fig. 2. Each distribution is also given separate bandwidth 3, 3. This adaptive mixture density PF (A-MDPF) allows the uncertainty in the estimation of the current latent state, and the degree of exploration in the particle resampling step, be decoupled and separately optimized during training. Note that training of separate posterior and resampling distributions is impossible for PFs that truncate temporal gradients; it is only feasible due to our unbiased and low-variance IWSG estimator of resampling gradients.
6 Experiments
We evaluate our MDPF and A- on a variety state estimation tasks in complex, but simulated, environments. We compare TG-PF [5], SR-PF [7], OT-PF [8], DIS-PF [6], as well as C-PF which uses the Concrete distribution [32, 31] to relax discrete resampling. We also compare to variants of MDPF: Truncated- -raaiet--DrrddeeMM- (TG-MDPF) is similar but stops gradients each particle resampling, and IRG-MDPF replaces our IWSG estimator with the unstable IRG estimator. Finally, we compare to a LSTM [45] which produces point predictions via uninterpretable latent states.
For all tasks, we estimate posterior distributions of a 3D (translation and angle) state of a robot, J, (x,, y, 0). We use Gaussian kernels for position components, and von Mises kemels for angular components, of all KDEs. As is common in recurrent neural network (RNN) training, we employ conservative gradient clipping [46], as well as Mancated-BPTT [47] where gradients are propagated for 4 time-steps. All methods are initialized as the true state with added noise.
We train and evaluate MDPFs to minimize the following negative-log -oggative-lo-liikllhoooood (NLL) loss: (x ()) w ,))).
CNLL
Log(m(z;
(18)
LET
We use sparsely labeled training data to highlight the need for effective multi-time-step gradients during training. Dense labeling of data is often expensive, so many real-world datasets have similarly sparse labels. During training. we label true states every 4th time-step (T = {4, 8, 12... and use densely labeled datasets (T 21,2,3.. .)) for evaluation. See the Appendix for details.
After training to minimize NLL, we optionally refine our MDPFs to minimize the weea-squured-erron (MSE) of the weighted particle mean, and report the induced Root-MSE (RMSE). Baseline methods do not estimate continuous posteriors, and thus Eq. (18) cannot be used for training. Instead we train and evaluate these methods with MSE and RMSE, before freezing the dynamics and measurement models. Then we construct KDEs, and optimize a bandwidth to report NLL values for comparison. Bearings- Only Tracking. In the bearings only tracking task, we track the state of a variable-velocis using noisy bearing observations from a radar station the generated
It ~ a Uniform( (- -, T) + (1 - a)- VonMises ((MMMisss(((())).....)))))) (xx), where (xx)) is the true bearing and a 0.15. Actions are not available. We use 5000 training and 1000 validation trajectories of length T = 17, and 5000 evaluation trajectories of length T = 150.
8

---

A-M00
Figure 6: Example trajectories from the Bearings-Only tracking task. The radar station (green circle) produces bearing observations (green line) from radar station to the car (red arrow). Note that observations are noisy and occasionally totally incorrect (sec 18). We plot the posterior distribution of the current state (blue cloud), the mean particle (yellow arrow), and the N = 25 particles (black dots). A A-MDPF is able to capture multiple modes in the state posterior (see 15, 17, 18, 19) while successfully tracking the true state of the car. Other models (DIS-PE, OT-PF, C-PF) spread their posterior distribution due poor tracking of the state.
Figure 7: Example trajectories (rows), with observations in corner, from the House3D task using A-MDPE. Top: Initialized with N = 50 particles set to the true state with moderate noise, the posterior (blue cloud) closcly tracks the robot. Middle: Multi-moda initialization with N 150 particles (bluc dots). Bottom: Naive initialization with observations have been seen to disambiguate the true state.
1000 particles drawn uniformly. A-MDPF maintains multiple modes until enough
Tike State
Moan Particle
Deaulty
Figure 8: Example posteriors three time steps (rows) House3D tracking, for several discriminative PFs given the same low-noise initialization. We show the current true state and state history (red arrow and dashes), the posterior distribution of the current state (blue cloud, with darker heing higher probability), and the estimated mean state and history (yellow arrow and dashes). Several baselines completely fail to track the robot.

---

Each method is evaluated and trained 11 times using N = 25 particles. To further highlight the stability of our IWSG estimator, we also train each method (except for IRG-MDPF) with the Epanechnikov kernel replacing the Gaussian in the KDE. Fig. 5 shows statisties of performance across 11 training runs. The very poor performance of IRG-MDPF due to unstable gradients; without aggressive gradient clipping, IRG-MDPF fails to optimize at all. The benefits of TWSG are highlighted by the inferior performance of TG-MDPE. Stable multi-time- time-step gradients clearly benefit learning and are enabled by IWSG. Qualitative results are shown in Fig. 6.
Deepmind-Man Maze Tracking. In the Deepmind-Maze tracking task, adapted from [5], we wish to track the 3D state of robot as it moves through modified versions of the maze environments from Deepmind Lab [48] using noisy odometry actions, and images (from the robot's perspective) as observations. We alter the experimental setup of [5] by increasing the noise of the actions by five times, and using N 25 particles for training and evaluation. We train and evaluate from the 3 unique mazes separately, using 5000 trajectories of length T = 17 from each maze for training, and 1000 trajectories of length T 99 for evaluation. We show results over multiple training and evaluation runs in Fig. 5, and include qualitative results in the Appendix.
trajectories
House3D Tracking. This 3D state tracking task is
Table 1: Mean + inter-quartile of evaluation metrics for all evaluation trajectories from the House3D task.
adapted from [7], where robot navigates single-level apartment environments from the SUNCG dataset [49] using the House3D simulator [50], with noisy odom- etry as action and RGB images as observations (see Fig. 7). Floor plans are input into measurement model via a modern Spatial Transformation Network architec- ture [44]. Training data consists of 74800 trajectories (length T = 24) from 199 unique environments, with evaluation data being 820 trajectories (length T 100) from 47 previously unseen floorplans. We train and evaluate with N 50 particles.
Method
NLL
RMSE
LSTM
12.54 + 3.43
48.51 27.16
TG-PF
15.50 + 2.21
206.32 = 109.44
OT-PF
14.87 + 2.31
159.57 = 109.65
SR-PF
14.92 -: 2.57
191.32 129,28
DIS-PF
15.06 2.02
212.58 = 133.69
TG-MDPF
8.38 1 2.53
35.77 = 16.67
MDPF
8.18 + 2.72
30,59 = 12.98
A-MDPF
8.09 + 3.49
30.51 = 12.32
Due to the larger computational demands of House3D, we train each method once, reporting results in Table 1. Interestingly we find that TG-PF, OT-PF, SR-PF, and DIS-PF are unable to learn useful dynamics or measurement models. The LSTM model achieves better performance, but looking deeper we discover that it ignores observations completely, and simply blindly propagates the estimated state using actions with good dynamics (i.e., dead- reckoning). This strategy is effective for noise-free actions. but in reality the actions are noisy and thus the LSTMs estimated state diverges from the true state after a moderate number of time-steps. In contrast, our MDPF methods are able to learn good dynamics and measurement models that generalizes well to the unseen environments.
We also investigate MDPFs capacity for multimodal tracking in Fig. 7, by initializing A-MDPF with particles spread throughout the state space. When initialized with random particles or with particles clustered around regions with similar observations, A-MDPF does not have sufficient information to collapse the posterior and thus maintains multiple posterior modes. As the robot moves, more evidence is collected, allowing A-MDPF collapse the state posterior to fewer distinct modes, before eventually collapsing its estimate into one mode containing the true state. Note that such estimation of multiple posterior modes is impossible for standard RNNs like the LSTM.
Limitations. A known limitation of all PFs is there inability to scale to very high-dimensional states. Sparsity increases as the dimension grows, and thus more particles (and consequently computation) are required to maintain the expressiveness and accuracy of the state posterior. Our MDPF is not immune to this issue, but we conjecture that end-to to-deoddddddddee training of discriminative PFs will allow particles to be more effectively allocated within large state spaces, scaling better than classic PFs.
7 Discussion
We have developed a novel importance-sampling estimator for gradients of samples drawn from continuous mixture model. Our results highlight fundamental flaws in the application of (implicit) reparameterization gradients to any mixture model with multiple modes. Applying our gradient estimator to the resampling step of regularized discriminative PF, we obtain a fully end-to-ent to-end differentiable MDPF that robustly learns accurate models across multiple training runs, and has much greater accuracy than the biased PFs proposed in prior work. While our experiments have focused on synthetic tracking problems in complex virtual environments, future applications of our MDPF to real-world vision and robotics data are very promising.
10

---

Acknowledgements
We thank Alexander Ihler for helpful discussions, and Zain Farhat for his assistance with the LSTM baseline. This research supported in part by NSF Robust Intelligence Award No. IIS-1816365 and ONR Award No. N00014-23-1-2712.
References
[1] Neil Gordon, David Salmond, and Adrian FM Smith. Novel approach to nonlinear/non- gaussian bayesian state estimation. IEE proceedings F-Radar and Signal Processing, 1993.
[2] K. Kanazawa, D. Koller, and S. Russell. Stochastic simulation algorithms for dynamic proba- bilistic networks. In UAI 11, pages 346-351. Morgan Kaufmann, 1995.
[3] A. Doucet, N. de Freitas, and N. Gordon, editors. Sequential Monte Carlo Methods in Practice. Springer-Verlag New York, 2001.
[4] M. Arulampalam, S. Maskell, N. Gordon, and Clapp. tutorial on particle filters for online nonlinear/non-Gane -Gaussian Bayesian tracking. IEEE Trans. Signal Proc., 50(2):174-188, February 2002.
[5] Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End- to-end learning with algorithmic priors. Proceedings of Robotics: Science and Systems (RSS), 2018.
[6] Adam Seibior, Vaden Masrani, and Frank Wood. Differentiable particle filtering without modi- fyyng the forward pass. International Conference on Probabilistic Programming (PROBPROG), 2021.
[7] Peter Karkus, David Hsu, and Wee Sun Lee, Particle filter networks with application to visual localization. Conference Robot Learning (CORL), 2018
[8] Adrien Corenflos, James Thornton, George Deligiannidis, and Amaud Doucet. Differentiable particle filtering via entropy-regue reguoay-egglariized optimal transport. International Conference on Machine Learning (ICML), 2021.
[9] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASurnJluraal of Basic Engineering, 1960.
[10] T. Kailath, A. H. Sayed, and B. Hassibi. Linear Estimation. Prentice Hall, New Jersey, 2000.
[11] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, 2005. ISBN 0262201623.
[12] S. Julier and K. Uhlmann. Unscented filtering and nonlinear estimation. Proc. IEEE, 92(3): 401 401-422, March 2004.
[13] Victor Elvira, Luca Martino, and Christian Robert. Rethinking the effective sample size. International Statistical Review, 2022.
[14] Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In ICML., page 11-1791-1799, 2014.
[15] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. International Confer- ence on Learning Representations (ICLR), 2014.
[16] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, volume 32, pages 1286-128. 2014.
[17] N. Oudjane and C. Musso. Progressive correction for regularized particle filters. International Conference Information Fusion, 2000.

---

[18] C. Musso, N. Oudjane, and F. Le Gland. Improving regularized particle filters. In A. Doucet, N. de Freitas, and N. Gordon, editors, Sequential Monte Carlo Methods in Practice, pages 247-271. Springer-Verlag, 2001.
[19] B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, 1986.
[20] Adrian W Bowman, An alternative method of cross-validation for the smoothing of density estimates. Biometrika, 1984.
[21] M Chris Jones, James S Marron, and Simon J Sheather. A brief survey of bandwidth selection for density estimation. Journal of the American Statistical Association, 1996.
[22] Nikolas Kantas, Arnaud Doucet, Sumeetpal S Singh, Jan Maciejowski, and Nicolas Chopin. On particle methods for parameter estimation in state-space models. Statistical Science, 30(3): 128-351,22 2015.
[23] Tuomas Haarooja, Anurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop kf: Learning discriminative deterministic state estimators. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[24] Minyoung Kim and Vladimir Pavlovic. Conditional state space models for discriminative motion estimation. IEEE International Conference on Computer Vision, 2007.
[25] John D. Lafferty, Andrew McCallum, and Fernando Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. International Conference on Machine Learning (ICML), 2001.
[26] Benson Limketkai, Dieter Fox, and Lin Liao. Crf-filters: Discriminative particle filters for sequential state estimation. IEEE International Conference on Roboties and Automation (ICRA), 2007.
[27] Rob Hess and Alan Fern. Discriminatively trained particle filters for complex multi-object tracking. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2009.
[28] Michael Zhu, Kevin P. Murphy, and Rico Jonschkowski. Towards differentiable resampling. ArXiv, abs/2004. 11938, 2020.
[29] Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 1990.
[30] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 1529-1538, 2018.
[31] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- International Conference on Learning Representations (ICLR), 2016.
[32] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. International Conference on Learning Representations (ICLR), 2016
[33] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 2013.
[34] V. A. Epanechnikov. Non- parametric estimation of a multivariate probability density, Theory of Probability & Its Applications, 1969.
[35] Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparameterizztioo trick. International Conference on Machine Learning (ICML), 2018.
[36] Martin Jankowiak and Theofanis Karaletsos. Pathwise derivatives for multivariate distributions. International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.
[37] Alex Graves. Stochastic backpropagation through mixture density distributions. arXiv preprint arXiv:1607.05660, 2016.
12

---

[38] Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Advances Neural Information Processing Systems (NeurIPS), 31, 2018.
[39] Ludger Rüschendorf. Copulas, Sklar's Theorem, and Distributional Transform. Springer Berlin Heidelberg, 2013.
[40] D. Alspach and H. Sorenson. Nonlinear bayesian estimation using Gaussian sum approximations. IEEE Transactions on Automatic Control, 1972.
[41] Mike Klaas, Nando de Freitas, and Arnaud Doucet. Toward practical N2 Monte Carlo: The marginal particle filter. In Proceedings of the Twenty-Fine First Conference on Uncertainty in Artificial Intelligence, pages 308-315, 2005.
[42] Jinlin Lai, Justin Domke, and Daniel Sheldon. Variational marginal particle filters. International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
[43] Keiron O' Shea and Ryan Nash. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458, 2015.
[44] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
[45] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[46] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty training recurrent neural networks. International Conference on Machine Learning (ICML), 2013.
[47] Herbert Jaeger. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the" echo state network" approach. GMID-Forrchungsseettrn Informatiionseehhik Bonn, 2002.
[48] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefraneq. Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv: 1612.0800, 2016.
[49] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[50] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint trXiv:/801.02200) 2018
[51] Marco Frei and Hans R. Künsch. Mixture ensemble kalman filters. Computational Statistics and Data Analysis, 2013.
[52] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
13

---

A Parameterization of the Neural Dynamics and Measurement Models
In our implementation of MDPF and A-MDPF, we define the dynamics and measurement models as set of small learnable neural networks, Fig. 2. The dynamics model uses two encoder networks which encode the particles and the actions into latent dimensions. The particle encoder is applied individually to each particle, allowing for parallelization. The encoded actions and particles are used as input, along with zero-mean unit variance Gaussian noise, into residual network to produce new particles.
The measurement model is also comprised of several small neural networks. Particles and observations are encoded into latent representations before being used as input into a simple feed-forward network. Our method makes no restrictions on the network architectures allowing modern architectures be used. For high dimensional observations, such images, for more complex problems a sophisticated architecture such as Convolutional Neural Networks [43] or Spatial Transformer Networks [44] can be implemented.
Since particle states are interpretable, they are often a representation not suitable for neural networks. For example angles contain discontinuities (at and 2T) which hinders their direct with neural networks. To address this, non-learned transforms are applied to the particles, converting them into representations that are better suited for neural networks. In the of angles transforms convert angles to and from unit vectors:
T(0)= (cos(0), (in(
Tu, v) atan2(u, v)
(19)
B Synthetic Learning Problem Details
In section 4, simple synthetic PF learning problem was presented to illustrate the smooth and stable gradient estimates produced by our Importance Weighted Samples Gradient (IWSG) estimator while also highlighting the instability of reparameterization methods when applied to mixture models. Here we give additional details about this learning problem.
In this learning problem we define several regularized PFs with linear Gaussian dynamics and bimodal observation likelihoods:
p(ax xt-1, = Norm(Aa, + Bai,a?)
(20)
p(y: Ix) Normm(Cii 72, + uggormm(222 72)
(21)
For each PF method, we set the dynamics and measurement models to the true linear Gaussian dynamics and bimodal observation with perturbed parameters with the goal of learning the true values of the parameters. In this problem we hold and y fixed and only perturb and learn the other parameters: A, B, C1, Cz. C1, C22 Wi and W22
We train three variants of the regularized PF with differing gradient estimation procedures for resampling. IWSG-PF uses our IWSG estimator, IRG-PF uses the implicit reparameterization gradients estimator, and TG-PF does not use any gradient estimator for mixture resampling; instead it simply truncates gradients to zero at resampling steps. We compare the stability and convergence of these three methods when learning the parameter values. For this problem, we decouple the resampling and posterior bandwidth parameters as our A-MDPF model. For resampling we set 0.05, and for the posterior we set 3 = 0.5.
We also train a mixture Kalman Filter (KF) [11, 40] with the same true dynamics and measurement models, also initializing with perturbed parameters. Since this model has linear dynamics with Gaussian noise, and each component of the observation distribution mixture is also linear and Gaussian, can apply mixture KFs compute the exact posterior a mixture of Gaussians. Dynamics are applied to each Gaussian component of the mixture using the KF update equations [1 []. keeping the number of components the same. When applying the measurement model, we simply multiply the KF mixture with the observation likelihood mixture, increasing the number of mixture components in the KF posterior. The number of components grows exponentially with time, so the mixture KF is not practical general inference algorithm, but the computation remains tractable for short sequences. Special care must be taken when computing the weights of each mixture component after applying the observation likelihoods; adapt the approach of Frei and Künsch [5
14

---

To make the learning problem identifiable and to enforce the constraints + W2 = 1, w1 2 0, W2 2 O, we parameterize wi and w2 within the PF and KF models via the logistic of single learnable parameter o:
W1 + exp(e)'
1 exp(-u)
(22)
To learn the dynamics and measurement model parameters, we setup synthetic training problem with dataset of 1000 sequences, each length 5 time-steps. Using stochastic gradient descent with batch size 64, we train to minimize the negative log-likelihood loss of the true state given the posterior distribution at only the final time-step. For PF models, we use kernel density estimation [19] to estimate mixture distribution using the final weighted particle set. For the mixture K.,, the posterior distribution is simply a mixture model and no further processing is needed to compute the loss.
Fig. 9 shows the convergence and gradient estimates for all of the learned parameters when using the different gradient estimators. IRG's unstable gradients prevents convergence all parameters, while IWSG converges smoothly and more rapidly than truncated gradients. Further we see that for parameters e₁ and c2, the TG-PF method does not converge to the true value. TG-PF truncates gradients at the resampling, resulting in biased gradients. When the loss is computed at the final time, only information gained from that final sime is present when updating the parameters, resulting in biased updates.
15

---

Learned Estimate
E.iimated Grudient Value
Gradients Estimate for
1.0
Training Baration
Training Heration
1.25
Learned Estimate
Extinated Gradien Value
Gradients Estimate
0.50
107
30
3P Training
101
14
30
Training
Learned Estimate w
Extimotedd Gradion Value
Gradients Estimate for
0.6
-3
Training Reration
Training ftorationn
2.00
C2
Extinutel Gradkint Valeo
53-
Training
Training
Learned Estimate of C2
Extimaked Grudient Value
Gradients Estimate for C
Training Baration
Training Iteration
Estimated Greatient Volue
Gradients
53
Learned Estimate of
Extimated Gradiien Value
Gradients Estimate for C
-03
10 20
32
16P
3.00
Training Baration
Training Baration
Figure 9: A simple temporal prediction problem where IRG has highly unstable gradient estimates. We 25 particles when training IRG-PF, IWSG-PF, truncated gradients (TG-PF). show the learned values for all lcarnable parameters and their gradients during training. IRG produces unstable gradients which prevent parameters from converging at all, but IWSG allows for smooth convergence of all parameters. IWSG is faster than biased TG, and nearly as effective as (expensive, and for general models intractable) mixture KF. TG-PF fails to learn correct values for and eg due to biases from truncating gradients resampling.
B.1 Computation Requirements
The computational requirements for our MDPF and A-MDPF methods are quadratic with the number of partiele during training, O(N2), due to gradient computation. During inference, the computation
16

---

TG MDPF
A-MDPF TG-FF
C-PF
SR-PF
MDPF
DIS-PF
OT-FF
Training
Inference
Time per Batch (Seconds)
10a
10
10a
10"
10-1
10-1
200
400
600
800
1000
0
200
400
600
800
1000
Number of Particles
Number of Particles
Figure 10: Computational time for various particle filters versus the number of particles the Deepmind Maze task. batch size 24 used. During training, the quadratic run-time MDPF and A-MDPF is due to gradient computation. During inference MDPF and A-MDPF have linear run-time requirements. In contrast OT-PF must solve complex optimal transport problem at both training and inference resulting in overall slow run-times.
requirement is linear in the number of particles, O(N), since we mearly have to sample from the resampling mixture distribution rather than computing gradients for the samples. This is highlighted in fig. 10. Other methods except OT-PF are have computational complexity O(N) during both inference and training. OT-PF requires solving optimal transport problem with complexity ( during both training and inference [8]. In practice we find that OT-PF has significant computational requirements resulting in slow run-times both at inference and testing.
C Additional Experiment Results
In this section we give additional qualitative experiment results for the various tracking tasks.
C.1 Bearings Only Tracking Task
In fig. 11 show additional qualitative results for various PF methods where it is apparent that A-MDPF and MDPF both track the true state well whereas other methods have wide posteriors due their poor tracking ability. Figs. 12 13 14, 15, 17 and 18 show additional trajectories for the various PF methods.
17

---

Figure 11: Example trajectory with various PF methods shown for the bearings only tracking task. The radar station (green circle) produces bearings observations (green line) from the radar station to the car (red arrow). Note that observations
noisy and occasionally totally incorrect. Shown is the posterior distribution of the (bluc cloud, with darker heing higher probabiliit), the mean particle (yellow arrow) and the N 25 particle set (black dots). A-MDPF and MDPF both have tight posterior distributions over the true state while maintaining tracking. Other methods such as DIS-PF, OT-PF and C-PF have wide posteriors showing their uncertainty due to their poor ability to track the true state.
Example trajectory using A-MDPF shown for bearings only tracking task. radar station (green circle) produces noisy (and occasionally incorrect) bearings observations (green linc) from the radar station to the car (red arrow). A-MDPF maintains a tight posterior distribution (blue cloud, with darker being higher probability) over the true state (red arrow) over time showing the effectiveness of our method. The mean particle (yellow arrow) and the N 25 particle set (black dots) also shown.
Figure 12:
Figure 13: Another example trajectory using A-MDPF shown for the bearings only tracking task.
18

---

Figure 14: Example trajectory using MDPF shown for the bearings only tracking task. Similar A-MDPF in
figs. 12 and 13, MDPF accurately tracks the state (red arrow) with tight posterior density (blue cloud).
Fow
Figure 15: Example trajectory using DIS-PF shown for the bearings only tracking task. Due to the poorly learned dynamics and measurement models, DIS-PF often loses track of the true state (red arrow) seen in time-steps 18 28. This results in large estimated posterior density (blue cloud) account for the incorrectly placed particles (black dots) caused by poor dynamics particle weighting.
Figure 16: Example trajectory using SR-PF shown for the bearings only tracking task.
19

---

feory oa
Tewms
Figure 17: Example trajectory using OT-PF shown for the bearings only tracking task. Like DIS-PF fig. 15, OT-PF has trouble tracking the truc state (red) due to poorly learned dynamics and measurement models.
Figure 18: Example trajectory using C-PF shown for the bearings only tracking task. C-PF is unable accurately track the true state (red) as shown by relatively non-moving mean particle (yellow arrow).
20

---

C.2 Deepmind Maze Tracking Task
In figs. 19, 20 and 21 we show additional qualitative results for various PF methods for mazes 2 and 3, respectively. Again it is apparent that A-MDPF and MDPF both track the true state well whereas other methods have wide posteriors due to their poor tracking ability.
Figure 19: Example trajectory with various PF methods shown for the Deepmind maze tracking task for maze 1. Shown is the current true state (red arrow), the posterior estimate of the current state (blue cloud, with darker being higher probability). the mean particle (yellow arrow) and the 25 particle (black dots).
Figure 20: Example trajectory with various PF methods shown for the Deepmind maze tracking task for maze 2. Shown is the current true state (red arrow). the posterior estimate of the current state (blue cloud, with darker being higher probability). the particle (yellow arrow) and the 25 particle (black dots).
21

---

Figure Example trajectory with various PF methods shown the Deepmind maze tracking task maze Shown is the current true state (red arrow), the posterior estimate of the current state (blue cloud, with darker being higher probability), the mean particle (yellow arrow) and the N = 25 particle set (black dots).
C.3 House3D Tracking Task
Figs. 22 and 23 show example trajectories for the House3D tracking task using A-MDPF and MDPF respectively. We see that our A-MDPF and MDPF methods both successfully track the true state over time evident by the tight posterior over the true state as well as the closeness of the mean state to the true state. Fig. 24 shows an example trajectory using the LSTM model. is clear that LSTM model is ignoring observations and instead is blind propagating its state estimate using good dynamics and the actions (dead-nec.oning).
later time-steps, the L.STM estimate drifts away from the true state however the predicted state trajectory looks similar to that of the true state, but with growing offset, indicating blind propagation.
Fig. 25 shows qualitative results when using OT-PF. OT-PF is unable to learn good dynamics and measurement models and quickly diverges away from true state. Figs. 26, 27 and 28 give example trajectories for TG-PF, DIS-PF and SR-PF respectively. These methods fail to learn usable dynamics or measurement models with their trajectories essentially being random.
22

---

Partils
Aosteter Deauty
Iess
I+st
Juif
Figure 22: Example trajectory for the House3D tracking task using A-MDPE: The posterior density of the state tight showing confidence A-MDPF has its estimate, The estimated posterior density (blue cloud, with darker being higher probabillity also converges on the true state (red arrow) showing its accuracy. The mean particle (yellow arrow) closely tracks the true state. The observations are shown as inset images in the plots.
Tres State
Mear Partici
Postartor Deaatty
Figure 23: Example trajectory for the House3D tracking task using MDPF. Similar fig. 22, MDPF accurately tracks the true state.
Figure 24; Example trajectory for the House3D tracking task using the LSTM model. The LSTM model does not us observations but rather blind- prodppropggates the estimated state using the available noisy actions. This is seen by the true state (shown in red) and the estimated state (shown in yellow) having similar shapes but with increasing emror.
23

---

Ix32
Figure 25: Example trajectory for the House3D tracking task using the OT-PE OT-PF is unable to learn good dynamics or measurement models and thus the estimated posterior (blue cloud) diverges away from the true state (shown in red) after couple time-steps.
1=22
be
Figure Example trajectory for the House3D tracking using the TG-PF. TG-PF is unable to usable dynamic or measurement model resulting fairly random output. This is reflected in large uncertainty of estimated posterior (blue cloud) as well as the random nature of the mean state (shown in yellow).
Tree Shate
Partill
Posturter Deautty
leat
E-I
I-44
Figure 27: Example trajectory for the House3D tracking task using the DIS-PE. Similar to TG-PF in fig. 26, DIS-PF unable learn usable dynamic measurement model and therefore is unable collapse the estimated posterior density (blue cloud) onto the true state (shown red).

---

Tris Stens
Mean Parbils
Postarer Desdtty
hal'
Figure 28: Example trajectory for the House3D tracking task using the SR-PF. Similar to TG-PF in fig. 26 and DIS-PF in fig. 27, SR-PF is unable learn usable dynamic measurement model.
25

---

be sensitive choice of D Experiment Details
In this section give additional details for the various experiments (tasks). We also provide specific neural architectures used in each task.
D.1 Mean Squared Error Loss
When training the comparison PF methods, we use the Mean-Squared Error (MSE) loss. This is computed as the mean squared error between the sparse true states 8, and the mean particle each time t:
LMSE Σ C
10)
W
2
(23)
LET
Here T are the indices of the labeled true states. During training, sparsely labeled true states are used and thus T = (4. 8, ... During evaluation we use dense labels, T = (1, (,, 2, .....)
For angular dimensions of s; compute the mean particle by converting the angle into unit vector, averaging the unit vectors (with normalization) and converting the mean unit vector back into angle. Further we take special care when computing square differences for angles due the discontinuity present at 0 and 2n.
D.2 Pre-Training Measurement Model
We pre-train the measurement model for each task using the sparse true states. To pre-train, a random state from the dataset is selected. A particle set is constructed by drawing samples from a distribution centered around the select true state and weights for this particle set are computed using the measurement model and the observation associated with the selected true state. A mixture distribution is then fit using KDE with a manually specified bandwidth. Negative Log-Likelihood loss of the true state is used as the training loss. We find pre-training bandwidth so long as it is not too small.
The dynamics model cannot be pre-traie pre-trained because of the lack of true state pairs due to the sparse true state training labels and thus must be trained for the first time within the full PF algorithm.
D.3 Additional General Task and Training Details
Table 2: Hyper- parameters use for various PF methods.
Parameter
Value
SR-PF [7] À
0.1
OT-PF [8] À (Bearings Only)
0.5
OT-PF [8] X (Deepmind)
0.01
OT-PF [8] À (House3D)
0.01
OT-PF [8] scaling
0.9
OT-PF [8] threshold
le-3
OT-PF [8] max iterations
500
C-PF [31, 32] X
0.5
For all tasks wish track the 3D (translation and angle) state of robot, s (I, y, 0). over time. This state space contains an angle dimension which is not suitable for neural networks. To address this issue, we convert particles into 4D using transform T(s) (z, y, sin( cos( before using them as input into a neural network. Further we convert back into the original 3D representation to retrieve the 3D particle after applying neural networks the particles.
We use Adam [52] optimizer with default parameters, tuning the learning rate for each task. We use batch size of 64 during training and decrease the learning rate by factor of 10 when the Shact-PropggaioonnTTouugg
validation loss reaches plateau. We use truncating the gradients every 4 time-steps.
(T-BPTT) [47],
26

---

For all tasks we use the Gaussian distribution for the positional components of the state and the Von Mises distribution for the angular components when applying KDE. Here 3 is dimension-speciin bandwidth corresponding to the two standard deviations for the Gaussians and concentration for the
Von Mises.
Some methods have non-lcarnable nyper-parameters that must be set. The values of the hyper- parameters are chosen via brief hyper-parameter search with the final chosen values shown in table 2. The hyper-parameler values are the same for all tasks unless otherwise specified. We find the regularization parameter À for OT-PF be sensitive the specific datasets.
We choose to make the dynamics position invariant. This prevents the learned PF models from memorizing the state space based on position but rather forces the learned dynamics to the true dynamics. Position invariance can be achieved by masking out the translational dimensions of the particle before input into the dynamics model.
D.4 Bearings Only Tracking Task
Inpul: Transformed Perticle
Mark out Transiational Companent
Iyper: Oboeration
Fully Connected (8)
Connected ReLU
Connected (1) ReLU
Fully Connected (1)
Fully Connected (8) ReLU
Pull
Fully Connected (8)
Fully Connacted (l)
+ Output
Output
Output
(a) Particle encoder for dy- namics Model.
(b) Particle encoder for mea- surement model.
(c) Observation encoder.
Iquut Frended Oosanation
Iqpur Receded
Donclornde
Fully Cooracted Ht ReLl
Fully Creracee ReLl
Siyncl
Scale cubovtooze |l 'aj
Oudyes
Output
(d) Dynamics model residual network, with actions omitted. Output scaling all dimensions to be within [-0.99, 0.99].
(e) Measurement model weights network.
Figure 29: Neural architecture for the networks within the dynamics and measurement models for the bearings only tracking task.
In this section give additional task specific information and network architecture details for the bearings only tracking task.
D.4.1 Additional Experiment Details
When training MDPF using the Negative Log-Likelihood (NLL.) loss, we set the learning rate (LR) of the neural networks to be 0.0005 and the bandwidth learning rate to 0.00005. When training
27

---

Figure 30: Example trajectories and observations from each of mazes Deepmind maze tracking lask.
using MSE loss, we use 0.0001 and 0.00001 for the neural network and bandwidth learning rates respectively. For the discrete PF methods, we use LR 0.0005 for training the neural network using the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the LSTM [45], we use 0.001 the learning rate for training the networks and LR 0.0005 when optimizing the bandwidth parameter using NLL.
We apply gradient norm clipping, clipping the norm of the gradients to be S 100.0.
In addition using the Gaussian distribution for KDE, we also train with the Epanechnikov distri- bution applied to the translational dimensions of the state. To do this we first train using Gaussian kernels before re-training with the Epanechnikov, using the already trained dynamies and measure- ment models as a starting point.
The observations
generated as noisy bearings from the radar station the car:
0₂ ~ a Uniform( -T, T, σ) + (1 - a) VonMises(els (+).. k)
where vi, is the true bearing, 0.15 and x 50. Observations or are angles and not suitable as input into a neural network. As such we convert the angular representation into a unit vector as is done with the angle dimensions of the particles.
D.4.2 Network Architectures
Fig. 29 gives details the neural network architectures used for the dynamics and measurement models within the PF for the bearings only tracking task. Since this task does not provide actions, the action encoder has been omitted.
D.5
Deepmind Maze Tracking Task
In this section give additional task specific information and network architecture details for the Deepmind maze tracking task.
D.5.1 Additional Experiment Details
The Deepmind maze tracking task was first proposed [5]. In this task, a robot moves through modified versions of the maze environments from Deepmind Lab [48]. Three distinct mazes are provided and we train and test on each maze individually. Example robot trajectories as well as example observations for each maze shown in fig. 30. We modify this task by increasing the noise applied to the actions by 5 fold as well as using sparsely labeled true states during training.
When training MDPF using the Negative Log-Likelihood (NLL) loss, we set the learning rate (LR) 0.0005 and the bandwidth learning rate to 0.00005. When training
of the neural networks using MSE loss, we use 0.0005 and 0.00005 for the neural network and bandwidth learning rates respectively. For the discrete the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the I.STM, we use 0.001 as the learning rate for training the networks and LR 0.0005 when optimizing the bandwidth parameter using NLL.
methods, we use LR = 0.0005 for training the neural network using
28

---

Iqual Tranffored Putioe
Maak cut Toressoca Componint
Fuly Gommentd W. RelU
Fuly Connested 88 IKU
Fulyy
Fully Connected & ReLU
Fully Conncted (R)
2ed Dam (10 - FaLu
Pulty Cormected (8)
Output
Output
Output
3dl Max Pool
(b) Particle encoder dynamics Model.
(c) Particle encoder for measurement model.
(d) Action encoder for dy- namies Model.
500 PMLU
2-d Mou Pool (9)
Dow (Ml) Relli
Flabean
Suecarad
My Canocial R0-
Fully Crenerted pit a ArL
Muly Canoriel P0-
Canedial 21
Output
Sigeod
(a) Observation
Stal oetanto ic (331
coder.
(e) Dynamics model residual network. Output scaling scales the transnational dimensions to be within [-7. 7] and the unit vector representing angles to be within [-1, 1].
(I) Measurement model weights network.
Figure 31: Neural architecture for the networks within the dynamics and measurement models for the Deepmind maze tracking task.
We apply gradient norm clipping, clipping the norm of the gradients to be S 500.0. Further we limit the bandwidth for the angle dimension such that 8 < 4.0. We do this only for this task as non-MDPF methods were prone to learning an unreasonably large bandwidth resulting in a near uniform distribution for the angular part of the state.
D.5.2 Network Architectures
Fig. 31 gives details on the neural network architectures used for the dynamics and measurement models within the PF for the Deepmind maze tracking task.
D.6
House3D Tracking Task
In this section give additional task specific information and network architecture details for the House3D tracking task.
D.6.1 Additional Experiment Details
When training MDPF using the Negative Log-Likelihowe (NLL) loss, we set the learning rate (LR) of the neural networks to be 0.0005 and the bandwidth learning rate to 0.00005. When training using MSE loss, we use 0.0001 and 0.00001 for the neural network and bandwidth learning rates respectively. For the discrete the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the LSTM, we use 0.001 as the learning rate for the L.TTM layers and LR 0.0005 when optimizing the bandwidth parameter using NLL.
methods, we use LR = 0.0005 for training the neural network using
29

---

We apply gradient norm clipping, clipping the norm of the gradients to be I 250.0 when training with NLL loss. When training with MSE loss, disable gradient norm clipping.
D.6.2 Network Architectures
Fig. 31 gives details on the neural network architectures used for the dynamics and measurement models within the PF for the House3I) tracking task.
Inputt
Oid Cone e1p PRaLU
Irpet Map
3ed Man Popl o2
Cork OE PRaLL
Core
3ed Man Pool
2ad Core (eC PRaLL
(c) Dynamics model residual network. Output scaling scales the transnational dimensions be within [-75, 75] and the unit vector representing angles be within [-0.99, 0.99].
Parten
Pully Comnoter 2011
Puly Comnwrzed you
Output
Output
(c) Particle encoder for measurement model.
(a)
Observation
coder.
Inpul: Action
Inpuit Thanfoones Partace
Convetaare
Connected PhLL
Trankaiional Compoeent
CalyCoeecddd (4 Fhaill
Pulyy
Output
Oulput
Ourput
(d) Action encoder for dy- namics Model.
Measurement model weights net- work.
(b) Particle encoder for dynamics Model.
Figure 32: Neural architecture for the networks within the dynamics and measurement models for the House3D tracking task.
D.7 LSTM Model Details
We train using LSTM [45] based recurrent neural network models for all tasks as a baseline compari- son. For the LSTM. use the same encoders the PF methods. Fig. 33 shows the LSTM network architecture used for all tasks. We use the same output scaling the PF methods.
30

---

Input:
Action
Encoded Observation
Concatenate
LSTM (256)
(256)
Input: Hidden States
Output: Hidden States
LSTM (256)
LSTM (258)
Fully (256) PReLU
Fully Connected (256) PRoLU
Fully Connected (256) PReLU
Fully Connected (4)
Input: Transformed Particle
Tanh
Output Scaling
Output
Figure 33: LSTM network architecture used for all tasks. The output is the 4D state which is then converted back into the 3D state using the inverse particle transform.
31


Reviews:
[{"review_id": "O15ucnmUQy", "reviewer": "Reviewer_fwhN", "summary": "This paper proposes an unbiased and low-variance gradient estimator for differentiable particle filters, where resampling steps incurs discrete changes that are non-differentiable in previous methods. The proposed method solved the problem by representing posteriors as continuous mixture densities, which is similar to the idea of regularized particle filters.", "questions": "1. As far as I can see, all the experiments presented in the paper assumed the true latent state is available during training, do you have any idea about how to train differentiable particle filters when true states are not available in training but we still want to track the latent state in testing?\n2. The paper claimed the proposed estimator is unbiased, but it is not obvious to me why it is unbiased and I expect to see a theoretical proof of this.", "limitations": "The proposed method is based on importance sampling, therefore some discussions on the variance of the estimator will improve the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well-presented, motivation and proposed method are clearly explained, experimental results demonstrated the effectiveness of the proposed gradient estimator for differentiable particle filters, especially when the posterior is multimodal.", "weaknesses": "The performance of the proposed method when used in differnetiable particle filters trained with unsupervised loss when labelled data are not available is not discussed in the experiments/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback.  To answer your specific questions:\n\n1. Classical particle filters assume known dynamics and measurement models, and thus also implicitly assume the latent state has been manually defined; typically it has an interpretable real-world meaning such as the position or velocity of the system. Our MDPF extends the classical particle filter by allowing the dynamics and measurement models to be learned, training them end-to-end from (possibly sparse) observations.  In general to allow prediction of latent states, the state space must be defined somehow, either by hand-crafting of models (as in classical particle filters) or by training data (as in our paper).\n\n2. Importance sampling is a broadly-applicable Monte Carlo method that, under very mild assumptions, produces unbiased estimates of expectations for any number of samples [1].  In general, importance sampling estimates the expectation with respect to some distribution $p(x)$ by drawing samples from some other distribution $q(x)$: $E_{p(x)}\\Big[f(x)\\Big] = \\int_{x} p(x) f(x) dx = \\int_{x}q(x) \\frac{p(x)}{q(x)} f(x) dx =  E_{q(x)}\\Big[\\frac{p(x)}{q(x)}f(x)\\Big]$. Our IWSG estimator is unbiased since it is derived from an unbiased importance-sampling estimator.  We will revise Sec. 4.2 to explain this more clearly.\n\n[1] C. P. Robert and G. Casella, “Monte Carlo Statistical Methods”, 2004.\n"}, {"review_id": "lpMmmXrGyq", "reviewer": "Reviewer_wjUs", "summary": "The paper consider the problem of particle filter (PF)-based state estimation in nonlinear models with unknown dynamics and (discriminative) observation models.  The key challenge they address is the typical inability of traditional gradient learning approaches, applied to PF, to deal with (backprop through) the non-differentiable particle resampling stage.   To do this, the authors propose a novel Importance Weighted Samples Gradient (IWSG) Estimator, based on a kernel-density representation of the resampling step.  The IWSG is fully differentiable but is also unbiased and, in practice, low variance, particularly compared to the previously proposed Implicit Reparameterization Gradients (IRG) estimator.  The authors conduct an extensive set of experiments on simulated data (synthetic + simulator object-in-a-maze tracking scenarios) to demonstrate the utility of their IWSG.  IWSG is shown to outperform the competing approaches.", "questions": "- How does the complexity of learning/inference depend on N?  Do you have experimental evidence that demonstrates this?\n- Fig. 3 (supplement) shows some increasing variance trends in both gradients and states for MDPF (and A-MDPF).  Can you comment on this / explain?  Was this tendency observed in experiments in the simulator data?", "limitations": "The authors has addressed limitations", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ Well-written paper, with sufficient details (in main paper + supplement) that clearly introduce the problem, describe prior PF-based approaches, and build the proposed Mixture Density Particle Filter (MDPF) using a novel IWSG framework, following the ideas of Scibior et al., but using a kernel density-based (mixture density) resampler.\n+ Addresses an important problem in the application of particle filter-based state estimators/trackers for models with unknown (parametric) dynamics and observation models (discriminative)\n+ Propose two variants of MDPF, a baseline with a single mixture density used for both particle resampling and state estimation, and an improved A-MDPF, which employs different mixtures for the two tasks. A-MDPF is generally demonstrated to be more effective at the added computational cost of having two mixtures.\n+ Extensive and convincing experiments conducted in simulated settings (both simple synthetic models and more complex 2D and 3D based simulators, c.f., Deepmaze and House3D with image based observations).  Experiments demonstrate that IWSG leads to consistently more accurate state estimates compared to either prior works / baselines (LSTM, TG-PF, OT-PF, SR-PF, DIS-PF, C-PF, TG-MDPF, and IRG-MDPF), together with lower estimator variances.", "weaknesses": "- Main paper does not clearly build the connection between Scibior et al. and the proposed approach, but this is clarified in the rather extensive supplement.  The proposed approach can be seen as perhaps rather obvious, once the KDE mixture model is employed for the resampler, but it is nevertheless a fairly ingenious combination of MD + IWSG.\n- The authors acknowledge that PF-based approaches are not effective for high-dimensional state spaces and/or when the number of particles is large.  However, they do not clearly investigate this dependency (all experiments are conducted on max 3D state spaces + bearing).  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and constructive feedback.  We agree that some of the discussion of baseline methods in the main text is not sufficiently clear, and will shift material from Appendix A to the main paper to address this issue.\n\nPlease see our global response to all reviewers for discussion of computational complexity, and the new plot in Figure 1 of the rebuttal pdf.  While the computational cost of MDPF gradient computation with N particles scales as O(N^2) during training, the cost of MDPF resampling at inference/test time is O(N log(N)), comparable or better than baseline methods.\n\nWith regard Fig. 3 (main paper) and Fig. 2 (supplement), the perceived increasing variance trends for both states and gradients of MDPF is a rendering artifact from using log scale on the horizontal axis.  We have re-rendered the plot with linear scale (shown in the 1-page rebuttal figure PDF, figure 2) to show that the variance of our MDPF does not tend to increase. "}, {"review_id": "rb5XWEoTPF", "reviewer": "Reviewer_53pr", "summary": "The paper studies the problem of sequential state estimation using discriminative particle filters. Particle filters (or SMC) methods are widely used in this setting owing to their flexibility. Recently, inspired by the success of deep learning enabled by end-to-end differentiable methods, there has been interest in imparting similar properties to particle filters for improving performance, in particular when modelling temporally extended systems. The authors first review some existing approaches for particle filtering with a focus on regularized particle filters as well approaches for making the non-differentiable resampling step differentiable. The authors highlight several drawbacks with the existing methods with differentiable resampling, including biased gradients, numerical instability and high variance gradient estimates. The proposed method is based on leveraging implicit reparameterization gradients coupled with an importance sampling scheme to obtain an unbiased gradient estimator for continuous mixture distributions. The implicit reparameterization avoids the inversion of the standardization function and the importance sampling scheme reduces the variance of the estimates. The approach avoid various pitfalls of previous attempts to make the resampling step differentiable. The authors leverage this estimator in their Mixture Density Particle Filter method, which is evaluated through a wide variety of experiments with thorough quantitative and qualitative analysis - achieving improved performance across tasks.", "questions": "* What would be potential solutions to scaling the method for higher dimensional problems? \n* Have you tried the approach on some standard SMC tasks [1]? Does the gradient estimator improve performance there? \n\n[1] An introduction to Sequential Monte Carlo by Nicolas Chopin and Omiros Papaspiliopoulos. https://particles-sequential-monte-carlo-in-python.readthedocs.io/en/latest/\n", "limitations": "The major limitations of the method are already highlighted in the paper - namely scaling to higher dimensional problems and reliance on labelled data. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Sequential Monte Carlo approaches are widely used across various domains. Improvements to the algorithm can have a broad impact. The importance-weighted gradient estimator proposed in the paper can improve the performance on a wide-variety of tasks, beyond those studied in the paper. \n* The proposed approach is principled and novel, leveraging recent advances in implicit reparameterization along with classical insights from the sampling literature to improve the performance of regularized particle-filters. \n* The resulting method MDPF is conceptually simple, and can be incorporated easily as an alternative to existing regularized PF approaches. \n* The paper is quite well written with a nice review of existing approaches and a lot of useful details about the experiments. I do think the introduction can be improved a bit. ", "weaknesses": "* A key weakness of the approach is scaling to higher dimensional problems. It is a well known problem that importance sampling based estimators can still have high variance induced by some bad samples with high weights. (This is already discussed in the paper)\n* While I appreciate the thorough experiments in the paper one issue is that most of the experiments are on relatively low dimensional problems. \n* Although the details of the experiments are covered fairly well, the absence of code could be a problem for reproducibility efforts. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your praise and feedback. We are grateful for the careful review of our work and appreciate your highlights to the broad applicability of our IWSG estimator beyond particle filtering and the simplicity and effectiveness of our MDPF method. Our intention is to release the code publicly (on Github) after some code cleanup and documentation.\n\nRegarding the application of particle filters in higher dimensions, we first note that our experimental domains are chosen to allow direct comparison to prior work, and are challenging in spite of their moderate dimension.  Scaling particle filters to higher-dimensional states does generally require more particles, but the number of required particles is heavily dependent on the posterior uncertainty; latent spaces with high posterior entropy require more particles [2]. By making the dynamics and measurement models of the particle filter learnable and effectively training them via our MDPF, the posterior state uncertainty can be reduced, allowing for fewer particles to be used. We conjecture that learning high-quality dynamics and measurement models, instead of hand-crafting models that only coarsely approximate reality, will allow particle filters to more efficiently scale to higher-dimensional problems.  As indirect evidence for this, we note that several of the baseline papers we cite used much-larger particle sets at test than during training, to compensate for poorly training models (due to biased gradients).  In contrast, our MDPF is effective when the test particle set size is exactly matched to training.\n\nWith regard to “standard” SMC tasks as in [1], these methods assume known dynamics and measurement models, and thus the gradient-estimation innovations that our MDPF focuses on are not needed. Our Bearings-Only Tracking Task is similar to the tracking problem shown in 2.4 of [1], but more challenging:  the latent state has an additional dimension, and the dynamics and measurement models must be learned from training data (rather than being fixed to the true process used to generate synthetic data).\n\n[1] Nicolas Chopin and Omiros Papaspiliopoulos. “An introduction to Sequential Monte Carlo”. Springer 2020. ISBN 978-3-030-47844-5\n\n[2] Dieter Fox. “KLD-Sampling: Adaptive Particle Filters”. Advances in Neural Information Processing Systems (NeurIPS), 2001.\n"}, {"review_id": "Z81lxpj03P", "reviewer": "Reviewer_FRJy", "summary": "This paper discusses the limitations of traditional particle filters in representing multiple posterior modes and their applicability to high-dimensional observations like images. Instead, the authors propose a method that leverages training data to learn particle-based representations of uncertainty using deep neural network encoders. The authors address the issue of biased learning and heuristic relaxations by representing posteriors as continuous mixture densities, allowing for unbiased and low-variance gradient estimates.\n\n", "questions": "Questions regarding the proposed importance weighted samples gradient (IWSG) estimator:\n\n1- How effective is the IWSG estimator in reducing variance compared to other gradient estimation techniques?\n\n2- Are there scenarios or specific mixture distributions where the IWSG estimator may still suffer from high variance?\n\n3- Have there been any empirical evaluations to quantify the variance reduction achieved by the IWSG estimator?\n\n4- How does the choice of the proposal distribution, q(z), affect the accuracy and efficiency of the IWSG estimator?\n\n5- What is the computational cost associated with using the IWSG estimator compared to other gradient estimation methods?\n\nQuestions regarding the Algorithms: \n\n1- How does the complexity of the deep neural networks used in MDPF and A-MDPF impact the computational requirements for training and inference?\n\n2- Are there any strategies or techniques to improve the computational efficiency of these models?\n\n3- How does the performance of MDPF and A-MDPF depend on the quality, size, and representativeness of the training data?\n\n4- What happens when there is limited or biased training data available?\n\n5- Can the learned parameters and decisions made by MDPF and A-MDPF be interpreted and understood?\n\n6- Are there any techniques or approaches to enhance the interpretability of these models?\n\n7- How does the performance of MDPF and A-MDPF depend on the choice and optimization of the bandwidth parameter (β) for kernel density estimation?\n\n8- Are there any alternative methods or strategies to optimize this parameter effectively?\n\n9- How does the decoupling of mixtures used for particle resampling and posterior state estimation affect the performance and training of A-MDPF?\n\n10- Are there specific scenarios or domains where A-MDPF offers significant advantages over MDPF?\n\n11- How well do MDPF and A-MDPF generalize to different problem domains or scenarios?\n\n12- Have these models been tested on real-world datasets, and how do they perform compared to alternative approaches in those contexts?\n", "limitations": "Please refer to the weakness and question. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper discusses an interesting problem.\n- It is easy to read and follow the train of thought.\n- Literature review is sufficient to grasp the idea of the idea of the paper. \n ", "weaknesses": "- As MDPF relies on DNN parameterizations for both dynamic and measurement models, this introduces significant complexity to the model which can make training and inference more computationally expensive. \n- Additional complexity of the model due to decoupling for A-MDPF as it requires careful tuning for separate bandwidth parameters. \n- While the paper mentions that bandwidth parameters can be learned through end-to-end training, optimizing this parameter efficiently can require careful tuning and thus is challenging. \n- I am still a bit uncertain about the authors' claim regarding the use of smaller NNs and non-learned operations in constructing the dynamic and measurement models. \n- Lack of comparison with alternative frameworks for tracking and robot localizations and mostly applying to synthetic data. \n- The limitations of the simulation setup in evaluating the MDPF and A-MDPF algorithms can be identified as follows:\n\n     * The state estimation tasks focus on a 3D state consisting of translation and angle components (s = (x, y, θ)). While this simplification allows for manageable simulations, it may not capture the full complexity of state estimation in more challenging real-world scenarios with higher-dimensional state spaces.\n\n     * The evaluation compares the MDPF and A-MDPF algorithms with a few selected particle filter variants (TG-PF, SR-PF, OT-PF, DIS-PF, C-PF) and an LSTM model. While this provides some insight into the performance of the proposed methods, a more comprehensive comparison with a wider range of state estimation algorithms would provide a better understanding of their strengths and weaknesses.\n\n     * The training data uses sparsely labeled true states every 4th time-step, which may not accurately represent the labeling constraints in real-world applications. Dense labeling is crucial for accurately assessing the performance of state estimation algorithms. Although the evaluation uses densely labeled datasets, the training process may not fully exploit the benefits of dense labeling.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback. We address various comments and reviewer questions below.\n\nWe would like to highlight Appendix A.4 (supplement) which gives additional details about the use of smaller neural networks and non-learned operations when constructing the dynamic and measurement models, and how this structure is motivated by particle filter mathematics.\n\nWith regard to the computational cost of parameterizing the dynamics and measurement models by deep neural networks, we first note that MDPF makes no restrictions on the parameterization of either model other than differentiability (see Sec. 3).  Hand-engineered parametric models with limited tunable parameters may be substituted for either model if deep neural networks are too computationally expensive, though the expressiveness and flexibility to match complex training data would be lost. \n\nWith regard to the additional bandwidth parameters in our A-MDPF methods, we find that careful manual tuning of the bandwidth for MDPF and A-MDPF is not necessary.  During end-to-end training, the bandwidth can be optimized simultaneously along with the dynamics and measurement models; this approach is successful in all of our experiments. \n\n\nIWSG Questions:\n\n1. IWSG offers much lower variance gradient estimates compared to other estimators like IRG. We highlight this in Sec. 4 (main text) and Appendix B (supplement).\n2. The only regime where IWSG has been observed to have high-variance is when the number of particles is extremely small (less than 20).\n3. We empirically show the variance reduction in Fig. 3 (main text) and Fig. 2 (supplement), where it is clear that our IWSG estimator is much lower variance than IRG.\n4. We frame IWSG as an estimator for estimating gradients for samples drawn from some mixture distribution $m(z)$.  The proposal distribution is always the “current” mixture distribution $q(z) = m(z)$, before gradient-based perturbation of the mixture parameters.  Thus, unlike some applications of importance sampling, there is no need to manually choose a proposal distribution.\n5. Please see our global response to all reviewers, and the additional plot in our 1-page rebuttal PDF (Fig. 1).\n\nAlgorithm Questions:\n\n1,2. The majority of the computation time of MDPF is neural network evaluation, and thus more computationally intensive networks will require more compute time. MDPF gives no constraints on neural network architecture (other than they must be differentiable), as stated in Sec. 3 (main text).  Thus, methods developed to increase computational efficiency of general neural networks can be incorporated into MDPF training.\n\n3,4. The quality of the learned dynamics and measurement models is dependent on the quality available data.  Our experiments show that training may be effective even when observations are temporally sparse, but of course, biased or insufficient data can lead to biased or inaccurate models. \n\n5,6. The latent space (particles, weights, bandwidths) of MDPF and A-MDPF are defined by the training data, and are thus interpretable with real world meaning as highlighted in sections 2.1 and 5 (main text).  The parameters of the neural networks are not directly interpretable, as is the case with most neural networks. MDPF does not place any restrictions (other than differentiability) on the models used, and therefore interpretable parameterizations could be used.\n\n7,8. The performance of MDPF (and A-MDPF) is dependent on the choice of bandwidth parameter $\\beta$ and thus we choose to set it as a learned parameter. Very-small bandwidths ($\\beta << 0.001$) can lead to numeric instability in float32 systems, and very-large bandwidths lead to poor MDPF performance. Classical methods, with known drawbacks, for setting the bandwidth exist and are referenced in section 2.2.  We find that learning the bandwidth end-to-end is simple and effective.\n\n9 . Decoupling posterior and resampling mixtures as done in A-MDPF gives superior performance as shown in the results in section 6.  Training A-MDPF requires additional computation (due to the additional networks) but a pre-trained MDPF can be used to initialize A-MDPF allowing for faster training.\n\n10 . We find that in general A-MDPF offers performance increases over MDPF due to the flexibility provided by decoupling the posterior and resampling distributions.  This can be seen in our results section 6 (specifically fig. 7).\n\n11 . Our MDPF and A-MDPF learn problem-specific dynamics and measurement models, and are easily applied to any domain where appropriate data is available.  Application to test data with altered dynamics/likelihoods, without first retraining, may lead to poor performance.\n\n12 . Prior work in this area has mainly been evaluated on synthetic datasets with high-dimensional observations, and we take the same approach. We evaluate on the same datasets as methods we directly compare to [1-3].  The House3D data is fairly realistic and the 3D environments it is based on data taken from real-world homes, but the observation images are still rendered with computer graphics.  Given the strong performance of MDPF, we believe experimentation with richer real-world datasets is a promising area for future research.\n\n\n[1] Adam Scibior, Vaden Masrani, and Frank Wood. “Differentiable particle filtering without modifying the forward pass”. International Conference on Probabilistic Programming (PROBPROG), 2021\n\n[2] Peter Karkus, David Hsu, and Wee Sun Lee. “Particle filter networks with application to visual localization”. Conference on Robot Learning (CORL), 2018.\n\n[3] Adrien Corenflos, James Thornton, George Deligiannidis, and Arnaud Doucet. Differentiable particle filtering via entropy-regularized optimal transport. International Conference on Machine Learning (ICML), 2021.\n"}, {"review_id": "ProtNt31dq", "reviewer": "Reviewer_FW5K", "summary": "The paper proposes the \"importance weighted samples gradient (IWSG)\" estimator and describes its integration into a \"mixture density particle filter (MDPF)\" for state space architectures. Similar to regularized particle filters, the MDPF framework represents the continuous state posterior with a continuous mixture density, which is differentiable and can be trained end-to-end using the proposed IWSG. As a discriminative approach, the proposed particle filter does not require the specification of a generative model and, in contrast to other discriminative particle filters, returns unbiased and low-variance gradient estimates.", "questions": "- Section 4: It is mentioned that the optimal transport methods described in [44,45] are compatible with Gaussian mixtures (l.154f), so I’m curious why they are not part of the evaluated methods?\n\n- Section 6: The design of the architecture used in the experiments (Sections 6.1-6.3) is not clear enough. Are all experiments based on Figure 4? If so, what are the encoders and transformations?\n\n- Section 6: One of the arguments against [5] is its use of truncated gradients (l.118f), however, l.244 mentions that the experiments use truncated BPTT. Doesn’t that bias the MDPF gradients as well and makes the arguments in favour of MDPF obsolete?\n\n- Section 6: I’m curious how the number of particles affects the reported performance metrics, and especially if the observed trends remain the same if the number of particles is significantly increased or decreased. If the authors have additional results along those lines it would be interesting to see them.", "limitations": "- I commend the authors for including a paragraph on the limitations of the proposed particle filter, such as its supervised nature and computational complexity on high-dimensional data.\n\n- The paper does not discuss any ethical concerns related to the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Efficient and accurate particle-based estimation of complex posteriors in sequence models is an important topic and has a long history in the machine learning community.\n\n- The paper includes an easy-to-follow recap of particle filtering and puts the proposed approach in context with many relevant discriminative particle filters, including a thorough discussion of their weaknesses that motivates both IWSG and MDPF.\n\n- The proposed discriminative particle filter tackles a number of well-known challenges, such as differentiation through the resampling step, biased or high-variance gradient estimates, and limited expressiveness of the posterior parameterization.\n\n- The experiments compare the proposed particle filter to a broad spectrum of relevant baselines and demonstrate MDPFs superior performance in three state estimation tasks.", "weaknesses": "- One of my concerns with this paper is its confusing presentation. The text often jumps between motivation, technical background, and related work, making it difficult to follow the intended train of thought. This problem is further aggravated by the fact that there are multiple technical streams: generative vs. discriminative particle filters, unbiased/low-variance gradient estimation, and inference in mixture models. There are many balls in the air and the paper has a hard time telling a streamlined story. Unfortunately, these are not the only problems with the presentation: Section 4 mixes technical background, the paper’s first main contribution (IWSG), and an experiment; Section 5 mixes the paper’s second main contribution (MDPF) with related work and contains no mathematical description of the model.  \n\n- The figures are weak for multiple reasons: (1) the font size is much too small; (2) the main text and the figure captions cannot be read independently, because the figures contain critical information that is not explained anywhere in the text; (3) the lack of subfigures ((a),(b),…) makes it difficult to map the information in the captions to individual subplots; and (4) the figures are not placed on the page they are referenced. Figure 4 is especially problematic as it describes the core of the paper’s contribution but is impossible to understand without further context (which the text does not provide either).\n\n- The paper’s weak presentation also makes it difficult to assess the two contributions (IWSG and MDPF): (1) l.170 mentions that the proposed IWSG estimator is a continuous variant of the discrete estimator used in [6], but the exact relationship between the two remains unclear. Are there non-trivial challenges that prevent a direct generalization of [6] to a continuous setting?; (2) l.89/l.209 mentions that the proposed MDPF is a variant of [14,15], but again the differences are not explained in enough detail. Are there non-trivial challenges that prevent a direct application of [14,15] in a discriminative setting?\n\n- One of the paper’s main claims is MDPF’s ability to compute unbiased and low-variance gradients. Unfortunately, neither the technical sections nor the experiments provide any *direct* evidence supporting this claim. I would have liked to see either a technical analysis of the proposed estimator's properties or experiments that go beyond the evaluation of NLL/RMSE and at least give empirical insights along those lines.\n\nOverall, I feel this paper would benefit from another round of polishing before publication, including a reworked presentation and better positioning and differentiation of its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough comments and helpful feedback, which we will use in future revisions.  \n\nWe concede that the presentation in the paper can be improved, and based on feedback we intend to move some technical details that are present in the supplement into the main paper text (see global response).  In the meantime, we would like to point the reviewer to Appendix A of the supplement, which we believe addresses many of your concerns.  Specifically, Appendix A contains information on prior works which clarifies how our IWSG extends work from [6], as well as the differences and similarities between our MDPF and classical regularized particle filters. Furthermore, in Appendix A.4 we give additional details on the general architecture/parameterization of MDPF and A-MPDF (details on encoders and transformations), and in Appendix D we give specific architectural details for all neural network components in MDPF and A-MDPF.\n\nWith regard to optimal transport methods [44, 45] mentioned in the main paper text but for which we do not compare against: both [44] and [45] proposed gradient estimators focused on static distributions, and thus cannot be immediately applied to our particle filter context.  The text citing these papers was unclear about this, we will revise.  (Aspects of this work are also specialized to Gaussian mixtures, and cannot handle the von Mises kernels used in our experiments.)  For this reason, we instead compare to the Entropy-Regularized Optimal-Transport [8] (OT-PF) method, which was an optimal transport method specifically engineered to integrate with particle filters.\n\nWith regard to truncated Back-Propagation-Through-Time (BPTT) biasing MDPF, it is true that truncating gradients biases all methods (including MDPF) to reduce variance. The main contribution of MDPF is the ability to propagate gradients temporally across multiple timesteps (between the sparse truncation points) which baselines like TG-PF [5] are unable to do.  With MDPF the BPTT truncations can be selected to balance bias and variance, as well as the computational requirements of training (frequent truncation is computationally cheaper, but more biased). TG-PF forces truncation at every time-step, and thus bias cannot be reduced even if additional computation is available.\n\nUnfortunately we did not run additional experiments with varying numbers of particles, as training multiple baseline methods for multiple trials (see box plots in Fig. 5,7) is already computationally demanding. But in the 1-page rebuttal figure PDF (Fig. 1), we do compare computational cost of the various methods with varying numbers of particles.\n\n[5] Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. Proceedings of Robotics: Science and Systems (RSS), 2018.\n\n[6] Adam Scibior, Vaden Masrani, and Frank Wood. Differentiable particle filtering without modifying the forward pass. International Conference on Probabilistic Programming (PROBPROG), 2021.\n\n[8] Adrien Corenflos, James Thornton, George Deligiannidis, and Arnaud Doucet. Differentiable particle filtering via entropy-regularized optimal transport. International Conference on Machine Learning (ICML), 2021.\n\n[44] Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparameterization trick. International Conference on Machine Learning (ICML), 2018.\n\n[45] Martin Jankowiak and Theofanis Karaletsos. Pathwise derivatives for multivariate distributions. International Conference on Artificial Intelligence and Statistics (AISTATS), 2019.\n"}]

Author rebuttal:
Thank you all for your feedback and helpful suggestions.  We want to address a few topics that were referenced by multiple reviewers.

First, for additional technical details about our methods as well as baselines, please refer to Appendix A of the supplement.  When drafting the paper, we put these details in Appendix A to allow more space for experimental details.  However, based on reviewer feedback, we now realize that this made it harder for readers to understand details of our mixture-density particle filter (MDPF) and gradient estimators.  In future revisions, we will move key parts of Appendix A back to the main paper (with space made available by shifting some parts of Sec. 6 to the appendix).

Several reviewers asked how the computational complexity of our MDPF scales with the number of particles N.  Figure 1 in the 1-page rebuttal PDF shows empirical trends for all methods. At training time, the complexity of MDPF gradient computation is O(N^2) per time step:  each of the N particles at the next time point could have been generated by any of the N mixture components at the previous time point, and the probabilities of these events must be computed.  Note that this overhead is not substantial for moderate numbers of particles, because for models of complex domains like images, resampling is much faster than neural-network likelihood evaluations whose cost remains O(N).  This extra computational cost is also key to avoiding the instability and poor performance of several baseline methods, which suffer from the “ancestor problem” (see Sec. 3).  Note also that this MDPF gradient computation is far cheaper than optimal-transport methods for aligning particles across time.

At inference or test time, gradient computation is not needed, and the MDPF has similar computational cost to “standard” particle filters.  Discrete resampling of N particles from the categorical distribution of particle weights requires O(N log(N)) time [1,2], and then particles may be perturbed by Gaussian noise (with learned bandwidth) in O(N) time.  This is in contrast with optimal-transport methods, which require expensive optimization at test as well as training.

We are currently in the process of cleaning and documenting our implementation of the MDPF, and will release code on Github when the paper becomes publicly available.

[1] https://en.wikipedia.org/wiki/Categorical_distribution#Sampling

[2] https://en.wikipedia.org/wiki/Alias_method


Meta-review: This submission was reviewed by five reviewers in the field. The reviewer consensus reads that the paper is well-motivated and interesting, and the experimental results demonstrate the effectiveness of the proposed gradient estimator for differentiable particle filters. All reviewers recommend accepting this work. The concern by reviewer FW5K related to the presentation was well covered in the author reply. It would do the paper good to address the concerns, but I don't recommend a complete rewrite. Take the constructive feedback from reviewer FW5K into account when preparing the camera-ready version.
Decision: Accept (poster)