Title: Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes
Venue: NeurIPS 2023
Accepted: yes
Keywords: particle

filter

mixture

belief propagation

nonparametric

deep learning

generative

discriminative

graphical model

multiple modes

mutli-modal
Abstract: Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods.  Such generative models may be inaccurate or unavailable for high-dimensional observations like images.  We instead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders.   While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities.  Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness.  On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, will also showing much greater stability across multiple training runs.

Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes

Abstract

Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods. Such generative models may be inaccurate or unavailable for high-dimensional observations like images. Weinstead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders. While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities. Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness. On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, while also showing much greater stability across multiple training runs.

1 Introduction

A particle filter (PF) [1-4] uses weighted samples to provide a flexible, nonparametric representation of uncertainty in continuous latent states. Classical PFs are generative models that typically require human experts to specify the latent state dynamics and measurement models. With the growing popularity of deep learning and abundance of time-series data, recent work has instead sought to discriminatively learn PF variants that (unlike conventional recurrent neural networks) capture uncertainty in latent states [5-8]. These approaches are especially promising for high-dimensional observations like images, where learning accurate generative models is extremely challenging.

The key challenge in learning discriminative PFs is the non-differentiable particle resampling step, which is key to robustly maintaining diverse particle representations, but inhibits end-to-end learning by preventing gradient propagation. Prior discriminative PFs have typically used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients. These methods severely compromise the effectiveness of PF training; prior work has often assumed exact dynamical models are known, or used very large numbers of particles for test data, due to these limitations.

After reviewing prior work on generative (Sec. 2) and discriminative (Sec. 3) training of PFs, Sec. 4 demonstrates dramatic failures of popular reparameterization-based gradient estimators for mixture models. This motivates our importance weighted samples gradient estimator, which provides the foundation for the mixture density particle filter of Sec. 5. Experiments in Sec. 6 show substantial advances over prior discriminative PF for several challenging tracking and visual localization tasks.

x

t

-1

y

t

y

t

-1

y

t

+1

a

t

a

t

-1

a

t

+1

x

t

x

t

+1

x

t

-1

y

t

y

t

-1

y

t

+1

a

t

a

t

-1

a

t

+1

x

t

x

t

+1

2 Sequential State Estimation via Generative Models

Algorithms for sequential state estimation compute or approximate the posterior distribution of the system state x t , given a sequence of observations y t and (optionally) input actions a t , at discrete times t = 1 , . . . , T . Sequential state estimation is classically formulated as inference in a generative model like the hidden Markov model (HMM) or state space model of Fig. 1, with dynamics defined by state transition probabilities p ( x t | x t -1 , a t ) , and observations generated via likelihoods p ( y t | x t ) .

Given a known Markov prior on latent state sequences, and a corresponding observation sequence, the filtered state posterior p ( x t | y t , y t -1 , . . . , y 1 ) can in principle be computed via Bayesian inference. When the means of state dynamics and observations likelihoods are linear functions, and noise is Gaussian, exact filtered state posteriors are Gaussian and may be efficiently computed via the Kalman filter (KF) [9-11]. The non-Gaussian state posteriors of general state space models may be approximated via Gaussians [12], but estimating the posterior mean and covariance is in general challenging, and cannot faithfully capture the multimodal posteriors produced by missing or ambiguous data.

2.1 Particle Filters

To flexibly approximate general state posteriors, particle filters (PF) [1-4] represent possible states nonparametrically via a collection of weighted samples or particles . The classical PF parameterizes the state posterior at time t by a set of N particles x (:) t = { x (1) t , . . . , x ( N ) t } with associated weights w (:) t = { w [1] t , . . . , w ( N ) t } . PFs recursively update the state posterior given the latest observation y t and action a t , and may be flexibly applied to a broad range of models; they only assume that the state dynamics may be simulated, and the observation likelihood may be evaluated.

Particle Proposal. To produce particle locations x (:) t at time t , a model of the state transition dynamics is applied individually to each particle x ( i ) t -1 , conditioned on external actions a t if available:

Measurement Update. After a new particle set x (:) t has been proposed, particle weights must be updated to account for the latest observation y t via the known likelihood function:

The updated weights are then normalized so that ∑ N i =1 w ( i ) t = 1 . This weight update is motivated by importance sampling principles, because (asymptotically, as N →∞ ) it provides an unbiased approximation of the true state posterior p ( x t | y t , . . . , y 1 ) ∝ p ( y t | x t ) p ( x t | y t -1 , . . . , y 1 ) .

Particle Resampling. Due to the stochastic nature of PFs, over time particles will slowly diverge to regions with low posterior probability, and will thus be given little weight in the measurement step. If all but a few particles have negligible weight, the diversity and effective representational power of the particle set is diminished, resulting in poor estimates of the state posterior.

To address this, particle resampling is used to maintain diversity of the particle set over time, and may be performed at every iteration or more selectively when the 'effective' sample size becomes too small [13]. During resampling, a new uniformly-weighted particle set ˆ x (:) t is constructed by discrete resampling with replacement. Each resampled particle is a copy of some existing particle, where copies are sampled with probability proportional to the particle weights:

After resampling, assigning uniform weights ˆ w ( i ) t = 1 N maintains an unbiased approximate state posterior. This discrete resampling is non-differentiable, preventing gradient estimation via standard reparameterization [14-16], and inhibiting end-to-end learning of dynamics and measurement models.

2.2 Regularized Particle Filters

The regularized PF [17, 18] acknowledges that a small set of discrete samples will never exactly align with the true continuous state, and thus estimates a continuous state density m ( x t | x (:) t , w (:) t , β ) by convolving particles with a continuous kernel function K with bandwidth β :

The kernel function could be a Gaussian, in which case β is a (dimension-specific) standard deviation. The extensive literature on kernel density estimation (KDE) [19] provides theoretical guidance on the choice of kernel. Quadratic Epanechnikov kernels take K ( u ; β ) = 3 4 ( 1 -( u/β ) 2 ) if | u/β | ≤ 1 , K ( u ; β ) = 0 otherwise, and asymptotically minimize the mean-squared-error in the estimation of the underlying continuous density. A number of methods have been proposed for selecting the smoothing bandwidth β [19-21], but they often have asymptotic justifications, and can be unreliable for small N .

Attractively, regularized PFs resample particles ˆ x ( i ) t ∼ m ( x t | x (:) t , w (:) t , β ) from the continuous mixture density rather than the discrete particle set. By resampling from a continuous distribution, regularized PFs ensure that no duplicates exist in the resampled particle set, increasing diversity while still concentrating particles in regions of the state space with high posterior probability. Our proposed mixture density PF generalizes regularized PFs, using bandwidths β that are tuned jointly with discriminative models of the state dynamics and observation likelihoods.

3 Conditional State Estimation via Discriminative Particle Filters

While classical inference algorithms like KFs and PFs are effective in many domains, they require faithful generative models. A human expert must typically design most aspects of the state dynamics and observation likelihoods, and algorithms that use PFs to learn generative models often struggle to scale beyond low-dimensional, parametric models [22]. For complex systems with high-dimensional observations like images, learning generative models of the observation likelihoods is often impractical, and arguably more challenging than estimating latent states given an observed data sequence. We instead learn discriminative models of the distribution of the state conditioned on observations (see Fig. 1), and replace manually engineered generative models with deep neural networks learned from training sequences with (potentially sparse) latent state observations.

Discriminative Kalman Filters. Discriminative modeling has been used to learn conditional variants of the KF [23, 24] where the posterior is parameterized by a Gaussian state space model, and state transition and observation emission models are produced from data by trained neural networks. Like conditional random field (CRF) [25] models for discrete data, discriminative KFs do not learn likelihoods of observations, but instead condition on them. Discriminative KFs desirably learn a posterior covariance, and thus do not simply output a state prediction like conventional recurrent neural networks. But, they are limited to unimodal, Gaussian approximations of state uncertainty.

Discriminative Particle Filters. It is attractive to integrate similar discriminative learning principles with PFs, but the technical challenges are substantially greater due to the nonparametric particle representation of posterior uncertainty, and the need for particle resampling to maintain diversity.

Like in generative PFs, discriminative PFs update the particle locations using a dynamics model as in Eq. (1). Unlike classic PFs, the particle weights are not updated using a generative likelihood function. Instead, discriminative PFs compute new weights w (:) t using a measurement model function ℓ ( x t ; y t ) trained to properly account for uncertainty in the discriminative particle posterior:

Here ℓ ( x t ; y t ) is a (differentiable) function optimized to improve the accuracy of the discriminative particle posterior, rather than a generative likelihood.

Creating a learnable discriminative PF requires parameterizing the dynamics and measurement models as differentiable functions, such as deep neural networks. This is straightforward for the measurement model ℓ ( x ( i ) t ; y t ) , which may be defined via any feed-forward neural network architecture like those typically used for classification (see Fig. 2). For the dynamics model, the neural network does not simply need to score particles; it must be used for stochastic simulation. Using reparameterization [1416], dynamics simulation is decomposed as sampling from a standard Gaussian distribution, and then

Resampling

Dynamics

Model

𝒙

𝒕"𝟏

(:)

, 𝒘

𝒕"𝟏

(:)

, 𝜷

𝒙

𝒕

(:)

, 𝒘

𝒕

(:)

, 𝜷

Measurement

Model

Learned

Bandwidth

Mixture	Density	Particle	Filter

Resampling

Dynamics

Model

Adaptive	-	Mixture	Density	Particle	Filter

𝒙

𝒕"𝟏

(:)

,

= 𝒘

𝒕"𝟏

(:)

,

> 𝜷

𝒙

𝒕

(:)

, 𝒘

𝒕

(:)

, 𝜷

𝒙

𝒕

(:)

,

= 𝒘

𝒕

(:)

,

> 𝜷

Measurement

Model

(Posterior)

Measurement

Model

(Resampling)

Learned

Bandwidth

(Posterior)

Learned

Bandwidth

(Resampling)

𝑦

?

𝑦

?

𝑦

?

𝑎

?

𝑎

?

𝒙

𝒕"𝟏

(𝒊)

𝒙

𝒕

(𝒊)

Particle

Encoder

Action

Encoder

Residual

Network

Inverse

Transform

Transform

𝑎

?

𝜼

𝒕

(𝒊)

Dynamics	Model

Measurement	Model

𝒙

𝒕

(𝒊)

𝒘

𝒕

(𝒊)

Particle

Encoder

Observation

Encoder

Feed-Forward

Network

Transform

𝑦

?

𝒘

𝒕"𝟏

(𝒊)

𝒘

𝒕

(𝒊)

= 𝒘

𝒕"𝟏

(𝒊)

R 𝒍(𝒙

𝒕

(𝒊)

; 𝒚

𝒕

)

:	Learned	Block	(Neural	Network)

:	Non-learned	Operation

transforming that sample using a learned neural network, possibly conditioned on action a t :

The dynamics model f ( η ; x t -1 , a t ) is a feed-forward neural network that deterministically processes noise η , conditioned on x t -1 and a t , to implement the dynamical sampling of Eq. (1). The neural network f may be flexibly parameterized because discriminative particle filters only require simulation of the dynamical model, not explicit evaluation of the implied conditional state density.

3.1 Prior Work on Discriminative Particle Filters

Specialized, heuristic variants of discriminative PFs have been previously used for tasks like robot localization [26] and multi-object tracking [27]. Principled end-to-end learning of a discriminative PF requires propagating gradients through the PF algorithm, including the discrete resampling step. Zhu et al. [28] explore replacing the PF resampling step with a learned (deterministic) particle transform, but find that exploding gradient magnitudes prohibit end-to-end training.

Truncated-Gradient Particle Filter (TG-PF) . The first so-called 'differentiable' particle filter [5] actually treated the particle resampling step as a non-differentiable discrete resampling operation, and simply truncated all gradients to zero when backpropagating through this resampling. Though simple, this approach leads to biased gradients, and often produces ineffective models because backpropagation through time (BPTT [29]) is not possible. Perhaps due to these limitations, experiments in Jonschkowski et al. [5] assumed a simplified training scenario where the ground-truth dynamics are known, and only the measurement model must be learned.

Discrete Importance Sampling Particle Filter (DIS-PF) . ´ Scibior et al. [6] propose a gradient estimator for generative PFs that may also be adapted to discriminative PFs. They develop their estimator by invoking prior work on score-function gradient estimators [30], but we provide a simpler derivation via importance sampling principles. Instead of discretely resampling particles as in Eq. (3), a separate set of weights v (:) t is defined and used to generate samples:

To account for discrepancies between these resampling weights v (:) and true weights w (:) , the resampled particle weights ˆ w ( i ) t are defined via importance sampling. The choice of v (:) t critically impacts the effectiveness of the resampling step. To maintain the standard PF update of Eq. (3), we set v ( i ) t = w ( i ) t , yielding the following resampled particle weights and associated gradients:

Intuitively, particle locations are resampled according to the current weights in the forward pass of DIS-PF. Then when computing gradients, perturbations of the observation and dynamics models are accounted for by changes in the associated particle weights.

A drawback of this discrete importance sampling is known as the ancestor problem : since the resampled particle set is constructed from a subset of the pre-resampled particles, no direct gradients exist for particles that were not chosen during resampling; they are only indirectly influenced by the weight normalization step. This increases the DIS gradient estimator variance, slowing training.

Soft Resampling Particle Filter (SR-PF) . Karkus et al. [7] propose a soft-resampling (SR) mixing of the particle weights with a discrete uniform distribution before resampling: v ( i ) t = (1 -λ ) w ( i ) t + λ/N .

Viewing particle locations as fixed, SR-PF evaluates the resampled particle weights and gradients as

While the SR-PF weight update is differentiable, it does not avoid the ancestor problem. By resampling low-weight particles more frequently, SR-PF will degrade overall PF performance when N is not large. More subtly, the gradient of Eq. (9) has (potentially substantial) bias: it assumes that perturbations of model parameters influence particle weights, but not the outcome of discrete resampling (7). Indeed in some experiments in [7], smoothing is actually disabled by setting λ = 0 .

Concrete Particle Filter (C-PF). The Gumbel-softmax or Concrete distribution [31, 32] approximates discrete sampling by interpolation with a continuous distribution, enabling reparameterized gradient estimation. Each resampled particle is a convex combination of the weighted particle set:

Gradients of (10) are biased with respect to discrete resampling due to the non-learned 'temperature' hyperparameter λ > 0 . When λ is large, the highly-biased relaxation will interpolate between modes, and produce many low-probability particles. Bias decreases as λ → 0 , but in this limit the variance is huge. Even with careful tuning of λ , Concrete relaxations are most effective for small N . Maddison et al. [32] focus on models with binary latent variables, and find that performance degrades even for N = 8 , far smaller than the N needed for practical PFs. While we provide C-PF as a baseline, we are unaware of prior work successfully incorporating Concrete relaxations in PFs.

Optimal Transport Particle Filter (OT-PF). OT-PF [8] modifies PFs by replacing stochastic resampling with the optimization-based solution of an entropy-regularized optimal transport (OT) problem. OT seeks a probabilistic correspondence between particles x (:) t with weights w (:) t as in (5), and a corresponding set of uniformly weighted particles, by minimizing a Wasserstein metric W 2 2 ,λ :

Entropy regularization is required for differentiability, and OT-PF accuracy is sensitive to the nonlearned hyperparameter λ > 0 . OT-PF uses this entropy-regularized mapping to interpolate between particles: ˆ x ( i ) t = ∑ N j =1 N ˜ α ij x ( j ) t . This assignment approximates the results of true stochastic resampling in the limit as N →∞ , but lacks accuracy guarantees for moderate N .

OT-PF solves a regularized OTproblem which relaxes discrete resampling, producing biased gradients that are reminiscent (but distinct) from C-PF. Minimization of Eq. (11) via the Sinkhorn algorithm [33] requires O ( N 2 ) operations. This OT problem must be solved at each step of both training and test, and in practice, OT-PF is substantially slower than all competing discriminative PFs (see Fig. 10).

4 Stable Gradient Estimation for Mixture Model Resampling

Training our discriminative mixture density particle filter (MDPF) requires unbiased and low-variance estimates of gradients of samples from continuous mixtures. Mixture sampling is classically decomposed into discrete (selecting a mixture component) and continuous (sampling from that component) steps. The Gumbel-softmax or Concrete distribution [31, 32] provides a reparameterizable relaxation of discrete resampling, but may have substantial bias. Some work has also applied OT methods for gradient estimation in restricted families of Gaussian mixtures [35, 36]. We instead develop importance-sampling estimators that are unbiased, computationally efficient, and low variance.

Implicit Reparameterization Gradients. Direct reparameterization of samples, as used for the latent Gaussian distributions in variational autoencoders [14-16], cannot be easily applied to mixture models. Reparameterized sampling requires an invertible standardization function S ϕ ( z ) = ϵ that transforms a sample z , drawn from a distribution parameterized by ϕ , to an auxiliary variable ϵ independent of ϕ . For mixture distributions, the inverse of S ϕ ( z ) cannot be easily computed.

The implicit reparameterization gradients (IRG) estimator [37, 38] avoids explicit inversion of the standardization function S ϕ ( z ) . Using implicit differentiation, reparameterization can be achieved so long as a computable (analytically or numerically) and invertible S ϕ ( z ) exists, without the need to

2

0

2

x

0.0

0.1

0.2

0.3

0.4

0.5

0.6

p(x)

0=

1.25

w

0 =0.5

1 =1.25

w

1 =0.5

Initial Samples

2

0

2

x

0.0

0.1

0.2

0.3

0.4

0.5

0.6

p(x)

0=

1.75

w

0 =0.25

1 =1.75

w

1 =0.75

Shift in Samples under IRG

3

2

1

0

1

2

3

x1

3

2

1

0

1

2

3

x2

Initial Samples

3

2

1

0

1

2

3

x1

3

2

1

0

1

2

3

x2

Shift in Samples under IRG

2

0

2

x

0.0

0.1

0.2

0.3

0.4

0.5

0.6

p(x)

0=

1.75

w

0 =0.25

1 =1.75

w

1 =0.75

Change in Sample Weights under IWSG

3

2

1

0

1

2

3

Original Sample Locations

3

2

1

0

1

2

3

New Sample Location

Change in Sample Location when using IRG

3

2

1

0

1

2

3

x1

3

2

1

0

1

2

3

x2

Change in Sample Weights under IWSG

3

2

1

0

1

2

3

x1

3

2

1

0

1

2

3

x2

Sample Shift Vector Field under IRG

explicitly compute its inverse. IRG gradients are expressed via the Jacobian of S ϕ ( z ) :

For univariate distributions, the cumulative distribution function (CDF) M ϕ ( z ) is a valid standardization function. For higher dimensions, the multivariate distributional transform [39] is used:

While IRG may be applied to any distribution with continuous CDF, and has been explicitly suggested (without experimental validation) for mixtures [37], we show that it may have enormous variance.

Importance Weighted Sample Gradient Estimator. We propose a novel alternative method for computing gradients of samples from a continuous mixture distribution. Our importance weighted sample gradient (IWSG) estimator employs importance sampling for unbiased gradient estimation. IWSG is related to the DIS-PF gradient estimator for discrete resampling [6], but we instead consider continuous mixture distributions with arbitrary component distributions. This resolves the ancestry problem present in ´ Scibior et al. [6], since particle locations could be sampled from multiple overlapping mixture components, whose parameters will all have non-zero gradients.

Formally, we wish to draw samples z ( i ) ∼ m ( z | ϕ ) from a mixture with parameters ϕ . Instead of sampling from m ( z | ϕ ) directly, IWSG samples from some proposal distribution z ( i ) ∼ q ( z ) . Importance weights w ( i ) , and associated gradients, for these samples then equal

Setting the proposal distribution q ( z ) = m ( z | ϕ 0 ) to be a mixture model with the 'current' parameters ϕ 0 at which gradients are being evaluated, the IWSG gradient estimator (14) becomes

While current samples are given weight one because they are exactly sampled from a mixture with the current parameters ϕ 0 , gradients account for how importance weights will change as mixture parameters ϕ deviate from ϕ 0 . Note that gradients are not taken with respect to the proposal q ( z ) = m ( z | ϕ 0 ) in the denominator of w ( i ) , since sample locations z ( i ) are not altered by gradient updates; changes in the associated importance weights are sufficient for unbiased gradient estimation.

Instability of Implicit Reparameterization Gradients. Reparameterization methods capture changes to a given distribution p ( z | ϕ ) , and thus to its parameters ϕ , by shifting the samples drawn from that distribution. When p ( z | ϕ ) is unimodal, this induces a smooth shift in the samples. When p ( z | ϕ ) contains multiple non-overlapping modes, these shifts are no longer smooth as samples

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Value

Learned Estimate of

B

10 0

10 1

10 2

10 3

10 4

Training Iteration

75

50

25

0

25

50

75

Gradients Estimate for

B

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.0

0.2

0.4

0.6

0.8

1.0

Learned Estimate of

w

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Gradients Estimate for

w

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Learned Estimate of

C

1

10 0

10 1

10 2

10 3

10 4

Training Iteration

75

50

25

0

25

50

75

Gradients Estimate for

C

1

IRG-PF

IWSG-PF

Mixture KF

Truth

jump from one mode to another; see Fig. 3. For mixtures with finitely supported kernels (like the Epanechnikov) and non-overlapping modes, discontinuities exist in S -1 ϕ ( ϵ ) , and IRG estimates may have infinite variance. These discontinuities correspond to samples shifting from one mode to another, and the resulting non-smooth sample perturbations will destabilize backpropagation algorithms for gradient estimation. While the inverse CDF is always continuous for Gaussian mixtures, it may still have near-infinite slope when modes are widely separated, and induce similar IRG instabilities.

In contrast, our IWSG estimator reweights particles instead of shifting them. This reweighting is smooth and does not suffer from any discontinuities. To illustrate this, we create a simple discriminative PF with linear-Gaussian dynamics (16) and a bimodal observation likelihood (17):

Exact posterior inference for this model is possible via a mixture KF [11, 40] which represents the state posterior as a Gaussian mixture. The number of mixture components grows exponentially with time, motivating PFs for long-term tracking, but remains tractable over a few time steps.

Using a fixed dataset and gradient descent, we train discriminative PFs using the IRG estimator, our IWSG estimator, as well as with biased gradients that are truncated at each resampling step [5]. We initialize with perturbed parameter values and aim to learn the values of all dynamics and likelihood parameters, except for σ and γ which are fixed to their true values. See Appendix for details.

Fig. 4 shows the estimates and gradients for a subset of the learned parameters, and several gradient estimators. IRG's unstable gradients prevent convergence of all parameters. For training (generative) marginal PFs [41] which approximate marginals with mixtures, Lai et al. [42] found that IRG gradient variance was so large that biased estimators converged faster, but provided no detailed analysis. In contrast, IWSG converges smoothly for all parameters, and more rapidly than truncated gradients.

5 Mixture Density Particle Filters

Our mixture density particle filter (MDPF) is a discriminative model inspired by regularized PFs, where resampling uses KDEs centered on the current particles. We apply IWSG to this mixture resampling, achieving unbiased and low-variance gradient estimates for our fully-differentiable PF.

MDPF does not incorporate human-specified dynamics or measurement models; some prior work assumed known dynamics to simplify learning [5, 7]. These models are parameterized as deep neural networks (NNs), and trained via stochastic gradient descent to minimize the negative-log-likelihood of (potentially sparsely labeled) states x t , given observations y t and (optionally) actions a t . As shown in Fig. 2, dynamics and measurement models are flexible composed from smaller NNs. Our MDPF places no constraints on the functional form of NNs, allowing for modern architectures such as CNNs [43] and Spatial Transformers [44] to be used. See Appendix for implementation details.

The MDPF uses KDE mixture distributions as in Eq. (4) to define state posteriors used to evaluate the training loss, as well as resampling. A bandwidth parameter β is thus required for each state dimension. Rather than setting β via the classic heuristics discussed in Sec. 2.2, we make β a learnable parameter, optimizing it using end-to-end learning along with the dynamics and measurement models. This contrasts with prior work including SR-PF [7] and C-PF and OT-PF [8], which all include relaxation hyperparameters that must be tuned via multiple (potentially expensive) training trials.

We also extend the MDPF by decoupling the mixtures used for particle resampling and posterior state estimation. Using two separate measurement models, we compute two sets of particle weights ˜ w (:) t

Bearings Only

LSTM

TG-PF

OT-PF

SR-PF

DIS-PF

C-PF

TG-MDPF

IRG-MDPF

MDPF

A-MDPF

6

7

8

9

10

11

Negative Log-likelihood

Epanechnikov

LSTM

TG-PF

OT-PF

SR-PF

DIS-PF

C-PF

TG-MDPF

IRG-MDPF

MDPF

A-MDPF

3

4

5

6

7

8

9

10

Root-Mean-Square-Error

Epanechnikov

No Distribution Needed

LSTM

TG-PF

OT-PF

SR-PF

DIS-PF

C-PF

TG-MDPF

IRG-MDPF

MDPF

A-MDPF

5.5

6.0

6.5

7.0

7.5

8.0

8.5

9.0

9.5

10.0

Negative Log-likelihood

Deepmind Maze

LSTM

10

12

14

IRG-MDPF

17.0

17.5

Maze 1

Maze 2

Maze 3

LSTM

TG-PF

OT-PF

SR-PF

DIS-PF

C-PF

TG-MDPF

IRG-MDPF

MDPF

A-MDPF

3.5

4.0

4.5

5.0

5.5

Root-Mean-Square-Error

Deepmind Maze

IRG-MDPF

10

15

Maze 1

Maze 2

Maze 3

Bearings Only

and w (:) t , one for resampling and the other for posterior state estimation; see Fig. 2. Each distribution is also given a separate bandwidth ˜ β , β . This adaptive mixture density PF (A-MDPF) allows the uncertainty in the estimation of the current latent state, and the degree of exploration in the particle resampling step, to be decoupled and separately optimized during training. Note that training of separate posterior and resampling distributions is impossible for PFs that truncate temporal gradients; it is only feasible due to our unbiased and low-variance IWSG estimator of resampling gradients.

6 Experiments

We evaluate our MDPF and A-MDPF on a variety state estimation tasks in complex, but simulated, environments. We compare to TG-PF [5], SR-PF [7], OT-PF [8], DIS-PF [6], as well as C-PF which uses the Concrete distribution [32, 31] to relax discrete resampling. We also compare to variants of MDPF: Truncated-Gradient-MDPF (TG-MDPF) is similar but stops gradients at each particle resampling, and IRG-MDPF replaces our IWSG estimator with the unstable IRG estimator. Finally, we compare to a LSTM [45] which produces point predictions via uninterpretable latent states.

For all tasks, we estimate posterior distributions of a 3D (translation and angle) state of a robot, x t = ( x, y, θ ) . We use Gaussian kernels for position components, and von Mises kernels for angular components, of all KDEs. As is common in recurrent neural network (RNN) training, we employ conservative gradient clipping [46], as well as truncated-BPTT [47] where gradients are propagated for 4 time-steps. All methods are initialized as the true state with added noise.

We train and evaluate MDPFs to minimize the following negative-log-likelihood (NLL) loss:

We use sparsely labeled training data to highlight the need for effective multi-time-step gradients during training. Dense labeling of data is often expensive, so many real-world datasets have similarly sparse labels. During training, we label true states every 4th time-step ( T = { 4 , 8 , 12 , . . . } ), and use densely labeled datasets ( T = { 1 , 2 , 3 , . . . } ) for evaluation. See the Appendix for details.

After training to minimize NLL, we optionally refine our MDPFs to minimize the mean-squared-error (MSE) of the weighted particle mean, and report the induced Root-MSE (RMSE). Baseline methods do not estimate continuous posteriors, and thus Eq. (18) cannot be used for training. Instead we train and evaluate these methods with MSE and RMSE, before freezing the dynamics and measurement models. Then we construct KDEs, and optimize a bandwidth to report NLL values for comparison.

Bearings-Only Tracking. In the bearings only tracking task, we track the state of a variable-velocity car using noisy bearing observations from a radar station to the car generated as

where ψ ( x t ) is the true bearing and α = 0 . 15 . Actions are not available. We use 5000 training and 1000 validation trajectories of length T = 17 , and 5000 evaluation trajectories of length T = 150 .

A-MDPF

Time-step: #02

Time-step: #06

Time-step: #10

Time-step: #14

Time-step: #15

Time-step: #17

Time-step: #18

Time-step: #19

Time-step: #22

Time-step: #26

Time-step: #30

Time-step: #34

Time-step: #38

Time-step: #42

Time-step: #46

Time-step: #48

TG-PF

DIS-PF

OT-PF

C-PF

True State

Mean Particle

Particles

True State

Multimodal Init

t=01

=02

t=03

t=04

t=07

t=10

t=13

=16

t=19

01

=03

10

=13

=16

4

t=04

t=07

=10

=13

=16

1

=24

Mean Particle

Particles / Posterior Density

Time-step #1

MDPF

A-MDPF

TG-MDPF

TG-PF

OT-PF

SR-PF

DIS-PF

LSTM

Time-step #15

Time-step #30

True State

Mean Particle

Posterior Density

Each method is evaluated and trained 11 times using N = 25 particles. To further highlight the stability of our IWSG estimator, we also train each method (except for IRG-MDPF) with the Epanechnikov kernel replacing the Gaussian in the KDE. Fig. 5 shows statistics of performance across 11 training runs. The very poor performance of IRG-MDPF is due to unstable gradients; without aggressive gradient clipping, IRG-MDPF fails to optimize at all. The benefits of IWSG are highlighted by the inferior performance of TG-MDPF. Stable multi-time-step gradients clearly benefit learning and are enabled by IWSG. Qualitative results are shown in Fig. 6.

Deepmind-Maze Tracking. In the Deepmind-Maze tracking task, adapted from [5], we wish to track the 3D state of a robot as it moves through modified versions of the maze environments from Deepmind Lab [48] using noisy odometry as actions, and images (from the robot's perspective) as observations. We alter the experimental setup of [5] by increasing the noise of the actions by five times, and using N = 25 particles for training and evaluation. We train and evaluate on trajectories from the 3 unique mazes separately, using 5000 trajectories of length T = 17 from each maze for training, and 1000 trajectories of length T = 99 for evaluation. We show results over multiple training and evaluation runs in Fig. 5, and include qualitative results in the Appendix.

House3D Tracking. This 3D state tracking task is adapted from [7], where a robot navigates single-level apartment environments from the SUNCG dataset [49] using the House3D simulator [50], with noisy odometry as action and RGB images as observations (see Fig. 7). Floor plans are input into measurement model via a modern Spatial Transformation Network architecture [44]. Training data consists of 74800 trajectories (length T = 24 ) from 199 unique environments, with evaluation data being 820 trajectories (length T = 100 ) from 47 previously unseen floorplans. We train and evaluate with N = 50 particles.

Due to the larger computational demands of House3D, we train each method once, reporting results in Table 1. Interestingly we find that TG-PF, OT-PF, SR-PF, and DIS-PF are unable to learn useful dynamics or measurement models. The LSTM model achieves better performance, but looking deeper we discover that it ignores observations completely, and simply blindly propagates the estimated state using actions with good dynamics (i.e., deadreckoning). This strategy is effective for noise-free actions, but in reality the actions are noisy and thus the LSTMs estimated state diverges from the true state after a moderate number of time-steps. In contrast, our MDPF methods are able to learn good dynamics and measurement models that generalizes well to the unseen environments.

We also investigate MDPFs capacity for multimodal tracking in Fig. 7, by initializing A-MDPF with particles spread throughout the state space. When initialized with random particles or with particles clustered around regions with similar observations, A-MDPF does not have sufficient information to collapse the posterior and thus maintains multiple posterior modes. As the robot moves, more evidence is collected, allowing A-MDPF to collapse the state posterior to fewer distinct modes, before eventually collapsing its estimate into one mode containing the true state. Note that such estimation of multiple posterior modes is impossible for standard RNNs like the LSTM.

Limitations. Aknown limitation of all PFs is there inability to scale to very high-dimensional states. Sparsity increases as the dimension grows, and thus more particles (and consequently computation) are required to maintain the expressiveness and accuracy of the state posterior. Our MDPF is not immune to this issue, but we conjecture that end-to-end training of discriminative PFs will allow particles to be more effectively allocated within large state spaces, scaling better than classic PFs.

7 Discussion

We have developed a novel importance-sampling estimator for gradients of samples drawn from a continuous mixture model. Our results highlight fundamental flaws in the application of (implicit) reparameterization gradients to any mixture model with multiple modes. Applying our gradient estimator to the resampling step of a regularized discriminative PF, we obtain a fully end-to-end differentiable MDPF that robustly learns accurate models across multiple training runs, and has much greater accuracy than the biased PFs proposed in prior work. While our experiments have focused on synthetic tracking problems in complex virtual environments, future applications of our MDPF to real-world vision and robotics data are very promising.

References

Neil J Gordon, David J Salmond, and Adrian FM Smith. Novel approach to nonlinear/nongaussian bayesian state estimation. IEE proceedings F-Radar and Signal Processing , 1993.

K. Kanazawa, D. Koller, and S. Russell. Stochastic simulation algorithms for dynamic probabilistic networks. In UAI 11 , pages 346-351. Morgan Kaufmann, 1995.

A. Doucet, N. de Freitas, and N. Gordon, editors. Sequential Monte Carlo Methods in Practice . Springer-Verlag, New York, 2001.

M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Proc. , 50(2):174-188, February 2002.

Rico Jonschkowski, Divyam Rastogi, and Oliver Brock. Differentiable particle filters: Endto-end learning with algorithmic priors. Proceedings of Robotics: Science and Systems (RSS) , 2018.

Adam ´ Scibior, Vaden Masrani, and Frank Wood. Differentiable particle filtering without modifying the forward pass. International Conference on Probabilistic Programming (PROBPROG) , 2021.

Peter Karkus, David Hsu, and Wee Sun Lee. Particle filter networks with application to visual localization. Conference on Robot Learning (CORL) , 2018.

Adrien Corenflos, James Thornton, George Deligiannidis, and Arnaud Doucet. Differentiable particle filtering via entropy-regularized optimal transport. International Conference on Machine Learning (ICML) , 2021.

Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering , 1960.

T. Kailath, A. H. Sayed, and B. Hassibi. Linear Estimation . Prentice Hall, New Jersey, 2000.

Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents) . The MIT Press, 2005. ISBN 0262201623.

S. J. Julier and J. K. Uhlmann. Unscented filtering and nonlinear estimation. Proc. IEEE , 92(3): 401-422, March 2004.

Víctor Elvira, Luca Martino, and Christian Robert. Rethinking the effective sample size. International Statistical Review , 2022.

Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In ICML , page II-1791-1799, 2014.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR) , 2014.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML , volume 32, pages 1278-1286, 2014.

N. Oudjane and C. Musso. Progressive correction for regularized particle filters. International Conference on Information Fusion , 2000.

C. Musso, N. Oudjane, and F. Le Gland. Improving regularized particle filters. In A. Doucet, N. de Freitas, and N. Gordon, editors, Sequential Monte Carlo Methods in Practice , pages 247-271. Springer-Verlag, 2001.

B. W. Silverman. Density Estimation for Statistics and Data Analysis . Chapman & Hall, 1986.

Adrian W Bowman. An alternative method of cross-validation for the smoothing of density estimates. Biometrika , 1984.

MChris Jones, James S Marron, and Simon J Sheather. A brief survey of bandwidth selection for density estimation. Journal of the American Statistical Association , 1996.

Nikolas Kantas, Arnaud Doucet, Sumeetpal S Singh, Jan Maciejowski, and Nicolas Chopin. On particle methods for parameter estimation in state-space models. Statistical Science , 30(3): 328-351, 2015.

Tuomas Haarnoja, Anurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop kf: Learning discriminative deterministic state estimators. Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Minyoung Kim and Vladimir Pavlovic. Conditional state space models for discriminative motion estimation. IEEE International Conference on Computer Vision , 2007.

John D. Lafferty, Andrew McCallum, and Fernando Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. International Conference on Machine Learning (ICML) , 2001.

Benson Limketkai, Dieter Fox, and Lin Liao. Crf-filters: Discriminative particle filters for sequential state estimation. IEEE International Conference on Robotics and Automation (ICRA) , 2007.

Rob Hess and Alan Fern. Discriminatively trained particle filters for complex multi-object tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2009.

Michael Zhu, Kevin P. Murphy, and Rico Jonschkowski. Towards differentiable resampling. ArXiv , abs/2004.11938, 2020.

Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 1990.

Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DiCE: The infinitely differentiable Monte Carlo estimator. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 1529-1538, 2018.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. International Conference on Learning Representations (ICLR) , 2016.

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. International Conference on Learning Representations (ICLR) , 2016.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NeurIPS) , 2013.

V. A. Epanechnikov. Non-parametric estimation of a multivariate probability density. Theory of Probability & Its Applications , 1969.

Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparameterization trick. International Conference on Machine Learning (ICML) , 2018.

Martin Jankowiak and Theofanis Karaletsos. Pathwise derivatives for multivariate distributions. International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Alex Graves. Stochastic backpropagation through mixture density distributions. arXiv preprint arXiv:1607.05690 , 2016.

Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Ludger Rüschendorf. Copulas, Sklar's Theorem, and Distributional Transform . Springer Berlin Heidelberg, 2013.

D. Alspach and H. Sorenson. Nonlinear bayesian estimation using Gaussian sum approximations. IEEE Transactions on Automatic Control , 1972.

Mike Klaas, Nando de Freitas, and Arnaud Doucet. Toward practical N 2 Monte Carlo: The marginal particle filter. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence , pages 308-315, 2005.

Jinlin Lai, Justin Domke, and Daniel Sheldon. Variational marginal particle filters. International Conference on Artificial Intelligence and Statistics (AISTATS) , 2022.

Keiron O'Shea and Ryan Nash. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458 , 2015.

Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems (NeurIPS) , 28, 2015.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 1997.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. International Conference on Machine Learning (ICML) , 2013.

Herbert Jaeger. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the" echo state network" approach. GMD-Forschungszentrum Informationstechnik Bonn , 2002.

Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801 , 2016.

Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209 , 2018.

Marco Frei and Hans R. Künsch. Mixture ensemble kalman filters. Computational Statistics and Data Analysis , 2013.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 2014.

A Parameterization of the Neural Dynamics and Measurement Models

In our implementation of MDPF and A-MDPF, we define the dynamics and measurement models as a set of small learnable neural networks, Fig. 2. The dynamics model uses two encoder networks which encode the particles and the actions into latent dimensions. The particle encoder is applied individually to each particle, allowing for parallelization. The encoded actions and particles are used as input, along with zero-mean unit variance Gaussian noise, into a residual network to produce new particles.

The measurement model is also comprised of several small neural networks. Particles and observations are encoded into latent representations before being used as input into a simple feed-forward network. Our method makes no restrictions on the network architectures allowing modern architectures to be used. For high dimensional observations, such as images, or for more complex problems a sophisticated architecture such as Convolutional Neural Networks [43] or Spatial Transformer Networks [44] can be implemented.

Since particle states are interpretable, they are often in a representation not suitable for neural networks. For example angles contain a discontinuities (at 0 and 2 π ) which hinders their direct use with neural networks. To address this, non-learned transforms are applied to the particles, converting them into representations that are better suited for neural networks. In the case of angles transforms convert angles to and from unit vectors:

B Synthetic Learning Problem Details

In section 4, a simple synthetic PF learning problem was presented to illustrate the smooth and stable gradient estimates produced by our Importance Weighted Samples Gradient (IWSG) estimator while also highlighting the instability of reparameterization methods when applied to mixture models. Here we give additional details about this learning problem.

In this learning problem we define several regularized PFs with linear Gaussian dynamics and bimodal observation likelihoods:

For each PF method, we set the dynamics and measurement models to the true linear Gaussian dynamics and bimodal observation with perturbed parameters with the goal of learning the true values of the parameters. In this problem we hold σ and γ fixed and only perturb and learn the other parameters: A , B , C 1 , C 2 , c 1 , c 2 , w 1 and w 2 .

We train three variants of the regularized PF with differing gradient estimation procedures for resampling. IWSG-PF uses our IWSG estimator, IRG-PF uses the implicit reparameterization gradients estimator, and TG-PF does not use any gradient estimator for mixture resampling; instead it simply truncates gradients to zero at resampling steps. We compare the stability and convergence of these three methods when learning the parameter values. For this problem, we decouple the resampling and posterior bandwidth parameters as in our A-MDPF model. For resampling we set ˜ β = 0 . 05 , and for the posterior we set β = 0 . 5 .

We also train a mixture Kalman Filter (KF) [11, 40] with the same true dynamics and measurement models, also initializing with perturbed parameters. Since this model has linear dynamics with Gaussian noise, and each component of the observation distribution mixture is also linear and Gaussian, we can apply mixture KFs to compute the exact posterior as a mixture of Gaussians. Dynamics are applied to each Gaussian component of the mixture using the KF update equations [11], keeping the number of components the same. When applying the measurement model, we simply multiply the KF mixture with the observation likelihood mixture, increasing the number of mixture components in the KF posterior. The number of components grows exponentially with time, so the mixture KF is not a practical general inference algorithm, but the computation remains tractable for short sequences. Special care must be taken when computing the weights of each mixture component after applying the observation likelihoods; we adapt the approach of Frei and Künsch [51].

To make the learning problem identifiable and to enforce the constraints w 1 + w 2 = 1 , w 1 ≥ 0 , w 2 ≥ 0 , we parameterize w 1 and w 2 within the PF and KF models via the logistic of a single learnable parameter v :

To learn the dynamics and measurement model parameters, we setup a synthetic training problem with a dataset of 1000 sequences, each of length 5 time-steps. Using stochastic gradient descent with batch size 64, we train to minimize the negative log-likelihood loss of the true state given the posterior distribution at only the final time-step. For PF models, we use kernel density estimation [19] to estimate a mixture distribution using the final weighted particle set. For the mixture KF, the posterior distribution is simply a mixture model and no further processing is needed to compute the loss.

Fig. 9 shows the convergence and gradient estimates for all of the learned parameters when using the different gradient estimators. IRG's unstable gradients prevents convergence of all parameters, while IWSG converges smoothly and more rapidly than truncated gradients. Further we see that for parameters c 1 and c 2 , the TG-PF method does not converge to the true value. TG-PF truncates gradients at the resampling, resulting in biased gradients. When the loss is computed at the final time, only information gained from that final time-step is present when updating the parameters, resulting in biased updates.

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.6

0.8

1.0

1.2

1.4

Estimate Value

Learned Estimate of

A

10 0

10 1

10 2

10 3

10 4

Training Iteration

50

0

50

Estimated Gradient Value

Gradients Estimate for

A

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.25

Estimate Value

Learned Estimate of

B

10 0

10 1

10 2

10 3

10 4

Training Iteration

50

0

50

Estimated Gradient Value

Gradients Estimate for

B

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.0

0.2

0.4

0.6

0.8

1.0

Estimate Value

Learned Estimate of

w

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

10

5

0

5

10

Estimated Gradient Value

Gradients Estimate for

w

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Estimate Value

Learned Estimate of

C

1

10 0

10 1

10 2

10 3

10 4

Training Iteration

50

0

50

Estimated Gradient Value

Gradients Estimate for

C

1

10 0

10 1

10 2

10 3

10 4

Training Iteration

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Estimate Value

Learned Estimate of

C

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

50

0

50

Estimated Gradient Value

Gradients Estimate for

C

2

10 0

10 1

10 2

10 3

10 4

Training Iteration

3.6

3.8

4.0

4.2

4.4

Estimate Value

Learned Estimate of

c

1

10 0

10 1

10 2

10 3

10 4

Training Iteration

50

0

50

Estimated Gradient Value

Gradients Estimate for

c

1

10 0

10 1

10 2

10 3

10 4

0.2

0.1

0.0

0.1

0.2

Estimate Value

Learned Estimate of

c

2

10 0

10 1

10 2

10 3

10 4

50

0

50

Estimated Gradient Value

Gradients Estimate for

c

2

IRG-PF

IWSG-PF

TG-PF

Mixture KF

Truth

Training Iteration

Training Iteration

B.1 Computation Requirements

The computational requirements for our MDPF and A-MDPF methods are quadratic with the number of particle during training, O ( N 2 ) , due to gradient computation. During inference, the computation

0

200

400

600

800

1000

Number of Particles

10

1

10 0

10 1

Time per Batch (Seconds)

Training

0

200

400

600

800

1000

Number of Particles

10

1

10 0

10 1

Inference

TG-MDPF

MDPF

A-MDPF

TG-PF

C-PF

DIS-PF

SR-PF

OT-PF

requirement is linear in the number of particles, O ( N ) , since we mearly have to sample from the resampling mixture distribution rather than computing gradients for the samples. This is highlighted in fig. 10. Other methods except OT-PF are have computational complexity O ( N ) during both inference and training. OT-PF requires solving an optimal transport problem with complexity O ( N 2 ) during both training and inference [8]. In practice we find that OT-PF has significant computational requirements resulting in slow run-times both at inference and testing.

C Additional Experiment Results

In this section we give additional qualitative experiment results for the various tracking tasks.

C.1 Bearings Only Tracking Task

In fig. 11 we show additional qualitative results for various PF methods where it is apparent that A-MDPF and MDPF both track the true state well whereas other methods have wide posteriors due to their poor tracking ability. Figs. 12 13 14, 15, 16, 17 and 18 show additional trajectories for the various PF methods.

A-MDPF

Time-step: #02

Time-step: #06

Time-step: #10

Time-step: #14

Time-step: #15

Time-step: #17

Time-step: #18

Time-step: #19

Time-step: #22

Time-step: #26

Time-step: #30

Time-step: #34

Time-step: #38

Time-step: #42

Time-step: #46

Time-step: #48

TG-PF

DIS-PF

OT-PF

C-PF

True State

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

True State

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

Mean Particle

Particles

Time-step #01

Time-step #02

Time-step #04

Time-step #06

Time-step #08

Time-step #10

Time-step #12

Time-step #14

Time-step #16

Time-step #18

Time-step #20

Time-step #22

Time-step #24

Time-step #26

Time-step #28

Time-step #30

Time-step #32

Time-step #34

Time-step #36

Time-step #38

Time-step #40

Time-step #42

Time-step #44

Time-step #46

Time-step #48

Time-step #50

Time-step #52

Time-step #54

Time-step #56

Time-step #58

Time-step #60

Time-step #62

Time-step #64

Time-step #66

Time-step #68

Time-step #70

Time-step #72

Time-step #74

Time-step #76

Time-step #78

True State

Mean Particle

Particles

C.2 Deepmind Maze Tracking Task

In figs. 19, 20 and 21 we show additional qualitative results for various PF methods for mazes 1, 2 and 3, respectively. Again it is apparent that A-MDPF and MDPF both track the true state well whereas other methods have wide posteriors due to their poor tracking ability.

Time-step #01

Time-step #03

Time-step #05

Time-step #07

Time-step #09

Time-step #11

Time-step #13

Time-step #15

Time-step #17

Time-step #19

Time-step #21

Time-step #23

A-MDPF

MDPF

TG-MDPF

TG-PF

DIS-PF

OT-PF

SR-PF

C-PF

True State

Mean Particle

Particles

Time-step #01

Time-step #03

Time-step #05

Time-step #07

Time-step #09

Time-step #11

Time-step #13

Time-step #15

Time-step #17

Time-step #19

Time-step #21

Time-step #23

A-MDPF

MDPF

TG-MDPF

TG-PF

DIS-PF

OT-PF

SR-PF

C-PF

True State

Mean Particle

Particles

True State

Mean Particle

Particles

Time-step #01

Time-step #03

Time-step #05

Time-step #07

Time-step #09

Time-step #11

Time-step #13

Time-step #15

Time-step #17

Time-step #19

Time-step #21

Time-step #23

A-MDPF

MDPF

TG-MDPF

TG-PF

DIS-PF

OT-PF

SR-PF

C-PF

C.3 House3D Tracking Task

Figs. 22 and 23 show example trajectories for the House3D tracking task using A-MDPF and MDPF respectively. We see that our A-MDPF and MDPF methods both successfully track the true state over time as evident by the tight posterior over the true state as well as the closeness of the mean state to the true state. Fig. 24 shows an example trajectory using the LSTM model. It is clear that the LSTM model is ignoring observations and instead is blind propagating its state estimate using good dynamics and the actions (dead-reckoning). In later time-steps, the LSTM estimate drifts away from the true state however the predicted state trajectory looks similar to that of the true state, but with a growing offset, indicating blind propagation.

Fig. 25 shows qualitative results when using OT-PF. OT-PF is unable to learn good dynamics and measurement models and quickly diverges away from true state. Figs. 26, 27 and 28 give example trajectories for TG-PF, DIS-PF and SR-PF respectively. These methods fail to learn usable dynamics or measurement models with their trajectories essentially being random.

↓

True State

Mean Particle

Posterior Density

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

1

t=31

t=32

t=33

t=34

t=35

t=37

t=38

6E=→

t=40

True State

Mean Particle

Posterior Density

↓

t=01

t=03

t=04

t=05

90=

t=07

t=08

t=09

t=10

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

t=31

t=32

t=33

t=34

t=35

t=37

t=38

t=39

t=40

皮

True State

Mean Particle

Posterior Density

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

AL

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

t=31

t=32

t=33

t=34

t=35

t=37

t=38

t=39

t=40

True State

Mean Particle

Posterior Density

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

t=31

t=32

t=33

t=34

t=35

t=37

t=38

t=39

t=40

→

True State

Mean Particle

Posterior Density

→

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

7.

v-ti

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

?

t=31

t=32

t=33

t=34

t=35

t=37

t=38

t=39

t=40

?

→

True State

Mean Particle

Posterior Density

↓

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

X

1.3

我!

t=31

t=32

t=33

t=34

t=35

t=37

t=38

6=

t=40

True State

Mean Particle

Posterior Density

↓

t=01

t=03

t=04

t=05

t=06

t=07

t=08

t=09

t=10

一t

t=11

t=12

t=13

t=14

t=15

t=16

t=17

t=18

t=19

t=21

t=23

t=24

t=25

t=26

t=27

t=28

t=29

t=30

2!

K.7

6.7

t=31

t=32

t=33

t=34

t=35

t=37

t=38

t=39

t=40

D Experiment Details

In this section give additional details for the various experiments (tasks). We also provide specific neural architectures used in each task.

D.1 Mean Squared Error Loss

When training the comparison PF methods, we use the Mean-Squared Error (MSE) loss. This is computed as the mean squared error between the sparse true states s t and the mean particle at each time t :

Here T are the indices of the labeled true states. During training, sparsely labeled true states are used and thus T = { 4 , 8 , 12 , . . . } . During evaluation we use dense labels, T = { 1 , 2 , 3 , 4 , . . . }

For angular dimensions of x ( i ) t , we compute the mean particle by converting the angle into a unit vector, averaging the unit vectors (with normalization) and converting the mean unit vector back into an angle. Further we take special care when computing square differences for angles due to the discontinuity present at 0 and 2 π .

D.2 Pre-Training Measurement Model

We pre-train the measurement model for each task using the sparse true states. To pre-train, a random true state from the dataset is selected. A particle set is constructed by drawing samples from a distribution centered around the select true state and weights for this particle set are computed using the measurement model and the observation associated with the selected true state. A mixture distribution is then fit using KDE with a manually specified bandwidth. Negative Log-Likelihood loss of the true state is used as the training loss. We find pre-training not to be sensitive to choice of bandwidth so long as it is not too small.

The dynamics model cannot be pre-trained because of the lack of true state pairs due to the sparse true state training labels and thus must be trained for the first time within the full PF algorithm.

D.3 Additional General Task and Training Details

For all tasks we wish to track the 3D (translation and angle) state of a robot, s = ( x, y, θ ) , over time. This state space contains an angle dimension which is not suitable for neural networks. To address this issue, we convert particles into 4D using a transform T ( s ) = ( x, y, sin( θ ) , cos( θ )) before using them as input into a neural network. Further we convert back into the original 3D representation to retrieve the a 3D particle after applying neural networks to the particles.

We use the Adam [52] optimizer with default parameters, tuning the learning rate for each task. We use a batch size of 64 during training and decrease the learning rate by a factor of 10 when the validation loss reaches a plateau. We use Truncated-Back-Propagation-Through-Time (T-BPTT) [47], truncating the gradients every 4 time-steps.

For all tasks we use the Gaussian distribution for the positional components of the state and the Von Mises distribution for the angular components when applying KDE. Here β is a dimension-specific bandwidth corresponding to the two standard deviations for the Gaussians and a concentration for the Von Mises.

Some methods have non-learnable hyper-parameters that must be set. The values of the hyperparameters are chosen via brief hyper-parameter search with the final chosen values shown in table 2. The hyper-parameter values are the same for all tasks unless otherwise specified. We find the regularization parameter λ for OT-PF to be sensitive to the specific datasets.

We choose to make the dynamics position invariant. This prevents the learned PF models from memorizing the state space based on position but rather forces the learned dynamics to the true dynamics. Position invariance can be achieved by masking out the translational dimensions of the particle before input into the dynamics model.

D.4 Bearings Only Tracking Task

Input: Transformed Particle

Mask out Translational Component

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (8)

Output

(a) Particle encoder for dy- namics Model.

Input: Transformed Particle

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (8)

Output

(b) Particle encoder for mea- surement model.

Input: Observation

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (8)

Output

(c) Observation encoder.

Input: Encoded Particle

Input:

η

Concatenate

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (4)

Output Scaling

Output

Input: Transformed Particle

Concatenate

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (1)

Sigmoid

Scale output to be in:

[1

-5

,1]

Output

Input: Encoded Observation

Input: Encoded Particle

(d) Dynamics model residual network, with actions omitted. Output scaling all dimensions to be within [-0.99, 0.99].

(e) Measurement model weights network.

Figure 29: Neural architecture for the networks within the dynamics and measurement models for the bearings only tracking task.

In this section we give additional task specific information and network architecture details for the bearings only tracking task.

D.4.1 Additional Experiment Details

When training MDPF using the Negative Log-Likelihood (NLL) loss, we set the learning rate (LR) of the neural networks to be 0 . 0005 and the bandwidth learning rate to 0 . 00005 . When training

using MSE loss, we use 0 . 0001 and 0 . 00001 for the neural network and bandwidth learning rates respectively. For the discrete PF methods, we use LR = 0 . 0005 for training the neural network using the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the LSTM [45], we use 0 . 001 as the learning rate for training the networks and LR = 0 . 0005 when optimizing the bandwidth parameter using NLL.

We apply gradient norm clipping, clipping the norm of the gradients to be ≤ 100 . 0 .

In addition to using the Gaussian distribution for KDE, we also train with the Epanechnikov distribution applied to the translational dimensions of the state. To do this we first train using Gaussian kernels before re-training with the Epanechnikov, using the already trained dynamics and measurement models as a starting point.

The observations o t are generated as noisy bearings from the radar station to the car:

where ψ t is the true bearing, α = 0 . 15 and κ = 50 . Observations o t are angles and not suitable as input into a neural network. As such we convert the angular representation into a unit vector as is done with the angle dimensions of the particles.

D.4.2 Network Architectures

Fig. 29 gives details on the neural network architectures used for the dynamics and measurement models within the PF for the bearings only tracking task. Since this task does not provide actions, the action encoder has been omitted.

D.5 Deepmind Maze Tracking Task

In this section we give additional task specific information and network architecture details for the Deepmind maze tracking task.

D.5.1 Additional Experiment Details

The Deepmind maze tracking task was first proposed in [5]. In this task, a robot moves through modified versions of the maze environments from Deepmind Lab [48]. Three distinct mazes are provided and we train and test on each maze individually. Example robot trajectories as well as example observations for each maze are shown in fig. 30. We modify this task by increasing the noise applied to the actions by 5 fold as well as using sparsely labeled true states during training.

When training MDPF using the Negative Log-Likelihood (NLL) loss, we set the learning rate (LR) of the neural networks to be 0 . 0005 and the bandwidth learning rate to 0 . 00005 . When training using MSE loss, we use 0 . 0005 and 0 . 00005 for the neural network and bandwidth learning rates respectively. For the discrete PF methods, we use LR = 0 . 0005 for training the neural network using the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the LSTM, we use 0 . 001 as the learning rate for training the networks and LR = 0 . 0005 when optimizing the bandwidth parameter using NLL.

Input: Observation

3x3 Conv  (16) + ReLU

Fully Connected  (64) + ReLU

Output

3x3 Conv  (32) + ReLU

3x3 Conv  (64) + ReLU

Fully Connected  (64)

(a) Observation en- coder.

Input: Transformed Particle

Mask out Translational Component

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (8)

Output

(b) Particle encoder for dynamics Model.

Input: Transformed Particle

Fully Connected  (32) + ReLU

Fully Connected  (32)

Output

(c) Particle encoder for measurement model.

Input: Action

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (8)

Output

(d) Action encoder for dy- namics Model.

Concatenate

Fully Connected  (8) + ReLU

Fully Connected  (8) + ReLU

Fully Connected  (4)

Output Scaling

Output

Input: Encoded Particle

Input:

η

Input: Transformed Particle

Input: Encoded Action

Concatenate

Fully Connected  (64) + ReLU

Fully Connected  (64) + ReLU

Fully Connected  (1)

Sigmoid

Scale output to be in:

[1

-5

,1]

Output

Input: Encoded Observation

Input: Encoded Particle

(e) Dynamics model residual network. Output scaling scales the transnational dimensions to be within [-7, 7] and the unit vector representing angles to be within [-1, 1].

(f) Measurement model weights network.

We apply gradient norm clipping, clipping the norm of the gradients to be ≤ 500 . 0 . Further we limit the bandwidth for the angle dimension such that β ≤ 4 . 0 . We do this only for this task as non-MDPF methods were prone to learning an unreasonably large bandwidth resulting in a near uniform distribution for the angular part of the state.

D.5.2 Network Architectures

Fig. 31 gives details on the neural network architectures used for the dynamics and measurement models within the PF for the Deepmind maze tracking task.

D.6 House3D Tracking Task

In this section we give additional task specific information and network architecture details for the House3D tracking task.

D.6.1 Additional Experiment Details

When training MDPF using the Negative Log-Likelihood (NLL) loss, we set the learning rate (LR) of the neural networks to be 0 . 0005 and the bandwidth learning rate to 0 . 00005 . When training using MSE loss, we use 0 . 0001 and 0 . 00001 for the neural network and bandwidth learning rates respectively. For the discrete PF methods, we use LR = 0 . 0005 for training the neural network using the MSE loss. We use the same bandwidth for optimizing the bandwidth using the NLL loss when the neural networks are held fixed. For the LSTM, we use 0 . 001 as the learning rate for the LSTM layers and LR = 0 . 0005 when optimizing the bandwidth parameter using NLL.

We apply gradient norm clipping, clipping the norm of the gradients to be ≤ 250 . 0 when training with NLL loss. When training with MSE loss, we disable gradient norm clipping.

D.6.2 Network Architectures

Fig. 31 gives details on the neural network architectures used for the dynamics and measurement models within the PF for the House3D tracking task.

Input: Observation

3x3 Conv  (16) + PReLU

Fully Connected  (64) + PReLU

Output

3x3 Conv  (32) + PReLU

3x3 Conv  (64) + PReLU

Fully Connected  (64)

(a)

Observation

en-

coder.

Input: Transformed Particle

Mask out Translational Component

Fully Connected  (32) + PReLU

Fully Connected  (32) + PReLU

Fully Connected  (32)

Output

(b)

Particle

encoder

Input: Local Map

3x3 Conv  (16) + PReLU

Fully Connected  (64) + PReLU

Output

3x3 Conv  (32) + PReLU

3x3 Conv  (64) + PReLU

Fully Connected  (64)

(c) Particle encoder for

measurement model.

Input: Action

Fully Connected  (32) + PReLU

Fully Connected  (32) + PReLU

Fully Connected  (32)

Output

(d) Action encoder for dy-

namics Model.

Concatenate

Fully Connected  (32) + PReLU

Fully Connected  (32) + PReLU

Fully Connected  (4)

Output Scaling

Output

Input: Encoded Particle

Input:

η

Input: Transformed Particle

Input: Encoded Action

(e) Dynamics model residual network. Output scaling scales the transnational dimensions to be within [-75, 75] and the unit vector representing angles to be within [-0.99, 0.99].

Input: Encoded Observation

Input: Particle

Concatenate

Fully Connected  (64) + PReLU

Fully Connected  (64) + PReLU

Fully Connected  (1)

Sigmoid

Scale output to be in:

[1

-5

,1]

Output

Create local Map

Encoder

(From Spatial Transformer Network)

(f) Measurement model weights net-

work.

for dynamics Model.

D.7 LSTM Model Details

We train using LSTM [45] based recurrent neural network models for all tasks as a baseline comparison. For the LSTM, we use the same encoders as the PF methods. Fig. 33 shows the LSTM network architecture used for all tasks. We use the same output scaling the PF methods.

Input: Encoded Action

Output Scaling

Output

Input: Transformed Particle

Input: Encoded Observation

Input: Encoded Particle

Concatenate

Fully Connected  (256) + PReLU

Fully Connected  (4)

Fully Connected  (256) + PReLU

Fully Connected  (256) + PReLU

LSTM  (256)

LSTM  (256)

LSTM  (256)

LSTM  (256)

Output: Hidden States

Input: Hidden States

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the \"importance weighted samples gradient (IWSG)\" estimator and describes its integration into a \"mixture density particle filter (MDPF)\" for state space architectures. Similar to regularized particle filters, the MDPF framework represents the continuous state posterior with a continuous mixture density, which is differentiable and can be trained end-to-end using the proposed IWSG. As a discriminative approach, the proposed particle filter does not require the specification of a generative model and, in contrast to other discriminative particle filters, returns unbiased and low-variance gradient estimates.\n\nSTRENGTHS:\n- Efficient and accurate particle-based estimation of complex posteriors in sequence models is an important topic and has a long history in the machine learning community.\n\n- The paper includes an easy-to-follow recap of particle filtering and puts the proposed approach in context with many relevant discriminative particle filters, including a thorough discussion of their weaknesses that motivates both IWSG and MDPF.\n\n- The proposed discriminative particle filter tackles a number of well-known challenges, such as differentiation through the resampling step, biased or high-variance gradient estimates, and limited expressiveness of the posterior parameterization.\n\n- The experiments compare the proposed particle filter to a broad spectrum of relevant baselines and demonstrate MDPFs superior performance in three state estimation tasks.\n\nWEAKNESSES:\n- One of my concerns with this paper is its confusing presentation. The text often jumps between motivation, technical background, and related work, making it difficult to follow the intended train of thought. This problem is further aggravated by the fact that there are multiple technical streams: generative vs. discriminative particle filters, unbiased/low-variance gradient estimation, and inference in mixture models. There are many balls in the air and the paper has a hard time telling a streamlined story. Unfortunately, these are not the only problems with the presentation: Section 4 mixes technical background, the paper’s first main contribution (IWSG), and an experiment; Section 5 mixes the paper’s second main contribution (MDPF) with related work and contains no mathematical description of the model.  \n\n- The figures are weak for multiple reasons: (1) the font size is much too small; (2) the main text and the figure captions cannot be read independently, because the figures contain critical information that is not explained anywhere in the text; (3) the lack of subfigures ((a),(b),…) makes it difficult to map the information in the captions to individual subplots; and (4) the figures are not placed on the page they are referenced. Figure 4 is especially problematic as it describes the core of the paper’s contribution but is impossible to understand without further context (which the text does not provide either).\n\n- The paper’s weak presentation also makes it difficult to assess the two contributions (IWSG and MDPF): (1) l.170 mentions that the proposed IWSG estimator is a continuous variant of the discrete estimator used in [6], but the exact relationship between the two remains unclear. Are there non-trivial challenges that prevent a direct generalization of [6] to a continuous setting?; (2) l.89/l.209 mentions that the proposed MDPF is a variant of [14,15], but again the differences are not explained in enough detail. Are there non-trivial challenges that prevent a direct application of [14,15] in a discriminative setting?\n\n- One of the paper’s main claims is MDPF’s ability to compute unbiased and low-variance gradients. Unfortunately, neither the technical sections nor the experiments provide any *direct* evidence supporting this claim. I would have liked to see either a technical analysis of the proposed estimator's properties or experiments that go beyond the evaluation of NLL/RMSE and at least give empirical insights along those lines.\n\nOverall, I feel this paper would benefit from another round of polishing before publication, including a reworked presentation and better positioning and differentiation of its contributions.\n\nQUESTIONS:\n- Section 4: It is mentioned that the optimal transport methods described in [44,45] are compatible with Gaussian mixtures (l.154f), so I’m curious why they are not part of the evaluated methods?\n\n- Section 6: The design of the architecture used in the experiments (Sections 6.1-6.3) is not clear enough. Are all experiments based on Figure 4? If so, what are the encoders and transformations?\n\n- Section 6: One of the arguments against [5] is its use of truncated gradients (l.118f), however, l.244 mentions that the experiments use truncated BPTT. Doesn’t that bias the MDPF gradients as well and makes the arguments in favour of MDPF obsolete?\n\n- Section 6: I’m curious how the number of particles affects the reported performance metrics, and especially if the observed trends remain the same if the number of particles is significantly increased or decreased. If the authors have additional results along those lines it would be interesting to see them.\n\nLIMITATIONS:\n- I commend the authors for including a paragraph on the limitations of the proposed particle filter, such as its supervised nature and computational complexity on high-dimensional data.\n\n- The paper does not discuss any ethical concerns related to the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the limitations of traditional particle filters in representing multiple posterior modes and their applicability to high-dimensional observations like images. Instead, the authors propose a method that leverages training data to learn particle-based representations of uncertainty using deep neural network encoders. The authors address the issue of biased learning and heuristic relaxations by representing posteriors as continuous mixture densities, allowing for unbiased and low-variance gradient estimates.\n\nSTRENGTHS:\n- This paper discusses an interesting problem.\n- It is easy to read and follow the train of thought.\n- Literature review is sufficient to grasp the idea of the idea of the paper.\n\nWEAKNESSES:\n- As MDPF relies on DNN parameterizations for both dynamic and measurement models, this introduces significant complexity to the model which can make training and inference more computationally expensive. \n- Additional complexity of the model due to decoupling for A-MDPF as it requires careful tuning for separate bandwidth parameters. \n- While the paper mentions that bandwidth parameters can be learned through end-to-end training, optimizing this parameter efficiently can require careful tuning and thus is challenging. \n- I am still a bit uncertain about the authors' claim regarding the use of smaller NNs and non-learned operations in constructing the dynamic and measurement models. \n- Lack of comparison with alternative frameworks for tracking and robot localizations and mostly applying to synthetic data. \n- The limitations of the simulation setup in evaluating the MDPF and A-MDPF algorithms can be identified as follows:\n\n     * The state estimation tasks focus on a 3D state consisting of translation and angle components (s = (x, y, θ)). While this simplification allows for manageable simulations, it may not capture the full complexity of state estimation in more challenging real-world scenarios with higher-dimensional state spaces.\n\n     * The evaluation compares the MDPF and A-MDPF algorithms with a few selected particle filter variants (TG-PF, SR-PF, OT-PF, DIS-PF, C-PF) and an LSTM model. While this provides some insight into the performance of the proposed methods, a more comprehensive comparison with a wider range of state estimation algorithms would provide a better understanding of their strengths and weaknesses.\n\n     * The training data uses sparsely labeled true states every 4th time-step, which may not accurately represent the labeling constraints in real-world applications. Dense labeling is crucial for accurately assessing the performance of state estimation algorithms. Although the evaluation uses densely labeled datasets, the training process may not fully exploit the benefits of dense labeling.\n\nQUESTIONS:\nQuestions regarding the proposed importance weighted samples gradient (IWSG) estimator:\n\n1- How effective is the IWSG estimator in reducing variance compared to other gradient estimation techniques?\n\n2- Are there scenarios or specific mixture distributions where the IWSG estimator may still suffer from high variance?\n\n3- Have there been any empirical evaluations to quantify the variance reduction achieved by the IWSG estimator?\n\n4- How does the choice of the proposal distribution, q(z), affect the accuracy and efficiency of the IWSG estimator?\n\n5- What is the computational cost associated with using the IWSG estimator compared to other gradient estimation methods?\n\nQuestions regarding the Algorithms: \n\n1- How does the complexity of the deep neural networks used in MDPF and A-MDPF impact the computational requirements for training and inference?\n\n2- Are there any strategies or techniques to improve the computational efficiency of these models?\n\n3- How does the performance of MDPF and A-MDPF depend on the quality, size, and representativeness of the training data?\n\n4- What happens when there is limited or biased training data available?\n\n5- Can the learned parameters and decisions made by MDPF and A-MDPF be interpreted and understood?\n\n6- Are there any techniques or approaches to enhance the interpretability of these models?\n\n7- How does the performance of MDPF and A-MDPF depend on the choice and optimization of the bandwidth parameter (β) for kernel density estimation?\n\n8- Are there any alternative methods or strategies to optimize this parameter effectively?\n\n9- How does the decoupling of mixtures used for particle resampling and posterior state estimation affect the performance and training of A-MDPF?\n\n10- Are there specific scenarios or domains where A-MDPF offers significant advantages over MDPF?\n\n11- How well do MDPF and A-MDPF generalize to different problem domains or scenarios?\n\n12- Have these models been tested on real-world datasets, and how do they perform compared to alternative approaches in those contexts?\n\nLIMITATIONS:\nPlease refer to the weakness and question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of sequential state estimation using discriminative particle filters. Particle filters (or SMC) methods are widely used in this setting owing to their flexibility. Recently, inspired by the success of deep learning enabled by end-to-end differentiable methods, there has been interest in imparting similar properties to particle filters for improving performance, in particular when modelling temporally extended systems. The authors first review some existing approaches for particle filtering with a focus on regularized particle filters as well approaches for making the non-differentiable resampling step differentiable. The authors highlight several drawbacks with the existing methods with differentiable resampling, including biased gradients, numerical instability and high variance gradient estimates. The proposed method is based on leveraging implicit reparameterization gradients coupled with an importance sampling scheme to obtain an unbiased gradient estimator for continuous mixture distributions. The implicit reparameterization avoids the inversion of the standardization function and the importance sampling scheme reduces the variance of the estimates. The approach avoid various pitfalls of previous attempts to make the resampling step differentiable. The authors leverage this estimator in their Mixture Density Particle Filter method, which is evaluated through a wide variety of experiments with thorough quantitative and qualitative analysis - achieving improved performance across tasks.\n\nSTRENGTHS:\n* Sequential Monte Carlo approaches are widely used across various domains. Improvements to the algorithm can have a broad impact. The importance-weighted gradient estimator proposed in the paper can improve the performance on a wide-variety of tasks, beyond those studied in the paper. \n* The proposed approach is principled and novel, leveraging recent advances in implicit reparameterization along with classical insights from the sampling literature to improve the performance of regularized particle-filters. \n* The resulting method MDPF is conceptually simple, and can be incorporated easily as an alternative to existing regularized PF approaches. \n* The paper is quite well written with a nice review of existing approaches and a lot of useful details about the experiments. I do think the introduction can be improved a bit.\n\nWEAKNESSES:\n* A key weakness of the approach is scaling to higher dimensional problems. It is a well known problem that importance sampling based estimators can still have high variance induced by some bad samples with high weights. (This is already discussed in the paper)\n* While I appreciate the thorough experiments in the paper one issue is that most of the experiments are on relatively low dimensional problems. \n* Although the details of the experiments are covered fairly well, the absence of code could be a problem for reproducibility efforts.\n\nQUESTIONS:\n* What would be potential solutions to scaling the method for higher dimensional problems? \n* Have you tried the approach on some standard SMC tasks [1]? Does the gradient estimator improve performance there? \n\n[1] An introduction to Sequential Monte Carlo by Nicolas Chopin and Omiros Papaspiliopoulos. https://particles-sequential-monte-carlo-in-python.readthedocs.io/en/latest/\n\nLIMITATIONS:\nThe major limitations of the method are already highlighted in the paper - namely scaling to higher dimensional problems and reliance on labelled data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper consider the problem of particle filter (PF)-based state estimation in nonlinear models with unknown dynamics and (discriminative) observation models.  The key challenge they address is the typical inability of traditional gradient learning approaches, applied to PF, to deal with (backprop through) the non-differentiable particle resampling stage.   To do this, the authors propose a novel Importance Weighted Samples Gradient (IWSG) Estimator, based on a kernel-density representation of the resampling step.  The IWSG is fully differentiable but is also unbiased and, in practice, low variance, particularly compared to the previously proposed Implicit Reparameterization Gradients (IRG) estimator.  The authors conduct an extensive set of experiments on simulated data (synthetic + simulator object-in-a-maze tracking scenarios) to demonstrate the utility of their IWSG.  IWSG is shown to outperform the competing approaches.\n\nSTRENGTHS:\n+ Well-written paper, with sufficient details (in main paper + supplement) that clearly introduce the problem, describe prior PF-based approaches, and build the proposed Mixture Density Particle Filter (MDPF) using a novel IWSG framework, following the ideas of Scibior et al., but using a kernel density-based (mixture density) resampler.\n+ Addresses an important problem in the application of particle filter-based state estimators/trackers for models with unknown (parametric) dynamics and observation models (discriminative)\n+ Propose two variants of MDPF, a baseline with a single mixture density used for both particle resampling and state estimation, and an improved A-MDPF, which employs different mixtures for the two tasks. A-MDPF is generally demonstrated to be more effective at the added computational cost of having two mixtures.\n+ Extensive and convincing experiments conducted in simulated settings (both simple synthetic models and more complex 2D and 3D based simulators, c.f., Deepmaze and House3D with image based observations).  Experiments demonstrate that IWSG leads to consistently more accurate state estimates compared to either prior works / baselines (LSTM, TG-PF, OT-PF, SR-PF, DIS-PF, C-PF, TG-MDPF, and IRG-MDPF), together with lower estimator variances.\n\nWEAKNESSES:\n- Main paper does not clearly build the connection between Scibior et al. and the proposed approach, but this is clarified in the rather extensive supplement.  The proposed approach can be seen as perhaps rather obvious, once the KDE mixture model is employed for the resampler, but it is nevertheless a fairly ingenious combination of MD + IWSG.\n- The authors acknowledge that PF-based approaches are not effective for high-dimensional state spaces and/or when the number of particles is large.  However, they do not clearly investigate this dependency (all experiments are conducted on max 3D state spaces + bearing).\n\nQUESTIONS:\n- How does the complexity of learning/inference depend on N?  Do you have experimental evidence that demonstrates this?\n- Fig. 3 (supplement) shows some increasing variance trends in both gradients and states for MDPF (and A-MDPF).  Can you comment on this / explain?  Was this tendency observed in experiments in the simulator data?\n\nLIMITATIONS:\nThe authors has addressed limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes an unbiased and low-variance gradient estimator for differentiable particle filters, where resampling steps incurs discrete changes that are non-differentiable in previous methods. The proposed method solved the problem by representing posteriors as continuous mixture densities, which is similar to the idea of regularized particle filters.

STRENGTHS:
The paper is well-presented, motivation and proposed method are clearly explained, experimental results demonstrated the effectiveness of the proposed gradient estimator for differentiable particle filters, especially when the posterior is multimodal.

WEAKNESSES:
The performance of the proposed method when used in differnetiable particle filters trained with unsupervised loss when labelled data are not available is not discussed in the experiments/

QUESTIONS:
1. As far as I can see, all the experiments presented in the paper assumed the true latent state is available during training, do you have any idea about how to train differentiable particle filters when true states are not available in training but we still want to track the latent state in testing?
2. The paper claimed the proposed estimator is unbiased, but it is not obvious to me why it is unbiased and I expect to see a theoretical proof of this.

LIMITATIONS:
The proposed method is based on importance sampling, therefore some discussions on the variance of the estimator will improve the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07