Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Users of sequential Monte Carlo algorithms have many choices to make, including the proposal distributions and the sequence of intermediate target distributions. An appealing approach is to attempt to learn optimal proposals and intermediate targets by 
stochastic gradient descent; this can be formulated within a variational inference framework, where the SMC algorithm can be understood
either as *being* the variational family, or as a technique for tightening an existing variational bound. Although researchers have had some success applying this approach to the problem of learning proposals, learning intermediate targets is more challenging: they affect the objective's value only through their impact on SMC's resampling steps, which are ignored by commonly used biased gradient estimators.

This paper presents a new method, SIXO, for learning the intermediate target distributions in sequential Monte Carlo (or rather, learning twisting functions that tilt filtering distributions toward the true smoothing distributions, which are the optimal intermediate targets in a state-space model). The key innovation is to recognize that the optimal twisting function in this setting is a density ratio, and to apply density ratio estimation techniques. In particular, the authors propose to train a classifier to distinguish between samples from $p(x_{t}) p(y_{t+1:T})$ and $p(x_{t}) p(y_{t+1:T} \mid x_t)$. The logits from the trained classifier can be understood as computing, up to a constant, the log density ratio needed for twisting. The authors further propose parameterizing the classifier with a recurrent architecture, reading the sequence of $y$'s backward from the end.

In experiments, the authors show that SIXO outperforms methods that do not use twisting, and that their density-ratio estimation scheme outperforms an unbiased gradient estimator for the joint optimization of proposal, model, and twist function parameters.
strengths_and_weaknesses: Strengths:

1. This is a very nice paper that proposes an appealing solution to a long-standing and important problem. Although framed in the context of model-learning and variational inference, the technique seems like it would also be useful for improving SMC inference even when the model is not learned.

2. The idea to treat the twisting function as a density ratio estimator gives rise to a simple and efficient training procedure.

3. Parameterizing twisting functions as recurrent networks is a smart choice that greatly reduces computational complexity.

4. The paper is very nicely written with well-made and informative figures. I especially liked Figure 2, which clearly shows the problem with FIVO: the good proposals it learns are rendered useless by resampling against inaccurate intermediate targets.


Weaknesses:

1. Because the twisting functions are trained on samples of (x, y) from the generative model, they may work less well when the true observations look quite different from the synthetic data. (This may be less of an issue in the setting where the model is learned to fit the true observations, however.)

2. I don't have a great sense of how hard the inference problems from the experiments are; it might have been nice to report results from the IWAE baseline on all of them, to help readers understand to what extent resampling is even necessary on these problems.


Not weaknesses, but minor suggestions: 

* The paper could do more to highlight the challenge that the DRE solution overcomes. The move from FIVO / VSMC to what the paper calls "SIXO-u" is quite small; FIVO/VSMC jointly optimize any parameters of the SMC algorithm, and so of course we could introduce learned parameters into the intermediate targets. The challenge is that learned targets affect the SMC algorithm only through the resampling steps, which are not differentiable; as a result, all signal for improving the intermediate targets' parameters comes from high-variance score function estimator terms, which previous papers have advocated dropping during training. This explains why there isn't a "SIXO-biased" that just optimizes the SIXO objective directly with a biased estimator -- the biased estimator does not enable training of the intermediate targets.

* On L145-148 a reference to the appendix where the architecture is more fully explained might be nice. (A possible misunderstanding readers might have is to think X is an input to the recurrent part of the network, but of course it is important that it not be--you run the recurrent part once to get the embeddings, then score many X's using those embeddings during SMC.)

* Possible connection for related work -- Nested Variational Inference (https://proceedings.neurips.cc/paper/2021/hash/ab49b208848abe14418090d95df0d590-Abstract.html) also attempts to learn intermediate targets in SMC-like algorithms, and also tries to do it using a separate objective for training them that is not understood as part of one overall variational bound. 
questions: 1. In Figure 3, why is resampling so much more frequent in the SIXO plot than in the BPF plot? From the trajectories I would guess that ESS is relatively high throughout the SIXO sweep, and sometimes dips quite low in the BPF sweep, which should lead to the opposite behavior (more resampling in BPF).

2. The related work draws connections to APF, iAPF, cSMC, etc. and explains that SIXO is novel with respect to them (e.g., it supports model learning). But if I am not trying to learn model parameters, and am just trying to get better intermediate targets, how should I understand the relative benefits and disadvantages of these methods, compared to SIXO?
limitations: Yes, I think the paper adequately addresses the limitations. (However, see Weakness 2 -- I think additional context could help readers make better sense of the experimental results.)
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a new form of sequential Monte Carlo for particle smoothing problems. The basic idea is to learn the twisting functions that convert the filtering distributions to approximate smoothing distributions using the density estimation ratio trick. This provides a potentially tight bound on the marginal likelihood.

strengths_and_weaknesses: Strengths:
- The basic idea is solid and simple
- The method has appealing theoretical properties. The tightness of the bound is a clear benefit over competing SMC approaches.
- The empirical evaluation covers several interesting timeseries problems and compares performance with relevant baselines. The method clearly outperforms the baselines.
- The paper is clear and well-written

Weaknesses:
- It would have been interesting to see the method applied to high dimensional problems

questions: Does the approach scale to high dimensional settings such as, for example, natural video processing?

limitations: The paper would benefit from a longer discussion section, discussing potential limitations and failure cases. I cannot see any negative societal impacts.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes Smoothing Inference for Twisted Objectives (SIXO) for sequential Monte Carlo inference using approximations of the full posterior as the target distribution at each stage.  Importance weights are computed by learning approximations of the posterior (up to proportionality).  These approximations are learned by using variational backwards methods to solve for twisting functions that approximate the predictive distribution on future measurements, given the current state.  The authors propose learning the twists by casting the ratio of backwards messages and latent marginals as a density ratio estimation problem, leading to SIXO-DRE.  Empirical results are presented on small-scale simulation models and are compared to Filtering Variational Objectives (FIVO).
strengths_and_weaknesses: **STRENGTHS**
The paper is well-written, well-motivated, and empirical results are sufficient for the stated claims.  The content is technically sound and appears to be without significant errors.  Figures are particularly well prepared, easily readable fonts, lines, legends, and colors.  Overall, it is a strong paper.

**WEAKNESSES**
The argument for choosing posteriors as the target—instead of filter distributions—largely relies on a non-gradual shift in the target distributions.  The authors motivate this with the pathological case of a drift diffusion whereby a single observation is obtained at the final timestep.  However, typical dynamical systems receive measurements along with latent states at each time, and thus filters gradually shift.  In such settings as the stochastic volatility model one does not observe a strong benefit over FIVO, for example.  As a result, it is unclear to what extent SIXO is beneficial in typical settings where observations are dense in the time domain.

The consistency results in Proposition 1 are largely vacuous.  Assumptions of the propositions state that the space of twists $\mathcal{R}$ must include the true predictive distribution, and that the space of proposals $\mathcal{Q}$ must include the true posterior (conditioned on the previous state).  These conditions cannot be verified in general and typically hold only in trivial settings.

Full posterior target distributions are impractical in the setting where data arrive in a streaming fashion with inference being updated as new data arrive.  Such a setting is the most common one for SMC and related inference for dynamical systems.  The proposed SIXO requires static data for updating backwards messages, and so cannot be applied in streaming scenarios.  This could not, for example, be used in a control scenario, where particle filters are often applied.  The authors do not discuss this limitation and instead rely on static data for experiments and model parameter learning.

Detailed comments below:
* Line 157 : Typo, change $p_{\theta}(x_t \mid y_{t:1:T})$ to $p_{\theta}(x_t \mid y_{t+1:T})$
* The authors propose SIXO-u then state that gradient estimators are high variance and thus impractical.  Yet there exist numerous variance reduction techniques that can be applied in this setting (indeed, FIVO uses such approaches).  It seems that the authors prematurely abandon this idea.

questions: The paper argues that target distributions in SMC is a design choice, yet the target distributions of SMC are uniquely defined to be the sequence of marginal or joint filter distributions (i.e. conditional on past observations only).  Under what circumstances are these targets a valid choice in algorithm development? 
limitations: The authors do not clearly discuss limitations of the methodology.  This reviewer sees no obvious negative societal impact of the proposed work.
review_title: null

Meta-review: metareview: This paper proposes a new method to adjust the proposal used by sequential Monte Carlo. Existing methods struggle for this problem since the resampling step poses difficulties for reparameterization-based methods. This paper proposes a "twisting" method to learn a density ratio by training a classifier to distinguish between two distributions. Reviewers agreed the method was novel, relevant, and sufficiently supported by experiments.
confidence: Certain
Decision: Accept