Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a novel extension of the LSTM which incorporates stochastic inputs at each timestep. These stochastic inputs are themselves dependent on the LSTM state at the previous timestep. Considering the stochastic dependencies, this then yields a highly flexible non-Markov state space model, where the latent variable transitions are partially parameterized by an LSTM update.

Naturally, the challenges are then efficiently estimating parameters and performing inference over the latent states. Here, SMC (and conditional SMC / particle Gibbs) are used for inference over the latent states z. A particularly nice touch is that even when the LSTM model is used for the transitions in the latent space, so long as the conditional distributions p(z_t | z_{1:t-1}) are conjugate with the emission distribution then it is possible to compute the optimal forward filtering proposal distribution in closed form, as done for the conditionally Gaussian (with affine Gaussian observations) and conditionally multinomial models considered here. Note that this really is a special feature of the models under consideration, though: for example, if the emission distribution p(x_t | z_t) is instead a *nonlinear* Gaussian, then one would have to fall back to bootstrap proposals. This probably deserves some mention: equations (13) are not, generally, tractable to integrate or normalize.

I think this paper is missing a few necessary details on how the overall optimization algorithm proceeds, which I would like to see in an update. I understand that particle Gibbs updates (or SMC) are used to approximate the posterior distribution in a Monte Carlo EM algorithm. However, this does leave some questions:

1. For the M step, how are the \omega parameters (of the LSTM) handled in equation (8)? I understand that due to the particular models considered, maximum likelihood estimates of \phi can be found in closed form. However, that’s not the case for \omega. Is a gradient descent algorithm run to convergence? Or is a single gradient step taken, interleaved with a single PG update? Or something else?

2. How reliably does the algorithm as a whole converge? Monte Carlo EM does not in general have convergence guarantees of “standard” EM (i.e. each step is not guaranteed to monotonically improve the lower bound). This might be fine! But, I think requires a bit of discussion.

3. Is it necessary to include a replenishing operation (or independent MCMC steps) in the particle Gibbs algorithm? A known issue when running an iterated conditional SMC algorithm like this is that path degeneracy can make it very difficult for the PG kernel to mix well over the early time steps in the LSTM. Does this issue appear here? How many particles P are needed to efficiently mix, when considering time series of length T?
review_title: A nice formulation of a stochastic LSTM

final_score: rating: 5: Marginally below acceptance threshold
review_content: [After author feedback]
I would suggest that the authors revise the literature study and contributions to more accurately reflect prior work.

[Original review]
The authors propose state space models where the transition probabilities are defined using an LSTM. For inference the authors propose to make use of Monte Carlo expectation maximization.

The model proposed seems to be a special case of previously proposed models that are mentioned in the 2nd paragraph of the related works section, and e.g. the Maddison et al. (2017) paper. The inference method has also been studied previously (but not to my knowledge applied to SSLs/SRNNs), see the following review papers and references therein:
Schön, Lindsten, Dahlin, W˚agberg, Naesseth, Svensson, Dai, "Sequential Monte Carlo Methods for System Identification", 2015
Kantas, Doucet,  Singh, Maciejowski, Chopin, "On Particle Methods for Parameter Estimation in State-Space Models", 2015

Given this it is unclear to me what the novel contributions are. Perhaps the authors can elaborate on this?

Minor comments:
- Note that generally a state space model only has the Markov assumption, there is no restrictions on the transition and observation models.
- EKF also requires Gaussian noise
- It is a bit unclear what is meant by "forward messages" e.g. below eq. (6). For this model I believe the exact would generally be unavailable (at least for continuous models) because they would depend on previous messages.
- Eq. (12) and (14) are exactly the same? The text seems to indicate they are not.
- The optimal proposal is only locally optimal, minimizing the incremental weight variance
- "w" should be "x" in eq. (20)

review_title: PMCMC EM for LSTM-based SSM with unclear contributions

final_score: rating: 3: Clear rejection
review_content: This article presents an approach for learning and inference in nonlinear state-space models (SSM) based on LSTMs. Learning is done using a stochastic EM where Particle PMCM is used to sample state trajectories.

The model is presented assuming that SSMs are linear. This is not necessarily the case since nonlinear SSMs have been used for a long time (see for example Ljung, 1999, "System Identification, Theory for the User"). The presented model is a nonlinear SSM with a particular structure that uses LSTMs.

The model described in the paper is Markovian: if one defines the variable sz_t = {s_t, z_t} there exists a Markov chain for the latent state sz:

sz_t -> sz_{t+1} -> sz_{t+2} -> ...

Marginalizing the latent variables s_t leads to a structure that, in general, is not Markovian. The authors claim that this marginalization "allows the SSL to have non-Markovian state transition". The word "allows" may mislead the reader in thinking that the model has gained some appealing property whereas the model is still essentially Markovian as evidenced by the Markov chain in sz. Any general algorithm for inference in nonlinear Markovian models could be used for inference of sz.

The algorithm used for inference and learning is stochastic EM with PMCMC but the authors do not cite important prior work such as: Lindsten (2013) "An efficient stochastic approximation EM algorithm using conditional particle filters"


Pros:

The model is sound.

The overall structure of the paper is good.


Cons:

The authors formulate the problem in such a way that they are forced to use an algorithm for non-Markovian models when they could have conserved the Markovian structure by choosing the appropriate parameterization.

The presentation of state-space models, filtering and smoothing shows some lack of familiarity with the literature. The control theory literature has dealt with nonlinear SSMs for decades and there is recent work in the machine learning community on nonlinear SSMs, e.g. Gaussian Process SSMs. 

I would advise against the use of non-English expressions unless they are used precisely:

   - sine qua non: LSTMs are not literally an indispensable model for sequence modeling nowadays. If the use of Latin was unavoidable, "de facto standard" would have been slightly more accurate.

   - bona fide: I am not sure what the authors wanted to say.

   - naívely: the correct spelling would be naïvely or naively.
review_title: Poor model manipulation and missing many important references.

Meta-review: Thank you for submitting you paper to ICLR. The consensus from the reviewers is that this is not quite ready for publication. The work is related to (although different from) Gu et al Neural Sequential Monte Carlo NIPS2015 and it would be useful to point this out in the related work section.
Decision: Reject