Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors identify issues with neural likelihood-free methods for joint density estimation in hidden variable models and propose a "post-processing" technique using samples from the posterior of the parameters that can be generically applied to mitigate these issues. The proposed approach is of interest primarily in applications in which estimation of the hidden variables important.  The approach is validated using simulations from three different biological HMMs against an SMC baseline.
strength_and_weaknesses: Strengths:  The work identifies and attempts to address weaknesses in current approaches to likelihood free inference.

Weaknesses:  Estimating the hidden variables isn't always so useful (so applicability of this approach is somewhat limited).  Some novelty issues related to the approach, e.g., the key concepts have appeared elsewhere (just not in this specific context).

clarity,_quality,_novelty_and_reproducibility: Clarity:  Some (obvious) minor typos here and there -- draft requires a careful editing pass.

Novelty/Quality:  One of the key equations (11) seems to be related to pseudolikelihood approximations, though little discussion appears in the main text about related work or even why (11) is a good approximation.  For me, this limits the novelty a bit as this reformulation seems to be the central contribution.  As far as a I can tell, after this observation, the remainder of the approach is applying existing tools/techniques.

Reproducibility:  Quite a few details are included in the main text and additional details in the Appendix appear to be enough to implement the proposed approach.


summary_of_the_review: An interesting approach the fills gaps in the literature, but the novelty might be limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about learning the full (over parameters and hidden states) posterior of a Hidden Markov Model. The motivation for inferring the hidden states as well is that it is either an end in itself or a means to better estimate the posterior-predictive to assess the model's goodness of fit. The key insight is to break down the full posterior into two parts: a distribution over parameters and a distribution over hidden states. The first is learnt using Simulation-Based Inference powered by Deep Learning. The second term is learnt by maximizing the likelihood of a normalizing flow architecture that follows the simple Markov factorization of the HMM's hidden states. Experiments are conducted to show that this combination of methods can efficiently estimate the full posterior and, as a consequence, quantifiably improves the estimation of the posterior-predictive. 
strength_and_weaknesses: The authors motivate the need to estimate the full posterior and discuss the limitations of existing methods. Their experiments seem to support their claim that estimating the hidden states improves the estimation of the posterior-predictive. The relation to prior work is thoroughly discussed. 

I am curious as to why MMD (versus another IPM or sample-based divergence) is used to evaluate the posterior-predictive. 



clarity,_quality,_novelty_and_reproducibility: The presentation is very clear; so is the contribution and motivation.
summary_of_the_review: The authors presented a convincing-enough argument for why current methods cannot efficiently learn the posterior over hidden states (high-dimensionality of x, or the HMM prior is a bad choice of MCMC proposal). Their method is clearly explained and can be applied after estimating the posterior over parameters, making it a convenient option. It seems broadly applicable: first learn the posterior over parameters, then use samples of these parameters to simulate hidden variables which are learnt by an architecture that mimicks the conditional dependency structure that is assumed for these hidden variables. The experiments seem to support the efficiency of this method in practice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of approximating the joint posterior -- p( parameters, hidden state variables | data) -- for a hidden Markov model with continuous-valued states (not discrete ones). The assumed context is that the generative process can be sampled from, but is not available analytically (e.g. PDFs of the prior over state transitions or the likelihood that generates observations given states may not be easily evaluated). Thus, likelihood-free methods are pursued.

The claimed contributions are:

1) Exposing the problem of using neural likelihood-free methods to estimate the posterior over parameters *and* hidden states simultaneously (discussed around Eq 5-6). Essentially, the problem is that including hidden states means too many parameters and difficult estimation.

2) Proposing a "post-processing" technique to remedy the issue, with 3 steps:

a) Estimate p(parameters | data) via standard NLFI
b) Approximating the posterior over states given parameters via an "incremental" factorization

$$
p( x_1, ... x_T | params, data) \approx \prod_t q( x_t | x_t-1, params, data)
$$

c) Train a neural density estimator (masked autoreg. flow, Sec. 4.3) to use for this incremental posterior $q$

The approach overall (essentially steps b,c above) is called the "Incremental Density Estimator" (IDE). 

Evaluations on 3 different models suggest that the proposed IDE delivers:

* posterior predictive samples for future observations that are closer to a gold-standard SMC than alternatives like ABC (see Tab 1)
* better estimates of latent states x (see Fig 3)
strength_and_weaknesses: 
# Strengths

* Likelihood-free inference for continuous-valued HMMs is an important/interesting topic
* Proposed IDE approach is rather elegant and clean
* Plots in Fig 3 (esp 3b) show clear gains from this approach at predicting latent state trajectories x_1:T


# Weaknesses

I highlight the main issues impacting my scoring here... see below comments under relevant heading (Quality/Clarity) for detailed elaboration 

* W1: The incremental posterior idea is still quite a lossy approximation
* <strike> W2: Scalability is difficult to assess, more experiments would help</strike>
* <strike> W3: Need to clearly state that IDE only applies to continuous-state HMMs</strike>
* <strike> W4: Initial state handling is unclear</strike>


**Update after discussion**:
* W2 has been addressed in Supp F
* W3 and W4 have been satisfactorily addressed.
clarity,_quality,_novelty_and_reproducibility: 
## Novelty

In my assessment, the paper does seem to point out a reasonable issue with using off-the-shelf NLFI to infer parameter-and-hidden-variable posteriors, and offer a useful (if still limited) fix that differs from prior approaches.

The fix is in a sense "straightforward" (I think many others would try this kind of "drop the future" approximation if pursuing the same problem), so I don't feel there's abundant *technical* novelty here, but that's not a problem in my view.


## Quality

### W1: The incremental posterior idea is still quite a lossy approximation

In Eq. 11, the approximate factorization of the posterior proposed here is lossy, as we "drop the conditioning on future datapoints".

Naturally, some inaccuracy is tolerable in exchange for tractability. However, I don't think the paper has done enough to study/clarify/mitigate the consequences of this assumption. I think of this approximation vs the ideal as the difference between Kalman filtering (which updates x given past only) versus Kalman smoothing (which benefits from past and future). 

I view this kind of limitation as one that should be acknowledged explicitly in the "Limitations" paragraph on page 5.

### W2: Scalability is difficult to assess, more experiments/analysis would help

A clear use case for this paper will be that a practitioner comes along and wonders: would IDE help with my problem? But the present paper could do more to help readers understand:

* the kinds of problems (dimensionality, number of timesteps, etc) the approach is well-suited for
* sensitivity to hyperparameters (how to train the MAF well, etc)

For example, if the problem of interest had 10-dimensional or 20-dimensional x, would this approach out of the box be effective? All 3 datasets tested here have x of only 2-4 dimensions if I understand correctly.



## Clarity 

There's two big issues related to clarity I want to see addressed in revision

### W3: Need to clearly state that IDE only applies to continuous-state HMMs

The methods here apply *only* for HMMs with continuous-valued latent states x_t. Discrete states are not easily handled because the MAF distribution over x in Eq. 13 assumes continuous values (it transforms samples from a Normal)

This is easy to fix, but important to do so.


###  W4: Initial state handling is unclear

Eq. 12 as written includes q terms for x_1, ... x_T, but does not seem to include x_0 (initial state). Please fix in revision and describe the fix in response text.

Eq. 11 has a similar issue... the initial state is not accounted for. 

I hope these are easy enough to address, but at present this impacts my correctness/completeness rating.


### Other issues on clarity

* Terminology: Many readers from ML backgrounds will assume that "HMM" refers to models with *discrete states* (e.g. Bishop's PRML textbook in Sec. 13.2 defines an HMM as "The hidden Markov model can be viewed as a specific instance of the state space model ... in which the latent variables are discrete"). I'd suggest in abstract and in background being clear about what your definition is (continuous states allowed), and acknowledging somewhere that it differs from how other authors use the term. I don't have a problem with calling the models here HMMs (they are hidden variables with markov transitions), but it differs from how I would use the term.

* Background: I think the assumptions about what makes the HMM "intractable" need to be stated more clearly to many readers. I think the key assumption is that you can sample from f and from g, but you cannot evaluate the PDF of one or both. I'd say this directly, rather than say they are "unavailable analytically" which is perhaps too vague and doesnt make clear sampling is possible.

* Notation indexing time as t_0, t_1, t_2 (Eq. 2 and beyond) seems unnecessarily complicated.... can't we just assume discrete time at standard intervals and index the times with integers like x_0, x_1, .... or y_0, y_1, ...., as in Eq 1?

* Eq 4 describing the posterior predictive doesn't define exactly what y_* is. Is it the observation at the single next timestep? at the next M timesteps?

* Can your notation in Eq 13-14 make it more obvious how \phi informs the function h? Always weird to me when something on left-hand-side of equation defining a function doesn't appear on right-hand-side

Minor:
* typo on page 7? "J=5 layers each of which has two hidden layers ..."

summary_of_the_review: Overall I think the approach has promise, but I worry about scalability to bigger models (all models here cover 2-4 dimensional x) and lossy approximations (dropping all future observations is a restrictive assumption). I hope revisions can address my concerns.

**Update after discussion**:

Revisions addressed my concerns about assessing scalability and improving clarity. I've raised my score from 5 to 7. Technically, I can either give a 6 or an 8. I will enter 6 since I am still a bit worried about the lossy approximation, but to clarify I'm a bit more of a 7 than the entered score 6.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses the use of neural density estimation problems for latent variables in hidden Markov models. To do so, the authors suggest to amend the neural posterior with a conditional density estimator that factorizes according to the Markov property of the hidden states. This enables more efficient learning. The method is demonstrated on three tasks where it is compared with classical ABC techniques and SNLE and SRE approaches using summary statistics. 
strength_and_weaknesses: __Strengths:__
- The approach provides a sensible complement to neural posterior estimation techniques, providing an additional estimate of the latent states
- The method is easy to understand and a straight-froward extension to neural SBI tools. 
- It can be implemented using existing conditional density estimators.
- The method seems to perform well in experiments (although with caveats, see below).

__Weaknesses:__
- The authors use hand-crafted summary statistics for SNRE. That's an inappropriate comparison. A RNN or other temporal neural network should be used. 
- Provided alternatives use summary statistics. If those are not sufficient (and they likely are not) a loss in performance is expected. However, methods that can deal with time-series data exist. Such comparisons would be important because it's impossible to disentangle the loss of information due to summary statistics and the general performance of the density estimation approach. E.g. why not compare to an ABC method that can deal with time-series data?
- Even for SNLE either the full data could be used (probably doesn't work too well) or approximately sufficient summary statistics could be learned [1].
- I am not convinced the MMD is the best measure for comparing time-series output.
- A completely tractable example would be helpful to see the performance vs. SMC. 

[1] Chen, Y., Zhang, D., Gutmann, M. U., Courville, A. C., & Zhu, Z. (2021, January). Neural Approximate Sufficient Statistics for Implicit Models. In ICLR.

clarity,_quality,_novelty_and_reproducibility: While the paper was generally relatively easy to follow, there are many formulations and statements that seem rushed and imprecise. 

> Recently, a new class of likelihood-free inference methods (Cranmer et al., 2020)

That suggests that the methods were introduced by Cranmer et. al. It should be made explicit that this is a review.

> neural network based emulator of the (unnormalised) posterior density,

Why unnormalised? Most methods are able to provide a normalized posterior / likelihood?

> Such methods were empirically shown to be much more sample
efficient in comparison to ABC. 

Needs a reference here. 

> We like to point out that these neural likelihood-free approaches (NLFI), by
relying on emulation of an intractable density, are designed to estimate the posterior of the parameters
only.

That seems misleading as well. NFLI methods can in principle estimate any (conditional) distribution where there is sampled data. Whether a particular architecture is designed for highly autocorrelated time-series data is a different question, but there is nothing in e.g. normalizing flows that is "designed" to estimate parameters. Looking at the appendix this seems to be what the authors actually mean. Perhaps a more apt description would be to say that those methods are "usually applied to" or alternatively, that a naive implementation performs unreliably, potentially for a lack of inductive biases. 

> We are interested in the case where either f(·) or g(·) or both is unavailable analytically. [...] However, when these densities are analytically intractable then we have to resort to likelihood-free methods.

I find this suggests that if any of the two is missing particle-filtering doesn't work. However, if only $g$ is available, one could use a bootstrap particle filter. Except you mean to say that _sampling_ from $f$ is also not possible, but then it's unclear how data is generated. Even the exact use of $g$ can be circumvented by using a generalized Bayesian approach [2].

> This is a non-standard approach of drawing from the joint distribution, simply because there did
not exist any method, before the advent of NLFI, which can target p(θ|y) without drawing x. Any
classical pseudo-marginal sampling method, targeting θ (Beaumont, 2003; Andrieu & Roberts, 2009),
that uses the marginal likelihood p(y|θ), or its unbiased estimate, in the MCMC acceptance step
invariably also draws $x$.

To estimate $p(\theta \mid y)$, one needs paired samples $(\theta_i, y_i: i = 1,\ldots,n)$, but how are those sampled without drawing $x_i$? The simulated data needs to first draw $x$ and then $y \sim g(y \mid x)$. Seems misleading, or am I missing something?

> The most common approaches to tackle the inference of an intractable HMM consist largely of ABC
method.

What do you mean with intractable HMM? It's also possible to just assume Gaussian noise in the sense of an error model to make the HMM tractable? 
The framing of $g$ being unavailable generally requires more elaboration: usually $g$ is _assumed_ to have a particular form as e.g. an error model. Are you suggesting that $g$ is defined by a simulator that is well-defined but intractable?
If so, an application where this is actually the case would be interesting.

> Given these training examples ϕ can be learnt, using gradient descent, through maximising the total likelihood

Sloppy writing: either one maximizes the likelihood using gradient ascent, or minimizes the negative log-likelihood using gradient descent. 


Minor:

- " (i) the Lotka-Volterra" -> stochastic Lotka-Volterra. The original Lotka-Volterra model is deterministic 

[2] Boustati, A., Akyildiz, O. D., Damoulas, T., & Johansen, A. (2020). Generalised Bayesian filtering via sequential monte carlo. Advances in neural information processing systems, 33, 418-429.
summary_of_the_review: The approach presented here might work well in practice, but the manuscript at the current stage seems rushed, containing many imprecise passages. The experiment section is weak, since important aspects for sources of differences haven't been properly investigated. For example, using the component-wise MMD as the only measure of the posterior predictive, or the use of summary statistics for competing models. I don't think the paper should be accepted before those points are adequately addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has received mixed reviews, with two reviewers recommending rejection (5-3) and two recommending acceptance (8-7). However, Reviewer DLUp (8) stated that he/she is not an expert in simulation-based inference, so the overall score is between 5.0 and 5.75.

The author-discussion was productive, and the authors addressed many of the reviewers' concerns. However, some concerns still remain, such as:

- The quality of the posterior on x is evaluated only indirectly through the posterior predictive y^rep|y and only via the MMD metric. Additionally, In Appendix F, the authors evaluate the actual posterior (on x) for a toy problem, but they do not provide a thorough assessment of the posterior approximations (they only use the MSE, which only captures the mean, and 90% empirical coverage, which only captures a percentile of the distribution).
- In Appendix G, it is unclear whether NPE used for joint inference also uses the same summary statistics. If the same summary statistic is used ("We retained the same architecture and optimisation settings that we used in other experiments"), poor inference results on the hidden states are expected. This needs to be clarified to demonstrate the proposed method's value better, as the main argument for the proposed method is the inability of vanilla NPE to work in these cases.
- The presentation of the manuscript appears to be rushed. Many typos remain, and the manuscript would benefit from a careful editing pass.

Therefore, I believe that the paper is not yet entirely ready for publication. The method is promising, and the authors are encouraged to submit a revised version to another venue.
justification_for_why_not_higher_score: The presentation of the manuscript could be improved, and the experimental results could be strengthened. Accounting for the lack of expertise of one of the reviewers, the paper's average score is closer to 5.0 than to its current score.
Decision: Reject