FlowHMM: Flow-based continuous hidden Markov models
Authors: Pawel Lorek
Rafał Nowak
Tomasz Trzcinski
Maciej Zieba
Venue: NeurIPS 2022 Accept (2022)
Keywords: Hidden Markov Models
Continuous Normalizing Flow models

Abstract: Continuous hidden Markov models (HMMs) assume that observations are generated from a mixture of Gaussian densities, limiting their ability to model more complex distributions. In this work, we address this shortcoming and propose  novel continuous HMM models, dubbed FlowHMMs, that enable learning general continuous observation densities without constraining them to follow a Gaussian distribution or their mixtures. To that end, we leverage deep flow-based architectures that model complex, non-Gaussian functions and propose two variants of training a FlowHMM model. The first one, based on gradient-based technique, can be applied directly to continuous multidimensional data, yet its application to larger data sequences remains computationally expensive. Therefore, we also present a second approach to training our FlowHMM that relies on the co-occurrence matrix of discretized observations and considers the joint distribution of pairs of co-observed values, hence rendering the training time independent of the training sequence length. As a result, we obtain a model that can be flexibly adapted to the characteristics and dimensionality of the data. We perform a variety of experiments in which we compare both training strategies with a baseline of Gaussian mixture models. We show, that in terms of quality of the recovered probability distribution, accuracy of prediction of hidden states, and likelihood of unseen data, our approach outperforms the standard Gaussian methods. 

FlowHMM:
Flow-based continuous hidden Markov models
Pawel Lorek
Rafal Nowak
Mathematical Institute
Institute of Computer Science
University of Wroclaw Tooploox
University of Wroclaw Tooploox
pawel. www.koorraahhh..m uni wrw..
rafal www.llnwwwwec..omm wni.wroc. .proc.pl.
Tomasz Trzeiński
Maciej Zięba
Warsaw University of Technology
Department of Artificial Intelligence
Jagiellonian University of Cracow
Wroclaw University of Science and Technology
Tooploox, IDEAS NCBR
Tooploox
tomasz. wwwcintkiiiiiii.
maciej zieba@pwr..r.o edu. pl
Abstract
Continuous hidden Markov models (HMMs) assume that observations are gener- ated from mixture of Gaussian densities, limiting their ability model more complex distributions. In this work, we address this shortcoming and propose novel continuous HMM models, dubbed FlowHMMs, that enable learning general continuous observation densities without constraining them follow Gaussian distribution or their mixtures. To that end, we leverage deep flow-based architec- tures that model complex, non-Gaussian functions and propose two variants of training a FlowHMM model. The first one, based on gradient- gradient-based technique, can be applied directly to continuous multidimensional data, yet its application to larger data sequences remains computationally expensive. Therefore, we also present second approach training FlowHMM that relies the co-occurrence matrix of discretized observations and considers the joint distribution of pairs co-observed values, hence rendering the training time independent of the training sequence length. As result, we obtain model that can be flexibly adapted to the characteristics and dimensionality of the data. We perform a variety of experiments in which we compare both training strategies with a baseline of Gaussian mixture models. We show, that in terms of quality of the recovered probability distribution. accuracy of prediction of hidden states, and likelihood of unseen data, our approach outperforms the standard Gaussian methods.
1 Introduction
Hidden Markov models (HMMs) are a standard tool modeling and analysis time series data. Although structurally simple, they have been successfully applied in wide variety of applica- tions, ranging from finance [I speech recognition to computational biology [3] and climate modeling [4]
HMMs are capable of solving complex problems, but their adoption limited due to several reasons. First, the training process of HMMs typically relies on the Baum-Welch algorithm 5 particular case of expectation expectation-mmxxiizaate maximization (EM) method, which offers relatively slow convergence to local maximum. To reduce this burden, several discrete HHMs introduce the so-called co-occurrence matrix that aggregates information about the probability of jointly observed values in the chain and estimate
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

it together with the parameters of the whole model 6 8 9 [0]. computed only once and then used significantly reduce convergence time during optimization, its application is strictly limited to discrete HMMs and cannot be easily generalized to their continuous variants.
Although the co-oceurrence matrix
Secondly, continuous HMM models are restricted follow standard parametrized distributions when modeling the observations. Most of them use either Gaussians or other parametric families of distribu- tions [T [2), while the others rely on the mixtures of Gaussians or apply semiparametrie distribution estimation [3) [4]. As result, the existing HMMs have limited ability to model complex observations that do not follow the distributions mentioned above. This, in turn, hinders their application in real-life use cases, e.g., in human action recognition [1S].
1
fa
(
$1
3
fB
2
A 1)
(3,
82
$3
A
In this work, address the shortcomings of the existing HMMs and introduce FlowHMM, a novel continuous hidden Markov model that learns a gen- eral continuous distribution of observations by ex- ploiting the properties of flow-based models [I6]. The core idea of our approach is to model the distri- butions of the observations with normalizing flows, instead of Gaussians or their mixtures. Flows map a simple Gaussian prior to more complex distribu- tions using a parametrized neural network. This formulation enables flow-based models to overpass parametric models in their flexibility to handle data samples of unknown distributions. Moreover, flow-based models naturally extend to multimodal setting, which effectively renders obsolete the tedious process of tuning the number of hidden states.
2
A(3,3)
Figure 1: The concept of FlowHMM for L = 3 states and transition matrix A. Each emis- sion distribution characterised by density fa,(.) is modeled using a separate flow component. Thanks to this, they can adjust to complex, non- Gaussian distributions.
Practically, training the model with flow-based components requires gradient- gradient-based optimization. To achieve that, we propose two variants of training the FlowHMM model. The first approach based on maximum likelihood (ML) technique and can be applied directly to continuous multidimensional data. the second method, we discretize the continuous values during training and leverage the co-occurrence matrix. As result, we provide end-to-end training procedure that jointly optimizes the flow-based component parameters and the co-occurrence matrix using standard gradient-bat based techniques. Since the model is trained in discretized form, the optimization process is simple and more efficient than the competing HMM training procedures, while during inference, we can still use continuous version of our FlowHMM model.
To summarize, the main contribution of our paper is a novel continuous HMM method dubbed FlowHMM with two alternative training scenarios, capable of modeling complex multimodal distri- butions of observations without constraining them follow Gaussians. Not only does outperform the competing approaches, but it also increases the efficiency of the required optimization procedure.
2 Background
2.1 Hidden Markoy models
Let [XA be an ergodic, time homogeneous Markov chain over hidden states S = {s1,- $L} with a stationary distribution π (M TT). and transition matrix A, i.e., for any k we have P(X &y 8;) = A(i, j). Let (Yr h be sequence of random variables taking values in V (the observation space), which can be continuous or discrete. Let us fix some time horizon
T. Random variables Your = (Yo,.., Yr) are independent conditionally on the state sequence Xo. (No,..., Xr), i.e.
T
pYo::r Yoor| Fo:T)
II P(yalra)
(1)
k=

---

where p(y(x) = P(Y = y|X = 2)) represents the so-called emission probabilities for discrete observations. We will shortly write p(yo:r) for p(Y.:T Vo:T) and similarly e.g., P(yo:T|zo:7 for p(Yo.r = your Xo.T To.T) to simplify the notation. Considering the continuous case, plyl represents the conditional emission density function for given state J. For a Gaussian Mixture HMM (which we simply call Gaussian HMM) we assume that p(y|æ) = Ex=1 0z,kN (y: Pz,k: 22,)) where ax,, is a distribution {1, ...., K.. The values of parameters {0k, M2, o #2,k/k= are determined by the conditioning value Note that we assume that each mixture has the same number of components K. For K we have classic Gaussian HMM.
HMM model can be parametrized by 0 = { A, 3],, where B = 1.3 and 31 stays behind the parameters of p(yl(), for given state For discrete case, Br represents the parameters for categorical distribution, while for a Gaussian HMM we have 3 = {au,Ks 14,,, a2 - The probability of observing the sequence of observations yo:r can be expressed as:
p(yo:r; 0)
p(yo:r To:1: : (pp(r A, A, 20.7 687-1
Σ
(2)
where p(uorr Rerr: 3) is given by Eq. ① and p( A, T) can be expressed as:
T
p(Zo.T: = PP(z))
II
Two II
A Ax
(3)
k=1
2.2 Normalizing Flows
Normalizing flows [16] are generative models that can be efficiently trained via direct likelihood estimation thanks to the application of the change-of-variable formula. Practically, they utilize sequence of parametric and invertible transformations: y hn o .... h1)2). The goal of the
transformation is to map z from the known, normal distribution (22; 0, I) to the more complex distribution described by a density function fy) from the observation domain. The hog-probabilityy for y can be expressed as:
N log det
ahn
log fvv = log (2; 0,I)
(4)
a2n
One of the main challenges while designing the normalizing flows is selection of proper form of transformation functions hn The sequence of discrete transformations can be replaced by continuous equivalent by application of Continuous Normalizing Flows (CNFs) 117, where the aim is solve differential equation of the form ds ga(2(t), f), where gs(2(t), E) represents the function of dynamics, described by parameters B. Our goal is to find solution of the equation in f1, y == 2tz). assuming the given initial state =tt)) with known prior. The transformation function ha and its inverse are defined as defined as:
h=(=)= + //e ga(=((),
(2) y -f" ga(z(1), dt.
(5)
The probabbbbyyl of y can be computed by (where hg (z) 2): / dgg(=((),() dz(t) dt.
log fa(y) log VV (v):0)))
(6)
The choice of using CNFs is motivated by the fact, that we are focused on modeling distributions of one or low-dimensional data. CNFs were successfully applied in such models as NGGP (18).
PointFlow [19]. or StyleFlow (20). where the dimensionality and characteristic of data are similar. is somehow confirmed by the empirical results provided by the authors of FFJORD ([17) Table
2), the proposed approach performs better than discrete flows like RealNVP 21 or Glow 22
for low-dimensional data in terms of normalized log- likeliioooood. Such models were used in Moreover, flows that use coupling layers (RealNVP, Glow) and autoregressive flows (MAF 24] do not make sense for 1D data. While operating on 1D data, we not care about simplifying the estimation of the Jacobian, and any invertible differentiable transformation can be applied. At the same time, we need a complex, well-parameterized transformation that is delivered by dynamic function of CNE.
3

---

3 FlowHMM
In this section we introduce FlowHMMs HMM variants of continuous flow capable to model the observations using complex, non Gaussan distributions. The idea behind this approach is to model each of conditional densities p(y|æ s1) = fag (y), for each of the considered states, $t E S,, using separate CNF module. In practice, plyl2 = 5t) can be calculated using formula given by Eq. (6) with the parameters 3r dedicated for the state 81. The idea of our approach is illustrated in Fig. [
In FlowHMM we assume that {Xx} is stationary. i.e., the stationary distribution я is also its initial distribution. In such a case, instead of using A, the model can be equivalently represented by the state joint probabilities:
(i. = P(X, = Si, X -5) P = s,|X, = s(X, = s;) = j(i,)),
(7)
Note that गy can be computed E, j(i, what can be written as 1,S17, where 1 denotes a vectors consisting of ones and 1; consists of 1 on position i and zeros otherwise. For such a formulation, the FlowHMM model can be parametrized by 0 ( (
We propose two variants of FlowHMM models, which differ mainly in a training process: gradient- based model j-L which trained with an maximum likelihood approach directly on continuous data, and co-occurrence matrix-based model FQ, that utilizes co-occurrence matrix and is trained in end-to-end setting using discretized sequences. The first of the proposed methods does not require discretisation step, but is costly and ineffective for larger sequences. JQ eliminates that problem, and makes the training time independent of the length of the training sequence (only estimating empirical co-occurennce matrix depends on T. but the time is marginal). In upcoming sections we are going to introduce both models in more details.
3.1 Training FlowHMM JMM model
Given a sequence of observations Vo:r (Jo.....9r)) the model is trained by optimizing the log- liglliilihood log p(yo:r; 0) (see Eq. 2 where we aim at finding the optimal values of S* (and thus A and π ) and parameters of flow models 37, such that 0 SS B3. 13,,....,,. = arg maxo log P(yo:T; 0). In order to satisfy the constraints on S (i.e., that 1S1 = 1 and each entry is non-negative, what we denote by S > 0) we parametrize the matrix S by a real-valued matrix S also of size L x L, using the following softmax function:
exp(S(s1, 82))
S(81, 82) E...e, exp(((s;.s5))
(8)
Thanks to that representation, train the S and Bi's 's iteratively with gradient-based approach, by maximizing the incomplete log-likelitte ligliiklioood log p(yorr: 0), calculated directly using the forward algorithm.
3.2 Training FlowHMM 2 model
Considering FR model we introduce the co-occurrence matrix that represents the joint distribution
of two consecutive observations: Q 32) p(Yx+1 y/2 Y, v1). The matrix represents categorical distribution, (ith) entry represents the probability of observing a pair of states (V;, 1y) some fixed two consecutive steps & and is + 1. Note that it is independent of k, since throughout the paper we assume that underlying Markov chain on hidden states is stationary. thus a bivariate distribution of (Xs, Xk+1) i) is independent of k. The matrix can be rewritten as:
Q(11.12)
Σ SL.SyES
Assuming the discrete set of observations, V (u....,, Uar }. it can be further expressed as Q SP, where collects probabilities of possible observations at each hidden state matrix P(s;, 13) = p(ujles (s:). In this case there are 2 possible observation pairs (v1, v2) E V x V. Note that matrices Q,S. P are of sizes M x M, x I. and I. x M, respectively. Moreover, we have En Q v,) Exye, S( 9,) 1 and Evey P(s v) 1 for all s; E S. In matrix

---

form, these can be written as 1Q1" - 1S1" = and P1 = 1 (note that 1 must be of an appropriate size). Given observations yo:r the matrix Q can be empirically estimated by:
T-1 E
Q U2)
My u: U vz),
(10)
for all pairs (Us; 11)) V x V. The problem of training the HMM is to find such parameters (matrices S and P) so that the co-occurrence matrix Q is close (in some sense) to the empirical co-occurrence
matrix Q. We can formulate the problem as (for some distance dist)
dist QPP'SP)) SP), subject to 1S1" 1, P1 1 P.. 0.
min MM SERAXE
(11)
This problem formulation has couple of advantages compared to standard likelihood-based op- timization. First, the empirical co-occurrence matrix Q is independent of the sequence length T. Second, the given objective can be easily optimized using gradient-basee techniques. On the other hand, the constraints for matrices P and S should be satisfied. The set V is discrete, while we aim to design the model for continuous observations. In order to satisfy the constraints for matrix S we use the representation given by Eq. (8). In order to represent matrix P we use Flow-based emission probabilities.
Flow-based emission probabilities. With each hidden state we associate the flow model described by a density function S/, that can be calculated from Eq. (6), where Br set of trainable parameters of the flow. We construct P based on these models (in such a case P = Pa) The i-th row of the matrix density fa. (+) evaluated at U.....,
........M......MM UM and normalized as follows:
(uy)
Ug) Σ (1,(1))
(12)
Similarly, the optimization problem becomes:
min 8,SeRLxL
dist dist(Q,, PSSP))
(13)
As distance function, we propose use Kullback- Leibler divergence,
L L= Qsl, Qa(i.)
min f, A.RERiXL
(14)
(i.5)
where Q,3 STSP3. We postulate to apply divergence instead to L distance used egg. . in 7 because is more natural measure for comparing two distributions. We observed during empirical evaluation, that using this divergence gives better stability and convergence of the training process.
The final training procedure of FlowHMM FQ model is as follows. Let train be the training set and let test, be the test set. For the fixed we construct the grid r = . using one train. of the approaches described in A.1. Next, we create the discretized training data yo:T where y; I arg min 11> grrain| || The values of the grid represent further the set V. V Next, we calculate the empirical co-occurrence matrix Q:
Q 0z) = T I(sk train.f = ey (1
train.f = vz),
(15)
and matrices S and P,8 (using Eqs. [4]. Practically, add Gaussian perturbations to the grid values, while calculating the matrix P:
that are further used to calculate the loss given by Eq.
(u, +e)
Pa(s v2) Ee, +c)
(16)
where e is a random sample from (O, a 2o ) and 2oii is hyperparameter the method.
This is one of the standard tricks applied while training normalizing flows, that imitates the situation where we have an access to an infinite number of training examples. These perturbations prevent from overfitting of the flow-based components caused by observing the same grid while training. Next, all of the parameters of FlowHMM are updated with gradient based approach. The procedure is repeated until convergence. The procedures of training is presented in Algorithm
5

---

3.3 Inference with FlowHMM Algorithm 1 Training using JQ technique
Require: Q: empirical co-occurrence matrix from Eq. (T)
As we postulated before, our model is designed for continuous problems, due to the fact, that both of the training techniques turn estimated S, and the param- eters, that represent emission dis- tributions for each of the states, Thus, during the inference stage,
Parameters: 33 [....... 31, flow parameters, S parame-
ters representing un-normalised co-occurrence matrix. peeppaaaamter::: L number of hidden states, a- step size, noise variance o nnise.
1: function TRAIN(Q, L, a)
2: Initialize B. and S.
3:
while not convergent do
we simply calculate A and π from S and apply forward proce-
4:
Calculate from Eq. (8) and P3 from Eq. (16).
dure on the test data y0:7" to cal- culate p Aest, : O). We can also determine the hidden state values
5:
Calculate loss function L from Eq. (14 using Q- vv
6:
5 S
7:
for each 1 ....... L} do
using the Viterbi procedure. Con-
8:
3 +- B 0 VV3. C
cluding, Flow PlowHMM can be used in the same applications as stan- dard continuous HMM models,
9:
end for end while
10:
11:
return 3I..... S
but with no restrictions to emis- sion distributions.
12:
end fuuction
FlowHMM for multidimensional observations. Our approach can be easily extended to multidi- mensional observations, i.e., to the case where train are from Rd. For such a case, the FMI, model straightforward, let us focus on description of QQ, On one hand the extension straightforward: we construct some grid I' = (7....., ".t....,,.a. 7M) of d-dimensional points, create discretized training set I and the empirical co-occurrence matrix Q from (15), afterwards
compute P/3 from Eq. (16) and the gradient V,, C of L given in (14). In other words we proceed with an Algorithm | Note that CNFs are well-suited for multivariate observations. On the other hand, there is challenging aspect of training F model for multidimensional data: an effective discretization technique. We elaborate on that in A.1 proposing new grid search method. We suggest using FQQ model for lower-dimensional data, especially with long observation sequence (recall, the matrix Q is computed once) - in such a case JML will usually be very slow, each epoch loops through a whole observation sequence. On the other hand, for FML is better suited for high-dimensional data and short observation sequence for longer observation sequences, trick, at each epoch we sample subsequence of fixed length, e.g., 10a, There is trade-off between execution time and performance, see Table 7 in A.7.
Vo.
order to shorten the execution time, we apply small
4 Related work
One of the most popular way to train HMM models is the Baum-Welch algorithm (a name for EM applied to HMMs). However, one of (several) drawbacks of using welnh algorithm for training that it is prohibitively slow for long sequences. The complexity of the newward-backwerr (which must be run for each epoch) is O(N2T), where the number of hidden states and is the length of the observation sequence. In order eliminate this issue, the authors [6] propose to pairwise co-occurrence probabilities (also higher order statistics are also discussed there) and prove that it is possible recover the structure of an HMM based only on co-occurrences. An interesting extension (both, of pairs and triplets) was considered in 25). Authors used there non-consecutive tuples, which outperforms consecutive tuples in some cases.
algorithm
Our approach is, in a sense, close to the technique used in 77,, with however significant differences: authors use alternating least square methods for optimization (in our case, all the matrices are real- valued and, softmax is applied whenever needed) and for the continuous case, they assume that the observation densities are mixture of the predefined number of kernels. The "softmax trick" is also used in 9 (their model learns so-called dense representations of hidden states and observation probabilities it designed only for discrete HMMs).
As already mentioned, typically continuous HMMs assume that observations were sampled from Gaussian distribution a mixture of such distributions. For non- Gaussian distribution, usually

---

assumes some parametric family of distributions. In [1] authors propose a decoupling method to learning the parametric HMMs which are stationary. Instead of estimating the parameters of hidden states and observations jointly, they learn the parameters of observation densities using some parametric mixture learner, and then hidden states by solving some convex quadratic programming problem. In [13] method is proposed for case where at least one state-dependent distribution is modeled with some nonparametric technique (e.g., maximum likelihood estimation under shape constraints), some broader review of methods is presented in [26). In (44 authors use Gaussian copula to model the dependence structure. A semiparametric data transformation also proposed to ensure one may indeed use such copulas (the final observation distribution is finite mixture of the copula models). In [27] authors present an efficient learning for parametric continuous HMMs sampled at finite irregular time instants (they incorporate some ideas from the theory of continuous-tima Markov chains).
In 28 the authors consider the extension of a Kalman filter (which you can think of as a version of a HMM with continuous both, observations and hidden states), where the pseudo-observation (as authors call it) are transformed through normalizing flow producing the actual observation.
The most similar model our approach was introduced in Ghosh al. 20 and further developed in 223. where the authors employ normalizing flows in observation densities and apply it to classification problems. However, they utilize the flow with coupling layers that cannot be directly applied to one-dimensional data. We use CNFs that can be easily used for any type of data, and are characterized by better quality for -ww-immeniional examples compared to coupling-buis based flows. Moreover, their approach relies on EM training, which makes it impractical for longer sequences of data. We propose two variants of training models: a) QQQ model which relies on co-occurrences of discretized data and can successfully be applied on very long sequences, and b) FML model which applies gradient- based optimization not only to the parameters of the flow (as in [29] and (23). but also to the transition matrix on hidden states (which because the underlying chain is stationary - is uniquely determined by the matrix representing joint states probabilities and parametrization given in Eqs. (8) and (9) consequently, in each iteration, only one forward step is needed, while Ghosh et al. model requires two forward passes (one to fit frozen weights (no gradient) and one gradient- based to update weights).
For mie-dimensional examples we show the advantages of the proposed JQQ empirically (compared to JML model). Finally, the authors of [29] apply their model only on sequence classification as black box. At the same time, we investigate the proprieties of specific examples in order to discover the capabilities of distribution adjustment and discover the hidden states of data unobserved during training.
5 Experimental results
J_L
FQ
Evaluation. Here we consider our FlowHMM models JML and LQ with training procedures described Sections 3.1 and 3.2 respectively.
3.2
We compare our results with classic HMM models utilizing Gaussian mixtures obser- vation densities with K 1, 10, 20 com- ponents; we denote them by ((K) and for simplicity we write g = g()) We used hmmlearn library for the computations of all g(K) models and provide the source code for FlowHMM model We evaluate the quality of the proposed methods computing Normalized log-likelihood (normLL) of un- seen observations. Total variation distance (dev) between learned and true emission probabilities, and accuracy of predicted hid- den states (accuracy).
Figure 2: Distributions learned by JML and F for T = 10 observations for Example (top row) and Ex- ample 2a (bottom row). Original distributions shaded.
htaape///gichub.com//mm///
7

---

Example
Example 2n
Example 2b
G
g(
g(
g
g(1e)
g(a0)
y-
g(0)
g(
5-
10
-2.21
2.46
.3.66
0.17
0.17
0.09
10
2.19
-2.99
-2.19
-2.28
-2.18
0.18
0.17
0.17
0.33
0.34
0.09
0.18
0.18
0.32
0.30
10a
-2.19
-2.19
-2.19
-2.28
-2.16
0.18
0.17
0.17
0.34
0.33
0.09
0.18
0.18
0.32
0.18
10a
0.09
0.09
0.09
0.13
0.39
0.28
0.31
0.31
0.38
0.43
10
0.07
0.07
0.07
0.11
0.05
0.29
0.32
0.32
0.30
0.15
0.07
0.07
0.07
0.11
0.04
0.29
0.32
0.32
0.28
Table 1: normLL and dy metrics for synthetics Examples 1, 2a-b. Best results bolded.
JQQ
JML
g
Figure 3: Results for Example 2b (three original distributions shaded) for = 104 observations.
We define all the metrics and motivate their choice in A.2, the experimental settings are described in details in A.3.
ID Synthetic sequences. We consider HMMs with L = 3 hidden states and experiment with transition matrices A (used in 7 Eq. (I1))) A2, where
0.0
0.9
0.1
0.4
0.2
0.4
A
0.0
0.0
1.0
A2
0.25
0.5
0.25
(17)
1.0
0.0
0.0
0.4
0.2
0.4
We consider examples. Example 1 transition matrix A and emission probabilities: two Gaussian and one Uniform. Example 2a transition matrix A2, emission probabilities: one Gaussian, one Uniform, and Beta distribution. In addition, consider Example 2b, where we aim at adjusting the models assuming only two hidden states. The examples are described in details Appendix A.4.
For each example we sample train and test observations of the same length T E { , 10"}. We train JMI. and F models using 500 (Example 1) and 1000 (Examples 2a and 2b) epochs. For FQ we used grid of size M 30. For each case (i.e., fixed example and 7) we perform 10 simulations and report means of normLLs and total variation distances in Table T For observations of length T 2 10s flow models clearly outperform Gaussian ones in all the examples. Note that in these examples we had a grid of size M = 30, thus we estimated the matrix Q with 900 entries (number of observation pairs). It is intuitively clear that T = 10 observations is not enough then for FQ, In this case JML outperforms JQ in terms of normLLs; models are comparable for larger T. For T 2 10a the JQQ model "recovers" original distributions much more accurately, the small values of d, from Table I are confirmed in Fig. 2 where distributions learned by our flow models trained with T 10 observations depicted (from single simulations). In A.4 in Fig. 5, the trained distributions for Examples 1, 2a-b trained with T 10x observations are depicted. In A.7 we investigate the impact of length T quality of the model.
We encourage the reader to confront Fig. for FQ and T 10 for Example with Fig. 4]. In Fig. we depict the distributions learned models with only I. = 2 hidden states for T = 10 observations (recall observations were sampled from model with different distributions). Since flow models are capable of fitting multimodal distributions, they outperform Gaussian ones. Moreover, co-occurrence matrix- based model l²Q outperforms the models FML. and _i assume that the underlying Markov
model
JMI Recall that, by construction, chain is stationary. We want to point out that in our examples we used at most L states. In such a case, even if the assumption of stationarity does not hold, it does not have large impact results. This is because for such chains the rate convergence to stationarity is exponential (in number of steps). To be more specific, e.g., for the chain with Lm. A₂ given in (17). after 10 steps the total variation between the stationary distribution (which is (1) = 5/14, (2) = 4//4, (3) = 5/14)
and the distribution of X to is at most (for any initial distribution) 4.22 10 -
8

---

S&P
Real datasets. Example 3. S&P 500 and Dow Jones
g
Gran
XT..
J4
10.476
10.157
10.686
6.850
-5.779
are popular measures of market performance. We retrieved
3
-10.800
-10.194
-10.806
-68837
-4.666
data [30] for period of 719977-82011 that consist of 2082 points. We trained models on = 1000 observations (and computed normLLs for the remaining data).
-12.462
-9.7641
-11.130
-6.908
-5.04
Dow Jones
-12.317
-141154
-13.138
8.730
7.120
-12.645
-13.747
-15.007
-8
-7.426
Example 4. We used mean air pressure from [31 dataset collected between 02/15/18 and 02/28/19. There were 54 531 observations, the models were trained on the first T= 40 000 ones and tested them on the remaining obser- vations. In both, Example 3 and Example 4, we applied standard difference transform (to remove trend), i.e., we considered time series V/44 = yls yk- We trained the models with J. E 3,44 hidden states, models JML and FQ were trained for 500 epochs, for the latter we used a uniform grid of size M 30. During training FML, model we randomly sampled subsequences of length 10 for each epoch. The values of normLL are reported in Table 2 As can be seen, in all cases FQ yields the best results and for Example 3 model FMM. is second best. Note also that although a number of observations for Example 3 is not large (we used only T = 1000 observations, while for M = 30 we have 900 observation pairs) Ja outperforms JML Fig. 6 in A.4 shows that the learned observation densities for I. E (2,3.4] are non-Gaussian. We want to remark one thing: in financial data one usually works with log-returns log(yk/ (): which tend be " "more Gaussian". This is also the case with the examples we considered: FMLL, F and Gaussian models returned very similar normLLs (which no surprise in case of Gaussian data, other models easily fitted to them). We chose and reported the differences yı Mk to compare the models in case of non-Gaussian data.
-13,956
-13.313
-14,669
-8.022
-7.514
Air pressure
-0.704
0.169
0.598
-0.984
0.823
3
-0,743
0.646
0.603
0,540
0.861
-0.758
0.692
0.661
0.783
Table 2: normLL for Examples 3 and 4.
2D synthetic sequences. We consider HMM models with -= 3 states. We consider 2 examples: Example 5 with emission probabilities: two "Moons" and one Uniform and Example with emission probabilities: one bivariate Gaussian, one Uniform and one related to geometric Brownian motion. Details provided in A.5. In both examples we consider the transition matrix A (variant (a)) and A (variant (b)), defined in (1
T
Example Sa
Example 5b
g
g(10)
(20)
Ghoch
J-M
9
g(10)
g(
Ghosh
J-M1.
accuracy nomtL.
10a
-1.6572
.0.6211
.066677
-0. 764644
-0.60033
-2.7090
-1.6500
-1.4459
-1.4486
40.7430
-1.3309
-2.6789
10
-1,6083
-0.6176
-0.6081
-0, 70522
-0,4822
-0.6711
-1.6574
-1,4392
-1.356
40.7036
-1,2286
-1.4185
10
-1.6117
-0,6001
-0,5988
-0.6740
-0,4898
-3,4500
-1.6510
-1,4100
-1.4178
-0.6728
-1.2538
-3,0717
16a
0.6010
1.00000
1.0000
1,0000
0.5600
0.5450
0.6560
0.6480
0.7390
0.5360
10a
0.6325
0.9995
0.9998
0,9998
0.9956
0.5680
0.6677
0.6723
0,7088
0.7414
10a
0.6274
1.0000
1,0000
0.9998
0.6097
0.5726
0.6775
0.6727
0.7448
0.5615
Example 6a
Example 6b
accuracy sorall
10a
0.8652
0,9443
0,9472
0,1600
1.0531
-2.1823
0.2270
0,3128
0,2915
0,1487
0.3636
-2.1891
10a
0.8755
1.0009
1.0014
0.1673
1.0807
0.9189
0.2378
0,3161
0,3173
0.1665
0.3551
0.2120
10a
0.8843
0.9978
0.9992
0.1713
1.0688
-1.8349
0.2186
0.3076
0.3103
0.1730
0.3496
-0.3917
10a
0.9960
0.9960
0.9960
0,9940
0.6700
0.8270
0.6580
0.6590
0.8550
0.6490
10
0.9971
0.9987
0.9987
0,9989
0.9934
0.8173
0.6533
0.6558
0.5419
0.8428
10a
0.9964
0.9994
0.9994
0.9987
0.9219
0.8077
0.6503
0.6443
0.8796
0.7376
Table 3: normLL and accuracy for Examples 5 and 6. Best results bolded.
g
J-L
Figure 4: Trained distributions for Example 5a for 7 = 1 original observations denoted as red dots e, grid presented as gray crosses +; samples from trained distributions fa, 1, 2, depicted as black green and blue dots (colors correspond to state l).

---

We compare Gaussian baseline models, our FMI, and FQ models, as well as model from Ghosh at al. 23 we used authors' official implementaionf) For the latter we computed only normLLs.
In Table 3 values of normLL (for all models) and accuracy (for all models except Ghosh at al. (23)
reported, whereas in Fig. 4 the trained distributions for Example 5a for T 10 are depicted. Sim- ilar plots for Example 6a are presented in A.5 in Fig. 7. We used non-uniform grids of size M = 30 (cartesian product of two one-dimensional grids of size 30 on each coordinate for other strategies for choosing a grid see 1. One can observe that for all examples except Example 5b model JML gives the best normLLs, Ghosh et al. outperforms our models only Example 5b. For 2D examples we can spot a drop of quality in terms of normLL- - for F model. We performed additional simulations for this model for T 100 observations and grids of sizes 35 and 402, the results are provided in Table
As we can see they are still worse than JM. model (which is no surprise: we have 30*, 35 and 40 pairs and only 10 but we see a tendency: increasing a grid size increases normLL in all cases. One can observe that in all the considered cases and variants, model FML gives the best normLL values. When it comes to the other metric, i.e. the accuracy, all the classic Gaussian mixture models, as well as FMLL, yield almost the same accuracy if transition matrix A was used (variant (a)). However, for the transition matrix A2 (variant (b)), the models JMI. and F outperform the classic models with significant margin (note that A₂ yields "more complex" " model compared to A1).
35".
M 40
Sa
-2.548
-2.499
5b -2.220
-2470
-2,042
-1.997
6b -2.02
-1,990
Table
Values of
normLL for FQ model for Examples 5a-b, 6a-
b for T 10s observa- tions.
High dimensional (6D) real dataset. Example 7: from
31 dataset (used in Example 4) we chose features re-
L
g(0))
g(20)
JML
lated to humidity (mean, stdev) and air pressure (min,
2
1097.49
1096.81
-1066.65
.571.8
max, mean, stdev). Similarly as in Example 4 we con- sidered differences of the observations. Again, the length of training set was set to 40k, the remaining 14k obser-
10997.45
1096.73
-1096.51
.571.97
-0097,39
-1096.65
-1096.09
-572.61
Table 5: Values of normLL for 6-
vations constituted a test set. We used FML model in each epoch we randomly sampled a subsequence of length 10 In all cases (I= 2, 3, 4) the model significantly outperfoms Gaussian models, can be seen in Table 5
dimensional data.
Conclusions and limitations
In this work, we proposed continuous hidden Markov model that leverages deep flow-based network architectures to model complex, non-Gaussian distributions. Although our approach outperforms several baselines and competing approaches, it relies on the co-occurrence matrix that can be only computed with the assumption that the Markov chain on a set of hidden states is stationary limitation not present in the EM-based approaches. Addressing this constraint, e.g., by re-computing temporally stationary co-occurrence matrix during training. is part of our future work. So is in-depth validating model multivariate time series or distributions, where potentially offer more benefits over the existing works. While we do not identify any straightforward negative societal implications of our work, acknowledge that its application to large data corpora can lead to significant energy consumption, despite our main contributions being focused on increased efficiency.
flexible approach can
Acknowledgements
This work supported by Foundation for Polish Science (grant carried out within the Team-Net program co-financed by the European Union under the Euro- pean Regional Development Fund, as well as the National Centre of Science (Poland) Grant No. 2020/39/B/ST6001511 The work conducted by Maciej Zieba was supported by the National Centre of Science (Poland) Grant No. 2021443/BSST600253.. For the purpose of Open Access, the author has applied a CC-BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission.
10

---

References
[1] Rogemar S Mamon and Robert James Elliott. Hidden Markov models in finance, volume 4. Springer, 2007.
[2] Mark Gales and Steve Young. The application of hidden Markov models in speech recognition. Now Publishers Inc, 2008.
[3] Mathukumalli Vidyasagar. Hidden Markov Processes: Theory and Applications to Biology.
Princcton University Press, 2014.
[4] Pierre Ailliot, Craig Thompson, and Peter Thomson. Space- time modelling of precipitation by using hidden markov model and censored gaussian distributions. Journal of the Royal Statistical Society: Series C (Applied Statistics), 58(3):405 4200054-4,,,60 2009.
[5] Leonard E Baum, Ted Petric, George Soules, and Norman Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. The annals of mathematical statistics, 41(1):164-11 171, 1970.
[6] Kejun Huang, Xiao Fu, and Nicholas Sidiropoulos. Learning hidden markov models from pairwise co-occurrences with application to topic modeling. In International Conference on Machine Learning, pages 1D68-2071 PMLR, 2018.
[7] Balaji Lakshminarayanan and Raviv Raich. Non-negative matrix factorization for parameter estimation hidden markov models. In 2010 IEEE International Workshop on Machine Learning for Signal Processing, pages 89-94, 2010.
[8] Daniel Hsu, Sham M Kakade, and Tong Zhang. A spectral algorithm for learning hidden markov models. Journal of Computer and System Sciences, 78(5):1460 1460-1480, 2012.
[9] Joachim Sicking. Maximilian Pintz, Maram Akila, and Tim Wirtz. DenscHMM: Learning hidden markov models by learning dense representations. CoRR, abs/2012.09783, 2020.
[10] Robert Mattila, Cristian R Rojas, Vikram Krishnamurthy, and Bo Wahlberg. Identification of hidden markov models using spectral learning with likelihood maximization. In 2017 IEEE 56th annual conference on decision and control (CDC), pages 5859-5864. IEEE, 2017.
[11] Aryeh Kontorovich, Boaz Nadler, and Roi Weiss. On learning parametric-output hmms. In Proceedings of the 30th International Conference International Conference Machine Learning Volume 28, ICML' 13, page I11-702-II-710. JMLR.org, 2013.
[12) Oscar Darwin and Stefan Kiefer. Equivalence of hidden markov models with continuous observations. arXiv preprint arXiv:2009. /2978, 2020.
[13] Jöm Dannemann. Semiparametric hidden markov models. Journal of computational and graphical statistics, 21(3):677-692, 2012.
[14] Hongyang Yu. A novel semiparametric hidden markov model for process failure mode identifi- cation. IEEE Transactions on Automation Science and Engineering, 15(2):506-518, 2018.
[15] Lu Xia, Chia-Chih Chen, and J. K. Aggarwal. View invariant human action recognition using histograms of 3d joints. In 2012 IEEE Computer Society Conference Computer Vision and Pattern Recognition Workshops, pages 20-27, 2012.
[16] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML. PMLR, 2015.
[17] Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv, 2018
[18] Marcin Sendera, Jacek Tabor, Aleksandra Nowak, Andrzej Bedychaj. Massimiliano Patacchiola, Tomasz Trzcinski, Przemyslaw Spurek, and Maciej Zieba. Non-gaussian gaussian processes for few-shot regression. In Advances in Neural Information Processing Systems. 2021.
11

---

[19] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541 4541-4550, 2019.
[20] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-conditton exploration of Transactions on Graphics (ToG), 40(3):1-22 1-,,,,111211000 2021.
images using conditional continuous normalizing flows. ACM
[21] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprini arXiv:
2016.
[22] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
[23] Anubhab Ghosh, Antoine Honoré, Dong Liu, Gustav Fje Henter, and Saikat Chatterjee. Normal- izing flow based hidden markov models for classification of speech phones with explainability. arXiv preprint arXiv: 100:02007000300 2021.
[24] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
[25] Robert Mattila, Cristian Rojas, Eric Moulines, Vikram Krishnamurthy, and Bo Wahlberg. Fast and consistent learning of hidden Markov models by incorporating non-consecutive correlations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6785-6796. PMLR, 13-18 Jul 2020.
[26] Jöm Dannemann, Hajo Holzmann, and Anna Leister. Semiparametric hidden markov mod- els: identifiability and estimation. Wiley Interdisciplinary Reviews: Computational Statislics, 6(6):418-45) 2014.
[27] Yu-Ying Liu, Shuang Li, Fuxin Li, Le Song, and James M Rehg. Efficient learning of continuous- time hidden markov models for disease progression. Advances in neural information processing systems, 28:3599, 2015.
[28] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. In NeurlPS, 2020.
[29] Anubhab Ghosh, Antoine Honoré, Dong Liu, Gustav Eje Henter, and Saikat Chatterjee. Robust classification using hidden markov models and mixtures of normalizing flows. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6, 2020.
[30] data. world. DowJones and S&P500 datasets, accessed 2022. chasewillden/stee http://tttccmmkkte....o market-from--eee hhrroo-aaahhgh--eeee//oooeeeeee
https://data.wa
[31] AmeriCEDSS. Wind-measuremmentss Papua New Guinea, 2019. data retrieved from https 1a8f-66--2118f-99988
1155--433a11883877
resourca/aBddbcec-BBe -f 4fe 4404cee6266222444455166666114
12


Reviews:
[{"review_id": "zn60jXq8b_N", "reviewer": "Reviewer_8Abb", "summary": "The authors propose to use continuous normalizing flows to extend Hidden Markov Models (HMM) by modeling the conditional distribution of the observables given the hidden state with a flow. They call their method FlowHMM and present two training algorithms: First, they point out that the Expectation-Maximization (EM) algorithm can be used for this purpose, which is a common procedure training HMMs. Second, the authors suggest training their FlowHMM model by finding parameters to match the empirically computed co-occurrence matrix of two subsequent observations. They applied their method to several synthetic and real world sequential datasets and thereby demonstrate the effectiveness of the two training algorithms and outperform the baselines given by Gaussians or mixture of Gaussians instead of a continuous normalizing flow modeling the distribution of the observables.", "questions": "* Why did you choose continuous normalizing flows as a density estimation method in your model and not another approach such as a discrete normalizing flow or an autoregressive model?", "limitations": "The authors highlight in the final section of their paper that the method relies on the co-occurrence matrix for training and is not validated sufficiently on multidimensional time series data.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nI appreciate that the article is well organized and easy to read. The authors have an illustrative figure of their procedure, i.e. Figure 1, which ameliorates the understandability of the presented material even further. They also clearly point out their contributions both in the abstract and in the main paper.\n\nMoreover, in their experiment section, the authors consider a variety of datasets, including synthetic once and real-world examples.\n\n## Weaknesses\n\nMy main criticism concerns the motivation of the method and its comparison to competing approaches.\n\nThe authors chose a continuous normalizing flow to model the conditional distribution of the observables given the latent variables, but this density could have been modeled by any model designed for density estimation, such as a discrete normalizing flow or an autoregressive model. They argue that a continuous normalizing flow is more expressive than a Gaussian or a mixture of Gaussians, which I agree with, but they give no reason why they did not choose another of the advanced density estimation models I mentioned.\n\nMoreover, Ghosh et al. (2020) developed a very similar approach and made a follow-up ([Ghosh et al, 2021](https://arxiv.org/abs/2107.00730)), which is not even mentioned in the paper. They claim that Ghosh's model cannot be applied in the cases they are considering since Real NVP cannot deal with one-dimensional data; however, instead of Real NVP they could just use a Neural Spline Flow or a Residual Flow and made a comparison. This should be done in my point of view to highlight the strength and novelty of the presented method.\n\n## Conclusion\n\nGiven that there are essential things missing in the paper as I argued above, I cannot vote for acceptance of this paper at the current stage. I ask the authors to clarify these points and are willing to update my score if they do so.\n\n**Edit**\n\nSince the authors addressed some of my concerns, I'll increase my score slightly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "xjP5mKwnaZK", "reviewer": "Reviewer_ujjf", "summary": "The paper presents the FlowHMM, a novel combination of a hidden Markov models (HMMs) and flow based deep network architectures, along with two methods of training the model. The first method is based on expectation-maximization (EM), and the second is based on matching co-occurrence matrices computed from data. Experiments are performed comparing both methods with a Gaussian mixture model baseline.", "questions": "Some typos I found:\n- Line 142: I believe it should be $\\sum_{v_i,v_j\\in\\mathcal{V}}$\n- Line 180: \"due to the fact\"\n\nQuestions:\n- In equation (14) you treat the matrices $Q_\\beta$ and $\\hat{Q}$ as probabilistic distributions. Given how these matrices are defined, is there a way to make this more rigorous (i.e. give a probabilistic interpretation of $Q_\\beta$ and $\\hat{Q}$ in their definition)?\n- In equation (16) random perturbations are added to the grid values for the computation of $P_\\beta$ to prevent overfitting. Is there a reason why the same is not done for $\\hat{Q}$ in equation (15)?\n- Have you tried discretization for higher dimensional data? Are the results competitive with mixture models/other flow-based models (Ghosh 2020)?\n- Could you clarify what you mean by \"as a black box\" (line 248) and \"discover the hidden states of the data unobserved during training\" (line 250)? Are you suggesting that it is hard to do inference in the model by Ghosh et. al.?", "limitations": "As I mentioned in the strengths and weaknesses section, the main limitation I see with this work is the difficultly of learning in higher dimensions, which the authors do not address in the main paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main ideas of the paper are clearly presented, and the details of the model along with two different training schemes are treated with care. The background on how continuous HMM models are trained in the past and their pros and cons are also made very clear to the reader. The experiment section provides comprehensive results on fitting several low-dimensional datasets, including a total of 6 different datasets/variants, including 2 real datasets and 2 synthetic ones.\n\nDespite the aforementioned strengths, however, the paper has a couple of weaknesses that are hard to overlook. One major weakness is the lack of comparison with similar continuous non-linear HMM models (e.g. structured variational autoencoders with HMM dynamics). Only comparing with Gaussian mixture HMMs doesn't seem to be sufficient and renders the results somewhat weak in my opinion. \n\nFurthermore, the $\\mathcal{F}^Q$ model, which seems to be the more novel of the two presented, yields unsatisfactory results in the majority of runs in examples 5 and 6 (table 3), which featured 2 dimensional datasets. This huge drop of performance is not addressed or explained in the paper. \n\nTo me, it seems like the drop of performance for $\\mathcal{F}^Q$ in 2D is a consequence of the discretization of the observation space that is central to the approach. As dimensionality increases the discretization is going to incur an exponential cost which seems highly problematic. This point is breifly addressed on line 208 but I believe it deserves more attention and perhaps experimental verification. \n\nLastly, it is claimed that the EM training method is inefficient for long sequences, but the computational costs are not reported or analyzed. A suggestion would be to talk about these computational tradeoffs to give more information about the practical value of these algorithms.\n\nTo conclude, the paper is pretty well written and the model proposed is somewhat novel, but the experimental results are quite weak and the difficultly of fitting to high-dimensional data is an important problem that is missing from this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tyjvb8XxO9D", "reviewer": "Reviewer_eTi7", "summary": "This paper introduces FlowHMM for modeling time-series data. The authors explain that HMMs can either be trained using EM, which is slow and prone to being in poor local minima, or using a co-ocurrence matrix in the case of discrete HMMs. Additionally, HMMs model the observations via simple, tractable distributions like Mixtures of Gaussians. Instead, FlowHMM models these observations using normalizing flows. \n\nThe authors propose two techniques to train their model. The first is with the standard EM approach, and the second approach involves discretizing the continuous HMM into a discrete one, and then using a co-ocurrence matrix based approach.\n", "questions": "Can you provide some insight into how to decide on which training technique to use? (EM, or discretization and co-occurrence matrix). Specifically, can you show empirically what happens when you use co-occurrence matrix when “the assumption that the Markov chain on a set of hidden states is stationary - a limitation not present in the EM-based approaches” (line 330) does not hold. \n", "limitations": "I do not see any potential negative societal impact. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors very carefully describe how they can exploit the structure of their FlowHMM model (flow-based emission probabilities) to easily get a good co-occurence matrix based approach for training their model. I feel that this should be highlighted at the end of the introduction.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nTbtSJj_qDu", "reviewer": "Reviewer_idya", "summary": "The authors study the problem of fitting Hidden Markov Models (HMM) when the continuous observation model is non-Gaussian. Precisely, they consider a family (one for each state of the latent variable) of continuous normalising flow models as the observation model. To jointly fit the transition probabilities of the latent states as well as the continuous flow models, the authors propose two methodologies:\n* FlowHMM \\mathcal{F}^{EM} : maximising the log-likelihood, whilst parameterising the joint latent distribution through a soft-max. \n* FlowHMM \\mathcal{F}^{Q} : fit the observation co-occurrence matrix, whilst: \n    - parameterising the emission probabilities through a soft-max of the normalizing flow\n    - discretising the continuous observation, allowing the observation co-occurrence matrix to be substituted by the empirical variant. \n\nThe second method being motivated by the fact that directly optimising the log-likelihood is more computationally costly as the length of the time series increases. \nThe authors evaluate these two methods against Gaussian mixture model methods on both: 1 and 2 dimensional synthetic examples, as well as real world financial and air pressure datasets. \n\nPost Rebuttal Edit: Thank you to the authors for providing follow up experiments and discussion. I have updated my score.  ", "questions": "Questions:\n* **Computational Cost**: Following the weakness associated to the computational cost:\n\t- What is the impact of sub-sampling sequence length on the statistical performance of the EM method? Particularly, the air pressure data set where there are 40k time steps and only 1k is sampled at any given time. \n\t- Would the EM method be more competitive with longer sampled sequences and fewer epochs? \n\n* For the real datasets, S&P 500 and Dow Jones, are the datasets prices? If so:\n    - it is standard practice in financial applications to work with log returns \\log(y_k/ y_{k-1}), which tend to be more Gaussian (albeit heavy tailed).\n     - how do the Gaussian mixture models perform compare to the normalizing flow methods in this case?\n\n* When Gaussian noise is added to the observations in equation (16) \n\t- how is the Gaussian noise hyperparameter chosen? (does it depend on the grid resolution ?)\n\t- what is its impact on training performance ? \n\nNotations:\n\n* Within background, the subscript notation for the parameters in x i.e. \\alpha_{x,k}, \\mu_{x,k}, \\sigma^2_{x,k}, is overloaded when using \\beta, as \\beta_l = \\alpha_{\\ell,k}, \\mu_{\\ell,k}, \\sigma^2_{\\ell,k}. This should be made clearer. \n* Equation (4) z_{n-1} needs to be defined \n* Section 3.3 : \"due ti the fact\" ti -> to \n\n", "limitations": "* **Curse of dimensionality**: The method used to discretize the observations for higher-dimensional observations seems ad-hoc, with the experiments suggesting the EM method is better when the observation model is more than 2 dimensional.  ", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n* Extending HMM to non-Gaussian observation domains is likely an important problem, where normalizing flows potentially offer a natural and flexible solution. \n* The manuscript is overall clearly written, with good introductions to the background and problem setting. \n\nWeaknesses \n* **Curse of dimensionality**: The approach of discretising the response tends to be limited to one (or low) dimensional observations for the co-occurrence method. \n* **Computational cost**: Experiments provide few details on how the two proposed methods are fairly compared from a computational cost perspective. That is, the Appendix states (where T is the time length of the sequence) that \"... for T > 1000 we randomly [for the EM model] sample sub-sequences of a given size T = 1000 at each step\". The impact of this decision is neither discussed or investigated within the main body. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AqB0A4k6ycs", "reviewer": "Reviewer_5RVg", "summary": "The paper proposes a method that utilizes NFs to construct a more general distribution over observations.\nThe paper provides two different training methods based on the EM algorithm or the co-occurence method.\nThe paper demonstrates the efficacy of both methods on synethetic and real datasets.", "questions": "See above", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited\n\nNothing presented is particularly novel in its own right but, to my knowledge, the combination of ideas is novel and the paper explores the combinations thoroughly (though additional baselines would have been nice wrt the real datasets).\n\n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nThe paper appears to be technically sound and the results are consistent with the claims. Current limitations and directions for future work are (briefly) discussed. These directions seem reasonable as a separate work and I do not think they need to be included in this effort.\n\n\nClarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nThe paper is well-written and well-cited. I have no major complaints. There are a few minor typos (e.g., line 180 \"...due *ti* the fact...\"). The figures and synthetic experiments do an excellent job of demonstrating and clarify the ideas discussed in the text.\n\n\nGeneral comments:\n\nOverall, I found this paper very accessible and the proposed method intuitive. The paper presents its ideas clearly and demonstrates them reasonably well. It would have been nice to see additionaly baselines on the real datasets and might have been nice to see higher dimension examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept