Hidden Markov models are recurrent neural networks: A disease progression modeling application
Authors: Matthew Baucum
Anahita Khojandi
Theodore Papamarkou
Venue:  (2021)
Keywords: hidden markov models
recurrent neural networks
disease progression

Abstract: Hidden Markov models (HMMs) are commonly used for disease progression modeling when the true state of a patient is not fully known. Since HMMs may have multiple local optima, performance can be improved by incorporating additional patient covariates to inform parameter estimation. To allow for this, we formulate a special case of recurrent neural networks (RNNs), which we name hidden Markov recurrent neural networks (HMRNNs), and prove that each HMRNN has the same likelihood function as a corresponding discrete-observation HMM. As a neural network, the HMRNN can also be combined with any other predictive neural networks that take patient covariate information as input. We first show that parameter estimates from HMRNNs are numerically close to those obtained from HMMs via the Baum-Welch algorithm, thus empirically validating their theoretical equivalence. We then demonstrate how the HMRNN can be combined with other neural networks to improve parameter estimation and prediction, using an Alzheimer's disease dataset. The HMRNN yields parameter estimates that improve disease forecasting performance and offer a novel clinical interpretation compared with a standard HMM.

Under review as a conference paper at ICLR 2021
HIDDEN MARKOV MODELS ARE RECURRENT NEURAL NETWORKS: A DISEASE PROGRESSION MODELING AP- PLICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Hidden Markov models (HMMs) are commonly used for disease progression modeling when the true patient health state is not fully known. Since HMMs may have multiple local optima, performance can improved by incorporating additional patient covariates to inform estimation. To allow for this, we formulate a special case of recurrent neural networks (RNNs), which we name hidden Markov recurrent neural networks (HMRNNs), and prove that each HMRNN has the same likelihood function as a corresponding dusscret--oseevattie HMRNN be combined with any other predictive neural networks that take patient covariate information as input. We show that HMRNN parameter estimates are numerically close those obtained from via the Baum- Welch algorithm, validating their theoretical equivalence. We then demonstrate how the HMRNN can be combined with other neural networks to improve parameter estimation, using an Alzheimer's 's disease dataset. The HMRNN's solution improves disease forecasting performance and offers a novel clinical interpretation compared with a standard HMM.
HMM. The
INTRODUCTION
Hidden Markov models (HMMs: Baum & Petrie 1966) are commonly used for modeling disease progression, because they allow researchers to conceptualize complex (and noisy) clinical measure- ments as originating from smaller set of latent health states. Each latent health state is characterized by an emission distribution that specifies the probabilities of each measurement/observaiooo given that state. This allows HMMs to explicitly account for uncertainty or measurement error, since the system's 's true state is not fully observable. Because of their intuitive parameter interpretations and flexibility, HMMs have been used to model biomarker changes in HIV patients (Guihenneuc-Jouy) et al.) 2000), Alzheimer's 's disease progression (Liu et al., 2015), breast cancer screening decisions
(Ayer et al. 2012). and patient response to blood anticoagulan lants (Nemati et al. (0166).
Researchers may wish to integrate HMMs with other disease progression models and/or data sources. For instance, researchers in [gl et al." (2018) jointly trained parameters for an HMM and a rein- forcement learning policy maximize patient returns. Other researchers have attempted learn or initialize HMM parameters based on additional sources of patient data (Gupta 2019 Zhou et al.) 2019). Such modifications typically require multiple estimation steps (e.g.. Zhou et al.) (019)) or changes to parameter interpretation (e.g.. Igl et al. 2018). This is because the standard algorithm for fitting HMMs, the Baum- Welch algorithm (Baum & Petrie 1966), maximizes the likelihood of data sequence without consideration of additional covariates.
We introduce Hidden Markov Recurrent Neural Networks (HMRNNs) neural networks that mimic the computation of hidden Markov models while allowing for substantial modularity with other predictive networks. Unlike past work combining neural networks and HMMs (e.g.. Bridle 1990). HMRNNs are designed to maximize the most commonly-used HMM fit criterion the likelihood of the data given the parameters. In doing so, our primary contributions are as follows: (1) We prove how recurrent neural networks (RNNs) can be formulated to optimize the same likelihood function as HMMs, with parameters that can be interpreted as HMM parameters (section 3): (2) We empirically demonstrate that model yields statistically similar parameter solutions compared with the Baum-

---

Under review as a conference paper at ICLR 2021
Welch algorithm (section 4.1 (3) We demonstrate model's utility in disease progression application, in which combining it with other predietive neural networks improves predictive accuracy and offers unique parameter interpretations not afforded by simple HMMs (section 4.2
2 RELATED WORK
small number of studies have attempted to formally model HMMs in a neural network context. Wessels & Omlin (2000) proposes using neural networks to approximate Gaussian emission distri- butions in HMMs; however, their method requires pre-training of the HMM. Similar to our work, Bridle (1990) demonstrates how HMMs can be reduced to recurrent neural networks for speech recognition, though it requires that neurons be computed via products (rather than sums), which are not commonly used in modern neural networks. This model also maximizes the mutual information between observations and hidden states; this is a commonly used criterion in speech recognition, but less common than likelihood maximization in other domains (e.g., disease progression modeling). Lastly, Bridle (1990) and Wessels & Omlin (2000) present only theoretical justification, with no empirical comparisons with the Weluh algorithm.
A limited number of studies have also explored connections between neural networks and Markov models in the healthcare domain. For instance, Nemati et al. (2016) employs discriminative hidden Markov model to estimate 'hidden states' underlying patients' ICU measurements, though these hidden states are not mathematically equivalent to HMM latent states. [Estebanez et al. (2012) compares HMM and neural network effectiveness in training robotic surgery assistant, while Baucum et al. (2020) proposes a generative neural network for modeling ICU patient health based on the mathematical intuition of the HMM. Although these studies showcase the of value of pairing neural networks and Markov models
the healthcare domain, they differ from our approach of directly formulating HMMs as neural networks, which maintains the interpretability of HMMs while allowing for joint estimation of the HMM with other predictive models.
In summary, studies have shown the promise of incorporating elements of HMMs into deep learning tasks, there are no existing methods for optimizing HMM log-likelihood a neural network context. While past works have also used gradient descent to learn HMM parameters (e.g. Yildirim al. 2013). we demonstrate how specifically implementing HMMs as neural networks allows additional data sources (e.g., patient covariates) to steer model estimation to fetter-fitting solutions. We thus develop the first neural network formulation of an HMM that maximizes the observed data likelihood, employs widely-used neural network operations, and compares favorably to the Baum- Welch algorithm when tested on real-world datasets.
METHODS
In this section, we briefly review HMM preliminaries, formally define the HMRNN, and prove that it optimizes the same likelihood function as a corresponding HMM.
3.1 HMM PRELIMINARIES
Formally, an HMM models a system over a given time horizon T. where the system occupies hidden state E 5 = 21.2...., k} at any given time point t E {0, ",.., T); that is. = i indicates that the system in the ith state time t. For any state 2, S and any time point t {0, 1, ... T}. the system emits an observation according to an emission distribution that is uniquely defined for each state. We consider the case of categorical emission distributions, which are commonly used in healthcare (c.g., Liu et al., 2015 Leon 2015) Ayer et al. 20122) Stanculessuu et al. 2013). These systems emit one c distinct observations at each time point; that is, for any time f, we observe Ve E O, where [0| and O f1.....c).
Thus, an HMM is uniquely defined by a k-llngth initial probability vector π, & x & transition matrix P. and & x emission matrix V. Entry in the vector π is the probability of starting in state i, row i in the matrix P is the state transition probability distribution from state i, and row i of the matrix w is the emission distribution from state
2

---

Under review as a conference paper at ICLR 2021
HMMs fit via the Helu--Welcha algorithm, which identifies the parameters that (locally) maximize the likelihood of the observed data (Baum & Petrie 1966). The likelihood of an observation sequence is function of HMM's initial state distribution (T)), transition probability matrix (P), and emission matrix ( (Jurafsky & Martin 2009). Iet diag((;; be a k x & diagonal matrix with the ith column of v as its entries i.e., the probabilities of observation i from each state. We then have
Pr(y n. P. 1) n
diag L 20) III P diag( v v.)) 1xx1-
The likelihood function can also be expressed in terms of are (1), the probability of being in state at time and having observed (yo...,,Bt) We denote at the (row) vector of all (x( for i E S, with
ax π T ..ddagg v vo)
II P diag( (
(2)
for t E {L...., T}. with ag = π diag( L mo)) Note that equation 3.1 also implies that at Q4-1 -.. P.diag( v y1 for t E {1...., T), and that
Pr(y ,. P. aT 1xx1
(3)
3.2 DEFINITION OF MIDDEN MARKOV RECURRENT NEURAL NETWORKS (HMRNNs)
An HMRNN is a recurrent neural network whose parameters directly correspond to the initial state, transition, and emission probabilities of HMM. As such, training HMRNN optimizes the joint log-likelihood of the N T-length observation sequences given these parameters.
Definition 3.1. An HMRNN is recurrent neural network with trainable parameters ι (a k-length stochastic vector), P (a k l stochastic matrix), and L (a k x c stochastic matrix). It is trained on T 1 input matrices of size N x C, denoted by Y, for t E {o,1......T.
where the n-th row of matrix Y, is one-hot encoded vector of observation y: (n) for sequence E {l..... :.....,N. N). The HMRNN consists of an inner block of hidden layers that is Tooped'T++ times (fort E {0, (0..,...,27 TY),
with each loop containing hidden layers (1) (1) and h and a c-length input layer (t) through which the input matrix Y, enters the model. The HMRNN has single output unit (T) whose value is the joint negative log-likelihood of the N observation sequences under an HMM with parameters n, P. and ;; the summed value of (T)) across all N observation sequences is also the loss which is minimized through any neural network optimizer (e.g., gradient descent).
(1) h2 hug and (T) are defined in the following equations. Note that the block matrix Layers 1
(1) (t)
in equation (5) is a e x (kc) block matrix of c 11xgvectors, arranged diagonally, while the block matrix in equation 6 is (ke) is row-wise concatenation of k x k identity matrices.
t= O, P, 1> O.
h
(1-1))
(4)
1ixk
01xk
h = ReLu 11 [diag( ( ())... diag(Wi Y,
lnx(ke)
(5)
0
1ix
h = h [Ix I+)"
(6)
(T) log (h (T) 1xx1).
(7)
Fig. outlines the structure of the HMRNN. Intuitively, operations within each recurrent block mimic matrix multiplication by diag(V) (V y.) (i.e., (0 = h(t) diag( )), while connections between blocks mimic multiplication by P. In each block, layer n contains k units that represent the joint probability of being in each state 1-k and all observations up to time t- note that this is equivalent
to the values in traditional HMM notation. Layer (t) expands each unit in h (4) into units via
connections with weights 2, for i (I,...,k) k} and j E {l...., c), resulting in c units; this is
equivalent to multiplying h (in row-vector form) by column-wise concatenation of diag( (V))
3

---

Under review as a conference paper at ICLR 2021
hg(0)
hy (0)
Figure 1: Structure of the hidden Markov recurrent neural network (HMRNN). Solid lines indicate learned weights that correspond to HMM parameters; dotted lines indicate weights fixed to 1. The inner block initializes with the initial state probabilities then mimics multiplication by diag V connections between blocks mimic multiplication by P.
for j (I.....c). The resulting units represent the probabilities of all possible state/outcome combinations at time I.
Each column in Y, is connected to all units in h22 that correspond to that column's observation, with connection weights set to 1. A bias of -1 and subsequent ReLu activation are then applied to layer h this leaves the units that correspond to Y, unchanged, while all other units (i.e., probabilities for non-oceurring observations) are made negative by the 1 bias, then forced to zero through the ReLu activation. Thus, layer h! identifies the joint probability of being in each state and observing Y..
Layer () then sums across all e units for each of the states (all which are zero except for those corresponding to Yo. yielding k unitsrepresenting the probabilities of being in each state given all previous observations and the observation time f, i.e., h h diag( y.
(3) to h ((+1) equivalent to matrix multiplication by P. i.e., h (((++)) = h, P, forcing the rows of P to sum to Lastly, activations for hg (T) are summed into unit (T)) and subject to negative logarithmic activation function, yielding the negative bog-likelihood of the data given the model's parameters. Note that the hidden layers may suffer from underflow for long sequences. This can be addressed by normalizing layer h to sum to at each time point, then simply subtracting the logarithm of the normalization term (i.c., the log-sum of the activations) from network's output log(o!T))
We then apply fully-connected layer of weights to transform h
The HMRNN is special case of an RNN, due to its time-dependent layers and shared weights between time points. Note that use of recurrent blocks (each of which containing three layers) differs from many RNNs that use only single layer at each time point; this distinction allows the HMRNN to mimic HMM computations of an HMM at each time point.
3.3 PROOF OF HMM/HMRNN EQUIVALENCE
We now formally establish that the HMRNN's output unit, oTT), is the negative log-likelihood of an observation sequence under an HMM with parameters æ, P, and L. We prove this for the case of N and drop notational dependence on n (i.e., we write y/ as yt), though extension to > is trivial since the log-likel lidhooo of multiple independence sequences is simply the sum of their individual log- likelioooo..s. We first rely on the following lemma.
Lemma 3.1. If all units h (9) (j) are between 0 and I (inclusive), then h h1()) diag( (
Proof. Let ( and h (j) represent the jth units of layer ht and h respectively. Showing (t) (1) diag( V ) is equivalent to showing that (1)(() T jay. h (1) (j) for j E k1,
4

---

Under review as a conference paper at ICLR 2021
To show this, recall that h22 contains xxe units, which we index with a tuple (1, for E (I,...,e) and m E [1....,k]. The connection matrix between h and hg is (diag(V)) diag( v I. Thus, the connection between units h (4 ()) and h (1, 2 when j m, and equals otherwise. Also recall that matrix Y, enters the model through a c-length input layer hy where the jth unit is when V/ = j. and equals 0 otherwise. This layer is connected to h by a x (kc) diagonal block matrix of (1 k) row vectors of ones. Thus, the connection between the jth unit of this input layer and unit (4, m)) of h22 is I when j L. and equals otherwise. Lastly, a bias of -1 is added to all units in hg (t) which is then subject to a ReLu activation, resulting in the following expression for each unit in h2 )
(1) (1, m) ReLu P hy (0) (m) + +n h. - 1)-
h
Because foy (1) is when y: I, and equals 0 otherwise, then if all units in 1 are between 0 and we have
ReLu v m. h (m)) v m. h (m), ye+ ReLu v m. h (m) 1) 0,
h 10 (l.
otherwise.
The connection matrix between h2 and hg is (ke) row-wise concatenation k identity matrices; thus, the connection between hg (1, m) and hg (j) is if m, and 0 otherwise. Hence,
h ())
Σ
122 (4) (1,5) = L j. h 1)
(10)
1=0
Thus, hg = 11 diag( L y.
Theorem 3.1. An HMRNN with parameters π. (1 x stochastic vector), P (k x k stochastic matrix), and & (k x stochastic matrix), and with layers defined as in equations 44 7 produces output neuron oTT) for sequence E {l,....N. N} whose value is the negative log- leleielldhood of corresponding HMM as defined in equation
Proof. Note that, based on Lemma
3.1 and equation 4 h2 = h2 (t-1), diag( V. for t e
( [0 for j kl, Since ar 04-1 P diag( ( y.). then if ft (t-1) Qr-1, then h (t) (j) E [0, for j E {l,..,k] k} and therefore hg -2, We show ag, since h ( π implies that h
{I,.., ..., T}, assuming that h (0)
the initial condition that hg
diag( L ,..) ag- ar, and (T) log(ar lex1), which is the logarithm of the HMM likelihood from equation
Therefore, by induction, h2 (T)
4 EXPERIMENTS AND RESULTS
In this section, we compare HMRNNs to Baum- Welch through computational experiments with synthetic data and case study of Alzheimer's disease patients. The experiment section [.]] demonstrates that Baum-Ward Welch and and the HMRNN yield similar parameter solutions and predictive accuracy. Section demonstratss how augmenting the HMRNN with additional neural networks yields HMM parameters with improved predictive ability real-world data.
4.1 EMPIRICAL VALIDATION OF HMRNN
We demonstrate that an HMRNN trained via gradient descent yields statistically similar solutions to the Baum- Welch algorithm. We show this with serhheiccally-geneeee observations sequences for which the true HMM parameters are known, allowing to validate Theorem 3.1 empirrical.
5

---

Under review as a conference paper at ICLR 2021
We simulate systems with state spaces S = 1, 2.,, that begin in state 1, using k = 5. 10, or 20 states. These state sizes are consistent with disease progression HMMs, which often involve less than 10 states (Zhou al. 2019 Jackson et al. 2003 (ukkarr et al., 2012 Martino et al.] 2020). We assume that each state 'corresponds' to one observation, implying the same number of states and observations (c = k). The probability of correctly observing state (P(y) x44) is tha which is the diagonal of v and is the same for all states. We simulate systems with th = 0.6, 0.75, and 0.9.
We test three variants of the transition probability matrix P. Each is defined by their same-state transition probability Pis, which is the same for all states. For all P the probability of transitioning to higher states increases with state membership; this is known as 'increasing failure rate' and is a common property for Markov processes. As Pis decreases, the rows of P stochastically increase, i.e. lower values of Pa imply greater chance of moving to higher states. We use values of ps: = 0.4, 0.6, and 0.8, for 27 total simulations (k- {5, 10, 20} x v = {0.6, 0.75, 0.9} x Pai (0.4, 0.6, 0.8)).
For each of the 27 simulations, we generate 100 trajectories of length T 60; this time horizon might practically represent one hour of data collected each minute or two months of data collected each day. Initial state probabilities are fixed at 1 for state I and 0 otherwise. Transition parameters are initialized based on the observed number of transitions in each dataset, using each observation as proxy for its corresponding state. Since transition probabilities are initialized assuming no observation error, the emission matrices are correspondingly initialized using 14 0.95 (with the remaining 0.05 distributed evenly across all other states). In practice, gradient descent on the HMRNN rarely yields parameter values that are exactly zero. To facilitate comparability between the HMRNN and Baum- Welch, we post-process all HMRNN results with one iteration of the Baum-Wele al algorithm, which forces low-probability entries to zero. For Baum-Welee and HMRNN, training ceased when all parameters ceased to change by more than 0.001. For each simulation, we compare Baum Welch's 's and the HMRNN's s average Wasserstein distance between the rows of the estimated and ground truth P and V matrices. This serves as a measure of each method's ' ability to recover the true data geatra-enratiigg parameters. We also compare the Welch and HMRNN solutions' liglliilsss using a separate hold-out set of 100 trajectories.
Across all simulations, the average Wasserstein distance between the rows of the true and estimated transition matrices was 0.191 for Baum-Welch and 0.178 for HMRNN (paired t-test p-value of 0.483), For the emission matrices, these distances were 0. 160 for Baum-Wedd t-test p- p-vuuue of 0.262). This suggests that Baum- Welch and the HMRNN recovered the ground truth parameters with statistically similar degrees of accuracy. This can be seen in Figure 2 which presents the average estimated values of pa and the under each model. Both models' estimated pa values are, on average, within 0.05 of the ground truth values, while they tended to estimate the values of around 0.8 regardless of the true that Note that, while Baum- Baum-Welch was slightly more accurate estimating pi and tha the overall distance between the ground truth and estimated parameters did not significantly differ between Baum-Welch and the HMRNN.
and 0.137 for HMRNN (paired
For each simulation, we also compute the log-likelihood of a held-out set of 100 sequences under the Baum- Welch and HMRNN parameters, as measure of model fit. The average holdout log- likelihoods under the ground truth, Geluh, and HMRNN parameters are -9250.53, 9299.03, and -9303.27, respectively (paired f-test p-value for Baum-Welchie Welch/HMMNNN difference of 0.440). Thus, Baum-Weled and HMRNN yielded similar degrees of model fit on held-out sequence data.
4.2 ALZHEIMER 'S DISEASE SYMPTOM PROGRESSION APPLICATION
We also demonstrate how incorporating additional data into an HMRNN can improve parameter fit and offer novel clinical interpretations. We test our HMRNN on clinical data from n = 426 patients with mild cognitive impairment (MCT), collected over the course of three (n 91), four (n = 106), or five (n- = 229) consecutive annual clinical visits (Initiative). Given MCI patients' heightened risk of Alzheimer's. 's. modeling their symptom progression is of considerable clinical interest (Hirao et al.. 2005| Hansson et al. 20100] Rabin et al., (2009). We analyze patients' overall cognitive functioning based on the Mini Mental Status Exam (MMSE; Folstein et al. 1975).
MMSE scores range from 0 to 30, with scores of 27 - 30 indicating no cognitive impairment, 24 - 26 indicating borderline cognitive impairment, and 17 77-22 23 indicating mild cognitive impairment (Chopral et al.] 2007; Monroe & Carter 2012). Scores below 17 were infrequent (1.2%) and were treated as scores of 17 for analysis. We a 3-state latent space S = {0, 2) with x; 0 representing

---

Under review as a conference paper at ICLR 2021
Estimated vs. Ground Truth Parameters, Baum-Welch vs. HMRNN
Baun-Wekch
HMRNN
Aug Extirated
Extirated
0.4
0.6
Aa
80
Figure 2: Estimated Pa (left) and dhe (right) under Baum-Weleh and HMRNN, shown by ground truth parameter value. Results for each column are averaged across 9 simulations. Dashed lines indicate ground truth pa (left) and vha (right) values, and error bars indicate 95% confidence intervals (but do not represent tests for significant differences). In line with Theorem 3.1 Baum-Welch and the HMRNN produce mear-identical parameter solutions according the Wasserstein distance metrie.
"no cognitive impairment,' I; representing "borderline cognitive impairment," and x; representing 'mild cogmitive impairment.' The observation space is O {0, 2), using = 0 for scores of 27 30, yt for scores of 24 26, and y/ for scores of 17 23.
To showcase the benefits of the HMRNN's flexibility, we augment the HMRNN through two sub- stantive modifications. First, the initial state probabilities in the augmented HMRNN are predicted from patients' gender, age, degree of temporal lobe atrophy (Hua et al. 2008), and amyloid beta 42 levels (A,32,, a relevant Alzheimer's s biomarker (Canuet et al. 2015 Blennow 2004)) ), using single-layer neural network. Second, at each time point, the probability of being the most impaired
state, h! (2)) (2), used predict concurrent scores on the Clinical Dementia Rating (CDR, Morris 1993), global assessment of dementia severity, allowing another clinical metric to inform estimation. We use a single connection and sigmoid activation to predict patients' probability of receiving CDR score above 0.5 (corresponding to 'mild dementia'),
We compare HMRNN and Baum- Bauch parameter solutions' ability to predict patients' final MMSE score categories from their initial score categories, using 10-fold cross-validation. We evaluate performance using weighted log-loss (Guerrero-Pena et al. 2018: Stelmach & Chlebus 2020), i... the log-probability placed each final MMSE score category averaged across score categories. This metric accounts for class imbalance and rewards models' confidence in their predictions, an important component of medical decision support (Bussone al. 2015 Lim & Dey 2010). We also report p. the average probability placed on patients' final MMSE scores (computed from L). We train all models using relative log-likelihood tolerance of 0.001% (we do not use parameter convergence since the number of parameters differs between models). Converge runtimes for Baum-Welch and the HMRNN are 2.89 seconds and 15.24 seconds, respectively.
Model results appear in Table I Note that the HMRNN's weighted log-loss L is significantly lower than Baum-Welede ts (paired t-lest p-value= 2.396 10 -), implying greater predictive performance. This is supported by Figure 3] which shows p. the average probability placed on patients' final MMSE scores by score category. Note that error bars represent marginal sampling error and do not represent statistical comparisons between Baum-Welch and HMRNN. Interestingly, the HMRNN yields lower transition probabilities and lower MMSE accuracy (i.e., lower diagonal values Y than Baum-Welch, suggesting that score changes are more likely attributable to testing ervor opposed to true state changes.
DISCUSSION
We outline a flexible approach for HMM estimation using neural networks. The HMRNN produces statistically similar solutions to the Baum-Welch algorithm in its standard form, yet can be combined
7

---

Under review as a conference paper at ICLR 2021
Table I:
Results from Alzheimer's disease case study.
Baum- Welch
HMRNN
π
0.727
0.271
0.002
0.667
0.333
0.000
0.898
0.080
0.022
0.970
0.028
0.002
P
0.059
0.630
0.311
0.006
0.667
0.327
0.000
0.016
0.984
0.000
0.003
0.997
0.939
0.060
0.001
0.930
0.067
0,003
0.175
0.819
0.006
0.449
0.548
0.003
0.004
0.160
0.836
0.005
0.308
0.687
L
0.991
0.884
p
0.371
0.413
Average Probability Placed on Final MMSE Caetgory
0.5
Baum-Weich
HMMNNN
Avarage Probabiiky
0.1
0.0
Final MMSE Score Category
Figure Average probability placed on final MMSE scores, by score category. Recall that HMRNN'S average performance significantly outperforms Baum-Welch (paired (-test p-value= 2.396 x 10 10-"). As see in the Figure, this effect is consistent across score categories. Error bars indicate 95% confidence intervals, and do not represent tests for significant differences.
with other neural networks improve predictive accuracy when additional data is available. In our Alzheimer's case study, augmenting an HMRNN with two predictive networks improves the predictive performance of its parameter solution compared with Baum-Welch. Interestingly, the HMRNN yields a clinically distinct parameter interpretation compared with Baum-Welch, predicting relatively poor diagnostic accuracy for the 'borderline' and 'mild' cognitive impairment states of the MMSE. This suggests that fewer diagnostic categories might improve the MMSE's utility, which is supported by existing MMSE research (c.g.. Monroe & Carter 2012), and suggests the HMRNN might also be used to improve the interpretability of HMM parameter solutions.
In addition demonstrating the HMRNN's utility practical setting, we also make theoretical contribution by formulating discrete-observatiie HMMs as special case of RNNs and by proving coincidence their likelihood functions. Unlike past approaches, our formulation relies only on matrix multiplication and nonlinear activations, and is designed for generalized use by optimizing for maximum likelihood, which widely used for HMM training.
Future work may formally assess the time complexity of the HMRNN formulation. The HMRNN converges slower than Baum-Weles for the Alzheimer's case study. Yet packages for training neural networks are designed to efficiently handle large datasets, and therefore the HMRNN may converge faster than Baum- Welch for problems with larger sample sizes. Yet since sequence lengths in healthcare often considerably shorter than in other domains that employ HMMs (e.g.. speech analysis), runtimes will likely be reasonable for many healthcare datasets. We also limited our case study disease progression; future work might explore the HMRNN in other healthcare domains.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
T. Ayer, O. Alagoz, and N.K. Stout. A POMDP approach to personalize mammography screening decisions. Operations Research, 6065):019--1044 2012.
M. Baucum, A. Khojandi, and Vasudevan. Improving deep reinforcement learning with transitional variational autoencoders: A healthcare application. Journal of Biomedical & Health Informatics (forthcoming), 2020, doi: N0.1100//BH1.2220000277
L.E. Baum and Petrie. Statistical inference for probabilistic functions finite state Markov chains. The Annals of Mathematical Statistics, 37(6):1554-1500 1966.
K. Blennow. Cerebrospinal fluid protein biomarkers for Alzheimer's disease. NeuroRx, 1(2):113-25), 2004.
J.S. Bridle. Alpha-nets: A recurrent 'neural' network architecture with a hidden Markov model interpretation. Speech Communication, 9(1):83-91 008833922222 1990.
A. Bussone, S. Stumpf, and D. O' Sullivan. The role of exlpanations on trust and reliance in clinical decision support systems, pp. 160-169. 2015.
L. Canuet, S. Pusil, M.E. Lopez, R. Bajo, J.A. Pineda-Pardo, P. Cuesta, G. Galvez, J.M. Gatzelu, D. Lourido, G. Garcia-Ribas, and F, Maestu. Network disruption and cerebrospinal fluid amyloid- beta and phospho-tau levels in mild cognitive impairment. Journal of Neuroscience, 35(28): 10325-10330, 2015.
A. Chopra, T.A. Cavalieri, and D.J. Libon. Dementia screening tools for the primary care physician. Clinical Geriatrics, 15(1):38-45 2007.
B. Estebanez, P. del Saz-Orozer Rivas, E. Bauzano, V.F. Muñoz, and I. Garcia-Morales. Maneu- vers recognition in laparoscopic surgery: Artificial Neural Network and hidden Markov model approaches, pp. 1164-1169. 169.. IEEE, 2012.
M.F. Folstein, S.E. Folstein, and P.R. McHugh. Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3):189- 1975.
F.A. Guerrero-Pent Pena, P.D.M. Fernandez, T.I. Ren, M. Yui, E. Rothenberg, and A. Cunha. Multiclass weighted loss for instance segmentation of cluttered cells, pp. 2451-2455. IEEE, 2018.
C. Guihenneuc- by hidden Markov process: Application to characterizing CD4 cell decline. Biometrics, 56(3): 733-741, 2000.
S. Richardson, and L.M. Longini Jr. Modeling markers of disease progression
A. Gupta. Developing clinical decision support systems for sepsis prediction using temporal and non-temporal machine learning method. Unpublished doctoral dissertation, 2019.
Oskar Hansson, Henrik Zetterberg, Eugeen Vanmechelen, Hugo Vanderstichele, Ulf Andreasson, Elisabet Londos, Anders Wallin, Lennart Minthon, and Kaj Blennow. Evaluation of plasma a:340 and a-342 predictors of conversion to alzheimer's disease in patients with mild cognitive impairment. Neurobiology of aging, 3163)3557-367, 2010.
K. Hirao, T. Ohnishi, Y. Hirata, F. Yamashita, T. Mori, Y. Moriguchi, and T. Iwamoto. The prediction of rapid conversion to Alzheimer's 's disease in mild cognitive impairment using regional cerebral blood flow spect. Neuroimage,
2005.
X. Hua, A.D. Leow, S. Lee, A.D. Klunder, A.W. Toga, N. Lepore, Y. Chou, C. Brun, M. Chiang, M. Barysheva, C.R. Jack, M.A. Bernstein, P.J. Britson, C.P. Ward, J.L. Whitwell, B. Borowski. A. Fleisher, N.C. Fox, R.G. Boyes, J. Barnes, D. Harvey, J. Kornak, N. Schuff, L. Boreta, G.E. Alexander, M.W. Weiner, and P.M. Thompson. 3d characterization of brain atrophy in Alzheimer's disease and mild cognitive impairment using tensor-based morphometry. Neuroimage, 41(1): 19-34, 2008
M. Igl. L. Zintgraf, T.A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for POMDPs. arXiv preprint, 2018.

---

Under review as a conference paper at ICLR 2021
Alzheimer's Disease Neuroimaging Initiative. Alzheimer's disease neuroimaging initiative, adni loni. USC, edu
C.H. Jackson, L.D. Sharples, S.G. Thompson, S. S.W. Duffy, and E. Couto. Multistate markov models
for disease progression with classification error. Journal the Royal Statistical Society: Series D, 52(2):193-099, 2003.
D. Jurafsky and J. Martin. Speech and language processing. Pearson, 2009.
M.I. Namen Leon. A POMDP approach age-dependent primary screening policies for cervical cancer in colombia. Unpublished Master's 's thesis, 2015.
B. Y. Lim and A.K. Dey. Toolkit to support intelligibility in context-aware applications, pp. 13-22. 2010.
Y.Y. Liu, S. Li, Li, Song, and J.M. Rehg. Efficient learning of continuous-time hidden Markov models for disease progression. Advances in Neural Information Processing Systems, pp. 3600- 3608, 2015.
A. Martino, G. Guatteri, and A.M. Paganoni. Multivariate hidden markov models for disease progression. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(5):499-501 2020.
T. Monroe and M. Carter. Using the Folstein mini mental state exam (MMSE) to explore method- ological issues in cognitive aging research. European Journal of Ageing, 9(3):265-274, 2012.
J.C. Morris. Clinical dementia rating: Current version and scoring rules. Neurology, 43:2412-2414, 1993.
S. Nemati, M.M. Ghassemi, and G.D. Clifford. Optimal medication dosing from suboptimal clinical examples: deep reinforcement learning approach. 38th Annual International Conference of IEEE Engineering in Medicine and Biology Society, pp. 1978-29811 2016
L.A. Rabin, N. Pare, A.J. Saykin, M.J. Brown, H.A. Wishart, L.A. Flashman, and R.B. Santulli. Differential memory test sensitivity for diagnosing amnestic mild cognitive impairment and predicting conversion alzheimer's disease. Aging, Neuropsychology, and Cognition, 16(3): 357-376, 2009.
I. Stanculescu, C.K. Williams, and Y. Freer. Autoregressive hidden Markov models for the early detection of neonatal sepsis. IEEE Journal of Biomedical and Health Informatics, 1865):1560-1570, 2013.
M. Stelmach and M. Chlebus. Novel multilayer stacking framework with weighted ensemble approach for multiclass credit scoring problem application. Preprint, 2020.
R. Sukkar, E. Katz, Y. Zhang, D. Raunig, and B.T. Wyman. Disease progression modeling using hidden Markov models, pp. 2845-2848. IEEE, 2012.
Wessels and C.W. Omlin. Refining hidden Markov models with recurrent neural networks. In Proceedings of the IEEE-INNS-ENNS International Joint Conference Neural Networks, volume 2, pp. 271-276, 2000.
S. Yildirim, S.S. Singh, T. Dean, and A. Jasra. Parameter estimation in hidden markov models with intractable likelihoods using sequential monte carlo. Journal of Computational and Graphical Statistics, 24(3):446-865, 2015.
Z. Zhou, Y. Wang. H. Mamani, and D.G. Coffey. How do tumor cytogenetics inform cancer treatments? dynamic risk stratification and precision medicine using multi-armed bandits. Preprini, 2019. doi:
34000008.
10


Reviews:
[{"review_id": "ynBnG5Wc_to", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a novel architecture of recurrent neural network that mimics the working of a standard HMM. In particular, the proposed HMRNN learns model-parameters, which are statistically similar solutions to those of a standard HMM, within a general neural network learning framework. While it is shown the proposed network similarly to the HMM, there are many issues that should be considered critically.   \n\n- In Eq. (5), $Y_{t}$ should be $Y_{t}^{n}$ to indicate n-th column/sample and $1_{nx(kc)}$ be $1_{1x(kc)}$.\n- The notation of $Psi_{i,j}$ is undefined.\n- In the simulation study, it is recommended to initialize parameters with random values, instead of exploiting the background knowledge for the simulation environment.\n- It is unclear for the reason of post-processing to compare with the standard HMM.\n- The network learning hyper-parameter values need describing for reproducibility.\n- Page 7, in the middle: What is the meaning of $h_{t}^{(1)}(2)$?  Due to inconsistency in notations, it is hard to read the manuscript.\n- The experimental settings should be denoted in detail.\n- Note that the authors argued and stated in Section 3 that HMRNN mimics the operations of the standard HMM and produces statistically similar solutions to the B-W algorithm. If so, it does not make sense to this reviewer why the HMRNN’s performance was superior to the HMM? \n- The problem setting over Alzheimer’s disease case study is not clear. To this reviewer’s understanding the input to the models was a sequence of quantized MMSE values and the final output is the category of either CDR>0.5 or CDR<0.5. In Fig. 3, why it is about quantization categories of MMSE, i.e., 0, 1, and 2?\n- There should be more rigorous experiments and their results to better support the validity of the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tCmSKdGzfKF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a Hidden Markov Recurrent Neural Network (HMRNN) that mimics the behavior of traditional hidden Markov models (HMM) and shows that the proposed network can be trained to obtain a similar solution as HMM.\n\nSome questions: \n1. The topic is not significant. The paper proposes an RNN to mimic the behavior of traditional HMM. Although the paper proves that the model can be optimized to obtain similar parameters as HMM, it doesn't include any comparison or experiment for the usefulness of the model.\n\n2. The paper lacks the necessary discussions. In particular, the paper claims that the proposed model allows “for substantial modularity with other predictive networks”. Even though the paper has added an additional network on the Alzheimer dataset, the paper does not show any ablation experiment on the contribution of each component and eventually the effectiveness of the proposed work.\n\n3. The paper lacks baseline models for the disease progression modeling. There are several disease progression models based on HMM (e.g., [1] and [2]). The paper doesn't compare to such baseline models to demonstrate the effectiveness of the proposed work.\n\n[1] Yu-Ying Liu, Shuang Li, Fuxin Li, Le Song, James M Rehg. Efficientlearningofcontinuous- time hidden markov models for disease progression. In Advances in neural information processing systems, pages 3600–3608, 2015. \n[2] Ahmed M Alaa, Scott Hu, and Mihaela van der Schaar. Learning from clinical judgments: Semi-markov-modulated marked hawkes processes for risk prognosis. International Conference on Machine Learning, 2017. \n\n4. The experiment on Alzheimer's disease dataset is not well described (feature dimension, covariances, etc) and no code or pseudo algorithm is provided, making the experimental results and findings hard to be reproduced.\n\n5. The methodology is not well explained. In the traditional HMM, the states in the next timestamp are computed based solely on the state on the current timestamp and are independent of the observation. However, based on equation(4) – (6), the proposed RNN is computing the states based on the current observation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pr8ls-ZoiK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe an RNN architecture, the HMRNN, which models the log-likelihood in an HMM.  The authors provide theoretical results showing that the HMRNN objective function does, indeed, correspond to the log-likelihood of an HMM.  Synthetic results are presented which compare the learned parameters between the HMRNN and an HMM trained using the Baum-Welch algorithm.  Finally, HMRNN training is augmented and compared to an HMM trained using the Baum-Welch algorithm for the task of Alzheimer's progression prediction.\n\nFirstly, the idea of providing an RNN architecture which models the log-likelihood of an HMM is very interesting.  However, the authors make several important mistakes in the broadness of their claims.  For instance:\n\"we also make a theoretical\ncontribution by formulating discrete-observation HMMs as a special case of RNNs and by proving\ncoincidence of their likelihood functions\" <-This does not make the\ntwo models equivalent, which is a major misconception stated\nthroughout the paper.  For instance, while the authors proved that the\nHMM log-likelihood is modeled by the HMRNN, this does not mean HMRNNs\nprovide equivalent quantities for the beta-recursion, posterior\ninference, filtering/smoothing/prediction probabilities, or the\nViterbi path in an HMM.  This is important to distinguish, and I also think the title is incorrect by stating: \"HMMs are RNNs.\"  For the evidence provided in the paper, the following claim would be correct: \"RNNs can exactly model the log-likelihood of HMMs.\"\n\nThe empirical evidence, which is meant to support the presented theoretical results showing the equivalence of HMRNNs objective to HMMs log-likelihood, also seem dubious.  The following is claimed in the paper and meant to be supported in Section 4.1: \"We demonstrate that an HMRNN trained via gradient descent yields statistically similar solutions to the Baum-Welch algorithm.\"  However, the \"HMRNN trained via gradient descent\" is:\n\"In practice, gradient descent on the HMRNN rarely\nyields parameter values that are exactly zero. To facilitate comparability between the HMRNN and\nBaum-Welch, we post-process all HMRNN results with one iteration of the Baum-Welch algorithm,\nwhich forces low-probability entries to zero.\"\nEven if the resulting learned parameters between the two were exact, the extra iteration of Baum-Welch added to HMRNN results invalidates the statement that HMRNNs trained via gradient descent produce similar parameters to the Baum-Welch algorithm.  Furthermore, Wasserstein distance is used to compare the two sets of learned parameters, but mean-squared error is much easier to interpret and also the standard measure of similarity when comparing\nlearned graphical model parameters, e.g., see:\n-Noorshams, Nima, and Martin J. Wainwright. \"Stochastic belief propagation: A low-complexity alternative to the sum-product algorithm.\" IEEE Transactions on Information Theory 59.4 (2012): 1981-2000.\n\nIn Section 4.1, why is just a single-layer neural network used to\npredict the initial state distribution?  Does the Baum-Welch-trained\nparameters similarly use this initial state distribution?  If not,\nthis does not actually fairly compare to the Baum-Welch learned\nparameters, since this is something easily included when training an HMM using standard Baum-Welch.  Similarly, the second extension to HMRNN training in section 4.2 does not seem entirely exclusive to RNNs: \"Second, at each time point, the probability of being in the most impaired\nstate... is used to predict concurrent scores on the Clinical Dementia Rating\na global assessment of dementia severity, allowing another clinical\nmetric to inform estimation.\" <- This also seems like something that\npossible using a very simple, augmented Baum-Welch algorithm for\nHMMs.  All the overhead and complexity of HMRNN do not seem warranted.\n\nAlso, while the theoretical results do seem to be correct for discrete emission densities, the notation used in the paper is really difficult to understand and the authors are inconsistent in defining variables/notation, making the description of the model difficult to follow along with.  For instance, in definition 1, what is N?  The definition is very confusing without this\ndefinition.  Is N the number of training sequences?  This setup is\nalso weird, because different training sequences in an HMM may have\ndifferent lengths, whereas T is fixed in the paper.   Please state the notation prior to use, i.e., that h_1^{(t)}(j) just a column vector.  Statements involving \\mathbf{Y_t} and h_y^{(t)} are also very hard to understand, please consider some other way of describing how the observations are loaded into the model.  \n\nFinally, the description of previous work is very light; only one mention of combining ANNs and HMMs is mentioned, and it is largely dismissed that this work used the maximum mutual information training criterion as opposed to maximum likelihood.  However, there is a long history of HMMs\ncombined with DNNs, which are called hybrid HMMs (there are also\nhybrid dynamic Bayesian networks).  These should be discussed and the novelty of the\npresented work should be addressed in light of this previous work. Some examples:\n-Bourlard, Herv¨¦, and Christian J. Wellekens. \"Links between Markov models and multilayer perceptrons.\" Advances in neural information processing systems. 1989.\n-Dahl, George E., et al. \"Context-dependent pre-trained deep neural\nnetworks for large-vocabulary speech recognition.\" IEEE Transactions\non audio, speech, and language processing 20.1 (2011): 30-42.\n-Graves, Alex, Navdeep Jaitly, and Abdel-rahman Mohamed. \"Hybrid speech recognition with deep bidirectional LSTM.\" 2013 IEEE workshop on automatic speech recognition and understanding. IEEE, 2013.\n\nOther comments:\n\nIt seems warranted that some kind of complexity be given for the\nHMRNN, since one of the major selling points of HMMs is that the\nforward-recursion can be done in O(T k^2) time and O(T k) memory.\n\nIn Section 4.1, please list implementation details regarding how HMRNN\nand the Baum-Welch algorithms were run.  This is generally important,\nbut is even more so since timing numbers are reported in Section 4.2.\nAlso, since it not discussed\nat any length, it is unclear whether the authors ran the\nbeta-recursion along with the alpha-recursion (the only of the two\ndiscussed), which is necessary to perform the Baum-Welch algorithm.\n\n\"Since HMMs may have multiple local optima\" <- this statement is\nimproper.  Training HMMs using EM/Baum-Welch may have multiple local\noptima.  Note that other HMM parameter estimation algorithm exist, such as\nspectral learning algorithms, which are not subject to multiple local\noptima.\n\nIt looks like the range of T values in the Alzheimer's application is\n3, 4, and 5.  Is that right?  If so, that is incredibly small to be\ntesting an HMM in general.  While this application is undoubtedly\nimportant, given that the premise is to show that HMRNNs can do things\nbetter than HMMs, a much better benchmark would be for a sufficiently\nlong time-series, such as speech data.  The authors would probably\ngain more traction considering speech data, as:\n1.) HMMs have been most leveraged for this application domain\n2.) Kaldi exists, and you can easily modify its source\n3.) HMMs still play an important part in the training of DNN-speech\narchitectures, e..g., serving as one of the building blocks in\ntraining a CD-DNN-HMM.\nThere are many, many other applications of HMMs to much longer time\nsequences (with easily modifiable open-source implementations, or the\npomegranate graphical models package is also available) that\nthe authors can turn to to more convincingly demonstrate the\nsuperiority of their architecture to standard HMMs.\n\nThe parantheses around superscripts seems completely unnecessary.\n\nIn section 3.3., N is not still not defined, which is confusing.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SS6RKvIU_gM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nAuthors demonstrated that one can encode the data likelihood function of an HMM using a specialized RNN architecture. Unlike previous work where neurons from different layers were multiplied together, the new encoding strictly followed the classical architecture restrictions of a neural network , i.e. each layer was a weighted sum of the previous layer. Empirically, author showed that the parameter learned by applying gradient descent on the likelihood function is similar to the one that is obtained using the EM algorithm. In addition, authors demonstrated that such formulation enables an application in studying Alzheimer's disease Symptom Progression.\n\n### Strength\n* It is an interesting new connection between RNN and HMM that was found by authors. \n* I found the Alzheimer's application is very inspiring. Compared to the baseline method that uses a single HMM to model the progression of the Alzheimer for every patient, author proposed to accommodate each individual patient's attributes when predicting the progression. This leads to a more accurate prediction of the final Alzheimer's prediction for each patient.\n\n### Weakness\n* The motivation for the connection between RNN and HMM is not very clear, and it hinders the significance of the work. As author has pointed out, there are multiple work that tries to formulate the likelihood function of a graphical model, i.e. a HMM, as a computation graph. Besides the ones that are cited by the author, Chapter 12 of [1] also shows general method to generate a computation graph from a graphical model, and the computation graph evaluates the probability query on the graphical model. These different representations differ only in the syntactical representation of the likelihood function. Although authors managed to formulate the function using the structure of a classical RNN, the underlying function is semantically equivalent to the previous methods. Hence, the gradients w.r.t. the HMM parameters are going to the same in all different representations. Hence, performing gradient descent on a HMRNN does not seem to be different from performing gradient descent on other syntactical representations of the likelihood function.\n\n[1] Modeling and Reasoning with Bayesian Networks. Adnan Darwiche.\n\n### Questions\n* It was not clear to me how the supervisions from each patient's attributes were incorporated to model the Alzheimer's progression. As there was no ground truth on the hidden states, I was not sure how the initial probabilities were trained, and how the CDR helped to supervise the training of the hidden states. Are these supervisions incorporated through the loss function?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is consensus that the submission is not yet ready for publication. The reviews contain multiple comments and suggestions and I hope they can be useful for the authors.
Decision: Reject