Hidden Markov models are recurrent neural networks: A disease progression modeling application

1 INTRODUCTION . Hidden Markov models ( HMMs ; Baum & Petrie , 1966 ) are commonly used for modeling disease progression , because they allow researchers to conceptualize complex ( and noisy ) clinical measurements as originating from a smaller set of latent health states . Each latent health state is characterized by an emission distribution that specifies the probabilities of each measurement/observation given that state . This allows HMMs to explicitly account for uncertainty or measurement error , since the system ’ s true state is not fully observable . Because of their intuitive parameter interpretations and flexibility , HMMs have been used to model biomarker changes in HIV patients ( Guihenneuc-Jouyaux et al. , 2000 ) , Alzheimer ’ s disease progression ( Liu et al. , 2015 ) , breast cancer screening decisions ( Ayer et al. , 2012 ) , and patient response to blood anticoagulants ( Nemati et al. , 2016 ) . Researchers may wish to integrate HMMs with other disease progression models and/or data sources . For instance , researchers in Igl et al . ( 2018 ) jointly trained parameters for an HMM and a reinforcement learning policy to maximize patient returns . Other researchers have attempted to learn or initialize HMM parameters based on additional sources of patient data ( Gupta , 2019 ; Zhou et al. , 2019 ) . Such modifications typically require multiple estimation steps ( e.g. , Zhou et al. , 2019 ) or changes to parameter interpretation ( e.g. , Igl et al. , 2018 ) . This is because the standard algorithm for fitting HMMs , the Baum-Welch algorithm ( Baum & Petrie , 1966 ) , maximizes the likelihood of a data sequence without consideration of additional covariates . We introduce Hidden Markov Recurrent Neural Networks ( HMRNNs ) - neural networks that mimic the computation of hidden Markov models while allowing for substantial modularity with other predictive networks . Unlike past work combining neural networks and HMMs ( e.g. , Bridle , 1990 ) , HMRNNs are designed to maximize the most commonly-used HMM fit criterion - the likelihood of the data given the parameters . In doing so , our primary contributions are as follows : ( 1 ) We prove how recurrent neural networks ( RNNs ) can be formulated to optimize the same likelihood function as HMMs , with parameters that can be interpreted as HMM parameters ( section 3 ) ; ( 2 ) We empirically demonstrate that our model yields statistically similar parameter solutions compared with the Baum- Welch algorithm ( section 4.1 ) ; ( 3 ) We demonstrate our model ’ s utility in a disease progression application , in which combining it with other predictive neural networks improves predictive accuracy and offers unique parameter interpretations not afforded by simple HMMs ( section 4.2 ) . 2 RELATED WORK . A small number of studies have attempted to formally model HMMs in a neural network context . Wessels & Omlin ( 2000 ) proposes using neural networks to approximate Gaussian emission distributions in HMMs ; however , their method requires pre-training of the HMM . Similar to our work , Bridle ( 1990 ) demonstrates how HMMs can be reduced to recurrent neural networks for speech recognition , though it requires that neurons be computed via products ( rather than sums ) , which are not commonly used in modern neural networks . This model also maximizes the mutual information between observations and hidden states ; this is a commonly used criterion in speech recognition , but less common than likelihood maximization in other domains ( e.g. , disease progression modeling ) . Lastly , Bridle ( 1990 ) and Wessels & Omlin ( 2000 ) present only theoretical justification , with no empirical comparisons with the Baum-Welch algorithm . A limited number of studies have also explored connections between neural networks and Markov models in the healthcare domain . For instance , Nemati et al . ( 2016 ) employs a discriminative hidden Markov model to estimate ‘ hidden states ’ underlying patients ’ ICU measurements , though these hidden states are not mathematically equivalent to HMM latent states . Estebanez et al . ( 2012 ) compares HMM and neural network effectiveness in training a robotic surgery assistant , while Baucum et al . ( 2020 ) proposes a generative neural network for modeling ICU patient health based on the mathematical intuition of the HMM . Although these studies showcase the of value of pairing neural networks and Markov models in the healthcare domain , they differ from our approach of directly formulating HMMs as neural networks , which maintains the interpretability of HMMs while allowing for joint estimation of the HMM with other predictive models . In summary , studies have shown the promise of incorporating elements of HMMs into deep learning tasks , there are no existing methods for optimizing HMM log-likelihood in a neural network context . While past works have also used gradient descent to learn HMM parameters ( e.g . Yildirim et al. , 2015 ) , we demonstrate how specifically implementing HMMs as neural networks allows additional data sources ( e.g. , patient covariates ) to steer model estimation to better-fitting solutions . We thus develop the first neural network formulation of an HMM that maximizes the observed data likelihood , employs widely-used neural network operations , and compares favorably to the Baum-Welch algorithm when tested on real-world datasets . 3 METHODS . In this section , we briefly review HMM preliminaries , formally define the HMRNN , and prove that it optimizes the same likelihood function as a corresponding HMM . 3.1 HMM PRELIMINARIES . Formally , an HMM models a system over a given time horizon T , where the system occupies a hidden state xt ∈ S = { 1 , 2 , . . . , k } at any given time point t ∈ { 0 , 1 , . . . , T } ; that is , xt = i indicates that the system is in the ith state at time t. For any state xt ∈ S and any time point t ∈ { 0 , 1 , . . . , T } , the system emits an observation according to an emission distribution that is uniquely defined for each state . We consider the case of categorical emission distributions , which are commonly used in healthcare ( e.g. , Liu et al. , 2015 ; Leon , 2015 ; Ayer et al. , 2012 ; Stanculescu et al. , 2013 ) . These systems emit one of c distinct observations at each time point ; that is , for any time t , we observe yt ∈ O , where |O| = c and O = { 1 , . . . , c } . Thus , an HMM is uniquely defined by a k-length initial probability vector π , k × k transition matrix P , and k × c emission matrix Ψ . Entry i in the vector π is the probability of starting in state i , row i in the matrix P is the state transition probability distribution from state i , and row i of the matrix Ψ is the emission distribution from state i. HMMs are fit via the Baum-Welch algorithm , which identifies the parameters that ( locally ) maximize the likelihood of the observed data ( Baum & Petrie , 1966 ) . The likelihood of an observation sequence y is a function of an HMM ’ s initial state distribution ( π ) , transition probability matrix ( P ) , and emission matrix ( Ψ ) ( Jurafsky & Martin , 2009 ) . Let diag ( Ψi ) be a k × k diagonal matrix with the ith column of Ψ as its entries - i.e. , the probabilities of observation i from each state . We then have Pr ( y|π , P , Ψ ) = π > · diag ( Ψy0 ) · ( T∏ i=1 P · diag ( Ψyi ) ) · 1k×1 . ( 1 ) The likelihood function can also be expressed in terms of αt ( i ) , the probability of being in state i at time t and having observed { y0 , ... , yt } . We denote αt as the ( row ) vector of all αt ( i ) for i ∈ S , with αt = π > · diag ( Ψy0 ) · ( t∏ i=1 P · diag ( Ψyi ) ) ( 2 ) for t ∈ { 1 , ... , T } , with α0 = π > · diag ( Ψy0 ) . Note that equation 3.1 also implies that αt = αt−1 · P · diag ( Ψyt ) for t ∈ { 1 , ... , T } , and that Pr ( y|π , P , Ψ ) = αT · 1k×1 . ( 3 ) 3.2 DEFINITION OF HIDDEN MARKOV RECURRENT NEURAL NETWORKS ( HMRNNS ) . An HMRNN is a recurrent neural network whose parameters directly correspond to the initial state , transition , and emission probabilities of an HMM . As such , training an HMRNN optimizes the joint log-likelihood of the N T -length observation sequences given these parameters . Definition 3.1 . An HMRNN is a recurrent neural network with trainable parameters π ( a k-length stochastic vector ) , P ( a k × k stochastic matrix ) , and Ψ ( a k × c stochastic matrix ) . It is trained on T + 1 input matrices of size N × c , denoted by Yt for t ∈ { 0 , 1 , . . . , T } , where the n-th row of matrix Yt is a one-hot encoded vector of observation y ( n ) t for sequence n ∈ { 1 , . . . , N } . The HMRNN consists of an inner block of hidden layers that is looped T + 1 times ( for t ∈ { 0 , 1 , . . . , T } ) , with each loop containing hidden layers h ( t ) 1 , h ( t ) 2 , and h ( t ) 3 , and a c-length input layer h ( t ) y through which the input matrix Yt enters the model . The HMRNN has a single output unit o ( T ) whose value is the joint negative log-likelihood of the N observation sequences under an HMM with parameters π , P , and Ψ ; the summed value of o ( T ) across all N observation sequences is also the loss which is minimized through any neural network optimizer ( e.g. , gradient descent ) . Layers h ( t ) 1 , h ( t ) 2 , h ( t ) 3 , and o ( T ) are defined in the following equations . Note that the block matrix in equation ( 5 ) is a c× ( kc ) block matrix of c 11×kvectors , arranged diagonally , while the block matrix in equation ( 6 ) is a ( kc ) × k row-wise concatenation of c k × k identity matrices . h ( t ) 1 = { π > , t = 0 , h ( t−1 ) 3 P , t > 0 . ( 4 ) h ( t ) 2 = ReLu ( h ( t ) 1 [ diag ( Ψ1 ) . . . diag ( Ψc ) ] + Yt [ 11×k . . . 01×k . . . . . . . . . 01×k . . . 11×k ] − 1n× ( kc ) ) ( 5 ) h ( t ) 3 = h ( t ) 2 [ Ik . . . Ik ] > ( 6 ) o ( T ) = − log ( h ( T ) 3 1k×1 ) . ( 7 ) Fig . 1 outlines the structure of the HMRNN . Intuitively , operations within each recurrent block mimic matrix multiplication by diag ( Ψyt ) ( i.e. , h ( t ) 3 = h ( t ) 1 diag ( Ψyt ) ) , while connections between blocks mimic multiplication by P . In each block , layer h ( t ) 1 contains k units that represent the joint probability of being in each state 1–k and all observations up to time t - note that this is equivalent to the α values in traditional HMM notation . Layer h ( t ) 2 , expands each unit in h ( t ) 1 into c units via connections with weights Ψi , j for i ∈ { 1 , . . . , k } and j ∈ { 1 , . . . , c } , resulting in k · c units ; this is equivalent to multiplying h ( t ) 1 ( in row-vector form ) by a column-wise concatenation of diag ( Ψj ) for j ∈ { 1 , . . . , c } . The resulting units represent the probabilities of all possible state/outcome combinations at time t. Each column in Yt is connected to all units in h ( t ) 2 that correspond to that column ’ s observation , with connection weights set to 1 . A bias of −1 and subsequent ReLu activation are then applied to layer h ( t ) 2 ; this leaves the units that correspond to Yt unchanged , while all other units ( i.e. , probabilities for non-occurring observations ) are made negative by the −1 bias , then forced to zero through the ReLu activation . Thus , layer h ( t ) 2 identifies the joint probability of being in each state and observing Yt . Layer h ( t ) 3 then sums across all c units for each of the k states ( all of which are zero except for those corresponding to Yt ) , yielding k unitsrepresenting the probabilities of being in each state given all previous observations and the observation at time t , i.e. , h ( t ) 3 = h ( t ) 1 diag ( Ψyt ) . We then apply a fully-connected layer of weights to transform h ( 3 ) 3 to h ( t+1 ) 1 , equivalent to matrix multiplication by P , i.e. , h ( t+1 ) 1 = h ( t ) 3 P , forcing the rows of P to sum to 1 . Lastly , activations for h ( T ) 3 are summed into unit o ( T ) and subject to a negative logarithmic activation function , yielding the negative log-likelihood of the data given the model ’ s parameters . Note that the hidden layers may suffer from underflow for long sequences . This can be addressed by normalizing layer h ( t ) 3 to sum to 1 at each time point , then simply subtracting the logarithm of the normalization term ( i.e. , the log-sum of the activations ) from network ’ s output − log ( o ( T ) ) . The HMRNN is a special case of an RNN , due to its time-dependent layers and shared weights between time points . Note that its use of recurrent blocks ( each of which containing three layers ) differs from many RNNs that use only a single layer at each time point ; this distinction allows the HMRNN to mimic HMM computations of an HMM at each time point .

Summary: Authors demonstrated that one can encode the data likelihood function of an HMM using a specialized RNN architecture. Unlike previous work where neurons from different layers were multiplied together, the new encoding strictly followed the classical architecture restrictions of a neural network , i.e. each layer was a weighted sum of the previous layer. Empirically, author showed that the parameter learned by applying gradient descent on the likelihood function is similar to the one that is obtained using the EM algorithm. In addition, authors demonstrated that such formulation enables an application in studying Alzheimer's disease Symptom Progression.