Hidden Markov models are recurrent neural networks: A disease progression modeling application

1 INTRODUCTION . Hidden Markov models ( HMMs ; Baum & Petrie , 1966 ) are commonly used for modeling disease progression , because they allow researchers to conceptualize complex ( and noisy ) clinical measurements as originating from a smaller set of latent health states . Each latent health state is characterized by an emission distribution that specifies the probabilities of each measurement/observation given that state . This allows HMMs to explicitly account for uncertainty or measurement error , since the system ’ s true state is not fully observable . Because of their intuitive parameter interpretations and flexibility , HMMs have been used to model biomarker changes in HIV patients ( Guihenneuc-Jouyaux et al. , 2000 ) , Alzheimer ’ s disease progression ( Liu et al. , 2015 ) , breast cancer screening decisions ( Ayer et al. , 2012 ) , and patient response to blood anticoagulants ( Nemati et al. , 2016 ) . Researchers may wish to integrate HMMs with other disease progression models and/or data sources . For instance , researchers in Igl et al . ( 2018 ) jointly trained parameters for an HMM and a reinforcement learning policy to maximize patient returns . Other researchers have attempted to learn or initialize HMM parameters based on additional sources of patient data ( Gupta , 2019 ; Zhou et al. , 2019 ) . Such modifications typically require multiple estimation steps ( e.g. , Zhou et al. , 2019 ) or changes to parameter interpretation ( e.g. , Igl et al. , 2018 ) . This is because the standard algorithm for fitting HMMs , the Baum-Welch algorithm ( Baum & Petrie , 1966 ) , maximizes the likelihood of a data sequence without consideration of additional covariates . We introduce Hidden Markov Recurrent Neural Networks ( HMRNNs ) - neural networks that mimic the computation of hidden Markov models while allowing for substantial modularity with other predictive networks . Unlike past work combining neural networks and HMMs ( e.g. , Bridle , 1990 ) , HMRNNs are designed to maximize the most commonly-used HMM fit criterion - the likelihood of the data given the parameters . In doing so , our primary contributions are as follows : ( 1 ) We prove how recurrent neural networks ( RNNs ) can be formulated to optimize the same likelihood function as HMMs , with parameters that can be interpreted as HMM parameters ( section 3 ) ; ( 2 ) We empirically demonstrate that our model yields statistically similar parameter solutions compared with the Baum- Welch algorithm ( section 4.1 ) ; ( 3 ) We demonstrate our model ’ s utility in a disease progression application , in which combining it with other predictive neural networks improves predictive accuracy and offers unique parameter interpretations not afforded by simple HMMs ( section 4.2 ) . 2 RELATED WORK . A small number of studies have attempted to formally model HMMs in a neural network context . Wessels & Omlin ( 2000 ) proposes using neural networks to approximate Gaussian emission distributions in HMMs ; however , their method requires pre-training of the HMM . Similar to our work , Bridle ( 1990 ) demonstrates how HMMs can be reduced to recurrent neural networks for speech recognition , though it requires that neurons be computed via products ( rather than sums ) , which are not commonly used in modern neural networks . This model also maximizes the mutual information between observations and hidden states ; this is a commonly used criterion in speech recognition , but less common than likelihood maximization in other domains ( e.g. , disease progression modeling ) . Lastly , Bridle ( 1990 ) and Wessels & Omlin ( 2000 ) present only theoretical justification , with no empirical comparisons with the Baum-Welch algorithm . A limited number of studies have also explored connections between neural networks and Markov models in the healthcare domain . For instance , Nemati et al . ( 2016 ) employs a discriminative hidden Markov model to estimate ‘ hidden states ’ underlying patients ’ ICU measurements , though these hidden states are not mathematically equivalent to HMM latent states . Estebanez et al . ( 2012 ) compares HMM and neural network effectiveness in training a robotic surgery assistant , while Baucum et al . ( 2020 ) proposes a generative neural network for modeling ICU patient health based on the mathematical intuition of the HMM . Although these studies showcase the of value of pairing neural networks and Markov models in the healthcare domain , they differ from our approach of directly formulating HMMs as neural networks , which maintains the interpretability of HMMs while allowing for joint estimation of the HMM with other predictive models . In summary , studies have shown the promise of incorporating elements of HMMs into deep learning tasks , there are no existing methods for optimizing HMM log-likelihood in a neural network context . While past works have also used gradient descent to learn HMM parameters ( e.g . Yildirim et al. , 2015 ) , we demonstrate how specifically implementing HMMs as neural networks allows additional data sources ( e.g. , patient covariates ) to steer model estimation to better-fitting solutions . We thus develop the first neural network formulation of an HMM that maximizes the observed data likelihood , employs widely-used neural network operations , and compares favorably to the Baum-Welch algorithm when tested on real-world datasets . 3 METHODS . In this section , we briefly review HMM preliminaries , formally define the HMRNN , and prove that it optimizes the same likelihood function as a corresponding HMM . 3.1 HMM PRELIMINARIES . Formally , an HMM models a system over a given time horizon T , where the system occupies a hidden state xt ∈ S = { 1 , 2 , . . . , k } at any given time point t ∈ { 0 , 1 , . . . , T } ; that is , xt = i indicates that the system is in the ith state at time t. For any state xt ∈ S and any time point t ∈ { 0 , 1 , . . . , T } , the system emits an observation according to an emission distribution that is uniquely defined for each state . We consider the case of categorical emission distributions , which are commonly used in healthcare ( e.g. , Liu et al. , 2015 ; Leon , 2015 ; Ayer et al. , 2012 ; Stanculescu et al. , 2013 ) . These systems emit one of c distinct observations at each time point ; that is , for any time t , we observe yt ∈ O , where |O| = c and O = { 1 , . . . , c } . Thus , an HMM is uniquely defined by a k-length initial probability vector π , k × k transition matrix P , and k × c emission matrix Ψ . Entry i in the vector π is the probability of starting in state i , row i in the matrix P is the state transition probability distribution from state i , and row i of the matrix Ψ is the emission distribution from state i. HMMs are fit via the Baum-Welch algorithm , which identifies the parameters that ( locally ) maximize the likelihood of the observed data ( Baum & Petrie , 1966 ) . The likelihood of an observation sequence y is a function of an HMM ’ s initial state distribution ( π ) , transition probability matrix ( P ) , and emission matrix ( Ψ ) ( Jurafsky & Martin , 2009 ) . Let diag ( Ψi ) be a k × k diagonal matrix with the ith column of Ψ as its entries - i.e. , the probabilities of observation i from each state . We then have Pr ( y|π , P , Ψ ) = π > · diag ( Ψy0 ) · ( T∏ i=1 P · diag ( Ψyi ) ) · 1k×1 . ( 1 ) The likelihood function can also be expressed in terms of αt ( i ) , the probability of being in state i at time t and having observed { y0 , ... , yt } . We denote αt as the ( row ) vector of all αt ( i ) for i ∈ S , with αt = π > · diag ( Ψy0 ) · ( t∏ i=1 P · diag ( Ψyi ) ) ( 2 ) for t ∈ { 1 , ... , T } , with α0 = π > · diag ( Ψy0 ) . Note that equation 3.1 also implies that αt = αt−1 · P · diag ( Ψyt ) for t ∈ { 1 , ... , T } , and that Pr ( y|π , P , Ψ ) = αT · 1k×1 . ( 3 ) 3.2 DEFINITION OF HIDDEN MARKOV RECURRENT NEURAL NETWORKS ( HMRNNS ) . An HMRNN is a recurrent neural network whose parameters directly correspond to the initial state , transition , and emission probabilities of an HMM . As such , training an HMRNN optimizes the joint log-likelihood of the N T -length observation sequences given these parameters . Definition 3.1 . An HMRNN is a recurrent neural network with trainable parameters π ( a k-length stochastic vector ) , P ( a k × k stochastic matrix ) , and Ψ ( a k × c stochastic matrix ) . It is trained on T + 1 input matrices of size N × c , denoted by Yt for t ∈ { 0 , 1 , . . . , T } , where the n-th row of matrix Yt is a one-hot encoded vector of observation y ( n ) t for sequence n ∈ { 1 , . . . , N } . The HMRNN consists of an inner block of hidden layers that is looped T + 1 times ( for t ∈ { 0 , 1 , . . . , T } ) , with each loop containing hidden layers h ( t ) 1 , h ( t ) 2 , and h ( t ) 3 , and a c-length input layer h ( t ) y through which the input matrix Yt enters the model . The HMRNN has a single output unit o ( T ) whose value is the joint negative log-likelihood of the N observation sequences under an HMM with parameters π , P , and Ψ ; the summed value of o ( T ) across all N observation sequences is also the loss which is minimized through any neural network optimizer ( e.g. , gradient descent ) . Layers h ( t ) 1 , h ( t ) 2 , h ( t ) 3 , and o ( T ) are defined in the following equations . Note that the block matrix in equation ( 5 ) is a c× ( kc ) block matrix of c 11×kvectors , arranged diagonally , while the block matrix in equation ( 6 ) is a ( kc ) × k row-wise concatenation of c k × k identity matrices . h ( t ) 1 = { π > , t = 0 , h ( t−1 ) 3 P , t > 0 . ( 4 ) h ( t ) 2 = ReLu ( h ( t ) 1 [ diag ( Ψ1 ) . . . diag ( Ψc ) ] + Yt [ 11×k . . . 01×k . . . . . . . . . 01×k . . . 11×k ] − 1n× ( kc ) ) ( 5 ) h ( t ) 3 = h ( t ) 2 [ Ik . . . Ik ] > ( 6 ) o ( T ) = − log ( h ( T ) 3 1k×1 ) . ( 7 ) Fig . 1 outlines the structure of the HMRNN . Intuitively , operations within each recurrent block mimic matrix multiplication by diag ( Ψyt ) ( i.e. , h ( t ) 3 = h ( t ) 1 diag ( Ψyt ) ) , while connections between blocks mimic multiplication by P . In each block , layer h ( t ) 1 contains k units that represent the joint probability of being in each state 1–k and all observations up to time t - note that this is equivalent to the α values in traditional HMM notation . Layer h ( t ) 2 , expands each unit in h ( t ) 1 into c units via connections with weights Ψi , j for i ∈ { 1 , . . . , k } and j ∈ { 1 , . . . , c } , resulting in k · c units ; this is equivalent to multiplying h ( t ) 1 ( in row-vector form ) by a column-wise concatenation of diag ( Ψj ) for j ∈ { 1 , . . . , c } . The resulting units represent the probabilities of all possible state/outcome combinations at time t. Each column in Yt is connected to all units in h ( t ) 2 that correspond to that column ’ s observation , with connection weights set to 1 . A bias of −1 and subsequent ReLu activation are then applied to layer h ( t ) 2 ; this leaves the units that correspond to Yt unchanged , while all other units ( i.e. , probabilities for non-occurring observations ) are made negative by the −1 bias , then forced to zero through the ReLu activation . Thus , layer h ( t ) 2 identifies the joint probability of being in each state and observing Yt . Layer h ( t ) 3 then sums across all c units for each of the k states ( all of which are zero except for those corresponding to Yt ) , yielding k unitsrepresenting the probabilities of being in each state given all previous observations and the observation at time t , i.e. , h ( t ) 3 = h ( t ) 1 diag ( Ψyt ) . We then apply a fully-connected layer of weights to transform h ( 3 ) 3 to h ( t+1 ) 1 , equivalent to matrix multiplication by P , i.e. , h ( t+1 ) 1 = h ( t ) 3 P , forcing the rows of P to sum to 1 . Lastly , activations for h ( T ) 3 are summed into unit o ( T ) and subject to a negative logarithmic activation function , yielding the negative log-likelihood of the data given the model ’ s parameters . Note that the hidden layers may suffer from underflow for long sequences . This can be addressed by normalizing layer h ( t ) 3 to sum to 1 at each time point , then simply subtracting the logarithm of the normalization term ( i.e. , the log-sum of the activations ) from network ’ s output − log ( o ( T ) ) . The HMRNN is a special case of an RNN , due to its time-dependent layers and shared weights between time points . Note that its use of recurrent blocks ( each of which containing three layers ) differs from many RNNs that use only a single layer at each time point ; this distinction allows the HMRNN to mimic HMM computations of an HMM at each time point .

Summary: This paper introduces a novel architecture of recurrent neural network that mimics the working of a standard HMM. In particular, the proposed HMRNN learns model-parameters, which are statistically similar solutions to those of a standard HMM, within a general neural network learning framework. While it is shown the proposed network similarly to the HMM, there are many issues that should be considered critically.