INFERENCE, PREDICTION, AND ENTROPY RATE OF CONTINUOUS-TIME, DISCRETE-EVENT PROCESSES

The inference of models , prediction of future symbols , and entropy rate estimation of discrete-time , discrete-event processes is well-worn ground . However , many time series are better conceptualized as continuous-time , discrete-event processes . Here , we provide new methods for inferring models , predicting future symbols , and estimating the entropy rate of continuous-time , discrete-event processes . The methods rely on an extension of Bayesian structural inference that takes advantage of neural network ’ s universal approximation power . Based on experiments with simple synthetic data , these new methods seem to be competitive with state-ofthe-art methods for prediction and entropy rate estimation as long as the correct model is inferred . 1 INTRODUCTION . Much scientific data is dynamic , meaning that we see not a static image of a system but its time evolution . The additional richness of dynamic data should allow us to better understand the system , but we may not know how to process the richer data in a way that will yield new insight into the system in question . For example , we have records of when earthquakes have occurred , but still lack the ability to predict earthquakes well or estimate their intrinsic randomness ( Geller , 1997 ) ; we know which neurons have spiked when , but lack an understanding of the neural code ( Rieke et al. , 1999 ) ; and finally , we can observe organisms , but have difficulty modeling their behavior ( Berman et al. , 2016 ; Cavagna et al. , 2014 ) . Such examples are not only continuous-time , but also discreteevent , meaning that the observations belong to a finite set ( e.g , neuron spikes or is silent ) and are not better-described as a collection of real numbers . These disparate scientific problems are begging for a unified framework for inferring expressive continuous-time , discrete-event models and for using those models to make predictions and , potentially , estimate the intrinsic randomness of the system . In this paper , we present a step towards such a unified framework that takes advantage of : the inference and the predictive advantages of unifilarity– meaning that the hidden Markov model ’ s underlying state ( the so-called “ causal state ” ( Shalizi & Crutchfield , 2001 ) or “ predictive state representation ” ( Littman & Sutton , 2002 ) ) can be uniquely identified from the past data ; and the universal approximation power of neural networks ( Hornik , 1991 ) . Indeed , one could view the proposed algorithm for model inference as the continuous-time extension of Bayesian structural inference Strelioff & Crutchfield ( 2014 ) . We focus on time series that are discrete-event and inherently stochastic . In particular , we infer the most likely unifilar hidden semi-Markov model ( uhsMm ) given data using the Bayesian information criterion . This class of models is slightly more powerful than semi-Markov models , in which the future symbol depends only on the prior symbol , but for which the dwell time of the next symbol is drawn from a non-exponential distribution . With unifilar hidden semi-Markov models , the probability of a future symbol depends on arbitrarily long pasts of prior symbols , and the dwell time distribution for that symbol is non-exponential . Beyond just model inference , we can use the inferred model and the closed-form expressions in Ref . ( Marzen & Crutchfield , 2017 ) to estimate the process ’ entropy rate , and we can use the inferred states of the uhsMm to predict future input via a k-nearest neighbors approach . We compare the latter two algorithms to reasonable extensions of state-of-the-art algorithms . Our new algorithms appear competitive as long as model inference is in-class , meaning that the true model producing the data is equivalent to one of the models in our search . In Sec . 3 , we introduce the reader to unifilar hidden semi-Markov models . In Sec . 4 , we describe our new algorithms for model inference , entropy rate estimation , and time series prediction and test our algorithms on synthetic data that is memoryful . And in Sec . 5 , we discuss potential extensions and applications of this research . 2 RELATED WORK . There exist many methods for studying discrete-time processes . A classical technique is the autoregressive process , AR-k , in which the predicted symbol is a linear combination of previous symbols ; a slight modification on this is the generalized linear model ( GLM ) , in which the probability of a symbol is proportional to the exponential of a linear combination of previous symbols ( Madsen , 2007 ) . Previous workers have also used the Baum-Welch algorithm ( Rabiner & Juang , 1986 ) , Bayesian structural inference ( Strelioff & Crutchfield , 2014 ) , or a nonparametric extension of Bayesian structural inference ( Pfau et al. , 2010 ) to infer a hidden Markov model or probability distribution over hidden Markov models of the observed process ; if the most likely state of the hidden Markov model is correctly inferred , one can use the model ’ s structure to predict the future symbol . More recently , recurrent neural networks and reservoir computers can be trained to recreate the output of any dynamical system through simple linear or logistic regression for reservoir computers ( Grigoryeva & Ortega , 2018 ) or backpropagation through time for recurrent neural networks ( Werbos et al. , 1990 ) . When it comes to continuous-time , discrete-event predictors , far less has been done . Most continuous-time data is , in fact , discrete-time data with a high time resolution ; as such , one can essentially sample continuous-time , discrete-event data at high resolution and use any of the previously mentioned methods for predicting discrete-time data . Alternatively , one can represent continuoustime , discrete-event data as a list of dwell times and symbols and feed that data into either a recurrent neural network or feedforward neural network . We take a new approach : we infer continuous-time hidden Markov models ( Marzen & Crutchfield , 2017 ) and predict using the model ’ s internal state as useful predictive features . 3 BACKGROUND . We are given a sequence of symbols and durations of those symbols , . . . , ( xi , τi ) , . . . , ( x0 , τ+0 ) . This constitutes the data , D. For example , seismic time series are of this kind : magnitude and time between earthquakes . The last seen symbol x0 has been seen for a duration τ+0 . Had we observed the system for a longer amount of time , τ+0 may increase . The possible symbols { xi } i are assumed to belong to a finite set A , while the interevent intervals { τi } i are assumed to belong to ( 0 , ∞ ) . We assume stationarity– that the statistics of { ( xi , τi ) } i are unchanging in time . Above is the description of the observed time series . What now follows is a shortened description of unifilar hidden semi-Markov models , notated M , that could be generating such a time series ( Marzen & Crutchfield , 2017 ) . The minimal such model that is consistent with the observations is the -Machine . Underlying a unifilar hidden semi-Markov model is a finite-state machine with states g , each equipped with a dwell-time distribution φg ( τ ) , an emission probability p ( x|g ) , and a function + ( g , x ) that specifies the next hidden state when given the current hidden state g and the current emission symbol x . This model generates a time series as follows : a hidden state g is randomly chosen ; a dwell time τ is chosen according to the dwell-time distribution φg ( τ ) ; an emission symbol is chosen according to the conditional probability p ( x|g ) ; and we then observe the chosen x for τ amount of time . A new hidden state is determined via + ( g , x ) , and we further restrict possible next emissions to be different than the previous emission– a property that makes this model unifilar– and the process repeats . See Fig . 1 for illustrations of a unifilar hidden semi-Markov model . 4 ALGORITHMS . We investigate three tasks : model inference ; calculation of the differential entropy rate ; and development of a predictor of future symbols . Our main claim is that restricted attention to a special type of discrete-event , continuous-time model called unifilar hidden semi-Markov models makes all three tasks easier . 4.1 INFERENCE OF UNIFILAR HIDDEN SEMI-MARKOV PROCESSES . The unifilar hidden semi-Markov models described earlier can be parameterized . Let M refer to a model– in this case , the underlying topology of the finite-state machine and neural networks defining the density of dwell times ; let θ refer to the model ’ s parameters , i.e . the emission probabilities and the parameters of the neural networks ; and let D refer to the data , i.e. , the list of emitted symbols and dwell times . Ideally , to choose a model , we would do maximum a posteriori by calculating argmaxM Pr ( M|D ) and choose parameters of that model via maximum likelihood , argmaxθ Pr ( D|θ , M ) . In the case of discrete-time unifilar hidden Markov models , Strelioff and Crutchfield ( Strelioff & Crutchfield , 2014 ) described the Bayesian framework for inferring the best-fit model and parameters . More than that , Ref . ( Strelioff & Crutchfield , 2014 ) calculated the posterior analytically , using the unifilarity property to ease the mathematical burden . Analytic calculations in continuous-time may be possible , but we leave that for a future endeavor . We instead turn to a variety of approximations , still aided by the unifilarity of the inferred models . The main such approximation is our use of the Bayesian inference criterion ( BIC ) Bishop ( 2006 ) . Maximum a posteriori is performed via BIC = max θ logPr ( D|θ , M ) − kM 2 log |D| ( 1 ) M∗ = argmax M BIC , ( 2 ) where kM is the number of parameters θ . To choose a model , then , we must calculate not only the parameters θ that maximize the log likelihood , but the log likelihood itself . We make one further approximation for tractability involving the start state s0 , for which Pr ( D|θ , M ) = ∑ s0 π ( s0|θ , M ) Pr ( D|s0 , θ , M ) . ( 3 ) As the logarithm of a sum has no easy expression , we approximate max θ logPr ( D|θ , M ) = max s0 max θ logPr ( D|s0 , θ , M ) . ( 4 ) Our strategy , then , is to choose the parameters θ that maximize maxs0 logPr ( D|s0 , θ , M ) and to choose the modelM that maximizes maxθ logPr ( D|θ , M ) − kM2 log |D| . This constitutes an inference of a model that can explain the observed data . What remains to be done , therefore , is approximation of maxs0 maxθ logPr ( D|s0 , θ , M ) . The parameters θ of any given model include p ( s′ , x|s ) , the probability of emitting x when in state s and transitioning to state s′ , and φs ( t ) , the interevent interval distribution of state s. Using the unifilarity of the underlying model , the sequence of x ’ s when combined with the start state s0 translate into a single possible sequence of hidden states si . As such , one can show that logPr ( D|s0 , θ , M ) = ∑ s ∑ j log φs ( τ ( s ) j ) + ∑ s , x , s′ n ( s′ , x|s ) log p ( s′ , x|s ) ( 5 ) where τ ( s ) j is any interevent interval produced when in state s. It is relatively easy to analytically maximize with respect to p ( s′ , x|s ) , including the constraint that ∑s′ , x p ( s′ , x|s ) = 1 for any s , and find that p∗ ( s′ , x|s ) = n ( s ′ , x|s ) n ( s ) . ( 6 ) Now we turn to approximation of the dwell-time distributions , φs ( t ) . The dwell-time distribution can , in theory , be any normalized nonnegative function ; inference may seem impossible . However , artificial neural networks can , with enough nodes , represent any continuous function . We therefore represent φs ( t ) by a relatively shallow ( here , three-layer ) artificial neural network ( ANN ) in which nonnegativity and normalization are enforced as follows : • the second-to-last layer ’ s activation functions are ReLus ( max ( 0 , x ) , and so with nonnegative output ) and the weights to the last layer are constrained to be nonnegative ; • and the output is the last layer ’ s output divided by a numerical integration of the last layer ’ s output . The log likelihood ∑ j log φs ( τ ( s ) j ) determines the cost function for the neural network . Then , the neural network can be trained using typical stochastic optimization methods . ( Here , we use Adam Kingma & Ba ( 2014 ) . ) The output of the neural network can successfully estimate the interevent interval density function , given enough samples , within the interval for which there is data . See Fig . 2 . Outside this interval , however , the estimated density function is not guaranteed to vanish as t → ∞ , and can even grow . Stated differently , the neural networks considered here are good interpolators , but can be bad extrapolators . As such , the density function estimated by the network is taken to be 0 outside the interval for which there is data . To the best of our knowledge , this is a new approach to density estimation , referred to as ANN here . A previous approach to density estimation using neural networks learned the cumulative distribution function ( Magdon-Ismail & Atiya , 1999 ) . Typical approaches to density estimation include k-nearest neighbor estimation techniques and Parzen window estimates , both of which need careful tuning of hyperparameters ( k or h ) ( Bishop , 2006 ) . They are referred to here as kNN and Parzen . We compare the ANN , kNN , and Parzen approaches in inferring an interevent interval density function that we have chosen , arbitrarily , to be the mixture of inverse Gaussians shown in Fig . 2 ( left ) . The k in k-nearest neighbor estimation is chosen according to the criterion in Ref . Fukunaga & Hostetler ( 1973 ) , and h is chosen to as to maximize the pseudolikelihood Marron et al . ( 1987 ) . Note that as shown in Fig . 2 ( right ) , this is not a superior approach to density estimation in terms of minimization of mean-squared error , but it is parametric , so that BIC can be used . To test our new method for density estimation presented here– that is , by training a properly normalized ANN– we generated a trajectory from the unifilar hidden semi-Markov model shown in Fig . 3 ( left ) and used BIC to infer the correct model . As BIC is a log likelihood minus a penalty for a larger number of parameters , a larger BIC suggests a higher posterior . With very little data , a two-state model shown in Fig . 3 is deemed most likely ; but as the amount of data increases , the correct four-state model eventually takes precedence . See Fig . 3 ( right ) . The six-state model was never deemed more likely than a two-state or four-state model . Note that although this methodol- ogy might be extended to nonunifilar hidden semi-Markov models , the unifilarity allowed for easily computable and unique identification of dwell times to states in Eq . 5 .

Summary: The paper focuses on the problem of modeling, predicting and estimating entropy information over continuous-time discrete event processes. Specifically, the paper leverages unifilar HSMM's for model inference and then uses the inferred states to make future predictions. The authors also use inferred model with previously developed techniques for estimating entropy rate. The authors describe the methods and provide the evidence of the effectiveness of their method with experiments on a synthetic dataset.