INFERENCE, PREDICTION, AND ENTROPY RATE OF CONTINUOUS-TIME, DISCRETE-EVENT PROCESSES
Authors: Sarah Marzen
James P. Crutchfield
Venue:  (2020)
Keywords: continuous-time prediction

Abstract: The inference of models, prediction of future symbols, and entropy rate estimation of discrete-time, discrete-event processes is well-worn ground. However, many time series are better conceptualized as continuous-time, discrete-event processes. Here, we provide new methods for inferring models, predicting future symbols, and estimating the entropy rate of continuous-time, discrete-event processes. The methods rely on an extension of Bayesian structural inference that takes advantage of neural network’s universal approximation power. Based on experiments with simple synthetic data, these new methods seem to be competitive with state-of- the-art methods for prediction and entropy rate estimation as long as the correct model is inferred.

Under review as a conference paper at ICLR 2018
INFERENCE, PREDICTION, AND ENTROPY RATE OF CONTINUOUS-TING DISCRETE-EVENT PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
The inference of models, prediction of future symbols, and entropy rate estimation of discrete-time, discrete-event processes is well-worn ground. However, many time series better conceptualized continuous-time, discrete-event processes. Here, we provide new methods for inferring models, predicting future symbols, and estimating the entropy rate of continuous-tima discrete-event processes. The methods rely on an extension of Bayesian structural inference that takes advantage of neural network's universal approximation power. Based on experiments with simple synthetic data, these new methods seem to be competitive with state-of- the-art methods for prediction and entropy rate estimation as long as the correct model is inferred.
INTRODUCTION
Much scientific data is dynamic, meaning that not static image of system but its time evolution. The additional richness of dynamic data should allow us to better understand the system, but may not know how to process the richer data in a way that will yield new insight into the system in question. For example, we have records of when earthquakes have occurred, but still lack the ability predict earthquakes well or estimate their intrinsic randomness (Geller) 1997); know which neurons have spiked when, but lack an understanding of the neural code (Ricke et al.) 1999: and finally, we can observe organisms, but have difficulty modeling their behavior (Berman et al. 2016 Cavagna et al. 2014). Such examples are not only continuous-time but also discrete- event, meaning that the observations belong a finite set (e.g, neuron spikes or silent) and are not dester-described as a collection of real numbers. These disparate scientific problems are begging for a unified framework for inferring expressive continuous-time discrete-event models and for using those models to make predictions and, potentially, estimate the intrinsic randomness of the system.
In this paper, we present a step towards such unified framework that takes advantage of: the inference and the predictive advantages of unifilarity- meaning that the hidden Markov model's un- derlying state (the so-called "causal state" (Shalizi & Crutchfield (200I) or "predictive state repre- sentation" (Littman & Sutton 2002) can be uniquely identified from the past data; and the universal approximation power of neural networks (Hornik 1991). Indeed, one could view the proposed algo- rithm for model inference as the continuous-time extension of Bayesian structural inference Strelioffi Streeioc & Crutchfield (2014). We focus on time series that are discrete-event and inherently stochastic.
In particular, infer the most likely unifilar hidden semi-Markov model (uhsMm) given data using the Bayesian information criterion. This class of models is slightly more powerful than semi-Markov models, which the future symbol depends only on the prior symbol, but for which the dwell time of the next symbol is drawn from non-exponential distribution. With unifilar hidden semi-Markov models, the probability of a future symbol depends on arbitrarily long pasts of prior symbols, and the dwell time distribution for that symbol is non-exponential. Beyond just model inference, we can use the inferred model and the closed-form expressions in Ref. (Marzen Crutchfield (017) to estimate the process" entropy rate, and we can use the inferred states of the uhsMm to predict future input via a k-nearest neighbors approach. We compare the latter two algorithms to reasonable extensions of state-of-tt the- algorithms. Our new algorithms appear competitive as long as model inference is in-class, meaning that the true model producing the data equivalent to one of the models in our search.

---

Under review as a conference paper at ICLR 2018
In Sec. introduce the reader to unifilar hidden semi- models. In Sec. we describe our new algorithms for model inference, entropy rate estimation, and time series prediction and test our algorithms synthetic data that is memoryful. And in Sec. [ we discuss potential extensions and applications of this research.
RELATED WORK
There exist many methods for studying discrete-time processes. A classical technique is the autore- gressive process, AR-k, in which the predicted symbol is a linear combination of previous symbols; slight modification on this is the generalized linear model (GLM), which the probability of sym- bol is proportional to the exponential of linear combination of previous symbols (Madsen 2007). Previous workers have also used the Baum- Welch algorithm (Rabiner & Juang, 1986), Bayesian structural inference (Strelioff & Crutchfield] 2014). or a nonparametric extension of Bayesian struc- tural inference (Přau et al., 2010) to infer hidden Markov model or probability distribution over hidden Markov models of the observed process; if the most likely state of the hidden Markov model is correctly inferred, one can use the model's s structure to predict the future symbol. More recently, recurrent neural networks and reservoir computers can be trained recreate the output of any dy- namical system through simple linear or logistic regression for reservoir computers (Grigoryeva & Ortega 2018) backpropagation through time for recurrent neural networks (Werbos al. 1990).
When it comes to continuous-time discrete-event predictors, far less has been done. Most continuous-time data is, in fact, discrete-tiie data with high time resolution; as such, one sentially sample continuous-time, discrete-event data at high resolution and use any of the previously mentioned methods for predicting discrete-time data. Alternatively, one can represent continuous- time, discrete-event data as list of dwell times and symbols and feed that data into either recurrent neural network or feedforward neural network. We take new approach: we infer continuous- hidden Markov models (Marzen & Crutchfield 2017) and predict using the model's internal state as useful predictive features.
BACKGROUND
We are given a sequence of symbols and durations of those symbols, ..., (I3; ... To This constitutes the data, D. For example, seismic time series are of this kind: magnitude and time between earthquakes. The last seen symbol zo has been seen for duration To Had we observed the system for longer amount of time, ग may increase. The possible symbols { h are assumed to belong to a finite set A, while the interevent intervals {T,}e are assumed to belong to (0, oo). We assume stationarity- that the statistics of { Tc),; are unchanging in time.
Above is the description the observed time series. What now follows is a shortened description of unifilar hidden semi-Marker models, notated M, that could be generating such time series (Mazen & Crutchfield, 2017). The minimal such model that is consistent with the observations the c-Machine. Underlying unifilar hidden semi-Markov model is finite-state machine with states g. each equipped with dwell-time distribution (T)( an emission probability p(ælg), and function + (g,xr) that specifies the next hidden state when given the current hidden state y and the current emission symbol z. This model generates a time series as follows: a hidden state randomly chosen; a dwell time T is chosen according to the dwell-time distribution (T):: an emission symbol is chosen according to the conditional probability p(rlg): and we then observe the chosen x for amount of time. new hidden state determined via c (g, 2r), and we further restrict possible next emissions to be different than the previous emission- property that makes this model unifilar- and the process repeats. See Fig. [ for illustrations of unifilar hidden semi-Markov model.
4 ALGORITHMS
We investigate three tasks: model inference; calculation of the differential entropy rate; and devel- opment of a predictor of future symbols. Our main claim is that restricted attention special type of discrete-event, continuous-ting model called unifilar hidden semi-Markov models makes all three tasks easier.
2

---

Under review as a conference paper at ICLR 2018
S
S B.
2
S
Sc
B,0
1
3
0
3.0
B.0
25-
2.0
Symbol
1.5
1.01
112
1/3 r c2-
0.5
100, d8..0
$11,7 08.11
20
40
60
80
100
Time
Figure 1: Illustrations of unifilar hidden semi-Markov models (uhsMm). At left, two presen- tations uhsMms. left bottom, a generative model for discrete-alphabet, continuous-time stochastic process. Dwell times are drawn upon transitions between states, and the corresponding symbol is shown for that amount of time. At top, the corresponding "conveyer belt" representation of the process generated by the model beneath. Conveyer belts represent the time since last sym- bol based on the height along the conveyer belt traveled; each conveyer belt has symbol. To right of the two presentations of a uhsMm, example time series generated from the model at left, where A ΦB, &c are inverse Gaussian distributions with (μ, A) pairs of (1, 2), (2, 3), (1,3), respectively.

---

Under review as a conference paper at ICLR 2018
4.1 INFERENCE OF UNIFILAR HIDDEN SEMI MARKOY PROCESSES
The unifilar hidden semi-Markov models described earlier can be parameterized. Let M refer to model- in this case, the underlying topology of the finite-state machine and neural networks defining the density of dwell times; let refer to the model's parameters, i.c. the emission prob- abilities and the parameters of the neural networks; and let D refer to the data, ie., the list of emitted symbols and dwell times. Ideally, to choose model, we would do maximum posteriori by calculating arg max. Pr(M(D) and choose parameters of that model via maximum likelihood, arg maxg Pr (r(100, M).
In the case of discrete-time unifilar hidden Markov models, Strelioff and Crutchfield (Strelioff & Crutchfield 2014) described the Bayesian framework for inferring the best-fit model and parameters. More than that, Ref. (Strelioff & Crutchfield 2014) calculated the posterior analytically, using the unifilarity property to case the mathematical burden. Analytic calculations in continuous-timg timm may be possible, but we leave that for a future endeavor. We instead turn to a variety of approximations, still aided by the unifilarity of the inferred models.
The main such approximation our use of the Bayesian inference criterion (BIC) Bishop 2006). Maximum posteriori is performed via
BIC = max log M) Km log [D]
(1)
2
M arg max BIC,
where KM is the number of parameters 0. To choose model, then, we must calculate not only the parameters f that maximize the log likelihood, but the log likelihood itself. We make one further approximation for tractability involving the start state Sp, for which
Pr(DIo, Σ M(sp)B).MMPr(((ssss 0, M.
(3)
As the logarithm of a sum has no easy expression, we approximate
max Pr(D max max log Pr(D|so, O,M).
(4)
Our strategy, then, is to choose the parameters 0 that maximize max.0 log Pr(D s0. 0, and to choose the model M that maximizes maxe log Pr(DIO, 2 log [D]. This constitutes an inference model that can explain the observed data.
ky
What remains to be done, therefore, is approximation of maxs, max, log Pr(D|so, M). The parameters 9 of any given model include p(s', æ[s), the probability of emitting æ when in state s and transitioning to state s', and (t), the interevent interval distribution of state Using the unifilarity of the underlying model, the sequence of x's when combined with the start state so translate into a show that
single possible sequence of hidden states si. As such,
log Pr(D) M.
ΣΣ log dh (T((0)
Σ
n(s', 2|s) (o'gp(s',213)
(5)
where (e) is any interevent interval produced when in state s. It is relatively easy to analytically maximize with respect to p(s', æ/s), including the constraint that E. p(s', x(s) 1 for any s, and find that
n(ss,z(a) nlss)
p" p"(s' (ss,
(6)
Now turn to approximation of the dwell-time distributions, (t). The dwell-time distribution can, in theory, be any normalized nonnegative function; inference may seem impossible. However, artificial neural networks can, with enough nodes, represent any continuous function. We therefore
represent o, (t) by a relatively shallow (here, three-layer) artificial neural network (ANN) in which nonnegativity and normalization are enforced as follows:
4

---

Under review as a conference paper at ICLR 2018
10a
N 500
ANN
1.50
N 5000
KNN
Tree
Parzen
1.25
10:1
MSE
0.75
0.50
10-1
0.00
10 10a
N
Figure 2: The estimated density function from varying numbers of samples. Shown, at left, the inferred density function using the neural network approach described here compared to the true density function (dotted, green) when given 500 samples (blue) and 5000 samples (orange). As the amount of data increases, the inferred density function becomes closer to ground truth. An interevent interval distribution with modes of two inverse Gaussians. Shown, at right, is the mean-squared error between the estimated density and the true density as we use more training data for three different estimation techniques. The blue line denotes the ANN algorithm introduced here, in which we learn densities from a neural network; the orange line denotes the k-nearest neighbors algorithm (Bishop 2006;; Fukunaga Hostetler, 1973}: and the green line uses Parzen window estimates (Bishop) 2006; Marron et al. 1987). Our new method is competitive with the two standard methods for density estimation.
arbitrarily chosen by setting ((T) to mixture
the second-to-last layer's activation functions are ReLus (max(0, æ)), and so with nonnega- tive output) and the weights to the last layer are constrained to be nonnegative;
and the output is the last layer's output divided by numerical integration of the last layer's output.
determines the cost function for the neural network. Then, the neural network can be trained using typical stochastic optimization methods. (Here, we use Adam Kingma & Ba (2014).) The output of the neural network can successfully estimate the interevent interval density function, given enough samples, within the interval for which there data. See Fig. 2 Outside this interval, however, the estimated density function is not guaranteed to vanish as t 0o, and can even grow. Stated differently, the neural networks considered here are good interpolators, but can be bad extrapolators. As such, the density function estimated by the network taken to be 0 outside the interval for which there is data.
The log likelihood E, log (T(TT) (T)
To the best of our knowledge, this is approach density estimation, referred to ANN here. A previous approach to density estimation using neural networks Iearned the cumulative distribu- tion function (Magdon-Ismail & Atiya 1999). Typical approaches density estimation include k-nearest neighbor estimation techniques and Parzen window estimates, both of which need careful tuning of hyperparameters (k /) (Bishop 2006). They are referred to here kNN and Parzen. We compare the ANN, KNN, and Parzen approaches in inferring an interevent interval density function that we have chosen, arbitrarily, be the mixture of inverse Gaussians shown in Fig. leftt) The in k--nerrstt neighbor estimation is chosen according to the criterion in Ref. Fukumaga & Hostetler [1973) and is chosen as maximize the pseudolikelihood Marron et al. (1987.. 7). Note that as shown in Fig. rright), this is not a superior approach to density estimation in terms of minimization of mean-squared error, but is parametric, so that BIC can used.
To test our new method for density estimation presented here- that is, by training properly nor- malized ANN- we generated a trajectory from the unifilar hidden semi-Markov model shown in Fig, lefft) and used BIC to infer the correct model. As BIC is log likelihood minus penalty for a larger number of parameters, a larger BIC suggests a higher posterior. With very little data, two-state model shown in Fig. is deemed most likely; but as the amount of data increases. the correct four-state model eventually takes precedence. See Fig. Bligght). The six stats model was never deemed more likely than two-state four-state model. Note that although this methodol-
5

---

Under review as a conference paper at ICLR 2018
A
111, - da
10a
100,2 da
BIC
Il ~ dp
1|1.1 0g
102
10P
10a
10a
100,7 ~ de
data points
Figure 3: At left, the two-state model (top) and four-state uhsMm (bottom) for binary-alphabet, continuous-time data. At right, BIC as a function of the amount of data for the two-state, four-state, and stx-state uhsMms left. (The six-state uhsMm not shown.) Larger BIC implies higher posterior, and so a better fit.
ogy might be extended to nonunifilar hidden semi- Markov models, the unifilarity allowed for easily computable and unique identification dwell times to states in Eq. 5
4.2 IMPROVED CALCULATION OF DIFFERENTIAL ENTROPY RATES
One benefit of unifilar hidden Mariov models that one can use them obtain explicit formu- lae for the differential entropy rate (Marzen & Crutchfield, 2017). Such entropy rates are measure of the inherent randomness of a process (Crutchfield Feldman 2003), and many have tried find better algorithms for calculating entropy rates of complex processes (Egner et al., [0044] Arnold & Loeliger 2001 Nemenman et al., [2002] |Archer et al. 20.4). Setting aside the issue for now of why one would want to estimate the entropy rate, we simply ask how well one can estimate the entropy rate from finite data.
Differential entropy rates difficult to calculate directly from data, since the usual program in- volves calculating the entropy of trajectories of some length and dividing by T:
H [x T)
h1
lim T-+00
T
A better estimator, though, is the following (Crutchfield & Feldman 2003
(H,,))
hy lim T-
dT
T
the slope of the graph of H( dimension, this entropy is seemingly difficult to estimate from data. To calculate [[ 7) X7 ], we use
vs. T. As the entropy of mixed random variable of unknown
a trick like that of Ref. Victor (002) and condition on the number of events N:
H H( H + (z, [N].
(9)
We then further break the entropy into its discrete and continuous components:
(x, N = = [zonn /N n) + H (T 1 N = n]
(10)
and use the d-ncarest-neighhor entropy estimator (Kraskov et al. 2004) to estimate H [To: [a N = n). with chosen to be 3. We estimate both 20 N n and H us- ing plug-in entropy estimators, as the state space is relatively well-sampled. We call this estimator model-free, in that we need not infer model to calculate the estimate.
We introduce a model-based estimator, for which we infer a model and then use the inferred model's differential entropy rate the differential entropy rate estimate. To calculate the differential entropy

---

Under review as a conference paper at ICLR 2018
rate from the inferred model, use plug-in in estimator based on the formula in Ref. (Marzen & Crutchfield 2
h4 = -==--p(s) I'' (aoo(t) log (t)dt,
(11)
where the sum is over internal states of the model. The parameter pa is merely the mean interevent interval out of state Io to. (t) We find the distribution over internal states p(s), by solving the linear equations (Marzen Crutchfield 2017)
p(s) Σ Me' -sd- p(s').
(12)
Hs ni
We use the MAP estimate of the model described previously and estimate the interevent interval density functions (,(() using a Parzen window estimate, with smoothing parameter / chosen so as maximize the pseudolikelihood (Marron et al., 1987), given that those proved to have lower mean-squared сπтоr than the neural network density estimation technique in the previous subsection. In other words, we use neural network density estimation technique choose the model, but once the model is chosen, we use the Parzen window estimates to estimate the density for purposes of estimating entropy rate. A full mathematical analysis of the bias and variance is beyond the scope of this paper.
Fig. 4 shows a comparison between the model-free method (k-nearest neighbor estimator of en- tropy) and the model-based method (estimation using the inferred model and Eq. [T] as a function of the length of trajectories simulated for the model. In Fig. Top). the most likely (two-state) model is used for the model-basce plug-in estimator of Eq. the previous subsection; but in Fig. (/Bottom), the correct four-state model used for the plug-in es- timator. Hence, the estimate in Eq. iiss based the wrong model, and hence, leads to systematic overestimate of the entropy rate. When the correct four-state model is used for the plug-in estimator in Fig. 4 Bottom), the model-based estimator has much lower variance than the model-free method.
ascertained by the procedure in
To efficiently estimate the excess entropy (Crutchfield & Feldman 2003 Bialek et al.) 2001b a) an additional important informational measure, requires models of the time-reversed process. Fu- ture research will elucidate the needed retrodictive representations of unifilar hidden semi-Markov models, which can be determined from the 'fowward" unifilar hidden semi-Markov models.
4.3 IMPROVED PREDICTORS USING THE INFERRED CAUSAL STATES
There are wide array of techniques developed for discrete-time prediction, described carlier in the manuscript. We can develop continuous-timg -immuuessttemeee-ee techniques that build on these discrete-time techniques, e.g. by using dwell times and symbols inputs to RNN. However, based the experiments shown here, we seem to gain by first identifying continuous-tin time causal states.
The first prediction method call "predictive ANN" PANN (with risk of confusion with the ANN method for density estimation described earlier) takes, as input, (I- T=++) (Io, To" into feedforward neural network that is relatively shallow (six layers) and somewhat thin (25 nodes). Other network architectures were tried with little improvement. The weights of the network are trained to predict the emitted value æ at time T' later based on a mean-squared error loss function. For this method to work, the neural network must guess the hidden state g from the observed data, which can be done if the dwell-time distributions of the various states are dissimilar. Increases in can increase the ability of the neural network to correctly guess its hidden state and thus predict future symbols, assuming enough data to avoid overfitting; here, n is chosen via cross-validation.
The second of these methods that we will label as "RNN" will take (x T ...................... (ro, To- as input to an L.TM, though any RNN could have been chosen. The LSTM is asked to produce an estimate of at time T subject to mean-squared error loss function, similar to the first prediction method.
The third of these methods that we will label "uhsMm" preprocesses the input data using an inferred unifilar hidden semi-Markov model so that each time step is associated with a hidden state g, time since last symbol change 7 and current emitted symbol øg. In discrete- time applications, there is an explicit formula for the optimal predictor in terms of the e-M; but for continuous time applications, there is such formula, and so we use k-nearest neighbor estimate. More precisely,
7

---

Under review as a conference paper at ICLR 2018
8
Differential entropy rate
6
4
2-
0-
1000
2000
3000
4000
5000
N
Figure 4: Entropy rate estimation, showing model-free vs. model-based estimators. The synthetic dataset is generated from Fig, 3 topp with (t) = @D(t) as inverse Gaussians with mean 1 and scale and with @ = &c (T) as inverse Gaussians with mean and scale The ground truth entropy rate from the formula in (Marzen & Crutchfield 2017) is 1.85 nats, shown in green. In orange, the model-free estimator (combination of plug-in in entropy estimator and kNN (Kraskov et al.) 2004) entropy estimators) described in the text. In blue, the model- estimator assuming a two- state model, i.e., the top left of Fig. 3 In black, the model-based estimator assuming four-state model, i.e., the bottom left of Fig. Lines denote the mean, and various data point denote the estimated entropy rate for different data sets of a particular size N. The model-free method has much higher variance than the model-based methods, and the model-based method for which the correct (four-state) model is used also has lower bias.
8

---

Under review as a conference paper at ICLR 2018
0.3
uhsmm
uhsmm
ANN
ANN
0.4
0.4
RNN
(MSE)
0.3
MSB
0.3
0.2
0.1-
0.1
0.0
2.5 0.0
1.0
10a
10-1
10a
T
T
Figure 5: At left, mean-squared error of the predictor of the symbol at time T from data prior to time for when 500 data points are available and 300 epochs are used to train the ANN; right, the mean-squared enror of the predictor for when 5000 data points available and 3000 epochs are used to train the ANN. The generating uhs.m in Fig. Blefft). The uhsMm method infers the internal state of the unfilar hidden semi-Markov model; the PANN method uses the last data points (Ia, Ti) as input into a feedforward neural network; and the RNN method uses the past (#i, Ti) as input to LSTM.
we find the l closest data points in the training data to the data point under consideration, and estimate æT as the average of the future data points in the training set. In the limit of infinite data so that the correct model is identified, for correctly-chseen k, this method will output an optimal predictor; we choose & via vross-validation.
The synthetic dataset is generated from Fig. topp) with @a(() (D(() inverse Gaussians with mean and scale 5 and with (t) (t) as inverse Gaussians with mean 3 and scale 2. We chose these means and scales so that it would be easier, in principle, for the non-uhsMm methods (i.e., PANN and RNN) to implicitly infer the hidden state (A, B, C, and D). Given the difference in dwell time distributions for each of the hidden states, such implicit inference is necessary for accurate predictions. From experiments, shown Fig. [5] the feedforward neural network and the recurrent neural network are typically outperformed by the uhsMm method. The corresponding mean-squared errors for the three methods are shown in Fig. Bjbottom) for two different dataset sizes. Different network architectures, learning rates, and number of epochs were tried; the results shown in Fig. 5 are typical. Using k-nearest neighbor estimate on the causal states (i.e., the internal state of the uhsMm) predict the future symbol requires little hyperparameter tuning and outperforms compute- intensivnntessive feedforward and recurrent neural network approaches.
5 DISCUSSION
We have introduced new algorithm for inferring causal states (Shalizi & Crutchfield 2001 of a continuous-timm discrete-event process using the groundwork of Ref. (Marzen & Crutchfeld 2017). We have introduced new estimator of entropy rate that uses the causal states. And finally, we have shown that a predictor based on causal states more accurate and less compute-heavy than other predictors.
The new inference, estimation, and prediction algorithms could be used to infer a predictive model of complex continuous-time. discrete-event processes, such as animal behavior, and calculate estimates of the intrinsic randomness of such complex processes. Future research could delve into improving estimators of other time series information measures (James al. 2011), using something more accurate than BIC to calculate MAP models, or enumerating the topology of all possible uhsMm models for non-binary alphabets (Johnson al.).

---

Under review as a conference paper at ICLR 2018
ACKNOWLEDGMENTS
This material based upon work supported by, in part by, the U. S. Army Research Laboratory and the U. S. Army Research Office under contracts W911NF-13-1-0390 and I9- and the Moore Foundation.
REFERENCES
Evan Archer, II Memming Park, and Jonathan W Pillow. Bayesian entropy estimation for countable discrete distributions. The Journal of Machine Learning Research, 15(1):2833-2868. 2014.
Dieter Arnold and H-A Loeliger. On the information rate of binary-input channels with memory. In ICC 2001. IEEE International Conference on Communications. Conference Record (Cat. No. 01Ch37240), volume 9, pp.
IEEE, 2001.
Gordon Berman, William Bialek, and Joshua W Shaevitz. Predictability and hierarchy in drosophila behavior. Proceedings of the National Academy of Sciences, 113(42): 2016.
W. Bialek, Nemenman, and N. Tishby, Neural Comp., 13:2409-2463. 2001a.
W. Bialek, Nemenman, and N. Tishby, Complexity through nonextensivity. Physica A, 302:899-99 2001b. Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Andrea Cavagna, Irene Giardina, Francesco Ginelli, Thierry Mora, Duccio Piovani, Raffaele Tavarone, and Aleksandra M Walczak. Dynamical maximum entropy approach to flocking. Phys- ical Review E, 8944)0042777 2014.
Crutchfield and D. P. Feldman. CHAOS, 13(1):25-54, 2003.
Sebastian Egner, Vladimir B Balakirsky, Ludo Tolhuizen, Stan Baggen, and Henk Hollmann. On the entropy rate hidden markov model. In International Symposium oninformation Theory, 2004. ISIT 2004. Proceedings. pp. 12. IEEE, 2004.
Keinosuke Fukunaga and L Hostetler.
Optimization of k nearest neighbor density estimates. 1EEE Transactions on Information Theory, 1203):320-326, 1973.
Robert J Geller. Earthquake prediction: a critical review. Geophysical Journal International, 131 (3):425-450 1997.
Lyudmila Grigoryeva and Juan-Pa -ablo Ortega. Echo state networks are universal. Neural Networks, 108:495-508, 2018.
Kurt Homikk. Approximation capabilities of multilayer feedforward networks. Neural networks, (2):251-257, 1991.
R. G. James, C. J. Ellison, and J. P. Crutchfield. Anatomy of a bit: Information in time series observation. CHAOS, 21(3):037109 109, 2011. doi: 10.1063/1.3637494, URL htttt:/araaa
org/abs/ /at...115... 2988
B. D. Johnson, J. P. Crutchfield, C. Ellison, and C. S. McTagge. arxiv.org: 1011 1.0036.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rXiv:1412.6980) 2014.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E. 69(6))066133 2004.
Michael L Littman and Richard S Sutton. Predictive representations of state. Advances in neural information processing systems, pp. 1555-1561, 2002.
Henrik Madsen. Time series analysis. Chapman and Hall/CRC, 2007.
10

---

Under review as a conference paper at ICLR 2018
Malik Magdon-Ismail and Amir F Atiya. Neural networks for density estimation. In Advances in Neural Information Processing Systems, pp. 522-528, 1999.
JS Marron et al. A comparison of cross-validation techniques in density estimation. The Annals of Statistics, 15())152-112, 1987.
Sarah E Marzen and James Crutchfield. Structure and randomness of continuous-time, discrete- event processes. Journal of Statistical Physics, 169(2):303-3155 2017.
Ilya Nemenman, Fariel Shafee, and William Bialek. Entropy and inference, revisited. In Advances in neural information processing systems, pp. 471-478, 2002.
D. Pfau, N. Bartlett, and F. Wood. Probabilistic deterministic infinite automata. Neural Information Processing Systems, 23:1930-1938, 2010.
Rabiner and Juang. IEEE ASSP Magazine, January, 1986.
Rieke, Warland, R. de Ruyter van Steveninck, and Bialek. Spikes: Exploring the Neural Code. Bradford Book, New York, 1999.
C. R. Shalizi and J. P. Crutchfield. J. Stat. Phys., 104:817-879, 2001.
C. C. Strelioff and J. P. Crutchfield. Bayesian structural inference for hidden processes. Phys. Rev. E, 89:042119, Apr 2014. doi: 10. S...PP/yy.......40121111 URL lttt:://iikk.aa:::ttt org/
Bo1/10.1103/PaysRee
042119
Jonathan Victor. Binless strategies for estimation of information from neural data. Physical Review E, 66(5):051903, 2002.
Paul Werbos et al. Backpropagation through time: what it does and how to do it. Proceedings of the JEEE, 78(10) 1990.


Reviews:
[{"review_id": "SJl91FP2tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a model for time series which are represented as discrete events in continuous time and describe methods for doing parameter inference, future event prediction and entropy rate estimation for such processes. Their model is based on models for Bayesian Structure prediction where they add the temporal dimension in a rigorous way while solving several technical challenges in the interim. The writing style is lucid and the illustrations are helpful and high quality. \n\nHowever, the paper and the technical challenges which the authors faced have been extensively studied in the closely related field of (marked) temporal point process modeling, which also models processes with discrete events in continuous time. Hence, the claim \"When it comes to continuous time, discrete-event predictors, far less has been done\" is mildly erroneous. The literature in that field has provided several ways of handling the technical challenges which authors have tried to overcome. For example, excellent predictor for the next event as well as time has been proposed by Du. et. al. (2016), Mei and Eisner (2017) and, very recently, by Türkmen et. al (2019), among many many others. Also, probability model (i.e. \\phi_s(t)) of the next event time can be represented using intensity functions \\lambda(t). However, if the authors require an explicit likelihood function, Normalized Flows with re-parametrization, can provide that too (see Kobyzev (2019) for a review of recent work in the area). This while being a Neural Network based approach well-weathered models and does not require numerical integration. Without placing this work in context of these works, it is very difficult to judge the contributions of the paper well.\n\nThen there are a few unanswered questions about the model:\n\n 1. In the Background, where the authors describe the generative process, does the emitted symbol also depend on the dwell time \\tau (like Mei and Eisner, 2017), or is the distribution of the dwell time and the next symbol independent given the state (like Du et. al)?\n 2. Similarly, during entropy calculation, while breaking the entropy term from (9) to (10), shouldn't there be a dependence of the discrete symbols on the continuous dwell times of the prior step?\n 3. We need significantly more information about the training used for the LSTM/RNN models to be able to judge whether the performance of the methods is comparable or not. These could be provided in a supplementary material, if they do not fit within the main text.\n\n\nSome other ways of improving the paper:\n - The reference style changes frequently in the paper and is mutually inconsistent.\n - The \\pi in Eqn. (3) is not formally defined. If it is the initial state probability, then should it be dependent on the model \\Mcal and parameters \\theta?\n - The illustrations, though useful, are too large and use up space.\n - Though the task of determining the entropy is indeed intellectually satisfying, it would very much help motivate the reader if there were some applications which the authors could allude to.\n\n\nCitations:\n - Aalen, Odd, Ornulf Borgan, and Hakon Gjessing. Survival and event history analysis: a process point of view. Springer Science & Business Media, 2008.\n - Du, Nan, et al. \"Recurrent marked temporal point processes: Embedding event history to vector.\" Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.\n - Mei, Hongyuan, and Jason M. Eisner. \"The neural hawkes process: A neurally self-modulating multivariate point process.\" Advances in Neural Information Processing Systems. 2017.\n - Türkmen, Ali Caner, Yuyang Wang, and Alexander J. Smola. \"FastPoint: Scalable Deep Point Processes.\"\n - Kobyzev, Ivan, Simon Prince, and Marcus A. Brubaker. \"Normalizing flows: Introduction and ideas.\" arXiv preprint arXiv:1908.09257 (2019).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xR2dmTFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a model for continuous-time, discrete events prediction and entropy rate estimation by combining unifilar hidden semi-Markov model and neural networks where the dwell time distribution is represented by a shallow neural network. \n\nComments:\nThe literature review on previous work for continuous-time, discrete events prediction is not thorough enough. For this problem, there are continuous-time Markov networks [El-Hay et al. 2006], continuous-time Bayesian networks [Nodelman et al. 2002] and its counterpart in relational learning domain, i.e. relational continuous-time Bayesian networks [Yang et al. 2016]. The authors should have learned their work and addressed the difference between the proposed model and these work in the related work section.\n\nTal El-Hay, Nir Friedman, Daphne Koller, and Raz Kupferman. Continuous Time Markov Networks. In UAI, 2006.\nNodelman, U.; Shelton, C.; and Koller, D. Continuous Time Bayesian Networks. In Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI), pages 378–387, 2002.\nShuo Yang, Tushar Khot, Kristian Kersting, and Sriraam Natarajan. Learning continuous-time Bayesian networks in relational domains: A non-parametric approach. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2265–2271, 2016. \n\nThe approximations made to get equation (3) and equation (4) lacks theoretical proof for the up-bound of its influence on the model performance. If these assumptions have been made before, please cite the references; if not, please illustrate why it makes sense and how the approximation will influence the value of the objective function.\n\nPlease explicitly state the meaning of all the symbols used in the paper even it can be inferred by the readers. E.g. ‘n’, which first appears in Equation (5) and is never being explained.\n\nThe experiments are rather simple both in terms of the model used to generate the data and the number of different data sets being used. Hence, the experimental results are not strong enough to support the claims made by this paper. \nSpecifically, it claims “With very little data, a two-state model shown in Fig. 3 is deemed most likely; but as the amount of data increases, the correct four-state model eventually takes precedence”, but in Figure 3, the plot with the highest BIC score when the training sample is less is the green curve which stands for the six-state model according to the legend. \n“The corresponding mean-squared errors for the three methods are shown in Fig. 3(bottom) for two different dataset sizes.” I could not find it in Figure 3.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ljx_SEcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the problem of modeling, predicting and estimating entropy information over continuous-time discrete event processes. Specifically, the paper leverages unifilar HSMM's for model inference and then uses the inferred states to make future predictions. The authors also use inferred model with previously developed techniques for estimating entropy rate. The authors describe the methods and provide the evidence of the effectiveness of their method with experiments on a synthetic dataset.\n\nThe paper should be rejected as it doesn't appear to be fit for this conference. While the paper does build techniques that leverages classical statistical learning and combine them with deep learning, it falls short on various aspects. Specifically, the paper neither provides any rigorous analysis of the developed methods so as to be useful to theoretical learning community nor does it support its approaches with strong empirical evidence so as to be of practical importance. \n\n- While the authors cite lack of work in such data setting (which is actually incorrect as described below), they do not motivate or justify why such an approach of using either unifilar HSMM or Bayesian inference techniques are good starting point to model such information. The authors claim that unifilar HSMM is more of a restriction to make the three proposed tasks easier. But such a model assumption is very strict and will likely lead to large model mis-specification issues when used in real-world. The author does not describe what can be done in this scenario.\n\n- Page2, Section 3: the statistics of {(x_i, t_i)} are unchanging in time. What does this mean?\n- As stated by authors, to make the approach tractable, they have to adopt a slew of approximations which seems to put severe restriction on what kind of data can be modeled and inferred on using these approaches.\n- It is not clear why one wants to estimate the entropy rate in such settings and authors avoid discussing it in Section 4.2\nsimilarly the authors explicitly avoid discussing bias-variance tradeoff of such method, however, this makes the paper\ndevoid of any insights into the proposed approach.\n- How was k chosen to be 3 in Eq 10? \n- Another major miss for the paper is that there has been lot of work at the intersection of stochastic processes (e.g. temporal point processes) and machine learning techniques that aim to model exactly the continuous-time discrete event information and the paper does neither cites any of them nor discusses them or compare with them. We encourage the authors to look at  the relevant literature and position their work in comparison to these existing approaches. I provide a few references here as starting point, however, lot of follow-up work are available:\n\nPoint Processes and Machine Learning:\n------------------------------------\nModeling the Dynamics of Learning Activity on the Web\nSmart Broadcasting: Do You Want to be Seen?, Karimi et. al.\nUncovering the Temporal Dynamics of Diffusion Networks, Gomez-Rodriguez et. al.\n\nHidden semi-markov models:\n-------------------------\nRecurrent Hidden Semi-Markov Model, Dai et. al. \n\nDeep learning and temporal modeling:\n-----------------------------------\nRecurrent Marked Temporal Point Processes: Embedding Event History to Vector, Du et. al.\nThe Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process, Mei et. al.\n\n- Finally, the authors show only one synthetic experiment. This is far from providing convincing evidence of the efficacy of the method and certainly needs a lot more work for acceptance in any machine learning conference.  The authors, at the least, must use their approaches on couple of real-world dataset (e.g. earthquake information, animal behavior as they mention or others) and also compare with the above stochastic processes techniques to test how their methods fare. \n\n- Also, the analysis of efficiency of their method is required. It appears that their methods may not be scalable for instance to a multi-dimensional discrete label case.\n\n- Figure 4 is not correctly referenced in the paragraph right above Section 4.3. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a Bayesian model for time series which are represented as discrete events in continuous time and describe methods for doing parameter inference, future event prediction and entropy rate estimation for such processes. 
However, the reviewers find that the novelty of the paper is not high enough, and without sufficient acknowledgement and comparison to existing literature.
Decision: Reject