Informed Temporal Modeling via Logical Specification of Factorial LSTMs

Consider a world in which events occur that involve various entities . Learning how to predict future events from patterns of past events becomes more difficult as we consider more types of events . Many of the patterns detected in the dataset by an ordinary LSTM will be spurious since the number of potential pairwise correlations , for example , grows quadratically with the number of events . We propose a type of factorial LSTM architecture where different blocks of LSTM cells are responsible for capturing different aspects of the world state . We use Datalog rules to specify how to derive the LSTM structure from a database of facts about the entities in the world . This is analogous to how a probabilistic relational model ( Getoor & Taskar , 2007 ) specifies a recipe for deriving a graphical model structure from a database . In both cases , the goal is to obtain useful inductive biases by encoding informed independence assumptions into the model . We specifically consider the neural Hawkes process , which uses an LSTM to modulate the rate of instantaneous events in continuous time . In both synthetic and real-world domains , we show that we obtain better generalization by using appropriate factorial designs specified by simple Datalog programs . 1 INTRODUCTION . Temporal sequence data is abundant in applied machine learning . A common task is to impute missing events , e.g. , to predict the future from the past . Often this is done by fitting a generative probability model . For evenly spaced sequences , historically popular models have included hidden Markov models and discrete-time linear dynamical systems , with more recent interest in recurrent neural network models such as LSTMs . For irregularly spaced sequences , a good starting point is the Hawkes process , a self-exciting temporal point process ; many variations and enhancements have been published , including neural variants using LSTMs . All of these models can be described schematically by Figure 1a . Events ei , ei+1 , . . . are assumed to be conditionally independent of previous events , given the system state si ( which may or may not be fully known given events e1 , . . . , ei ) . That is , si is enough to determine the joint distribution of the ith event and the updated state si+1 , which is needed to recursively predict all subsequent events . Figure 1a and its caption show the three types of influence in the model . The update , affect , and depend arrows are characterized by parameters of the model . In the case of a recurrent neural network , these are the transition , input , and output matrices . Our main idea in this paper is to inject structural zeros into these weight matrices . Structural zeros are weights that are fixed at zero regardless of the model parameters . In other words , we will remove many connections ( synapses ) from both the recurrent and non-recurrent portions of the neural network . Parameter estimation must use the sparse remaining connections to explain the observed data . Specifically , we partition the neural state si ∈ Rd into a number of node blocks . Different node blocks are intended to capture different aspects of the world ’ s state at step i . By zeroing out rectangular blocks of the weight matrix , we will restrict how these node blocks interact with the events and with one another . An example is depicted in Figures 1b ( affect , depend ) and 1d ( update ) . In addition , by reusing nonzero blocks within a weight matrix , we can stipulate ( for example ) that event e affects node block b in the same way in which event e′ affects node block b′ . Such parameter tying makes it possible to generalize from frequent events to rare events of the same type . Although our present experiments are small , we are motivated by the challenges of scale . Real-world domains may have millions of event types , including many rare types . To model organizational behavior , we might consider a dataset of meetings and emails in a large organization . To model supply chains , we might consider purchases of goods and services around the world . In an unrestricted model , anything in the past could potentially influence anything in the future , making estimation extremely difficult . Structural zeroes and parameter tying , if chosen carefully , should help us avoid overfitting to coincidental patterns in the data . Analogous architectures have been proposed in the world of graphical models and causal models . Indeed , to write down such a model is to explicitly allow specific direct interactions and forbid the rest . For example , the edges of a Gaussian graphical model explicitly indicate which blocks of the inverse covariance matrix are allowed to be nonzero . Some such models reuse blocks ( Hojsgaard & Lauritzen , 2008 ) . As another example , a factorial HMM ( Ghahramani & Jordan , 1997 ) —an HMM whose states are m-tuples—can be regarded as a simple example of our architecture . The state si can be represented using m node blocks , each of which is a 1-hot vector that encodes the value of a different tuple element . The key aspect of a factorial HMM is that the stochastic transition matrix ( update in Figure 1d ) is fully block-diagonal . The affect matrix is 0 , since the HMM graphical model does not feed the output back into the next state ; the depend matrix is unrestricted . But how do we know which interactions to allow and which to forbid ? This is a domain-specific modeling question . In general , we would like to exploit the observation that events are structured objects with participants ( which is why the number of possible event types is often large ) . For example , a travel event involves both a person and a place . We might assume that the probability that Alice travels to Chicago depends only on Alice ’ s state , the states of Alice ’ s family members , and even the state of affairs in Chicago . Given that modeling assumption , parameter estimation can not try to derive this probability ( presumably incorrectly ) from the state of the coal market . These kinds of systematic dependencies can be elegantly written down using Datalog rules , as we will show . Datalog rules can refer to database facts , such as the fact that Alice is a person and that she is related to other people . Given these facts , we use Datalog rules to automatically generate the set of possible events and node blocks , and the ways in which they influence one another . Datalog makes it easy to give structured names to the events and node blocks . The rules can inspect these structures via pattern-matching . In short , our contribution is to show how to use a Datalog program to systematically derive a constrained neural architecture from a database . Datalog is a blend of logic and databases , both of which have previously been used in various formalisms for deriving a graphical model architecture from a database ( Getoor & Taskar , 2007 ) . 2 PRELIMINARIES1 . Our methods could be applied to RNN sequence models . In this setting , each possible event type would derive its unnormalized probability from selected node blocks of state si . Normalizing these probabilities to sum to 1 would yield the model ’ s distribution for event ei . Only the normalizing constant would depend on all node blocks . In this paper , we focus on the even more natural setting of real-time events . Here no normalizing constant is needed : the events are not in competition . As we will see in section 5.1 , it is now even possible for different node blocks to generate completely independent sequences of timestamped events . The observed dataset is formed by taking the union of these sequences . In the real-time setting , event ei has the form ki @ ti where ki ∈ K is the type of the event and ti ∈ R is its time . The probability of an event of type k at any specific instant t is infinitesimal . We will model how this infinitesimal probability depends on selected node blocks of si . There is no danger that two events will ever occur at the same instant , i.e. , the probability of this is 0 . 1Our conventions of mathematical notation mainly follow those given by Mei & Eisner ( 2017 , section 2 ) . We begin by describing our baseline model for this setting , drawn from Mei & Eisner ( 2017 ) . 2.1 BASELINE MODEL : THE NEURAL HAWKES PROCESS . In general , a multivariate point process is a distribution over possible sequences of events e1 = k1 @ t1 , e2 = k2 @ t2 , . . . where 0 < t1 < t2 < . . .. A common paradigm for defining such processes , starting with Hawkes ( 1971 ) , is to describe their temporal evolution as in Figure 1a . Each si is deterministically computed from si−1 ( update ) and ei−1 ( affect ) , according to some formula , so by induction , si is a deterministic summary of the first i − 1 events . ei = ki @ ti is then emitted stochastically from some distribution parameterized by si ( depend ) . The structure of the depend distribution is the interesting part . si is used , for each event type k ∈ K , to define some time-varying intensity function λk : ( ti−1 , ∞ ) → R≥0 . This intensity function is treated as the parameter of an inhomogeneous Poisson process , which stochastically generates a set of future events of type k at various times in ( ti−1 , ∞ ) .2 Thus , all these |K| Poisson processes together give us many events of the form e = k @ t. The first such event—the one with the earliest time t—is taken to be the next event ei . The remaining events are discarded ( or in practice , never generated ) . As our baseline method , we take the neural Hawkes process ( Mei & Eisner , 2017 ) to be our method for computing si and defining the intensity function λk from it . In that work , si actually describes a parametric function of the form h : ( ti−1 , ∞ ) → Rd , which describes how the hidden state of the system evolves following event ei−1 . That function is used to define the intensity functions via λk ( t ) = fk ( v > k h ( t ) ) > 0 ( 1 ) 2Under an inhomogenous Poisson process , disjoint intervals generate events independently , and the number of events on the interval ( a , b ] is Poisson-distributed with mean ∫ b a λk ( t ) dt . Thus , on a sufficiently narrow interval ( t , t+ dt ] , the probability of a single event is approximately λk ( t ) dt and the probability of more than one event is approximately 0 , with an error of O ( dt2 ) in both cases . so the parameters of depend are the vectors vk and the monotonic functions fk . Once ei = ki @ ti has been sampled , the parameters for si+1 are obtained by si+1 ← Ψ ( Uwki + Vh ( ti ) ) ( 2 ) where Ψ is inspired by the structure of an LSTM , the affect parameters are given by matrix U and the event embeddings wk , and the depend parameters are given by matrix V. In this paper , we will show an advantage to introducing structural zeroes into vk , U , and V .

Summary: This paper builds an interesting connection between Datalog rules and temporal point processes. The novelty of the approach is to factorize the latent state of LSTM into different blocks that represent three major interactions between temporal events, including: dependency, affects, and updates. The design of the node blocks within the hidden state allows the modeling of fine-grain structure of a given event type. Based on the Datalog program and the logic rules, the intensity function of the temporal point process can be formulated from facts in a database. The problem of enabling a flexible family of intensity functions is one of the most important topics in point processes, and a paper advancing knowledge in this area is certainly welcome.