Transformer Embeddings of Irregularly Spaced Events and Their Participants

1 INTRODUCTION . Event sequences are ubiquitous in real-world applications , such as healthcare , finance , education , commerce , gaming , audio , news , security , and social media . We propose an elegant way to embed events into a vector space RD . An actual or possible event at time t ∈ R is contextually embedded using attention over actual events at other times , in a way that considers their timing . Good event embeddings could be used in a variety of downstream tasks , similar to word token embeddings in BERT . In particular , we consider the task of predicting the next event ( generative modeling ) : at each time t ∈ R , all of the possible events are embedded using masked attention over the preceding actual events , and those embeddings yield the instantaneous probabilities at time t of the possible events . Our approach is a continuous-time generalization of the Transformer ( Vaswani et al. , 2017 ) . Transformers scale up well to very large datasets and have achieved astonishing success at generative modeling of discrete-time sequences , such as natural-language documents ( Radford et al. , 2019 ; Brown et al. , 2020 ) and proteins ( Rao et al. , 2021 ) . We show in this paper that they are also effective for irregularly spaced continuous-time sequences , even in lower-data regimes . We further show how to derive models from arbitrary NDTT programs . NDTT , or neural Datalog through time ( Mei et al. , 2020a ) , is a neural-symbolic framework for modeling event sequences . An NDTT program specifies logical rules that derive facts within a deductive database . Moreover , the facts in the database are embedded using a deep recurrent neural architecture that is automatically determined by the facts ’ provenance . Among the derived facts at time t are the possible events at time t. As a simple NDTT example , if u is a user and d is a document and u has appropriate permissions , then u may open d. The embedding and instantaneous probability of this possible event at time t depend on the embeddings at time t of u and d ( or more precisely , of the facts that u is a user and d is a document ) . If the event actually occurs , it may change what facts are true at times > t and how their embeddings evolve over times > t. For instance , the NDTT program can state that if u opens d , this makes editing events possible , and also affects the future embeddings of u and d. These updated embeddings feed forward to help predict future events involving u and/or d. ( They could also be exploited for other downstream predictive tasks , via multitask training or pretraining . ) All of these effects are controlled by the hand-crafted rules and learned parameters of the NDTT program , event ’ s embedding at layer ` depends ( ) on all preceding events at layer ` − 1 . ( The dashed arrows reflect residual connections as well as the fact that each event or fact also attends to itself . ) Section 4 explains that the ee ff influences can be prevented by dropping the rule ff < - ee . Figure ( b ) shows an A-NDTT model from section 5 , in which the company forum ’ s embedding at layer ` depends ( ) on all preceding events at layer ` − 1 ( via < - rules ) . The events or possible events at layer ` do not depend directly on preceding events ; instead , their embeddings at time t are derived ( ) from the forum ’ s embedding at time t ( via : - rules ) . which are analogous to the hand-crafted structure and learned parameters of a graphical model , in that they enforce domain-appropriate conditional independence and parameter sharing . We modify the NDTT architectures to use attention . In our architecture , a fact that can be derived in multiple ways , by different NDTT rules , uses multiple attention heads to examine the past . Du et al . ( 2016 ) ; Mei & Eisner ( 2017 ) ; Mei et al . ( 2020a ) instead summarize the past using recurrent neural networks ( RNNs ) . Our attentive approach has three advantages over theirs : ¬ A Transformer does not summarize the past . Our predictions at time t can examine an unboundedly large representation of the past ( Rd embeddings of every event before t ) , not merely a fixeddimensional summary that was computed greedily from left to right ( an RNN ’ s state at time t ) . A Transformer ’ s computation graph is broader and shallower . The breadth makes it easier to learn long-distance influences . The shallowness does make it impossible to represent inherently deep concepts such as parity , but it enables greater parallelism : the layer- ` embeddings can be computed in parallel during training , as they depend only on layer ` − 1 and not on one another . ® Our architecture is simpler and arguably more natural , while remaining competitive in our experiments . To describe how embeddings vary continuously during the period ( ti , ti+1 ] between two events , Mei & Eisner ( 2017 ) had to stipulate an arbitrary family of parametric decay functions on t , and Mei et al . ( 2020a ) had to design a complex method for pooling the parameters of these decay functions across multiple NDTT rules . In contrast , our embeddings are constructed “ from scratch ” at each time t simply by attending to the set of previous events , using t-specific query vectors that include a continuous positional embedding of t. As t ranges over ( ti , ti+1 ] , the attention weights over the first i events vary continuously with t , so the embeddings do so too . 2 CONTINUOUS-TIME TRANSFORMER FOR EMBEDDING EVENTS . Suppose we observe I events over a fixed time interval [ 0 , T ) . Each event is denoted mnemonically as e @ t ( i.e. , “ type e at time t ” ) , where e ∈ { 1 , 2 , . . . , E } is the event ’ s type . The observed event sequence is e1 @ t1 , e2 @ t2 , . . . , eI @ tI , where 0 < t1 < t2 < . . . < tI < T . For any observed or possible event e @ t , we can compute an embedding JeK ( t ) ∈ RD by attending to a set H ( e @ t ) of relevant events . ( For the moment , imagine that H ( e @ t ) consists of all the observed events ei @ ti . ) More precisely , JeK ( t ) is the concatenation of layer-wise embeddings JeK ( 0 ) ( t ) , JeK ( 1 ) ( t ) , . . . , JeK ( L ) ( t ) . For ` > 0 , the layer- ` embedding of e @ t is computed as JeK ( ` ) ( t ) def= JeK ( ` −1 ) ( t ) ︸ ︷︷ ︸ residual connection + tanh ∑ f @ s∈H ( e @ t ) v ( ` ) ( f @ s ) α ( ` ) ( f @ s , e @ t ) ∑ f @ s∈H ( e @ t ) α ( ` ) ( f @ s , e @ t ) ( 1 ) where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is α ( ` ) ( f @ s , e @ t ) def = exp ( 1√ D k ( ` ) ( f @ s ) > q ( ` ) ( e @ t ) ) ∈ R ( 2 ) In layer ` , v ( ` ) , k ( ` ) , and q ( ` ) are known as the value , key , and query vectors and are extracted from the layer- ( ` -1 ) event embeddings using learned layer-specific matrices V ( ` ) , K ( ` ) , Q ( ` ) : v ( ` ) ( e @ t ) def = V ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3a ) k ( ` ) ( e @ t ) def = K ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3b ) q ( ` ) ( e @ t ) def = Q ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3c ) As the base case , JeK ( 0 ) ( t ) def= JeK ( 0 ) is a learned embedding of the event type e. JtK denotes an embedding of the time t ; as proposed by Zuo et al . ( 2020 ) , we use the same sinusoidal embedding as in Vaswani et al . ( 2017 ) , but with t now being real instead of integer : JtKd = cos ( t/10000 d−1 D ) if d is odd and JtKd = sin ( t/10000 d D ) if d is even ( 4 ) We concatenate JtK to the rest of the embedding rather than adding it ( cf . Kitaev & Klein , 2018 ; He et al. , 2020 ) ; furthermore we do so again at every layer . According to equations ( 1 ) – ( 3 ) , to compute the layer- ` embedding of an event , we only need the layer ( ` −1 ) embeddings of the relevant events . This makes it possible to compute the layer- ` embeddings of all events in parallel . The set of relevant events H ( e @ t ) may be defined in a task-specific way . For example , to pretrain BERT-like embeddings ( Devlin et al. , 2018 ) , we might use a corrupted version of { e1 @ t1 , . . . , eI @ tI } in which some ei @ ti have been removed or replaced with mask @ ti . Such embeddings could be pretrained with a BERT-like objective and then fine-tuned to predict properties of the observed events . 3 GENERATIVE MODELING OF CONTINUOUS-TIME EVENT SEQUENCES . In this paper , we focus on the task of predicting future events given past ones . At any time t , we would like to know what will happen at that time , given the actual events that happened before t. Our generative model is analogous to a Transformer language model ( Radford et al. , 2019 ; Brown et al. , 2020 ) , which , at each time t ∈ N , defines a probability distribution over the words in the vocabulary . In our setting , however , t ∈ R. With probability 1 , nothing happens at time t. Each possible event e in our vocabulary has only an infinitesimal probability of occurring at time t. We write this probability as λe ( t ) dt where λe ( t ) ∈ R is called the ( Poisson ) intensity of type-e events at time t. More formally , the probability of such an event occurring during [ t , t+ ) approaches λe ( t ) as →+ 0 . We model λe ( t ) as a function of the top-layer embedding of the possible event e @ t : λe ( t ) def = softplus ( w > e [ 1 ; JeKL ( t ) ] , τe ) where softplus ( x , τ ) = τ log ( 1 + exp ( x/τ ) ) > 0 ( 5 ) with learnable parameters we and τe > 0 . We do this separately for each possible e @ t , computing the embedding JeKL ( t ) using equations ( 1 ) – ( 3 ) . To ensure that our model is generative , we compute JeKL ( t ) from only the previous events . Thus , we takeH ( e @ t ) in equation ( 1 ) to consist of all the previously generated events ei @ ti for ti < t , as well as the possible event e @ t itself . Including e @ t allows e @ t itself to draw most of the attention if none of the previous events are very relevant ( e.g. , they are too far in the past ) , or all of the attention in the special case where no previous events exist ( i.e. , it ensures thatH ( e @ t ) 6= ∅ , preventing division by 0 in equation 1 ) . Embedding each possible e @ t separately is somewhat expensive . The attention weights α ( 1 ) , . . . , α ( L ) used to compute its embedding and intensity must be recomputed for each e and t. This is more expressive but also more expensive than a standard Transformer language model , which instead models the probability of word e at time t as proportional to exp ( w > e [ 1 ; JfKL ( s ) ] ) where JfKL ( s ) is the top-layer embedding of the previous word f @ s , where t = s + 1 , and can be reused for all possible choices of e @ t. Why do we not follow that cheaper approach , using the previous event f @ s in place of e @ t in equation ( 5 ) ? Because then λe ( t ) would depend only on s and not on t ( a language model can rely on t = s+ 1 , but we may have to evaluate intensities at all real-valued t ≥ s since our events are irregularly spaced ) . Our more expressive design also allows the attention vector used to compute λe ( t ) to depend on e , to support the “ selective attention ” of section 4 below . As a speedup , however , we can associate each event type e with a coarse event type ē that has the same selective attention pattern , and replace JeKL ( t ) with JēKL ( t ) in equation ( 5 ) . ( However , equation ( 5 ) still uses the fine-grained we . ) Now to compute λe ( t ) , we only have to embed ē @ t , which is a speedup if many of the possible event types e are associated with the same ē . In the case where we do not use selective attention , we can get away with using only a single coarse event type for the whole model—saving a runtime factor of E as in the cheaper approach . Note that the historyH still uses fine-grained embeddings , so if e @ t actually occurs , we must then compute JeK0 ( t ) , . . . , JeKL ( t ) . Our model falls into the family of multivariate point processes . Its log-likelihood is given by I∑ i=1 log λei ( ti ) − ∫ T t=0 E∑ e=1 λe ( t ) dt ( 6 ) This formula has been derived and discussed in previous work ( Hawkes , 1971 ; Liniger , 2009 ; Mei & Eisner , 2017 ) . Intuitively , during parameter training , each log λei ( ti ) is increased to explain why the observed event ei happened at time ti , while ∫ T t=0 ∑E e=1 λe ( t ) dt is decreased to explain why no event of any possible type k ever happened at other times . That is why we need to embed possible events and compute their intensities .

Summary: A model for sequences of events occurring in the continuous time setting is presented. It uses the self-attention mechanism from the Transformer model, together with some domain-specific masking operations, to create event embeddings that are contextualized by their relation to other (relevant) events that occurred. The model is then extended towards a generative process, which in turn is integrated into a neural datalog through time framework. Experiments show the model to be perform equally or better than baselines based on RNNs or discrete time embeddings.