Transformer Embeddings of Irregularly Spaced Events and Their Participants

1 INTRODUCTION . Event sequences are ubiquitous in real-world applications , such as healthcare , finance , education , commerce , gaming , audio , news , security , and social media . We propose an elegant way to embed events into a vector space RD . An actual or possible event at time t ∈ R is contextually embedded using attention over actual events at other times , in a way that considers their timing . Good event embeddings could be used in a variety of downstream tasks , similar to word token embeddings in BERT . In particular , we consider the task of predicting the next event ( generative modeling ) : at each time t ∈ R , all of the possible events are embedded using masked attention over the preceding actual events , and those embeddings yield the instantaneous probabilities at time t of the possible events . Our approach is a continuous-time generalization of the Transformer ( Vaswani et al. , 2017 ) . Transformers scale up well to very large datasets and have achieved astonishing success at generative modeling of discrete-time sequences , such as natural-language documents ( Radford et al. , 2019 ; Brown et al. , 2020 ) and proteins ( Rao et al. , 2021 ) . We show in this paper that they are also effective for irregularly spaced continuous-time sequences , even in lower-data regimes . We further show how to derive models from arbitrary NDTT programs . NDTT , or neural Datalog through time ( Mei et al. , 2020a ) , is a neural-symbolic framework for modeling event sequences . An NDTT program specifies logical rules that derive facts within a deductive database . Moreover , the facts in the database are embedded using a deep recurrent neural architecture that is automatically determined by the facts ’ provenance . Among the derived facts at time t are the possible events at time t. As a simple NDTT example , if u is a user and d is a document and u has appropriate permissions , then u may open d. The embedding and instantaneous probability of this possible event at time t depend on the embeddings at time t of u and d ( or more precisely , of the facts that u is a user and d is a document ) . If the event actually occurs , it may change what facts are true at times > t and how their embeddings evolve over times > t. For instance , the NDTT program can state that if u opens d , this makes editing events possible , and also affects the future embeddings of u and d. These updated embeddings feed forward to help predict future events involving u and/or d. ( They could also be exploited for other downstream predictive tasks , via multitask training or pretraining . ) All of these effects are controlled by the hand-crafted rules and learned parameters of the NDTT program , event ’ s embedding at layer ` depends ( ) on all preceding events at layer ` − 1 . ( The dashed arrows reflect residual connections as well as the fact that each event or fact also attends to itself . ) Section 4 explains that the ee ff influences can be prevented by dropping the rule ff < - ee . Figure ( b ) shows an A-NDTT model from section 5 , in which the company forum ’ s embedding at layer ` depends ( ) on all preceding events at layer ` − 1 ( via < - rules ) . The events or possible events at layer ` do not depend directly on preceding events ; instead , their embeddings at time t are derived ( ) from the forum ’ s embedding at time t ( via : - rules ) . which are analogous to the hand-crafted structure and learned parameters of a graphical model , in that they enforce domain-appropriate conditional independence and parameter sharing . We modify the NDTT architectures to use attention . In our architecture , a fact that can be derived in multiple ways , by different NDTT rules , uses multiple attention heads to examine the past . Du et al . ( 2016 ) ; Mei & Eisner ( 2017 ) ; Mei et al . ( 2020a ) instead summarize the past using recurrent neural networks ( RNNs ) . Our attentive approach has three advantages over theirs : ¬ A Transformer does not summarize the past . Our predictions at time t can examine an unboundedly large representation of the past ( Rd embeddings of every event before t ) , not merely a fixeddimensional summary that was computed greedily from left to right ( an RNN ’ s state at time t ) . A Transformer ’ s computation graph is broader and shallower . The breadth makes it easier to learn long-distance influences . The shallowness does make it impossible to represent inherently deep concepts such as parity , but it enables greater parallelism : the layer- ` embeddings can be computed in parallel during training , as they depend only on layer ` − 1 and not on one another . ® Our architecture is simpler and arguably more natural , while remaining competitive in our experiments . To describe how embeddings vary continuously during the period ( ti , ti+1 ] between two events , Mei & Eisner ( 2017 ) had to stipulate an arbitrary family of parametric decay functions on t , and Mei et al . ( 2020a ) had to design a complex method for pooling the parameters of these decay functions across multiple NDTT rules . In contrast , our embeddings are constructed “ from scratch ” at each time t simply by attending to the set of previous events , using t-specific query vectors that include a continuous positional embedding of t. As t ranges over ( ti , ti+1 ] , the attention weights over the first i events vary continuously with t , so the embeddings do so too . 2 CONTINUOUS-TIME TRANSFORMER FOR EMBEDDING EVENTS . Suppose we observe I events over a fixed time interval [ 0 , T ) . Each event is denoted mnemonically as e @ t ( i.e. , “ type e at time t ” ) , where e ∈ { 1 , 2 , . . . , E } is the event ’ s type . The observed event sequence is e1 @ t1 , e2 @ t2 , . . . , eI @ tI , where 0 < t1 < t2 < . . . < tI < T . For any observed or possible event e @ t , we can compute an embedding JeK ( t ) ∈ RD by attending to a set H ( e @ t ) of relevant events . ( For the moment , imagine that H ( e @ t ) consists of all the observed events ei @ ti . ) More precisely , JeK ( t ) is the concatenation of layer-wise embeddings JeK ( 0 ) ( t ) , JeK ( 1 ) ( t ) , . . . , JeK ( L ) ( t ) . For ` > 0 , the layer- ` embedding of e @ t is computed as JeK ( ` ) ( t ) def= JeK ( ` −1 ) ( t ) ︸ ︷︷ ︸ residual connection + tanh ∑ f @ s∈H ( e @ t ) v ( ` ) ( f @ s ) α ( ` ) ( f @ s , e @ t ) ∑ f @ s∈H ( e @ t ) α ( ` ) ( f @ s , e @ t ) ( 1 ) where the unnormalized attention weight on each relevant event f @ s ∈ H ( k @ t ) is α ( ` ) ( f @ s , e @ t ) def = exp ( 1√ D k ( ` ) ( f @ s ) > q ( ` ) ( e @ t ) ) ∈ R ( 2 ) In layer ` , v ( ` ) , k ( ` ) , and q ( ` ) are known as the value , key , and query vectors and are extracted from the layer- ( ` -1 ) event embeddings using learned layer-specific matrices V ( ` ) , K ( ` ) , Q ( ` ) : v ( ` ) ( e @ t ) def = V ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3a ) k ( ` ) ( e @ t ) def = K ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3b ) q ( ` ) ( e @ t ) def = Q ( ` ) [ 1 ; JtK ; JeK ( ` −1 ) ( t ) ] ( 3c ) As the base case , JeK ( 0 ) ( t ) def= JeK ( 0 ) is a learned embedding of the event type e. JtK denotes an embedding of the time t ; as proposed by Zuo et al . ( 2020 ) , we use the same sinusoidal embedding as in Vaswani et al . ( 2017 ) , but with t now being real instead of integer : JtKd = cos ( t/10000 d−1 D ) if d is odd and JtKd = sin ( t/10000 d D ) if d is even ( 4 ) We concatenate JtK to the rest of the embedding rather than adding it ( cf . Kitaev & Klein , 2018 ; He et al. , 2020 ) ; furthermore we do so again at every layer . According to equations ( 1 ) – ( 3 ) , to compute the layer- ` embedding of an event , we only need the layer ( ` −1 ) embeddings of the relevant events . This makes it possible to compute the layer- ` embeddings of all events in parallel . The set of relevant events H ( e @ t ) may be defined in a task-specific way . For example , to pretrain BERT-like embeddings ( Devlin et al. , 2018 ) , we might use a corrupted version of { e1 @ t1 , . . . , eI @ tI } in which some ei @ ti have been removed or replaced with mask @ ti . Such embeddings could be pretrained with a BERT-like objective and then fine-tuned to predict properties of the observed events . 3 GENERATIVE MODELING OF CONTINUOUS-TIME EVENT SEQUENCES . In this paper , we focus on the task of predicting future events given past ones . At any time t , we would like to know what will happen at that time , given the actual events that happened before t. Our generative model is analogous to a Transformer language model ( Radford et al. , 2019 ; Brown et al. , 2020 ) , which , at each time t ∈ N , defines a probability distribution over the words in the vocabulary . In our setting , however , t ∈ R. With probability 1 , nothing happens at time t. Each possible event e in our vocabulary has only an infinitesimal probability of occurring at time t. We write this probability as λe ( t ) dt where λe ( t ) ∈ R is called the ( Poisson ) intensity of type-e events at time t. More formally , the probability of such an event occurring during [ t , t+ ) approaches λe ( t ) as →+ 0 . We model λe ( t ) as a function of the top-layer embedding of the possible event e @ t : λe ( t ) def = softplus ( w > e [ 1 ; JeKL ( t ) ] , τe ) where softplus ( x , τ ) = τ log ( 1 + exp ( x/τ ) ) > 0 ( 5 ) with learnable parameters we and τe > 0 . We do this separately for each possible e @ t , computing the embedding JeKL ( t ) using equations ( 1 ) – ( 3 ) . To ensure that our model is generative , we compute JeKL ( t ) from only the previous events . Thus , we takeH ( e @ t ) in equation ( 1 ) to consist of all the previously generated events ei @ ti for ti < t , as well as the possible event e @ t itself . Including e @ t allows e @ t itself to draw most of the attention if none of the previous events are very relevant ( e.g. , they are too far in the past ) , or all of the attention in the special case where no previous events exist ( i.e. , it ensures thatH ( e @ t ) 6= ∅ , preventing division by 0 in equation 1 ) . Embedding each possible e @ t separately is somewhat expensive . The attention weights α ( 1 ) , . . . , α ( L ) used to compute its embedding and intensity must be recomputed for each e and t. This is more expressive but also more expensive than a standard Transformer language model , which instead models the probability of word e at time t as proportional to exp ( w > e [ 1 ; JfKL ( s ) ] ) where JfKL ( s ) is the top-layer embedding of the previous word f @ s , where t = s + 1 , and can be reused for all possible choices of e @ t. Why do we not follow that cheaper approach , using the previous event f @ s in place of e @ t in equation ( 5 ) ? Because then λe ( t ) would depend only on s and not on t ( a language model can rely on t = s+ 1 , but we may have to evaluate intensities at all real-valued t ≥ s since our events are irregularly spaced ) . Our more expressive design also allows the attention vector used to compute λe ( t ) to depend on e , to support the “ selective attention ” of section 4 below . As a speedup , however , we can associate each event type e with a coarse event type ē that has the same selective attention pattern , and replace JeKL ( t ) with JēKL ( t ) in equation ( 5 ) . ( However , equation ( 5 ) still uses the fine-grained we . ) Now to compute λe ( t ) , we only have to embed ē @ t , which is a speedup if many of the possible event types e are associated with the same ē . In the case where we do not use selective attention , we can get away with using only a single coarse event type for the whole model—saving a runtime factor of E as in the cheaper approach . Note that the historyH still uses fine-grained embeddings , so if e @ t actually occurs , we must then compute JeK0 ( t ) , . . . , JeKL ( t ) . Our model falls into the family of multivariate point processes . Its log-likelihood is given by I∑ i=1 log λei ( ti ) − ∫ T t=0 E∑ e=1 λe ( t ) dt ( 6 ) This formula has been derived and discussed in previous work ( Hawkes , 1971 ; Liniger , 2009 ; Mei & Eisner , 2017 ) . Intuitively , during parameter training , each log λei ( ti ) is increased to explain why the observed event ei happened at time ti , while ∫ T t=0 ∑E e=1 λe ( t ) dt is decreased to explain why no event of any possible type k ever happened at other times . That is why we need to embed possible events and compute their intensities .

Summary: The paper introduces A-NHP and A-NDTT that tackle the problem of continuous time sequences and neural-symbolic computing with a time component using transformers and attention. The paper explains their model and techniques mainly with equations. The work seems primarily motivated by previous work on Hawkes processes and neural datalog through time NDTT. They improve by adding a probability model that allows them to embed all possible events and using transformers instead of LSTMs for parallelization. They achieve strong experimental results, especially in StackOverflow and RoboCup.