HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences
Authors: Siqiao Xue
Xiaoming Shi
James Y. Zhang
Hongyuan Mei
Venue: NeurIPS 2022 Accept (2022)
Keywords: probabilistic model
event sequences
energy-based model
long-horizon prediction

Abstract: In this paper, we tackle the important yet under-investigated problem of making long-horizon prediction of event sequences. Existing state-of-the-art models do not perform well at this task due to their autoregressive structure. We propose HYPRO, a hybridly normalized probabilistic model that naturally fits this task: its first part is an autoregressive base model that learns to propose predictions; its second part is an energy function that learns to reweight the proposals such that more realistic predictions end up with higher probabilities. We also propose efficient training and inference algorithms for this model. Experiments on multiple real-world datasets demonstrate that our proposed HYPRO model can significantly outperform previous models at making long-horizon predictions of future events. We also conduct a range of ablation studies to investigate the effectiveness of each component of our proposed methods.

HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences
Siqiao Xue, Xiaoming Shi, James Zhang Ant Group 569 Xixi Road Hangzhou, China si aiqiao. xsqêa a www.bbbbincc..... com (peter. sxm, james z @antgroup com
Hongyuan Mei
Toyota Technological Institute at Chicago 6045 S Kenwood Ave Chicago, IL 60637 hongyuandttic.edn
arXiv:22100015531 [cs.LG] 4 Oct 2022
Abstract
In this paper, we tackle the important yet under-investigated problem of making long-horizon prediction event sequences. Existing state-of-the-ard models do not perform well this task due to their autoregressive structure. We propose HYPRO, hybridly normalized probabilistic model that naturally fits this task: its first part an autoregressive base model that learns to propose predictions; its second part is an energy function that learns to reweight the proposals such that more realistic predictions end up with higher probabilities. We also propose efficient training and inference algorithms for this model. Experiments on multiple real-world datasets demonstrate that our proposed HYPRO model can significantly outperform previous models at making long-horizon predictions of future events. We also conduct range of ablation studies to investigate the effectiveness of each component of our proposed methods.
Introduction
Long-horizon prediction of event sequences is essential in various real-world applied domains:
Healthcare. Given a patient's symptoms and treatments so far, we would be interested in predicting their future health conditions over the next several months, including their prognosis and treatment.
Commercial. Given online consumer's previous purchases and reviews, we may be interested in predicting what they would buy over the next several weeks and plan our advertisement accordingly.
Urban planning. Having monitored the traffic flow of town for the past few days, we'd like to predict its future traffic over the next few hours, which would be useful for congestion management.
Similar scenarios arise in computer systems, finance,,, dialogue, music, etc.
Though being important, this task has been under-investigated. the previous work in this research area has been mostly focused on the prediction of the next single event (c.g., its time and type).
In this paper, we show that previous state-of-ther the-trr models suffer at making hong-horizon predictions, i.e. predicting the series of future events over a given time interval. That is because those models are all autoregressive: predicting each future event is conditioned on all the previously predicted events; an error can not be corrected after is made and any error will be cascaded through all the subsequent predictions. Problems of the same kind also exist in natural language processing tasks such as generation and machine translation (Ranzato et al., 2016; Goyal, 2021).
In this paper, we propose a novel modeling framework that learns to make long-horizon prediction of event sequences. Our main technical contributions include:
A new model. The key component of our framework is HYPRO, hybridly normalized neural probabilistic model that combines an autoregressive base model with an energy function: the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

base model learns to propose plausible predictions; the energy function learns to reweight the proposals. Although the proposals are generated autoregressively, the energy function reads each entire completed sequence (i.e., true past events together with predicted future events) and learns to assign higher weights to those which appear more realistic as a whole.
Hybrid models have already demonstrated effective at natural language processing such as detecting machine- generated machine-generated text (Bakhtin et al., 2019) and improving coherency in text generation (Deng et al., 2020). We are the first to develop a model of this kind for time-stamped event sequences. Our model can use any autoregressive event model as its base model, and we choose the state-of-the-art continuous-time Transformer architecture (Yang et al., 2022) as its energy function.
A family of new training objectives. Our second contribution a family of training objectives that can estimate the parameters of our proposed model with low computational cost. Our training methods are based on the principle of noise-contrastive estimation since the log-likelihood of our HYPRO model involves an intractable normalizing constant (due to using energy functions).
A new efficient inference method. Another contribution is a normalized importance sampling algorithm, which can efficiently draw the predictions of future events over given time interval from a trained HYPRO model.
2 Technical Background
2.1 Formulation: Generative Modeling of Event Sequences
We are given a fixed time interval [0, T) over which an event sequence observed. Suppose there < ts < T. We denote the sequence as J[o.T] = (11, ..),..., (f1, ky) where each k; E (1...., K} is a discrete event type.
are 1 events in the sequence at times 0 < 11
Generative models of event sequences are temporal point processes. They are autoregressive events are generated from left to right; the probability of (t,, ki) depends on the history of events that were drawn at times < 4, They are locally normalized: if
=(0.e.) = (41, ki),..., we use (1 ((oo))) to denote the probability that an event of type l occurs over the infinitesimal interval [t, (f)d). then the probability that nothing occurs will be 1 --EK Pte(t ()o,))). Specifically, temporal point processes define functions A that determine finite intensity As (z | mmo.)) 2 0 for each event type lo at each time > 0 such that pt I(joa)) Au zjo.s))dt. Then the log-likelihood of temporal point process given the entire event sequence I|v,2] is
log dR, (fa m(aas)
Aa I(o.))
(1)
Popular examples of temporal point processes include Poisson processes (Daley & Vere-Jones, 2007) as well as Hawkes processes (Hawkes, 1971) and their modern neural versions (Du et al., 2016; Mei & Eisner, 2017: Zuo et al., 2020; Zhang et al., 2020: Yang et al., 2022).
2.2 Task and Challenge: Long- Horizon Prediction and Cascading Errors
We are interested in predicting the future events over an extended time interval (T, T". We call this task long-horizon prediction the boundary T' is large that (with high probability) many events will happen over (T, T'). A principled way to solve this task works as follows: we draw many possible future event sequences over the interval (T, T'), and then this empirical distribution to answer questions such as "how many events of type k = 3 will happen over that interval".
A serious technical issue arises when we draw each possible future sequence. To draw an event sequence from an autoregressive model, we have to repeatedly draw the next event, append to the history, and then continue to draw the next event conditioned on the new history. This process is prone to cascading errors: any error in a drawn event is likely to cause all the subsequent draws to differ from what they should be, and such emrors will accumulate.
2.3 Globally Normalized Models: Hope and Difficulties
An ideal fix of this issue to develop a globally normalized model for event sequences. For any time interval [0, 7]. such model will give a probability distribution that is normalized over all the possible full sequences on [0, 7] rather than over the possible instantaneous subsequences within each (t, + dt). Technically, globally normalized model assigns to each sequence z[o.7]

---

score exp (=E((00)) where E is called energy function; the normalized probability of 2 is proportional to its score: i... p( exp (-E(x10,11).
Had we trained globally normalized model, we wish to enumerate all the possible x|T,T| for a given =[o, and select those which give the highest model probabilities p(11..r1) Prediction made this way would not suffer cascading errors: the entire T[o.r" was jointly selected and thus the overall compatibility between the events had been considered.
However, training such a globally normalized probabilistic model involves computing the normalizing constant Σ exp (((00.)) where the summation Σ is taken over all the possible sequences; it is intractable since there are infinitely many sequences. What's worse, is also intractable to exactly sample from such model; approximate sampling is tractable but expensive.
3 HYPRO: A Hybridly Normalized Neural Probabilistic Model
We propose HYPRO, hybridly normalized neural probabilistic model that combines a temporal point process and an energy function: it enjoys both the efficiency of autoregressive models and the capacity of globally normalized models. Our model normalizes over (sub)sequences: for any given interval [0, [ and its extension (I, ] of interest, the model probability of the sequence I(r,T) is
exp( (xp(-((z))
PHYPRO (I(T.TT) 2To.T)) Pauto (
(2)
Zo (maar)
where Pauto is the probability under the chosen temporal point process and Es is energy function with parameters 0. The normalizing constant sums over all the possible continuations T(T,T" for given prefix 20 Ze (M)..)
Expreyy
Pauto (I(T)T)) Z(o.r) exp (- -x(x))))))
The key advantage of our model over autoregressive models that: the energy function Eg is able to pick up the global features that may have been missed by the autoregressive base model Panto: intuitively, the energy function fits the residuals that are not captured by the autoregressive model.
Our model is general: in principle, Pauto can be any autoregressive model including those mentioned in section and Fgg can any function that able to encode an event sequence a real number. In section 5, we will introduce couple of specific Pauto and Eg and experiment with them.
In this section, we focus on the training method and inference algorithm.
3.1 Training Objectives
Training our full model PhYpro is to learn the parameters of the autoregressive model Pauto as well as those of the energy function Eg. Maximum likelihood estimation (MLE) is undesirable: the objective would be log PuYpro (T(T.T>) Z)),T) = log Pauto (I<T.T") (Toori) - Fo( - log Zo (T
where the normalizing constant Zo (I)OTT) is known to be uncomputable and inapproxinable for large variety of reasonably expressive functions Eg (Lin McCarthy, 2022).
We propose a training method that works around this normalizing constant. We first train Pauto just like how previous work trained temporal point processes.' Then we the trained Pauto as a noise distribution and learn the parameters 0 of Eg by noise-contrastive estimation (NCE). Precisely, we (N)
sample N noise sequences x/ (1) ' ................................ [T compute the "energy" ( for each completed
(n) sequence of " and then plug those energies into one of the following training objectives.
Note that all the completed sequences "ioTT1 share the same observed prefix T[o,T]-
Binary-NCE Objective. We train a binary classifier based on the energy function Eg to discriminate the true event sequence sequence-denoted as z " against the noise sequences by maximizing
(0)
N
Joinary- = log a (-Fg(=(0.r))
(
+ Σ log (Fo (Fa(m(1)21))))
#=1
II can be done by either maximum likelihood estimation or crise-contrastivn estimation: for the former. read Daley& & Vere-Jones (2007); for the latter, read Mei et al. (2020b) which also has an in de-depth discussion about the theoretical connections between these parameter estimation principles.
3

---

where (u) [+exp[-uj is the sigmoid function. By maximizing this objective, we are essentially pushing our energy function Eg such that the observed sequences have low energy but the noise sequences have high energy. As a result, the observed sequences will be more probable under our full model PHYPRO while the noise sequences will be less probable: equation (2).
Theoretical guarantees of general Binary-NCE can be found in Gutmann & Hyvärinen (2010). For general conditional probability models like ours, Binary-NCE implicitly assumes self- normalization (Mnih & Teh, 2012; Ma & Collins, 2018): i.e., Zo (f(o,z))
satisfied.
This type of training objective has been used to train a hybridly normalized text generation model by Deng al. (2020); see section 4 for more discussion about its relations with our work.
Multi-NCE Objective. Another option is to use Multi-NCE objective", which means we maximize
Jmulu (o( log
exp (-Bo(e(orr))))
By maximizing this objective, we are pushing our energy function Fa such that each observed sequence has relatively lower energy than the noise sequences sharing the same observed prefix. In contrast, Joinary attempts make energies absolutely low (for observed data) high (for noise data) without considering whether they share prefixes. This effect is analyzed in Analysis-IlI of section 5.2.
This Jmmita objective also enjoys better statistical properties than Junsryy since it doesn't It assume self- normalization: the normalizing constant Ze (z) is neatly cancelled out in its derivation; Appendix A.1 for a full derivation both Binary-NCe and Multi-NCE.
Theoretical guarantees of Multi-NCE for discrete-ting time models were established by Ma & Collins (2018); Mei et al. (2020b) generalized them to temporal point processes.
Considering Distances Between Sequences. Previous work (LeCun et al., 2006; Bakhtin et al., 2019) reported that energy functions may be better learned if the distances between samples are considered. This has inspired us to design a regularization term that enforces such consideration.
Suppose that we can measure well-defined "distance" between the true sequence "io.r" and any
(0)
[n] noise sequence / "i.Trg' we denote d(n). We encourage the energy of each noise sequence be higher than that of the observed sequence by a margin; that is, we propose the following regularization:
N
Ω max (o, (d)))( + (0) - (0(
(5)
where 8 0 is hyperparameter that tune the held-out development data. With this regularization, the energies of the sequences with larger distances will be pulled farther apart: this will help discriminate not only between the observed sequence and the noise sequences, but also between the noise sequences themselves, thus making the energy function Eg more informed.
This method is general so the distance d can be any appropriately defined metric. In section 5, we will experiment with an optimal transport distance specifically designed for event sequences.
Note that the distance d in the regularization may be the final test metric. In that case, our method is directly optimizing for the final evaluation score.
Generating Noise Sequences. Generating event sequences from an autoregressive temporal point process has been well-studied in previous literature. The standard way is to call the thinning algorithm (Lewis & Shedler, 1979; Liniger, 2009). The full recipe for our setting is in Algorithm 1.
3.2 Inference Algorithm
Inference involves drawing future sequences T(T,Ty from the trained full model phypro: due to the uncomputability of the normalizing constant 2 (2)0.T)) exact sampling is intractable.
We propose normalized importance sampling method to approximately draw 2[T,Ty] from
Phypro: it is shown Algorithm 2. We first use the trained Pauto to be our proposal distribu-
(1) tion and call the thinning algorithm (Algorithm 1) to draw proposals TIr.r.'...
(M)
... .p... Then
It was named as Ranking NCE by Ma & Collins (2018), but we think Multi-NCE is more appropriate name since it constructs multi-class classifier over correct answer and multiple incomect answers.

---

Algorithm 1 Generating Noise Sequences.
Input: an event sequence T] over the given interval [0, 7' and an interval (T, T') of interest; trained autoregressive model Pamto and number of noise samples N
Output: a collection of noise sequences
1: procedure DRAWNOISE((TO.TT) Pauto. N)
2: for 1 to :
3: use the thinning algorithm to draw each noise sequences from the autoregressive model Punto
4: in particular, call the method in Algorithm 3 that described in Appendix A.2
5: (T,T") T. T", Paanta
(n)
6: return (T.T (T.T))
reweight those proposals with the normalized weights (m)) that are defined w (m)
Parenuo
[m] (1.0 3/Pano (rrr
exp Eo (2 (0.211)
(6)
-1 Privero (1r (mn')
(1. / //
(nd
EM Em'= exp Fig (o.71)
This collection of weighted proposals used for the long-horizon prediction over the interval (T, T']:
if we want the most probable sequence, we return the "[r,Ty with the largest weight ((();; if we want minimum Bayes risk prediction (for specific risk metric), we use existing methods (e.g., the consensus decoding method in Mei et al. (2019)) to compose those weighted samples into a single sequence that minimizes the risk. In our experiments (section 5), we used the most probable sequence.
(m)
Note that our sampling, method is biased since the weights (m)) are normalized. Unbiased sampling in our setting is intractable since that will need our weights be unnormalized: i.e.,
w = PHyPRO / Pauto = exp (-Ee) Z which circles back to the problem of Z's 's uncomputability. Exper- imental results section 5 show that our method indeed works well in practice despite that is biased.
def
Algorithm 2 Normalized Importance Sampling for Long- Horizon Prediction.
Input: an event sequence [o. over the given interval [0, T] and an interval (T, T"] of interest; trained autoregressive model Pauto and engergy function Eg, number of proposals M Output: a collection of weighted proposals
1: procedure NIS(I T' Paatos Eg. .)
2: use normalized importance sampling to approximately draw M proposals from Phnrro (M)
3: "ITrp" " 2[T,TT) +- DRAWNOISECETO.TT) T', Pauta, M)
(1)
see Algorithm /
4: construct completed sequences "io,r''......................... x 10,771 by appending each 2 ufل.y 2[o,T]
(1)
(M)
(m)
5: compute the exponential of minus energy (m)) eexp
(m) (-Eo(=jorn)) for each proposal
6: compute the normalized weights (m)) = ((m)) //M e(mx))
7: return (wi x (1) (.r).......... ..... (w) (Mf)
, -
D return the collection of weighted proposals
Related work
Over the recent years, various neural temporal point processes have been proposed. Many of them are built on recurrent neural networks, or LSTMs (Hochreiter Schmidhuber, 1997); they include et al. (2016); Mei & Eisner (2017); Xiao et al. (2017a,b); Omi et al. (2019); Shchur et al. (2020); Mei et al. (2020a); Boyd et al. (2020). Some others use Transformer architectures (Vaswani et al., 2017: Radford et al., 2019): in particular, Zuo et al. (2020); Zhang et al. (2020); Enguchard et al. (2020); Sharma et al. (2021); Zhu et al. (2021); Yang et al. (2022). All these models all autoregressive: they define the probability distribution over event sequences in terms of a sequence of locally noomlly-normalized conditional distributions over events given their histories.
Energy-based models, which have long history in machine learning (Hopfield, 1982; Hinton, 2002; LeCun et al., 2006; Ranzato et al., 2007; Ngiam al., 201 Xie et al., 2019), define the distribution over sequences in a different way: they use energy functions to summarize each possible sequence into a scalar (called energy) and define the unnormalized probability of each sequence terms of its energy, then the probability distribution is normalized across all sequences; thus, they are also
5

---

called globally normalized models. Globally normalized models are a strict generalization of locally normalized models (Lin al., 2021): all the locally normalized models are globally normalized; but the converse is not true. Moreover, energy functions are good at capturing global features and structures (Pang et al., 2021; Du & Mordatch, 2019; Brakel et al., 2013). However, the normalizing constants of globally normalized models are often uncomputable (Lin & McCarthy, 2022). Existing work that is most similar to ours is the energy-based text generation models of Bakhtin et al. (2019) and Deng et al. (2020) that train energy functions to reweight the outputs generated by pretrained autoregressive models. The differences are: we work on different kinds of sequential data (continuous- time event sequences vs. discrete-time natural language sentences), and thus the architectures of our autoregressive model and energy function are different from theirs; additionally, we explored a wider range of training objectives (e.g., Multi-NCE) than they did.
The task of long-horizon prediction has drawn much attention in several machine learning areas such as regular time series analysis (Yu et al., 2019; Le Guen & Thome, 2019), natural language processing (Guo et al., 2018; Guan al., 2021), and speech modeling (Oord et al., 2016). Deshpande al. (2021) I) is the best-performing to-date in long-horizon prediction of event sequences: they adopt hierarchical architecture similar to ours and use a ranking objective based on the counts of the events. Their method can be regarded as special case of our framework (if we let our energy function read the counts of the events), and our method works better in practice (see section 5).
5 Experiments
We implemented our methods with PyTorch (Paszke et al., 2017). Our code can be found at https: // gighhb.. com/an al ppay/hypro_tppp and https: /ggttbbtttbuu//.. htwcchub..oomaecccmmmmmmmcee http:/aaaaaapaarraaaaaa/ Im- plementation details can be found in Appendix B.2.
5.1 Experimental Setup
Given train set of sequences, we use the full sequences to train the autoregressive model Pauto by maximizing equation (1). To train the energy function Eg. we need to split each sequence into a prefix T[o,T] and continuation #(T,zy] we choose T and 7" such that there are 20 event tokens within (T, 7'I on average. During testing, for each prefix I[o,T), we draw 20 weighted samples (Algorithm 2) and choose the weeghstt-eighted one as our prediction p(r,r.] We evaluate our predictions by:
The root of mean square error (RMSE) of the number of the tokens each event type: for each type k, we count the number of type-k tokens in the true continuation dentte as C2 well as that in the mediction--enon
Cj: then the mean square error V K ΣK (Cx C2
The optimal transport distance (OTD) between event sequences defined by Mei et al. (2019): for any given prefix d.o.7]. the distance defined as the minimal cost of editing the prediction ((T,Ty) (by inserting or deleting events, changing their occurrence times, and changing their types) such that it becomes exactly the same as the true continuation ((T,T))-
We did experiments on two real-world datasets (see Appendix B.1 for dataset details):
Taobao (Alibaba, 2018). This public dataset was created and released for the 2018 Tianchi Big Data Competition. It contains time-stamped behavior records (e.g.. browsing, purchasing) of anonymized users on the online shopping platform Taobao from November 25 through December 03, 2017. Each category group (e.g., men's 's clothing) is an event type, and we have K 17 event types. We use the browsing sequences of the most active 2000 users; each user has sequence. Then we randomly sampled disjoint train, dev and test sets with 1300, 200 and 500 sequences. The time unit is 3 hours; the average inter-arrival time is 0.06 (i.e., 0.18 hour), and we choose the prediction horizon T T to be 1.5 that approximately covers 20 event tokens.
Taxi (Whong. 2014). This dataset tracks the time- stampe taxi pick-up and drop-off events across the five boroughs of the New York city; each (borough, pick-up drop-off) combination defines an event type, so there are K = 10 event types total. We work on a randomly sampled subset of 2000 drivers and each driver has a sequence. We randomly sampled disjoint train, dev and test sets with 1400, 200 and 400 sequences. The time unit is hour; the average inter-arrival time 0.22, and set the prediction horizon to be 4.5 that approximately covers 20 event tokens.

---

Mou Square Entor
Nore Hean Square Emon
12
te'n oh
hy pro-an
bypro-em
qualiee
hypmo
hyppe
mosp she
aniig
Werrgge OTD.
ou3P /9
strig
byonom
typo
myero-mm
5u500 10-0
(a) Taobao Data
(b) Taxi Data
(c) StackOverflow Data
Figure 1: Performance of all the methods on Taobao (la), Taxi (Ib) and StackOverflow (Ic) datasets, measured by RMSE (up) and OTD (down). In each figure, the models from left right are: DualTPP (dualtpp): NHP (nhp); NHP with more parameters (nhp-Ig); AuNHP (att); AuNHP with more parameters (att-Ig): our HYPRO with Transformer energy function trained via Binary-NCE (hypro-a-b) and Multi-NCE (hypro-a-m).
StackOverflow (Leskovec & Krevl, 2014). This dataset has two years of user awards on a question- answering website: each user received sequence of badges and there K 22 different kinds of badges in total. We randomly sampled disjoint train, dev and test sets with 1400, 400 and 400 sequences from the dataset. The time unit 11 days; the average arrivall time is 0.95 and we set the prediction horizon to be 20 that approximately covers 20 event tokens.
We choose two strong autoregressive models as our base model Pauto:
Neural Hawkes process (NHP) (Mei & Eisner, 2017). It is an LSTM-basee autoregressive model that has demonstrated effective at modeling event sequences in various domains.
Attentative neural Hawkes process (AttNHP) (Yang et al., 2022).
an attention- based autoregressive model-like Transformer language model (Vaswani et al., 2017; Radford et al., performance comparable to or better than that of the NHP as well as other attention-based models (Zuo et al., 2020; Zhang et al., 2020).
For the energy function Fgg adapt the continuous-time Transformer module of the AttHH model: the Transformer module embeds the given sequence of events x into a fixed-dimensional vector (see section-2 of Yang al. (2022) for details), which is then mapped to a scalar E R via multi-layer perceptron (MLP); that scalar is the energy value (x))
We first train the two base models NHP and AtNHH; they are also used as the baseline methods that we will compare to. To speed up energy function training, we use the pretrained weights of the AttNHP to initialize the Transformer part of the energy function; this trick (2020) to bootstrap the energy functions for text generation models. As the full model Phypro has significantly more parameters than the base model Pauta- also trained larger NHP and ArNHP with comparable amounts of parameters as extra baselines. Additionally, we also compare to the DualTPP model of Deshpande et al. (2021). Details about model parameters are in Table 2 of Appendix B.3.
also used in Deng et al.
5.2 Results and Analysis
The main results are shown Figure 1. The OTD depends on the hyperparameter Cdes, which the cost of deleting adding event token of any type, the averaged OTD in Figure 1; OTD for each specific C'ael can be found Appendix B.4. As we can see, NHP and AtNHP work the worst in most cases. DualTPP doesn't seem to outperform these autoregressive baselines even though it learns to rerank sequences based on their macro statistics; we believe that it is because DualTPP's base autoregressive model is not powerful the state-of- the- art NHP and AtNHH and using macro statistics doesn't help enough. Our HYPRO method works significantly better than these baselines.
used range of values of Caa and report
Analysis-I: Does the Cascading Error Exist? Handling cascading errors key motivation for framework. On the Taobao dataset, empirically confirmed that this issue indeed exists.
7

---

0.5
0.5
Song sequance
samp sequarnce
sample from nose burton
samp from nolee distrbutiins
0.2
0.2
0.1
0.3
-10
Enemy score Blrary-KCE
Energy score with MaTI-NCE (a) Without the using distance-basee regularization of equation (5).
HIPRO sdiibuttion observed nohe dicitbution
tample
sampie ditribution
33
53
0.
6.1
-15
Energy score with Brary-iCE
Energy
(b) Using the distance-based regularization of equation (5).
Figure Energy scores computed on the held-out development data Taobao dataset.
We first investigate whether the event type prediction errors are cascaded through the subsequent events. We grouped the sequences based on how early the first event type prediction error was made and then compared the event type prediction error rate on the subsequent events:
when the first error is made on the first event token, the base AttNHP model has 71.28 11288 error rate on the subsequent events and our hybrid model has a much lower 66.99% error rate.
when the first error is made the fifth event token, the base AuNHH model has 58 58.88% error rate on the subsequent events and our hybrid model has a much lower 51.89% error rate.
when the first error is made on the tenth event token, the base AttHH model has 44.48% error rate on the subsequent events and our hybrid model has a much lower 35.58% eтror rate.
Obviously, when mistakes are made earlier in the sequences, tend to end up with higher error rate on the subsequent predictions: that means event type prediction errors are indeed cascaded through the subsequent predictions. Moreover, in each group, hybrid model enjoys lower prediction error; that means it indeed helps mitigate this issue.
We then investigate whether the event time prediction errors are cascaded. For this, we performed linear regression: the independent variable is the absolute error
prediction on the time first event token; the dependent variable is the averaged absolute error of the prediction of the subsequent event tokens. Our fitted linear model is y = 0.7965x + 0.3219 where the p-value of the coefficient of x is R 0.0001 < 0.01. It means that the time prediction errors are also cascaded through the subsequent predictions.
the time
Analysis-II: Energy Function or Just More Parameters? The larger NHP and AtNHH have almost the same numbers parameters with HYPRO, but their performance is only comparable to the smaller NHP and Att That is to say, simply increasing the number of parameters in an autoregressive model will achieve the performance of using an energy function.
To further verify the usefulness of the energy function, we also compared our method with another baseline method that ranks the completed sequences based their probabilities under the base model, from which the continuations were drawn. This baseline is similar to our proposed HYPRO framework but its scorer is the base model itself. our experiments, this baseline method is not better than our method; details can be found in Appendix B.5.
Overall,
conclude that the energy function Eg is essential to the success of HYPRO.
Analysis-III Binary-NCE vs. Multi-NCE. Both Binary-NCE and Multi-NCE objectives aim to match our full model distribution PHyPRO with the true data distribution, but Multi-NCE enjoys better statistical properties (see section 3) and achieved better performance our experiments (see Figure 1). In Figure 2a, we display the distributions of the energy scores of the observed sequences. Phypro- genero-generated sequences, and Pauto -generate noise sequences: as we can see, the distribution of PhyPRO different from that of noise data and indeed closer to that of real data.

---

Reot Mean Square Ener
Root Moan Square
Avrerage OTD
37
weerage OTO
20.0
2.4
1.30
Average
19.55
19.0
2.2
hypro-a-6
hy hypro-a-b-rpp reg
hypre-a m
hypro-a-m-ren
hypro-a46 b-b
hypro-a a-pro--b-reg
hypro-a-1 m
hypro- m-res
1.20
hypro-aa
hypro-a- b-reg
hypro-a m
typro-a-m-ren
18.5
hypro-a44
hypro-a b-ro-a-b-rgg
hypro-a -
hypro-a-m-me
(a) Taohao Dataset.
(b) Taxi Dataset.
Figure 3: Adding the regularization term Q. In each figure, the suffix -reg denotes "with regularization"
0.765
1.38
10.0
att
hypro-a-b
1.37
2.450
3.475
37.5
57.0
hypro-a
0.755
1.36
2.425
3.450
9.6
19.4
37.0
0.750
3.425
9.4
56.5
1.35
0.745
2.375
3.400
36.5
0.740
1.34
2.350
19.0
56.0
0.735
1.33
18.8
36.0
Modtoon m 0.3 Mortoon m C.B.
Mortnon monmm11. 1.5
Monzon
Figure 4: HYPRO vs. AttNHP different horizons on Taobao Dataset.
Analysis-IV: Efffects of the Distance- Regularization Q. We experimented with the proposed distance-based regularization S2 in equation (5); as shown in Figure 3, it slightly improves both Binary- NCE and Multi-NCE, As shown Figure 2b, the regularization makes larger difference in the Binary-NCE case: the energies of PuypRo-generatee generated sequences are pushed further to the left.
We did the paired permutation test verify the statistical significance of regularization technique; see Appendix B.6 for details. Overall, we found that the performance improvements of using the regularization strongly significant in the Binary-NCE case (p-value < 0.05 ) but not significant in the Multi-NCE case (p-value s 0.1). This finding is consistent with the observations in Figure 2.
Analysis-V V: Effects of Prediction Horizon. Figure 4 shows how well method performs for different prediction horizons. On Taobao Dataset, we experimented with horizon being 0.3, 0.8, 1.5, 2, corresponding to approximately 10, 20, 30 event tokens, and found that our HYPRO method improves significantly and consistently over the autoregressive baseline AttHHP.
Root Mean Square Emor
Enor
37.0
1.35
20.00
2.4
36.5
1.30
36.0
2.3
35.5
1.25
19.0
Root
1.20
(a) Using different energy functions on Taobao hyprom-m
hypro-ac. b-b
hypro-n-B
hypro-a-m
hypro-n-m
hypro-a-b
hypro-n n-b
hypro-a-m
hypro-a-4
hypro-n-
hypro-2-m m
hypro-n-m
hypro-ac b
hypro-ami amm
hypro-n-m
hypro
(b) Using different energy functions on Taxi Data
Figure Using different energy functions. each figure, the suffix -n-b and -n-m denote using continuous-time LSTM as the energy function trained by Binary-NCE and Multi-NCE, respectively.
Analysis-Vis VI: Different Energy Functions. So far we have only shown the results and analysis of using the continuous time Transformer architecture the energy function Eg. We also experimented with using a continuous-ting LSTM (Mei & Fisner, 2017) as the energy function and found that never outperformed the Transformer energy function in our experiments; see Figure We think

---

it is because the Transformer architectures are better at embedding contextual information than LSTMs (Vaswani et al., 2017: Pérez et al., 2019; O'Connor & Andreas, 2021).
Analysis-VIL VII: Negative Samples. On the Taohao dataset, we analyzed how the number of negative samples affects training and inference. We experimented with the Binary-NCI, objective without the distance regularization (i.e., hypro-a-b). The results are in Figure 6. During training, increasing the number of negative samples from to 5 has brought improvements but further increasing it to 10 does not. During inference, the results are improved when we increase the number of samples from 20, but they stop improving when we further increase it. Throughout the paper, we used 5 in training and 20 in inference (Appendix B.3).
Root Mean Mean Square Entor
Reof Mean Square Enror
Average OTD
36.5
36.5
2.40
2.40
2.35
35.5
2.35
20
35.5
samples
samples
samples
(a) Training.
(b) Inference.
Figure 6: Effects of the number of negative samples in training and inference the Taobao dataset.
6 Conclusion
We presented HYPRO, a hybridly normalized neural probabilistic model for the task of long-horizon prediction of event sequences. Our model consists of autoregressive base model and an energy function: the latter learns to reweight the sequences drawn from the former such that the sequences that appear more realistic as a whole can end up with higher probabilities under our full model. We developed two training objectives that can train our model without computing its normalizing constant, together with efficient inference algorithm based on normalized importance sampling. Empirically, our method outperformed current state-of-the-ar autoregressive models as well as recent non autorrrrgrree model designed specifically for the same task.
Limitations and Societal Impacts
Limitations. Our method uses neural networks, which typically data-hungry. Although it worked well in our experiments, might still suffer compared to non-neural models if starved of data. Additionally, method requires the training sequences be sufficiently long it can learn to make long-horizon predictions; may suffer if the training sequences are short.
Societal Impacts. Our paper develops novel probabilistic model for long-horizon prediction of event sequences. By describing the model and releasing code, we hope to facilitate probabilistic modeling of continuous-ting sequential data in many domains. However, like many other machine learning models, our model may be applied to unethical ends. For example, abilities of better fitting data and making more accurate predictions could potentially be used for unwanted tracking of individual behavior, e.g. for surveillance.
Acknowledgments
This work was supported by a research gift to the last author by Adobe Research. We thank the anonymous NeurlPS reviewers and meta-reviewer for their constructive feedback. We also thank colleagues at Ant Group for helpful discussion.
10


Reviews:
[{"review_id": "tlpJw6_eyfp", "reviewer": "Reviewer_WY8n", "summary": "This paper presents HYPRO, a novel model for long-horizon prediction of event sequences. HYPRO its a hybrid model, first an autoregressive model learns to propose predictions, and a second energy function weights them to produce more realistic final predictions. Two auto autoregressive models are used, NHP and AttNHP, and a continuous-time Transformer is used as the energy function. HYPRO can be trained with two losses: a binary-NCE objective and a multiple-NCE objective. Authors also propose a novel regularization to enforce larger distances between true and noise sequences. The paper compares HYPRO against baselines on two benchmark datasets. ", "questions": "- Why were other benchmark datasets not included in the experiments? The AttNHP paper [1] uses MIMIC-II, StackOverflow, RoboCup, and IPTV datasets, some of which have longer time between events which would be more adequate to test the proposed model on the long-horizon setting.\n- How does training and inference time compare against baselines? ", "limitations": "Limitations are societal impacts are addressed in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The hybrid approach proposed in the paper was not used before for the prediction of event sequences.\n- Authors propose a novel regularization technique to enforce larger distances between sequences.\n- Authors test the model with multiple training objectives, results confirm the superior properties of the multi-NCE objective.\n- HYPRO achieves better performance than baselines on two benchmark datasets.\n- The experimental details are explained in detail in the appendix. \n- The paper is well written and clear.\n\nWeaknesses:\n- The core idea of HYPRO was already studied in several previous works for text generation. The paper's novelty is limited to applying it to a different task and using different architectures.\n- The improvements of the novel regularization technique are not statistically significant in any of the datasets.\n- While the paper's primary focus is long-horizon predictions, the experiments on the prediction horizon are weak. The authors only include the analysis on one dataset. The longest horizon on the main results seems short (6 hours) compared to the motivation problems stated in the introduction (several months).\n- One of the motivations for the training objectives and inference method is the \"low computational cost\"; however, the experiments do not evaluate training times.\n- Limited benchmark datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Qiixk6Vyw3d", "reviewer": "Reviewer_urPE", "summary": "The paper proposes a hybrid approach using an autoregressive + energy based model to model event sequences. Prior approaches model event sequences data using just an autoregressive model. This paper trains an energy-based model to rank the samples from the autoregressive model. The sample with the highest weight across a number of samples is then used as the model output.\n\n2 different objective functions (Binary-NCE + Multi-NCE) are used to train the energy-based model. These functions assign lower energy to ground-truth sequences and higher energy to noise. An additional regularization that enforces a margin based on “edit distance” between the real and model outputs are also explored. The authors outperform prior works (NHP and AttNHP) with 2x parameters and DUALTPP on the Taobao and Taxi dataset on the RMSE and OTD metrics.\n", "questions": "Major Questions:\n* [Experiments:] In L207 and L 209, the authors train the transformer model on 1300 and 1440 sequences on the Taobao and Taxi dataset. Transformers are generally known to be data hungry, so this is a bit surprising that the model works well on such a small dataset. What are the tricks (if any) that the authors employ to get good results on such a small dataset?\n* [Experiments:] It is a bit intriguing that the 2x larger models in Figure 2a and Figure 2b get the same results as the smaller baselines. Do the authors have intuition on this? Does the performance on the training set saturate as well?\n* [Experiments:] Instead of training a separate energy based model to assign weights to the samples in Eq 6), a simple baseline is to assign weights using the likelihood of the autoregressive model itself. What are the results using this?\n* [Claims:] I appreciate that the authors are transparent in L121 and L183-L184 that similar approaches have been used in text-generation. I suggest that the authors be more upfront about their contributions in this paper. For example, the technical innovations in the introduction from L31 - L44 imply that this paper is the first to propose the hybrid model. In my opinion, It should be rephrased to say that this paper is the first to adapt the hybrid model to event sequence data with XYZ architectural + training modifications, since the hybrid model itself is not the main contribution of the paper.\n* [Claims:] L183 says that \"the architectures of our autoregressive model and energy function are different from theirs;\". What exactly are the architectural changes? What changes were made to get this model to work on event sequence data?\n* [Claims:] Is the importance sampling method the same as Deng et al or were changes made to this to adapt this model to event sequence data as well?\n\nSome more details are required to make the draft self-contained.\n* What exactly is the output distribution? Is $\\lambda_k$ a Gaussian distribution per k?\n* How is the 2nd term optimized in practice? Is the integral over time approximated with uniform sampling or a discrete grid?\n* The paper says from L73 - L78 that energy-based models give a probability normalized across all sequences. But so do, autoregressive models, since the exact density at a point is computable. Can the authors please clarify this? In that case, the motivation from L72 - L78 might be a bit handwavy.\n* How is the upper bound in Line 7 of Algorithm 1 computed in practice since the domain of $t$ is infinite, i.e $t \\in (t_0, \\inf)$?\n\nNot crucial but Good to have/know:\n* [Experiments:] The authors perform the experiments with 20 samples. Do the results get better if the number of samples are increased?\n* [Experiments:] The regularization term explicitly incorporates the edit distances in the loss. So the Figure 3a) and 3b) Right could be misleading since the test metric is used in optimization.\n* [Experiments:] The BinaryNCE objective has one positive sample and N negative samples per gradient update. What is the effect of this class imbalance?\n* If I understand correctly, the entire framework can be viewed as a GAN with the autoregressive model as the generator and the energy based model as a discriminator and a single round of updates. Do the authors have thoughts on this?\n* Should Eq2) be proportional to rather than equal? Even though the energy model and autoregressive model independenly output probabilities (that are normalized across all sequences), their product might not.\n\nMinor:\n* I think there is an Integral missing at L56.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The entire paper is well written and easy to read. The empirical results are extremely strong. \n\nI have a few questions on the experiments, claims and the draft (see below). I am currently borderline accept because of the strong emprical results. I am happy to adjust my rating if the authors can answer the makor questions below convincingly\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DslRFpJL2Yt", "reviewer": "Reviewer_XUS4", "summary": "The authors propose a two-stage generative model for event sequences, with an emphasis on event forecasting over (relatively) long horizons. The first stage is simply a standard autoregressive model for event data in continuous time. There is already a substantial literature on such models - both classical temporal point processes and their neural counterparts - but the authors argue that this autoregressive structure leads to compounding errors for long-horizon prediction. Therefore the authors propose a second stage that reweights the probability of an autoregressively generated sequence using an energy function intended to capture plausible *global* structure. The authors propose corresponding training and conditional generation methods for this model, and they compare its performance to some baselines on two multivariate event datasets.", "questions": "1. Can the authors provide some citations or evidence for compounding errors in long-horizon event forecasting?\n\n2. The details of the NCE training of the energy function lead me to wonder whether the second stage of the HYPRO model is really just enforcing an after-the-fact regularization of the learned autoregressive distribution towards the empirical distribution of the training data. Can the authors comment on whether this is a reasonable or fair characterization? Can they comment on how the NCE training is expected to result in an energy function that prefers globally coherent / plausible sequences?\n\n3. Can the authors address the third bullet point under \"Weaknesses\" - specifically, whether the baseline autoregressive models also use \"best-of-$M$\" decoding, and if not, why not? My main concern is that taking the best of $M$ autoregressive generations is an advantage on its own, for a potentially wide variety of choices as to how we define \"best,\" and it would be informative for readers to be able to compare the HYPRO method here (which uses the energy function to define \"best\") against some baseline.\n\nMinor / misc.: \n\n1. It seems like Alg. 1 is just the standard thinning algorithm for simulation of a non-homogeneous (multivariate) point process. Could the authors clarify if there's a novel contribution here?\n", "limitations": "The authors have adequately addressed limitations and potential impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n\n- This is a well-written paper. The motivation, background, method, and experimental analysis are informative and presented in a lucid style. I would rate this paper highly in terms of \"clarity.\"\n\n- The experiments are plausible and involve a thorough analysis of the various modeling degrees of freedom in the proposed approach, even those with modest or inconclusive results. The authors study the performance of the HYPRO method over multiple time horizons, quantify uncertainty in the results, and are able to show consistent improvement over their baselines. They provide code for all experiments.\n\nWeaknesses.\n\n- The authors repeatedly claim that autoregressive models fail at long-horizon prediction due to the problem of compounding / cascading errors. This claim seems plausible, but there is essentially no evidence provided to support it;  $\\S$2.2., for example, has no references. Given that the premise of this paper is to \"fix\" that problem, it would be useful to see a stronger argument or evidence here.\n\n- The energy function is trained by minimization of a noise-contrastive loss, where the \"noise\" is distribution is just that of the stage-one autoregressive model. This seems to encode the assumption that *every* generation of the autoregressive model is bad, a much stronger (and less plausible) claim than \"autoregressive generations are prone to cascading errors.\" It is not clear to me how the \"contrast\" established in this noise-contrastive loss could lead the energy function to learn some notion of global coherence or plausibility of sequences. \n\n- Generation from the HYPRO model involves selecting the \"best\" (according to the energy function-based weights) among $M$ generated sequences from the autoregressive model. This raises some questions for fair comparison to the autoregressive baselines. First, it's not clear whether this \"best-of-$M$\" decoding method is also extended to the baseline autoregressive models, or (assuming it is not) how much better these baselines would perform if it had been. Second, there don't appear to be any ablations that attempt to isolate the impact of using *specifically* the learned energy function to select among the $M$ best baselines vs. some baseline criterion.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept