SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoming Shi $^{1}$  Siqiao Xue $^{1}$  Kangrui Wang $^{3}$  Fan Zhou $^{1}$  James Y. Zhang $^{1}$  Jun Zhou $^{1}$  Chenhao Tan $^{2}$  Hongyuan Mei $^{3}$ $^{1}$ Ant Group  $^{2}$ UChicago  $^{3}$ TTIC
{peter.sxm,siqiao.xsq,hanlian.zf, james.z, jun.zhoujun}@antgroup.com chenhao@uchicago.edu {kangrui,hongyuan}@ttic.edu
Large language models have shown astonishing performance on a wide range of reasoning tasks. In this paper, we investigate whether they could reason about real-world events and help improve the prediction performance of event sequence models. We design LAMP, a framework that integrates a large language model in event prediction. Particularly, the language model performs abductive reasoning to assist an event sequence model: the event model proposes predictions on future events given the past; instructed by a few expert-annotated demonstrations, the language model learns to suggest possible causes for each proposal; a search module finds out the previous events that match the causes; a scoring function learns to examine whether the retrieved events could actually cause the proposal. Through extensive experiments on several challenging real-world datasets, we demonstrate that our framework—thanks to the reasoning capabilities of large language models—could significantly outperform the state-of-the-art event sequence models.
Prompting large language models (LLMs) such as GPT-3.5 has recently become a standard approach to perform text-based reasoning tasks. In this paper, we investigate their capabilities in reasoning about real-world events and improving event prediction. Particularly, we focus on the problem of modeling sequences of time-stamped events and predicting future events given the past. For example, in the healthcare domain, we would like to model patients' sequences of time-stamped hospital visits and predict their future symptoms given their past diagnosis and treatments. It has been a long-standing and important problem in machine learning. Large language models are potentially useful for advancing solutions to this problem because event sequences are often accompanied with rich text information which large language models excel at handling. For example,
- Healthcare. Each hospital visit will have a doctor note summarizing this visit, including the department that the patient visits, the clinical measurements and treatments, and any future medical plans. By reading such textual information, a large language model may be elicited to recall the medical knowledge that it has read during pretraining and then reason about the future hospital visits such as what symptoms or treatments that the patient may have.
- Political. Each political event may generate a series of news articles describing the political agents involved in it and discussing its possible influences. A language model reading these articles may recall its knowledge—which is acquired from pretraining—about these agents, their relations, and fundamental principles in politics such that it could reason about future political events.
- Similar scenarios arise in commercial, dialogue, finance, etc.
In this paper, we propose LAMP, a framework that integrates a large language model in event prediction. The overview of our framework is illustrated in Figure 1. Given a history of previous
Figure 1: An overview of our framework that leverages a large language model to reason about events. Firstly, an event sequence model proposes predictions: in this example, we predict the predicate of the structured event type given its time, subject, and object. Secondly, a language model suggests cause events, which will pattern-match against actual previous events and retrieve the most relevant. In the end, a neural model learns to assign high scores to the proposed predictions that are strongly supported by the retrieved evidence.
events, we use a pretrained event sequence model to propose predictions on the future events, which are then examined with the assistance of an LLM. The LLM learns to perform abductive reasoning: it is instructed by a few expert-annotated demonstrations, and generates possible causes that may explain the possible occurrence of each proposal. Each generated cause serves as a query to search for similar or relevant events that have actually happened. Then another neural model learns to embed these retrievals and examine whether they could really justify the corresponding proposal.
We are the first—to the best of our knowledge—to integrate large language models into event sequence modeling. Our modeling and prediction framework is general: it can incorporate all kinds of event sequence models and large language models. We experimented with a range of model choices and demonstrate that large language models could indeed help improve the prediction performance of event sequence models. On several challenging real-world datasets, our framework significantly outperforms the current state-of-the-art event sequence models.
2 Problem Formulation and Technical Background
Now we give a formal introduction to our problem setting and review the background knowledge.
Event sequence modeling. The problem is to model event sequences  $(t_{1},k_{1}),(t_{2},k_{2}),\ldots$  , where  $0 < t_1 < t_2 < \dots$  are times of occurrence and each  $k_{i}\in \mathcal{K}$  is a discrete event type. The goal is to predict the next event for a given history of events  $\mathcal{H}_i = (t_1,k_1),\ldots ,(t_{i - 1},k_{i - 1})$  . Precisely, it consists of two subtasks: the first is to predict the time  $t_i$  of the next event; the second is to predict the type  $k_{i}$  of the next event with the knowledge of its time  $t_i$
The standard approach is to build a probabilistic model over the sequences. Such models typically define an intensity function  $\lambda_{k}$ : the intensity value  $\lambda_{k}(t)$  is the instantaneous rate that an event of type  $k$  occurs at time  $t$ . Given the function  $\lambda_{k}$ , one could obtain the minimum Bayes risk (MBR) prediction of the next event given the history. Particularly, the MBR time prediction  $\hat{t}_i$  is
$$
\hat {t} _ {i} = \int_ {t _ {i - 1}} ^ {\infty} t \lambda (t) \exp \left(- \int_ {t _ {i - 1}} ^ {t} \lambda (s) d s\right) d t \text {w h e r e} \lambda (t) = \sum_ {k \in \mathcal {K}} \lambda_ {k} (t) \tag {1}
$$
and it could be approximated by averaging samples given by the thinning algorithm (Lewis & Shedler, 1979; Liniger, 2009). The MBR type prediction  $\hat{k}_i$  given time  $t_i$  is
$$
\hat {k} _ {i} = \operatorname {a r g m a x} _ {k} \lambda_ {k} \left(t _ {i}\right) \tag {2}
$$
The intensity function  $\lambda_{k}$  is typically learned by maximizing the log-likelihood of the model. For a time period  $(0,T)$  that contains observed events  $(t_1,k_1),\ldots ,(t_I,k_I)$ , the log-likelihood is
$$
\sum_ {i = 1} ^ {I} \log \lambda_ {k _ {i}} \left(t _ {i}\right) - \int_ {t = 0} ^ {T} \sum_ {k \in \mathcal {K}} \lambda_ {k} (t) d t \tag {3}
$$
Rich text information. In real-world data, each type  $k$  may be represented as a text-based identifier: in the example of Figure 1, each  $k$  is one of the possible interactions between the political entities (organizations and individuals) in the G20 countries, which can be represented with a structured name
such as Tesla-cooperate-Australia. In addition, each event may have a text mark m that contains additional information about the event: in Figure 1, each m is a news headline about the event (e.g., "EV battery prices go up"). For notation simplicity, we will only mention the mark m of an event when necessary. While reading such text information, a human may recall their relevant domain knowledge (e.g., influence of battery prices on Tesla and Australia) and increase their estimate on the probability that an event of Tesla-cooperate-Australia happens in the near future. An important way that humans learn such knowledge is reading text such as textbooks, research publications, and news articles. But event sequence models can not directly leverage this kind of information.
Large language models. Language models learn by reading text. Over the past years, large language models that have read nearly the entire internet have shown astonishing performance on many challenging tasks such as arithmetic reasoning and multi-turn dialogue (Wei et al., 2022b; OpenAI, 2023). So it seems tempting to pair a large language model with an event sequence model to improve its prediction performance: the language model has consumed a tremendous amount of information that the event model may not have seen but should be able to benefit from.
3 LAMP: Large Language Model in Event Prediction
Now we present our LAMP framework, in which an LLM is leveraged to enhance the prediction process of an event sequence model. As shown in Figure 1, LAMP has three key components:
- A base event sequence model. This model is pretrained and we use it to propose candidate predictions. Section 3.1 is the discussion of this phase.
- A large language model. Its duty in the framework is to perform abductive reasoning, a form of logical inference seeking the most plausible explanations for a given observation (Russell & Norvig, 2010). Particularly, the language model reads each proposed prediction and suggests possible cause events for it. Then we pattern-match each LLM-generated cause against the actual previous events in the history, and retrieve those which are most similar. Section 3.2 discusses this phase.
- A ranking model. The ranking model learns to examine each combination of the candidate prediction and its retrieved events—or, in other words, its evidence—and assign high scores to the candidates that are strongly supported by the retrieved evidence. Section 3.3 discusses this phase.
3.1 Phase-I: Proposing Predictions
Given a history of previous events  $\mathcal{H}_i = (t_1, k_1), \ldots, (t_{i-1}, k_{i-1})$ , the base event sequence model is used as a proposer to generate candidate predictions on the time and type of the next event.
For time prediction, we draw  $L$  i.i.d. samples  $\hat{t}_i^{(1)},\dots,\hat{t}_i^{(L)}$  from the base model via the thinning algorithm. If we were to only use this base model but not our LLM-enhanced framework, the final MBR time prediction would be the average of the samples, i.e.,  $\hat{t}_i = \frac{1}{L}\sum_{\ell = 1}^L\hat{t}_i^{(\ell)}$ . However, the MBR prediction may not be accurate since the base model is imperfect. Therefore, our LAMP framework treats all the  $L + 1$  samples—with  $\hat{t}_i^{(L + 1)}$  denoting the MBR prediction—as candidates, and utilize the LLM and ranking model to score them in later phases. If any of the  $L$  draws is actually a better prediction than the MBR estimate, our framework has a chance to rank it higher.
As we'll show shortly in sections 3.2 and 3.3, the LLM and ranking model work on full events. So we find the most probable  $M$  full events  $\{(\hat{t}_i^{(\ell)},\hat{k}_i^{(\ell,m)})\}_{m=1}^M$  for each time proposal  $\hat{t}_i^{(\ell)}$ , where  $\hat{k}_i^{(\ell,m)}$  is the event type that has the  $m$ -th highest intensity at time  $\hat{t}_i^{(\ell)}$ .
For type prediction given the ground-truth time  $t_i$ , we find  $M$  event types  $\hat{k}_i^{(1)},\dots,\hat{k}_i^{(M)}$  where  $\hat{k}_i^{(m)}$  has the  $m$ -th highest intensity at time  $t_i$  under the base model. If we were to only use this base model for prediction, the MBR type prediction would be the event type  $\hat{k}_i^{(1)}$  with the highest model intensity. However, our full framework will use the LLM and ranking model to examine each of the top  $M$  full events  $\{(t_i,\hat{k}_i^{(m)})\}_{m = 1}^M$  at time  $t_i$  in order to make a more informed prediction.
In practice, the event types often have structures and we may be interested in predicting an attribute of the structured type. Figure 1 shows an example in which we are trying to predict the predicate of the structured event type given its time, subject, and object. In such cases, we just need to select as proposals the most probable  $M$  event types whose other attributes are the same as the known information (e.g., Tesla and Australia in Figure 1).
I want you to do the reasoning over social events. I given you an effect event and you give me four or five cause events. An effect event is an event that happens. A cause event is believed to be one of the causes that have triggered the effect event to happen. Each event consists of a time, a type (that includes subject, predicate, object), and a news headline describing the event.
The predicates are restricted to the 20 options below.  
1. MAKE STATEMENT
// Full list are in Appendix E.4. 20. ENGAGE IN MASS VIOLENCE
Now I give you 10 examples. In each example, the first event is the effect and the next several events are the causes that happened earlier.
: // Examples are in Listing 2. Now please generate possible causes for
effect predicate: CONSULT time: 2022-07-05 subject: CHINA PM object: YELLEN
Listing 1: Format of our LLM prompt.
predicate: APPEAL
time: 2022-04-23
subject:GERMANY
object: GREEN PROJECT
reasoning:
cause event 1
predicate:REDUCE RELATIONS
time: 2022-04-21
subject: EUROPE
object: RUSSIA
headline: Europe determined to ban Russian energy exports.
cause event 2
predicate: DISAPPROVE
time: 2022-03-16
object: RUSSIAN
headline: Europe can endure painful
transition to live without Russian oil.
// Other causes are in Appendix E.4.
// Other examples in Appendix E.4.
Listing 2: Few-shot examples in our prompt.
The  $L$  and  $M$  are hyperparameters. Ideally, we would like to analyze all the possible candidates (i.e.,  $L = \infty$  and  $M = |\mathcal{K}|$ ), which is intractable for time prediction and expensive for type prediction. In practice, we focus on the most plausible candidates to maintain a low computation cost. In our experiments, our framework already performs remarkably well with small  $L$  and  $M$ .
3.2 Phase-II: Prompting LLM to Perform Abductive Reasoning
For each proposed event  $(t,k)$ , our framework selects a set of previous events from its full history as its supporting evidence  $\mathbf{e}(t,k)$ . The selection is guided by an LLM (e.g., GPT-3.5). Technically, we prompt the LLM to imagine some possible cause events that—under the LLM's belief—would be able to explain the occurrence of this proposal. The imaginary cause events may not exactly match any actual event in the history, but we could use them as queries to search for the most similar ones.
Prompting is a widely used technique to extract knowledge from an LLM. A prompt is a concise statement designed to elicit a response from the LLM. It typically includes the task description and a few demonstrations. Our prompt follows the format in Listings 1 and 2.
Each LLM-generated cause event is used as a query to search for  $D$  most similar events in the history, where  $D > 0$  is a hyperparameter. The overall evidence  $\mathbf{e}(t,k)$  is then defined to be the union of the previous events retrieved by the LLM-generated causes. Retrieval is vector-based: we construct a query embedding  $\mathbf{v}_{\mathbf{q}}$  for the query event, and a key embedding  $\mathbf{v}_{\mathbf{k}}$  for each actual previous event; the similarity is measured by cosine between  $\mathbf{v}_{\mathbf{q}}$  and  $\mathbf{v}_{\mathbf{k}}$ , i.e.,  $\frac{\mathbf{v}_{\mathbf{q}}^{\top}\mathbf{v}_{\mathbf{k}}}{\|\mathbf{v}_{\mathbf{q}}\|_{1}\|\mathbf{v}_{\mathbf{k}}\|}$ . We use a pretrained SentenceBERT (SBERT) (Reimers & Gurevych, 2019) as our embedding model. The model takes as input a text string concatenating the event time  $t$ , the text identifier (e.g., subject-predicate-object) of the event type  $k$ , and—if any—the textual mark  $m$  of the event. It returns an embedding for each token, and we take the event embedding  $\mathbf{v}$  to be the average of the SBERT embeddings.
3.3 Phase-III: Ranking Proposals
In this phase, our framework scores each proposed event  $(t,k)$  based on the compatibility with its retrieved evidence  $\mathbf{e}(t,k)$ . Precisely, the score is defined to be
$$
s _ {\text {e v e n t}} (t, k) \stackrel {\text {d e f}} {=} \exp \left(c \left(\left(t, k\right), \mathbf {e} (t, k)\right)\right) \tag {4}
$$
The function  $c$  takes as input the proposed event  $(t, k)$  as well as its evidence  $\mathbf{e}(t, k)$ , and returns a scalar  $\in \mathbb{R}$ . A high value of  $c$  means that this proposal is strongly supported by its retrieved evidence, and thus is more likely to be an actual event at time  $t$ ; a low value means that this proposal has no strong evidence even after we have tried our best to search from the history.
Given the most probable  $M$  events  $\{(t, k^{(m)})\}_{m=1}^{M}$  at time  $t$ , we sum their  $s_{\text{event}}$  scores to measure the overall belief of our framework in an event occurring at time  $t$ . That is,
$$
s _ {\text {t i m e}} (t) \stackrel {\text {d e f}} {=} \sum_ {m = 1} ^ {M} s _ {\text {e v e n t}} (t, k ^ {(m)}) \tag {5}
$$
Intuitively, this score is high when any of the top-ranked event types at this time can be strongly supported by the retrieved evidence. Otherwise, even the top-ranked event types have no strong evidence in the history, which implies that the next event is unlikely to occur at this time.
For time prediction, each proposed time  $\hat{t}^{(\ell)}$  ( $\ell = 1, \dots, L + 1$ ) has a score  $s_{\mathrm{time}}(\hat{t}^{(\ell)})$ —more precisely,  $\sum_{m=1}^{M} s_{\mathrm{event}}(\hat{t}^{(\ell)}, \hat{k}^{(\ell, m)})$  and our final prediction is the proposal with the highest score.
For type prediction given time  $t$ , each proposed type  $\hat{k}^{(m)}$  ( $m = 1, \dots, M$ ) has a score  $s_{\mathrm{event}}(t, \hat{k}^{(m)})$ , and our framework takes the final prediction to be the type with the highest score.
Model architecture. Our function  $c$  is an energy function with a continuous-time Transformer architecture (Xue et al., 2022). It reads the proposal  $(t, k)$  followed by its evidence events in the chronological order, and returns a compatibility score  $\in \mathbb{R}$ .
We choose this architecture because its continuous-time attention is suitable for our setting. First, the attention mechanism may learn to disregard any of the retrieved events that are not really relevant and focus on those which can actually support the proposal. Second, its sophisticated handling of time may capture how the time of an evidence event may influence its relevance to the proposal (e.g., a recent evidence event may be more important than an ancient event).
Training. We train the ranking model by maximizing the objective  $J \stackrel{\mathrm{def}}{=} J_{\mathrm{actual}} + \beta J_{\mathrm{no}}$  where  $\beta \geq 0$  is a hyperparameter. The first term  $J_{\mathrm{actual}}$  is defined to be
$$
J _ {\text {a c t u a l}} \stackrel {\text {d e f}} {=} \sum_ {i = 1} ^ {I} \left(\log s _ {\text {e v e n t}} \left(t _ {i}, k _ {i}\right) - \log \left(s _ {\text {e v e n t}} \left(t _ {i}, k _ {i}\right) + \sum_ {m = 1} ^ {M} s _ {\text {e v e n t}} \left(t _ {i}, k _ {i} ^ {(m)}\right)\right)\right) \tag {6}
$$
where  $(t_1, k_1), \ldots, (t_I, k_I)$  is a sequence of events over the time interval  $(0, T)$ , and each  $k_i^{(m)}$  is the event type with the  $m$ -th highest intensity under the base model at time  $t_i$ . By maximizing  $J_{\mathrm{actual}}$ , the function  $c$  learns to increase the scores of the events that have actually happened, but suppress the scores of the non-events at times  $t_1, \ldots, t_I$ . The second term  $J_{\mathrm{no}}$  is defined to be
$$
J _ {\mathrm {n o}} \stackrel {\text {d e f}} {=} - \sum_ {n = 1} ^ {N} \log s _ {\text {t i m e}} \left(t _ {n}\right) = - \sum_ {n = 1} ^ {N} \log \sum_ {m = 1} ^ {M} s _ {\text {e v e n t}} \left(t _ {n}, k _ {n} ^ {(m)}\right) \tag {7}
$$
where each  $t_n$  is a time point uniformly sampled from  $(0, T)$  and each  $k_n^{(m)}$  is the event type with the  $m$ -th highest intensity at time  $t_n$ . By maximizing  $J_{\mathrm{no}}$ , the function  $c$  learns to decrease the scores of the non-events at times over  $(0, T)$  other than  $t_1, \ldots, t_I$ .
Although not explicitly mentioned in equations (6) and (7), computing  $J_{\mathrm{actual}}$  and  $J_{\mathrm{no}}$  involves searching for the evidence of the actual and proposed events.
Our code is at https://github.com/iLampard/lamp. This repository includes CSV files containing numerical results of our experiments. It also includes qualitative results such as LLM-generated cause events. Experiment details (e.g., hyperparameters) are in Appendix E.
We conducted experiments on three real-world datasets (see Appendix E.1 for dataset details).
GDELT (Leetaru & Schrodt, 2013). The GDELT Project monitors events all over the world, with live datasets updated every 15 minutes. We only focused on the political events that happened in G20 countries from 2022-01-01 to 2022-07-31, ending up with a corpus of 109000 time-stamped event tokens. This choice of time range guarantees that our data was not included in the training data of the most recent GPT. The event type  $k$  of each token has a structured name of the format subject-predicate-object. Each predicate is one of the twenty CAMEO codes such as CONSULT and INVESTIGATE (see Appendix E.4 for a full list); each subject or object is one of the 2279 political entities (individuals, groups, and states) such as Tesla and Australia. So there are about 104M event types in total, making this dataset extremely challenging. Each event token has a news headline that concisely describes the event. We split the dataset into disjoint train, dev, and test sets based on their dates: the 83100 events that happened before 2022-07-05 are training data; the 16650 events after 2022-07-19 are test data; the 9250 events between these dates are development data.
ICEWS (Boschee et al., 2015). Similar to GDELT, this dataset logs interactions between social-political entities. We collected 79410 event tokens from 2022-10-11 to 2023-02-28. Its event types have the same structure as GDELT: each predicate is one of the twenty CAMEO codes; each subject or object is one of the 2981 political entities. We split the dataset into disjoint train, dev, and test sets based on their dates: the 41600 events that happened before 2023-01-16 are training data; the 22030 events after 2023-02-01 are test data; the 15780 events between these dates are development data.
Amazon Review (Jianmo Ni, 2019). This dataset contains user reviews on Amazon shopping website from 2014-01-04 to 2016-10-02. We focused on the most active 2500 users and each user has a sequence of product review events. The type  $k$  is the category of the product: we selected the most frequently-reviewed 23 categories and grouped all the others into a special OTHER category, ending up with 24 categories in total. Each review event also has a mark  $m$  which is the actual content of the review. Each of the 2500 sequences is cut into three segments: the events that happened before 2015-08-01 are training data; those after 2016-02-01 are test data; the events between these dates are dev data. Then we have 49,680 training tokens, 7,020 dev tokens, and 13,090 test tokens.
We experimented with four state-of-the-art event sequence models: NHP (Mei & Eisner, 2017a), Know-Evovle (KE) (Trivedi et al., 2017), DyRep (Trivedi et al., 2019), and ANHP (Yang et al., 2022). For each of them, we evaluated it as a baseline method as well as integrated it into our LAMP framework. KE and DyRep require domain-specific knowledge to configure their structural-sparse architectures: we evaluated them on GDELT since their GDELT-specific architectures are available in the original papers; we didn't evaluate them on Amazon Review since we do not have such knowledge on this data. ANHP can take domain knowledge into its architecture but it is optional, so we evaluated it on both GDELT and Amazon data: on GDELT, we adapt the knowledge used in KE and DyRep into its structure; on Amazon Review, we use the generic architecture. On Amazon Review, we also experimented with NHP since it doesn't require any domain-specific structure knowledge.