Title: Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning
Venue: NeurIPS 2023
Accepted: yes
Keywords: event sequences

irregular time series

event prediction

large language model

reasoning

few-shot prompting
Abstract: Large language models have shown astonishing performance on a wide range of reasoning tasks. In this paper, we investigate whether they could reason about real-world events and help improve the prediction performance of event sequence models. We design LAMP, a framework that integrates a large language model in event prediction. Particularly, the language model performs abductive reasoning to assist an event sequence model: the event model proposes predictions on future events given the past; instructed by a few expert-annotated demonstrations, the language model learns to suggest possible causes for each proposal; a search module finds out the previous events that match the causes; a scoring function learns to examine whether the retrieved events could actually cause the proposal. Through extensive experiments on several challenging real-world datasets, we demonstrate that our framework---thanks to the reasoning capabilities of large language models---could significantly outperform the state-of-the-art event sequence models.

Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning

Abstract

Large language models have shown astonishing performance on a wide range of reasoning tasks. In this paper, we investigate whether they could reason about real-world events and help improve the prediction performance of event sequence models. We design LAMP, a framework that integrates a large language model in event prediction. Particularly, the language model performs abductive reasoning to assist an event sequence model: the event model proposes predictions on future events given the past; instructed by a few expert-annotated demonstrations, the language model learns to suggest possible causes for each proposal; a search module finds out the previous events that match the causes; a scoring function learns to examine whether the retrieved events could actually cause the proposal. Through extensive experiments on several challenging real-world datasets, we demonstrate that our framework-thanks to the reasoning capabilities of large language modelscould significantly outperform the state-of-the-art event sequence models.

1 Introduction

Prompting large language models (LLMs) such as GPT-3.5 has recently become a standard approach to perform text-based reasoning tasks. In this paper, we investigate their capabilities in reasoning about real-world events and improving event prediction. Particularly, we focus on the problem of modeling sequences of time-stamped events and predicting future events given the past. For example, in the healthcare domain, we would like to model patients' sequences of time-stamped hospital visits and predict their future symptoms given their past diagnosis and treatments. It has been a long-standing and important problem in machine learning. Large language models are potentially useful for advancing solutions to this problem because event sequences are often accompanied with rich text information which large language models excel at handling. For example,

Healthcare. Each hospital visit will have a doctor note summarizing this visit, including the department that the patient visits, the clinical measurements and treatments, and any future medical plans. By reading such textual information, a large language model may be elicited to recall the medical knowledge that it has read during pretraining and then reason about the future hospital visits such as what symptoms or treatments that the patient may have.

Political. Each political event may generate a series of news articles describing the political agents involved in it and discussing its possible influences. A language model reading these articles may recall its knowledge-which is acquired from pretraining-about these agents, their relations, and fundamental principles in politics such that it could reason about future political events.

Similar scenarios arise in commercial , dialogue , finance , etc.

In this paper, we propose LAMP, a framework that integrates a large language model in event prediction. The overview of our framework is illustrated in Figure 1. Given a history of previous

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

Subject: TESLA

Object: EUROPE

Predicate:

COOPERATE

Date, News …

Subject: COMPANY

Object: CHINA

Predicate:

MAKE STMT

Date, News …

…

Subject: FDA

Object: AMERICAN

Predicate

: YIELD

Date, News …

History

Time

Subject: TESLA

Object: GERMANY

Predicate:

CONSULT

Date, News …

Subject: TESLA

Object: CHINA

Predicate:

MAKE STMT

Date, News …

…

Generated cause events

Retrieve relevant events

Subject: BYD

Object: INDUSTRY

Predicate:

APPEAL

Date, News …

Prompting

Event Sequence Model

Subject:

TESLA

Object:

EUROPE

Predicate:

COOPERATE

Date: 2022-01-01

News

: Tesla model

3 is Europe's

best-selling EV in

2021.

Subject:

COMPANY

Object:

CHINA

Predicate:

MAKE STMT

Date: 2022-01-05

News: EV battery

prices go up

facing Lithium

shortage.

Subject:

FDA

Object:

AMERICAN

Predicate

:

YIELD

Date: 2022-01-03

News: FDA

reportedly close to

authorizing third

Pfizer shot for kids.

TESLA

AUSTRALIA

2022-01-16

Predicate?

Time

History

Future

…

Subject: TESLA

Object: AUSTRALIA

Predicate:

COOPERATE

Date:2022-01-16

Subject: TESLA

Object: AUSTRALIA

Predicate:

FIGHT

Date:2022-01-16

Subject: TESLA

Object: AUSTRALIA

Predicate:

REJECT

Date:2022-01-16

Subject: TESLA

Object: AUSTRALIA

Predicate:

COOPERATE

Date:2022-01-16

Subject: TESLA

Object: AUSTRALIA

Predicate:

FIGHT

Date:2022-01-16

Subject: TESLA

Object: AUSTRALIA

Predicate:

REJECT

Date:2022-01-16

…

1

3

2

events, we use a pretrained event sequence model to propose predictions on the future events, which are then examined with the assistance of an LLM. The LLM learns to perform abductive reasoning: it is instructed by a few expert-annotated demonstrations, and generates possible causes that may explain the possible occurrence of each proposal. Each generated cause serves as a query to search for similar or relevant events that have actually happened. Then another neural model learns to embed these retrievals and examine whether they could really justify the corresponding proposal.

Weare the first-to the best of our knowledge-to integrate large language models into event sequence modeling. Our modeling and prediction framework is general: it can incorporate all kinds of event sequence models and large language models. We experimented with a range of model choices and demonstrate that large language models could indeed help improve the prediction performance of event sequence models. On several challenging real-world datasets, our framework significantly outperforms the current state-of-the-art event sequence models.

2 Problem Formulation and Technical Background

Now we give a formal introduction to our problem setting and review the background knowledge.

Event sequence modeling. The problem is to model event sequences ( t 1 , k 1 ) , ( t 2 , k 2 ) , . . . , where 0 < t 1 < t 2 < . . . are times of occurrence and each k i ∈ K is a discrete event type. The goal is to predict the next event for a given history of events H i = ( t 1 , k 1 ) , . . . , ( t i -1 , k i -1 ) . Precisely, it consists of two subtasks: the first is to predict the time t i of the next event; the second is to predict the type k i of the next event with the knowledge of its time t i .

The standard approach is to build a probabilistic model over the sequences. Such models typically define an intensity function λ k : the intensity value λ k ( t ) is the instantaneous rate that an event of type k occurs at time t . Given the function λ k , one could obtain the minimum Bayes risk (MBR) prediction of the next event given the history. Particularly, the MBR time prediction ˆ t i is

and it could be approximated by averaging samples given by the thinning algorithm (Lewis & Shedler, 1979; Liniger, 2009). The MBR type prediction ˆ k i given time t i is

The intensity function λ k is typically learned by maximizing the log-likelihood of the model. For a time period (0 , T ) that contains observed events ( t 1 , k 1 ) , . . . , ( t I , k I ) , the log-likelihood is

Rich text information. In real-world data, each type k may be represented as a text-based identifier: in the example of Figure 1, each k is one of the possible interactions between the political entities (organizations and individuals) in the G20 countries, which can be represented with a structured name

such as Tesla-cooperate-Australia. In addition, each event may have a text mark m that contains additional information about the event: in Figure 1, each m is a news headline about the event (e.g., 'EV battery prices go up'). For notation simplicity, we will only mention the mark m of an event when necessary. While reading such text information, a human may recall their relevant domain knowledge (e.g., influence of battery prices on Tesla and Australia) and increase their estimate on the probability that an event of Tesla-cooperate-Australia happens in the near future. An important way that humans learn such knowledge is reading text such as textbooks, research publications, and news articles. But event sequence models can not directly leverage this kind of information.

Large language models. Language models learn by reading text. Over the past years, large language models that have read nearly the entire internet have shown astonishing performance on many challenging tasks such as arithmetic reasoning and multi-turn dialogue (Wei et al., 2022b; OpenAI, 2023). So it seems tempting to pair a large language model with an event sequence model to improve its prediction performance: the language model has consumed a tremendous amount of information that the event model may not have seen but should be able to benefit from.

3 LAMP: Large Language Model in Event Prediction

Now we present our LAMP framework, in which an LLM is leveraged to enhance the prediction process of an event sequence model. As shown in Figure 1, LAMP has three key components:

A base event sequence model. This model is pretrained and we use it to propose candidate predictions. Section 3.1 is the discussion of this phase.

A large language model. Its duty in the framework is to perform abductive reasoning , a form of logical inference seeking the most plausible explanations for a given observation (Russell & Norvig, 2010). Particularly, the language model reads each proposed prediction and suggests possible cause events for it. Then we pattern-match each LLM-generated cause against the actual previous events in the history, and retrieve those which are most similar. Section 3.2 discusses this phase.

A ranking model. The ranking model learns to examine each combination of the candidate prediction and its retrieved events-or, in other words, its evidence -and assign high scores to the candidates that are strongly supported by the retrieved evidence. Section 3.3 discusses this phase.

3.1 Phase-I: Proposing Predictions

Given a history of previous events H i = ( t 1 , k 1 ) , . . . , ( t i -1 , k i -1 ) , the base event sequence model is used as a proposer to generate candidate predictions on the time and type of the next event.

For time prediction, we draw L i.i.d. samples ˆ t (1) i , . . . , ˆ t ( L ) i from the base model via the thinning algorithm. If we were to only use this base model but not our LLM-enhanced framework, the final MBR time prediction would be the average of the samples, i.e., ˆ t i = 1 L ∑ L ℓ =1 ˆ t ( ℓ ) i . However, the MBR prediction may not be accurate since the base model is imperfect. Therefore, our LAMP framework treats all the L +1 samples-with ˆ t ( L +1) i denoting the MBR prediction-as candidates, and utilize the LLM and ranking model to score them in later phases. If any of the L draws is actually a better prediction than the MBR estimate, our framework has a chance to rank it higher.

As we'll show shortly in sections 3.2 and 3.3, the LLM and ranking model work on full events. So we find the most probable M full events { ( ˆ t ( ℓ ) i , ˆ k ( ℓ,m ) i ) } M m =1 for each time proposoal ˆ t ( ℓ ) i , where ˆ k ( ℓ,m ) i is the event type that has the m -th highest intensity at time ˆ t ( ℓ ) i .

For type prediction given the ground-truth time t i , we find M event types ˆ k (1) i , . . . , ˆ k ( M ) i where ˆ k ( m ) i has the m -th highest intensity at time t i under the base model. If we were to only use this base model for prediction, the MBR type prediction would be the event type ˆ k (1) i with the highest model intensity. However, our full framework will use the LLM and ranking model to examine each of the top M full events { ( t i , ˆ k ( m ) i ) } M m =1 at time t i in order to make a more informed prediction.

In practice, the event types often have structures and we may be interested in predicting an attribute of the structured type. Figure 1 shows an example in which we are trying to predict the predicate of the structured event type given its time, subject, and object. In such cases, we just need to select as proposals the most probable M event types whose other attributes are the same as the known information (e.g., Tesla and Australia in Figure 1).

The L and M are hyperparameters. Ideally, we would like to analyze all the possible candidates (i.e., L = ∞ and M = |K| ), which is intractable for time prediction and expensive for type prediction. In practice, we focus on the most plausible candidates to maintain a low computation cost. In our experiments, our framework already performs remarkably well with small L and M .

3.2 Phase-II: Prompting LLM to Perform Abductive Reasoning

For each proposed event ( t, k ) , our framework selects a set of previous events from its full history as its supporting evidence e ( t, k ) . The selection is guided by an LLM (e.g., GPT-3.5). Technically, we prompt the LLM to imagine some possible cause events that-under the LLM's belief-would be able to explain the occurrence of this proposal. The imaginary cause events may not exactly match any actual event in the history, but we could use them as queries to search for the most similar ones.

Prompting is a widely used technique to extract knowledge from an LLM. A prompt is a concise statement designed to elicit a response from the LLM. It typically includes the task description and a few demonstrations. Our prompt follows the format in Listings 1 and 2.

Each LLM-generated cause event is used as a query to search for D most similar events in the history, where D > 0 is a hyperparameter. The overall evidence e ( t, k ) is then defined to be the union of the previous events retrieved by the LLM-generated causes. Retrieval is vector-based: we construct a query embedding v q for the query event, and a key embedding v k for each actual previous

⊤

event; the similarity is measured by cosine between v q and v k, i.e., v q v k ∥ v q ∥∥ v k ∥ . We use a pretrained SentenceBERT (SBERT) (Reimers & Gurevych, 2019) as our embedding model. The model takes as input a text string concatenating the event time t , the text identifier (e.g., subject-predicate-object) of the event type k , and-if any-the textual mark m of the event. It returns an embedding for each token, and we take the event embedding v to be the average of the SBERT embeddings.

3.3 Phase-III: Ranking Proposals

In this phase, our framework scores each proposed event ( t, k ) based on the compatibility with its retrieved evidence e ( t, k ) . Precisely, the score is defined to be

The function c takes as input the proposed event ( t, k ) as well as its evidence e ( t, k ) , and returns a scalar ∈ R . A high value of c means that this proposal is strongly supported by its retrieved evidence, and thus is more likely to be an actual event at time t ; a low value means that this proposal has no strong evidence even after we have tried our best to search from the history.

Given the most probable M events { ( t, k ( m ) ) } M m =1 at time t , we sum their s event scores to measure the overall belief of our framework in an event occurring at time t . That is,

Intuitively, this score is high when any of the top-ranked event types at this time can be strongly supported by the retrieved evidence. Otherwise, even the top-ranked event types have no strong evidence in the history, which implies that the next event is unlikely to occur at this time.

For time prediction, each proposed time ˆ t ( ℓ ) ( ℓ = 1 , . . . , L +1 ) has a score s time ( ˆ t ( ℓ ) ) -more precisely, ∑ M m =1 s event ( ˆ t ( ℓ ) , ˆ k ( ℓ,m ) ) -and our final prediction is the proposal with the highest score.

For type prediction given time t , each proposed type ˆ k ( m ) ( m = 1 , . . . , M ) has a score s event ( t, ˆ k ( m ) ) , and our framework takes the final prediction to be the type with the highest score.

Model architecture. Our function c is an energy function with a continuous-time Transformer architecture (Xue et al., 2022). It reads the proposal ( t, k ) followed by its evidence events in the chronological order, and returns a compatibility score ∈ R .

We choose this architecture because its continuous-time attention is suitable for our setting. First, the attention mechanism may learn to disregard any of the retrieved events that are not really relevant and focus on those which can actually support the proposal. Second, its sophisticated handling of time may capture how the time of an evidence event may influence its relevance to the proposal (e.g., a recent evidence event may be more important than an ancient event).

Training. We train the ranking model by maximizing the objective J def = J actual + βJ no where β ≥ 0 is a hyperparameter. The first term J actual is defined to be

where ( t 1 , k 1 ) , . . . , ( t I , k I ) is a sequence of events over the time interval (0 , T ) , and each k ( m ) i is the event type with the m -th highest intensity under the base model at time t i . By maximizing J actual , the function c learns to increase the scores of the events that have actually happened, but suppress the scores of the non-events at times t 1 , . . . , t I . The second term J no is defined to be

where each t n is a time point uniformly sampled from (0 , T ) and each k ( m ) n is the event type with the m -th highest intensity at time t n . By maximizing J no, the function c learns to decrease the scores of the non-events at times over (0 , T ) other than t 1 , . . . , t I .

Although not explicitly mentioned in equations (6) and (7), computing J actual and J no involves searching for the evidence of the actual and proposed events.

4 Experiments

Our code is at https://github.com/iLampard/lamp . This repository includes CSV files containing numerical results of our experiments. It also includes qualitative results such as LLMgenerated cause events. Experiment details (e.g., hyperparameters) are in Appendix E.

4.1 Experimental Setup

We conducted experiments on three real-world datasets (see Appendix E.1 for dataset details).

GDELT (Leetaru & Schrodt, 2013) . The GDELT Project monitors events all over the world, with live datasets updated every 15 minutes. We only focused on the political events that happened in G20 countries from 2022-01-01 to 2022-07-31, ending up with a corpus of 109000 time-stamped event tokens. This choice of time range guarantees that our data was not included in the training data of the most recent GPT. The event type k of each token has a structured name of the format subject-predicate-object. Each predicate is one of the twenty CAMEO codes such as CONSULT and INVESTIGATE (see Appendix E.4 for a full list); each subject or object is one of the 2279 political entities (individuals, groups, and states) such as Tesla and Australia. So there are about 104M event types in total, making this dataset extremely challenging. Each event token has a news headline that concisely describes the event. We split the dataset into disjoint train, dev, and test sets based on their dates: the 83100 events that happened before 2022-07-05 are training data; the 16650 events after 2022-07-19 are test data; the 9250 events between these dates are development data.

ICEWS (Boschee et al., 2015) . Similar to GDELT, this dataset logs interactions between socialpolitical entities. We collected 79410 event tokens from 2022-10-11 to 2023-02-28. Its event types have the same structure as GDELT: each predicate is one of the twenty CAMEO codes; each subject or object is one of the 2981 political entities. We split the dataset into disjoint train, dev, and test sets based on their dates: the 41600 events that happened before 2023-01-16 are training data; the 22030 events after 2023-02-01 are test data; the 15780 events between these dates are development data.

Amazon Review (Jianmo Ni, 2019) . This dataset contains user reviews on Amazon shopping website from 2014-01-04 to 2016-10-02. We focused on the most active 2500 users and each user has a sequence of product review events. The type k is the category of the product: we selected the most frequently-reviewed 23 categories and grouped all the others into a special OTHER category, ending up with 24 categories in total. Each review event also has a mark m which is the actual content of the review. Each of the 2500 sequences is cut into three segments: the events that happened before 2015-08-01 are training data; those after 2016-02-01 are test data; the events between these dates are dev data. Then we have 49,680 training tokens, 7,020 dev tokens, and 13,090 test tokens.

We experimented with four state-of-the-art event sequence models: NHP (Mei & Eisner, 2017a), Know-Evovle (KE) (Trivedi et al., 2017), DyRep (Trivedi et al., 2019), and ANHP (Yang et al., 2022). For each of them, we evaluated it as a baseline method as well as integrated it into our LAMP framework. KE and DyRep require domain-specific knowledge to configure their structural-sparse architectures: we evaluated them on GDELT since their GDELT-specific architectures are available in the original papers; we didn't evaluate them on Amazon Review since we do not have such knowledge on this data. ANHP can take domain knowledge into its architecture but it is optional, so we evaluated it on both GDELT and Amazon data: on GDELT, we adapt the knowledge used in KE and DyRep into its structure; on Amazon Review, we use the generic architecture. On Amazon Review, we also experimented with NHP since it doesn't require any domain-specific structure knowledge.

We experimented with three strong LLMs: GPT-3-davinci (Brown et al., 2020) which we also denote as G3.0; GPT-3.5-turbo (Brown et al., 2020; Stiennon et al., 2020; Gao et al., 2022) which we also denote as G3.5, and Llama-2-chat with 13B parameters (Touvron et al., 2023) which we also denote as llama. For GDELT and ICEWS data, we used 10-shot prompts; for Amazon Review data, we used 8-shot prompts. Each 'shot' is a demonstration that contains an effect event followed by one or more expert-annotated cause events. Prompt examples can be found in Appendix E.4.

4.2 Main Results on Type and Time Prediction

Our main results are displayed in Figure 2. Figure 2a shows the result of each method on GDELT data. GDELT data is updated every fifteen minutes so the time intervals are regular and thus it is not interesting to predict them. For type prediction, we focus on predicting certain attributes given the others, which is more practical than full type prediction. In practice, predicting 'which of the hundreds of millions of events is the most probable' is too difficult and existing models will all perform disastrously. But answering questions like 'what will A do to B' and 'to whom A will do this' is usually useful enough for real applications. Note that attribute prediction is still very challenging: e.g., there are 45580 distinct predicate-object combinations in GDELT data.

2

5

10

15

20

30

40

50

M

0

5

10

15

20

25

Mean Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

10

25

50

75

100

M

10

20

30

40

Mean Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

(a) GDELT.

10

25

50

75

100

M

0

20

40

60

80

100

Mean Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

2

3

4

5

L

1.5

1.6

1.7

1.8

1.9

2.0

RMSE

×10 1

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

(b) ICEWS.

2

3

4

5

M

1.00

1.25

1.50

1.75

2.00

2.25

Mean Rank

NHP

ANHP

NHP-G3.5

ANHP-G3.5

2

3

4

5

L

1.25

1.30

1.35

1.40

1.45

1.50

RMSE

×10 2

NHP

ANHP

NHP-G3.5

ANHP-G3.5

(c) Amazon Review.

We evaluate each model by the quality of its top-ranked predictions. For each baseline model (KE, DyRep, or ANHP), the list of top predictions contains the top M event types (with known attributes filled, if any) that have the highest intensities; see section 3.1. Our LLM-enhanced framework takes the list given by its base model, and sorts it based on the s event scores of the proposals. Our primary evaluation metric is the mean rank (MR). This metric has a straightforward interpretation: given a sorted list of proposed predictions, it measures the average rank of the ground-truth type in the list; a smaller MR means a higher rank, and thus a better result (e.g., MR = 1 means 'ranked at the top on average'). We also used the mean reciprocal rank (MRR), which is less interpretable but more robust to bad predictions than MR. Appendix E.2 includes full procedures for computing these metrics. In our experiments, MR and MRR results yield the same conclusions. So we present MR results in this section for its straightforward interpretation, but leave MRR results to Appendix F.

On ICEWS and Amazon Review, we evaluate each model on time prediction (in addition to type prediction), which is measured by the root of mean squared error (RMSE). For each held-out token, each base model proposes a list of scored predictions; see section 3.1. Our LAMP framework reranks the list given by its base model. In either case, the final prediction is the highest-ranked proposal.

In each evaluation setting shown in Figure 2, our LLM-enhanced framework substantially and consistently outperforms the corresponding baseline model across a range of L and M values. All the results throughout the paper have 95% bootstrap confidence intervals but they are too tight to be visually obvious in most cases, implying that our improvements are significant. When we draw more proposals from the base model (i.e., L and M are larger), our framework tends to enjoy a larger improvement over the base model. For predicate-object joint prediction on GDELT, the ground-truth type ranks at around 40 in the lists given by the base models, but is moved to around 20 by our LLM-based adjustment. For object prediction on ICEWS, our method improved the MR results from around 90 to about 20. Note that it is not fair to compare the same method across L or M .

The MR and MRR are only evaluated on the held-out tokens whose ground-truth event types fall into the top proposals. But how many are there such tokens? This solely depends on how well the base event sequence model works. A detailed analysis can be found in Appendix F.2.

4.3 Analysis

Now we present our analysis on GDELT. On GDELT, we focus on the predicate and object prediction. It is less expensive than the time and predicate-object prediction (which requires an order of magnitude more GPT hours and GPT API calls) but the results are well correlated in our pilot experiments. More analysis can be found in Appendix F, including analysis on the other datasets.

Analysis-I: About LLMs. Section 4.2 only shows the results of the GPT-3.5 version of our framework. Figure 3 shows the results of the other versions with ANHP as the base model. These LLMs all help improve the prediction performance but GPT-3.5 and Llama work significantly better than GPT-3. Interestingly, the Llama version performs competitive to the GPT-3.5 version. Its strong performance demonstrates that our framework doesn't have to depend on black-box LLMs; indeed, it can excel with open-source LLMs like Llama. Note that Llama has considerably fewer parameters than GPTs. An important reason for its success may be the reinforcement learning from human

Mean Rank

2.75

2.50

2.25

2.00

1.75

1.50

1.25

2

2

5

10

15

20

M

2

4

6

8

Mean Rank

ANHP

ANHP-G3.0

ANHP-llama

ANHP-G3.5

ANHP

ANHP-G3.0

ANHP-llama

ANHP-G3.5

3

4

5

M

0

4

10

Num of Shots

2.3

2.4

2.5

2.6

2.7

Mean Rank

ANHP

ANHP-G3.0

ANHP-G3.5

0

4

10

Num of Shots

5

6

7

8

9

Mean Rank

ANHP

ANHP-G3.0

ANHP-G3.5

5

10

15

KE-G3.5

DyRep-G3.5

ANHP-G3.5

20

Num of Retrieved Evidence Events

5

10

15

KE-G3.5

DyRep-G3.5

ANHP-G3.5

20

Num of Retrieved Evidence Events

ANHP

G3.0-P1

G3.0-P2

G3.5-P1

G3.5-P2

2.2

2.3

2.4

2.5

2.6

2.7

Mean Rank

ANHP

G3.0-P1

G3.0-P2

G3.5-P1

G3.5-P2

6

7

8

9

Mean Rank

feedback (RLHF) technique, which is adopted by GPT-3.5 and Llama-2-chat, but not by GPT-3. Further investigation may be an interesting direction for future work.

Analysis-II: How many evidence events do we need? In our GDELT experiments, we varied the number of retrieved evidence events for each proposed prediction. As shown in Figure 4, the results improve when the number of retrievals increases from 5 to 10, but become worse when the number further goes up. Apparently, if we retrieve too many 'evidence' events, then most of them might not be actually helpful so the performance will degrade due to the decreased signal-to-noise ratio. 1 We also tried another retrieval criteria: instead of retrieving a fixed number of previous events, we only retrieve an event if its similarity score exceeds a prespecified threshold. It turns out that this criteria only had negligible impact on the results.

Analysis-III: Prompt design. How will the design of the prompt influence the performance of our LAMP framework? As shown in Figure 5, LAMP performs better when the prompt includes a larger number of demonstrations, and the GPT-3.5 version consistently outperforms the GPT-3 version. Interestingly, even 0-shot prompting yields a better result than the baseline ANHP model, emphasizing that the LLMs are indeed very helpful.

Figure 6 shows how sensitive our framework is to the choice of demonstrations. We use P1 to denote the set of demonstrations used throughout our main experiments. Then P2 refers to a new set of demonstrations obtained by randomly replacing half of the demonstrations in P1. As we can see, changing the demonstrations has only a slight effect on performance, and LAMP consistently outperforms ANHP, whether using P1 or P2.

We designed the prompt templates without prior experimentation or tuning on any dataset. In a post analysis, we tested several different templates on a small subset of GDELT data, and found only minimal variations in results, as long as the task description was clear. Interestingly, a new template that we tested doesn't include the full list of possible predicates (see Listing 1), yet the results with this template closely match those of our original version.

Analysis-IV: About generalization. The LLM in LAMP is instructed by a few demonstrations. On GDELT, these demonstrations cover 17 out of 20 (85%) predicates and 30 out of 2279 (1.31%) entities. However, GPT-3.5 is able generalize beyond the demonstrations, suggesting cause events that involve novel predicates and entities. Precisely, the LLM-generated cause events involve 4699 distinct predicates and 65554 distinct entities, covering all the 20 ground-truth predicates and more than 800 ground-truth entities. Appendix F.3 includes more details about this analysis.

LLMs are known for strong in-context learning capability. In our specific problem setting, an LLM may have already understood the meanings of the predicates and been familiar with many entities

Mean Rank

2.7

2.6

2.5

2.4

Mean Rank

6.8

6.6

6.4

6.2

6.0

5.8

5.6

2

3

4

5

M

1.5

2.0

2.5

3.0

Mean Rank

ANHP

ANHP-rnd

ANHP-rec

ANHP-bert

ANHP-rec-pre

ANHP-G3.5

2

5

10

15

20

M

2

4

6

8

10

Mean Rank

ANHP

ANHP-rnd

ANHP-rec

ANHP-bert

ANHP-rec-obj

ANHP-G3.5

2

3

4

5

M

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Mean Rank

ANHP

ANHP-G3.5

ANHP-G3.5-edit

5

10

15

20

M

2

4

6

8

10

Mean Rank

ANHP

ANHP-G3.5

ANHP-G3.5-edit

through its pretraining and fine-tuning processes. Consequently, when presented with a proposal, it can follow demonstrations and draw upon its internal knowledge to suggest plausible causes. Benefiting from the strong generalization capability of LLMs, our LAMP framework has a significant potential for broad applications.

Analysis-V: About retrieval methods. To further investigate the usefulness of LLMs, we tested several versions of our framework that do not involve the LLM in retrieval: rnd randomly samples 10 previous events from the history for each proposal; rec uses the most recent 10 past events; rec-pre retrieves the most recent 10 events that share the same predicate with the proposal; rec-obj retrieves the most recent 10 events that share the same object; bert uses SBERT to embed the text-based identifier of each event and retrieve the most similar 10 events based on the cosine similarity of SBERT embeddings. As shown in Figure 7, all these retrieval methods perform significantly worse than our LLM-based method. Noticeably, the 'most recent' retrievals work poorly. It suggests that 'recent' is not a good inductive bias for our problem setting. On dev data, we found that the cause events retrieved based on LLM-generated clues are often not 'recent'.

We also tested using the edit distance as the similarity score in our LLM-based retrieval method. For a pair of events, we compute the character-level edit distance (Jurafsky & Martin, 2009) between their text strings that are otherwise fed into the SBERT embedding model (section 3.2), and define the similarity to be the reciprocal of this distance. As shown in Figure 8, using this metric yields a better result than the baseline method, but performs worse than our original framework.

Analysis-VI: About data leakage. GPT models are trained on online text data up to 2021. 2 This dataset doesn't include the GDELT or ICEWS data used in our experiments. Our demonstrations span a wide time range in training data, and there is a large temporal gap between the training and test data. Precisely, the most recent demonstration is about an event on 2022-06-07, while the earliest test event occurred on 2022-07-05. Below are the percentiles for the time differences (in days) between the test events and the latest demonstration:

The GPT training data may have covered the Amazon Review data. Therefore, we had a conversation with GPT-3.5, checking whether it could recall information about specific users or specific reviews. GPT-3.5 could say nothing specific about the users or reviews, indicating that we do not have an issue of data leakage. Our conversation can be found in Appendix E.5.

5 Discussion and Related Work

Event reasoning and prediction is an important problem that arises in various real-world applications. A great number of event sequence models have been proposed and advanced this field, including the classical graphical models (Hawkes, 1971; Du et al., 2015a), recurrent neural models (Du et al., 2016; Mei & Eisner, 2017b; Xiao et al., 2017; Omi et al., 2019; Shchur et al., 2020; Boyd et al., 2020), and Transformer-based models (Zuo et al., 2021; Zhang et al., 2020; Enguehard et al., 2020; Sharma et al., 2021; Zhu et al., 2021; Yang et al., 2022; Xue et al., 2022; Liu et al., 2022). These models have been applied to a wide range of applications such as network analysis (Choi et al., 2015; Etesami

et al., 2016), recommendation systems (Du et al., 2015b), social analysis (Guo et al., 2015; Lukasik et al., 2016; Zhang et al., 2022), and healthcare (Hua et al., 2022; Zhang et al., 2023b).

Recently, there has been a growing body of research that has directed its focus towards the textual features of the real-world events, such as the reports and news articles about the events. This line of work includes Deng et al. (2020, 2021, 2022); Han & Ning (2022). Their methods all treat text as auxiliary features but do not consider reasoning about that text information. Our work is significantly different from this line of research since we focus on reasoning. Our framework reasons about the events and their text information (e.g., text identifiers of the event types, associated documents) by prompting a large language model. The large language model has consumed a massive amount of text during pretraining and is aware of diverse world knowledge, thus able to perform the kind of reasoning beyond the capacities of the aforementioned models.

By leveraging large language models, our framework induces domain-specific knowledge into the deployment of event sequence models. It is related to previous work of configuring domain-specific knowledge into model architectures (Trivedi et al., 2017, 2019; Mei et al., 2020; Yang et al., 2022). But our work is significantly different: it doesn't rely on human domain experts to write down a full set of knowledge; instead, we extract knowledge from language models via few-shot prompting. Prompting has become a standard way of applying large language models to reasoning tasks. There has been a recent surge of work that develops novel prompting techniques for a better performance (Wei et al., 2022a,b; Zhou et al., 2023). Our framework is general and can adopt any prompting methods.

This paper is closely related to research in logical reasoning, which primarily focuses on proving goals from known facts and rules. A major approach for this problem is backward chaining: it works backward from the goal, chaining through rules to find known facts that support the proof (Russell & Norvig, 2010). This approach has been applied to various application problems, including reasoning and planning in natural language (Ye et al., 2022; Weir & Van Durme, 2022; Kazemi et al., 2023; Kassner et al., 2023). In our setting, each proposed event is treated as a goal, and previous events serve as known facts. Like backward chaining, our method identifies previous events that support the proposal. But our method only performs a single step of reasoning, assuming complete data where all events are observable. A second step of reasoning is unnecessary since direct causes are observable and more temporally recent than indirect causes. In cases of incomplete data, multiple reasoning steps may be required to identify indirect causes. Handling incomplete data will be a non-trivial extension of our current framework, which we leave to future research.

This paper is also closely related to research in event-centric natural language processing (NLP) (Chen et al., 2021). Over the past decades, there has been a great amount of progress in this area, including many datasets and benchmarks (Ning et al., 2020; Li et al., 2020; Han et al., 2021; Wen et al., 2021; Feng et al., 2022) as well as a diversity of methods for key problems such as event detection and extraction (Ji & Grishman, 2008; Li et al., 2013; Feng et al., 2016; Lyu et al., 2021; Wang et al., 2021), relation extraction and prediction (Chan & Roth, 2011; Ning et al., 2018b; Wang et al., 2020; Wen & Ji, 2021; Li et al., 2022), event induction and summarization (Do et al., 2012; Saravanakumar et al., 2021; Li et al., 2021b,a; Jin et al., 2022), and temporal reasoning (Ning et al., 2018a; Ballesteros et al., 2020; Zhou et al., 2020, 2021). Recently, there has been research in leveraging LLMs to solve these problems (Dror et al., 2023; Li et al., 2023; Zhang et al., 2023a). Our work complements this line of research, focusing on the problem of event sequence modeling, which has been out of the scope of the classical event-centric NLP.

6 Conclusion

In this paper, we present LAMP, a general modeling and prediction framework that leverages the abductive reasoning ability of large language models to help improve the prediction performance of event sequence models. Empirical studies demonstrate that our LLM-enhanced framework can significantly outperform the state-of-the-art event sequence models. Our findings have significant implications for future research in the field of event sequence modeling. In principle, an event sequence model should benefit from a range of reasoning abilities of large language models such as deductive reasoning, inductive reasoning, commonsense reasoning, and arithmetic reasoning. Further exploration in this area may lead to impactful innovations.

Acknowledgments

This work was supported by a research gift to the last author by Adobe Research. We thank the anonymous NeurIPS reviewers and meta-reviewer for their constructive feedback. We also thank our colleagues at Ant Group, UChicago, and TTIC for helpful discussion.

References

Ballesteros, M., Anubhai, R., Wang, S., Pourdamghani, N., Vyas, Y., Ma, J., Bhatia, P., McKeown, K., and Al-Onaizan, Y. Severing the edge between before and after: Neural architectures for temporal ordering of events. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

Boschee, E., Lautenschlager, J., O'Brien, S., Shellman, S., Starz, J., and Ward, M. ICEWS Coded Event Data, 2015.

Boyd, A., Bamler, R., Mandt, S., and Smyth, P. User-dependent neural sequence models for continuous-time event data. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Chan, Y. S. and Roth, D. Exploiting syntactico-semantic structures for relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2011.

Chen, M., Zhang, H., Ning, Q., Li, M., Ji, H., McKeown, K., and Roth, D. Event-centric natural language processing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Choi, E., Du, N., Chen, R., Song, L., and Sun, J. Constructing disease network and temporal progression model via context-sensitive Hawkes process. In IEEE International Conference on Data Mining (ICDM) , 2015.

Deng, S., Rangwala, H., and Ning, Y. Dynamic knowledge graph based multi-event forecasting. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2020.

Deng, S., Rangwala, H., and Ning, Y. Understanding event predictions via contextualized multilevel feature learning. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management , pp. 342-351, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469.

Deng, S., Rangwala, H., and Ning, Y. Causality enhanced societal event forecasting with heterogeneous graph learning. In 2022 IEEE International Conference on Data Mining (ICDM) , 2022.

Do, Q., Lu, W., and Roth, D. Joint inference for event timeline construction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2012.

Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. In Findings of the Conference of the European Chapter the Association for Computational Linguistics (Findings of EACL) , 2023.

Du, N., Farajtabar, M., Ahmed, A., Smola, A. J., and Song, L. Dirichlet-Hawkes processes with applications to clustering continuous-time document streams. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2015a.

Du, N., Wang, Y., He, N., Sun, J., and Song, L. Time-sensitive recommendation from recurrent user activities. In Advances in Neural Information Processing Systems (NeurIPS) , 2015b.

Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2016.

Enguehard, J., Busbridge, D., Bozson, A., Woodcock, C., and Hammerla, N. Neural temporal point processes [for] modelling electronic health records. In Proceedings of Machine Learning Research , volume 136, pp. 85-113, 2020. NeurIPS 2020 Workshop on Machine Learning for Health (ML4H).

Etesami, J., Kiyavash, N., Zhang, K., and Singhal, K. Learning network of multivariate Hawkes processes: A time series approach. arXiv preprint arXiv:1603.04319 , 2016.

Feng, X., Huang, L., Tang, D., Ji, H., Qin, B., and Liu, T. A language-independent neural network for event detection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2016.

Feng, Y., Zhou, B., Wang, H., Jin, H., and Roth, D. Generic temporal reasoning with differential analysis and explanation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2022.

Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760 , 2022.

Guo, F., Blundell, C., Wallach, H., and Heller, K. The Bayesian echo chamber: Modeling social influence via linguistic accommodation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2015.

Han, R., Hsu, I.-H., Sun, J., Baylon, J., Ning, Q., Roth, D., and Peng, N. Ester: A machine reading comprehension dataset for reasoning about event semantic relations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Han, X. and Ning, Y. Text-enhanced multi-granularity temporal graph learning for event prediction. In Zhu, X., Ranka, S., Thai, M. T., Washio, T., and Wu, X. (eds.), IEEE International Conference on Data Mining, ICDM 2022, Orlando, FL, USA, November 28 - Dec. 1, 2022 , pp. 171-180. IEEE, 2022.

Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika , 1971.

Hua, W., Mei, H., Zohar, S., Giral, M., and Xu, Y. Personalized dynamic treatment regimes in continuous time: A Bayesian joint model for optimizing clinical decisions with timing. Bayesian Analysis , 2022.

Ji, H. and Grishman, R. Refining event extraction through cross-document inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2008.

Jianmo Ni, Jiacheng Li, J. M. Justifying recommendations using distantly-labeled reviews and fined-grained aspects. In Empirical Methods in Natural Language Processing (EMNLP) , 2019.

Jin, X., Li, M., and Ji, H. Event schema induction with double graph autoencoders. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2022.

Jurafsky, D. and Martin, J. H. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition . 2009.

Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schutze, H., and Clark, P. Language models with rationality. arXiv preprint arXiv:2305.14250 , 2023.

Kazemi, S. M., Kim, N., Bhatia, D., Xu, X., and Ramachandran, D. Lambada: Backward chaining for automated reasoning in natural language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2023.

Leetaru and Schrodt. Gdelt: Global data on events, location, and tone, 1979-2012. ISA annual convention , 2, 04 2013.

Lewis, P. A. and Shedler, G. S. Simulation of nonhomogeneous Poisson processes by thinning. Naval Research Logistics Quarterly , 1979.

Li, M., Zareian, A., Zeng, Q., Whitehead, S., Lu, D., Ji, H., and Chang, S.-F. Cross-media structured common space for multimedia event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2020.

Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., and Voss, C. Future is not onedimensional: Graph modeling based complex event schema induction for event prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021a.

Li, M., Ma, T., Yu, M., Wu, L., Gao, T., Ji, H., and McKeown, K. Timeline summarization based on event graph compression via time-aware optimal transport. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021b.

Li, Q., Ji, H., and Huang, L. Joint event extraction via structured prediction with global features. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2013.

Li, S., Ji, H., and Han, J. Open relation and event type discovery with type abstraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022.

Li, S., Zhao, R., Li, M., Ji, H., Callison-Burch, C., and Han, J. Opendomain hierarchical event schema induction by incremental prompting and verification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2023.

Liniger, T. J. Multivariate Hawkes processes . Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 18403, 2009.

Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Lukasik, M., Srijith, P. K., Vu, D., Bontcheva, K., Zubiaga, A., and Cohn, T. Hawkes processes for continuous time sequence classification: An application to rumour stance classification in Twitter. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2016.

Lyu, Q., Zhang, H., Sulem, E., and Roth, D. Zero-shot event extraction via transfer learning: Challenges and insights. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Mei, H. and Eisner, J. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems (NeurIPS) , 2017a.

Mei, H. and Eisner, J. The neural hawkes process: A neurally self-modulating multivariate point process. In Proceedings of the International Conference on Machine Learning (ICML) , 2017b.

Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Ning, Q., Feng, Z., Wu, H., and Roth, D. Joint reasoning for temporal and causal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2018a.

Ning, Q., Wu, H., and Roth, D. A multi-axis annotation scheme for event temporal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2018b.

Ning, Q., Wu, H., Han, R., Peng, N., Gardner, M., and Roth, D. Torque: A reading comprehension dataset of temporal ordering questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. 2017.

Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019.

Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach . Prentice Hall, 2010.

Saravanakumar, K. K., Ballesteros, M., Chandrasekaran, M. K., and McKeown, K. Event-driven news stream clustering using entity-aware contextual embeddings. In Proceedings of the Conference of the European Chapter the Association for Computational Linguistics (EACL) , 2021.

Sharma, K., Zhang, Y., Ferrara, E., and Liu, Y. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2021.

Shchur, O., Biloš, M., and Günnemann, S. Intensity-free learning of temporal point processes. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Trivedi, R., Dai, H., Wang, Y., and Song, L. Know-Evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the International Conference on Machine Learning (ICML) , 2017.

Trivedi, R., Farajtabar, M., Biswal, P., and Zha, H. DyRep: Learning representations over dynamic graphs. In Proceedings of the International Conference on Learning Representations (ICLR) , 2019.

Wang, H., Chen, M., Zhang, H., and Roth, D. Joint constrained learning for event-event relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

Wang, H., Zhang, H., Chen, M., and Roth, D. Learning constraints and descriptive segmentation for subevent detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Wei, J., Tay, Y ., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions of Machine Learning Research (TMLR) , 2022a.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS) , 2022b.

Weir, N. and Van Durme, B. Dynamic generation of interpretable inference rules in a neuro-symbolic expert system. arXiv preprint arXiv:2209.07662 , 2022.

Wen, H. and Ji, H. Utilizing relative event time to enhance event-event temporal relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Wen, H., Qu, Y., Ji, H., Ning, Q., Han, J., Sil, A., Tong, H., and Roth, D. Event time extraction and propagation via graph attention networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

Xiao, S., Yan, J., Farajtabar, M., Song, L., Yang, X., and Zha, H. Joint modeling of event sequence and time series with attentional twin recurrent neural networks. arXiv preprint arXiv:1703.08524 , 2017.

Xue, S., Shi, X., Zhang, Y. J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Xue, S., Shi, X., Chu, Z., Wang, Y., Zhou, F., Hao, H., Jiang, C., Pan, C., Xu, Y., Zhang, J. Y., et al. Easytpp: Towards open benchmarking the temporal point processes. arXiv preprint arXiv:2307.08097 , 2023.

Yang, C., Mei, H., and Eisner, J. Transformer embeddings of irregularly spaced events and their participants. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Ye, A., Cui, C., Shi, T., and Riedl, M. O. Neural story planning. arXiv preprint arXiv:2212.08718 , 2022.

Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, E. Self-attentive hawkes processes. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Zhang, T., Tham, I., Hou, Z., Ren, J., Zhou, L., Xu, H., Zhang, L., Martin, L. J., Dror, R., Li, S., et al. Human-in-the-loop schema induction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2023a.

Zhang, Y., Cao, D., and Liu, Y. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Zhang, Z., Mei, H., and Xu, Y. Continuous-time decision transformer for healthcare applications. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023b.

Zhou, B., Ning, Q., Khashabi, D., and Roth, D. Temporal common sense acquisition with minimal supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2020.

Zhou, B., Richardson, K., Ning, Q., Khot, T., Sabharwal, A., and Roth, D. Temporal reasoning on implicit events from distant supervision. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Zhu, S., Zhang, M., Ding, R., and Xie, Y. Deep fourier kernel for self-attentive point processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , 2021.

Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Appendices

A Societal Impacts

Our paper develops a novel method to integrate large language models into temporal modeling. By describing the model and releasing code, we hope to facilitate the modeling of continuous-time sequential data in many domains. However, like many other machine learning models, our method may be applied to unethical ends. For example, its abilities of better fitting data and making more accurate predictions could potentially be used for unwanted tracking of individual behavior, e.g. for surveillance.

B Limitations

Our framework utilizes LLMs such as GPT-3.5 and thus inherits the issues of those models such as hallucination and biased content. Therefore, the cause events generated by LLMs may be irrelevant, incorrect, or misleading, thus harming the overall performance of our framework. Also because of using LLMs, our approach requires human-crafted prompts, and the events need to have textual information (e.g., event types have textual identifiers or marks).

Our framework needs a pretrained event sequence model, and its overall capability is partially determined by this base model. When the base model can handle events with duration (i.e., events that last for a period of time, such as holding a tool), our framework will inherit this merit. But our framework will also inherit some of the technical limitations of its base event sequence model. If the base model can only handle a finite set of event types, then our framework will inherit this limitation. If the base model is misspecified or poorly trained, our framework may not work well.

Our compatibility function c is a Transformer-based neural network which is known to be data-hungry. Though it worked well in our experiments, it might struggle when starved of data.

C Possible Extensions

Our framework currently trains each component separately. A natural improvement will be to jointly train its three components. In principle, we can iteratively

refine the base event sequence model with the feedback from the LLM and ranking model;

learn to prompt the LLM so that it generates cause events of higher quality;

train the ranking model with proposals and evidence given by the improved base model and LLM.

Another interesting extension of our framework is to apply its propose-justify-rank pipeline to other problems. At the core of our framework is the LLM, which is able to examine each proposed prediction drawn from the base model, and suggest clues to justify its validity. This idea seems to be a natural fit to a wide range of reasoning problems. Question answering is an example: given a question, an existing model proposes multiple plausible answers; an LLM reads each proposed answer and reasons about its preconditions; a search module finds out evidence (from local knowledge bases or the open internet) which may match the LLM-suggested preconditions; a ranking model learns to score each combination of answer and evidence.

D Method Details

D.1 ANHP

On Amazon data, the ANHP model we used is directly from Yang et al. (2022). On GDELT and ICEWS data, we modified the definition of event embedding so that it is easier to scale up to hundreds of millions of event types. Particularly, ANHP defines a dynamic event embedding J k K ( t ) for the event type k at time t . On GDELT and ICEWS, each event type is a tuple of subject-predicate-object, so we redefine the event embedding to be

and keep everything else the same as in the original ANHP.

This modified version is more scalable since it needs to maintain a significantly smaller number of dynamic embeddings: there are hundreds of millions of possible event types, but there are only thousands of possible subjects and objects.

E Experimental Details

E.1 Dataset Details

Table 1 shows statistics about each dataset in our experiments.

E.2 Evaluation Metric Details

Now we give the formal definitions of MR and MRR. The primary evaluation metric is the mean rank (MR): for each method, we initialize N = R = 0 ; for each held-out token, the method provides a list of M proposed predictions sorted in the decreasing order of their scores; if the ground-truth type is among the proposals, we update N += 1 and R += r where r is the rank of the ground-truth type in this list; the mean rank is R/N and the lower is the better.

The mean reciprocal rank (MRR) is as follows: for each method, we initialize N = R = 0 ; for each held-out token, the method provides a list of M proposals sorted in the decreasing order of their scores; if the ground-truth type is among the proposals, we update N += 1 and R += 1 /r where r is the rank of the ground-truth type in this list; the MRR is R/N and the higher is the better.

Note that MR and MRR are not affected by the fact that there might be multiple ground-truth attributes. That is because the contribution of j th ground-truth (among the multiple) to the final number is only dependent on its rank in the list but not its j index.

Next we give the precise definitions of MAP and MAR.

MAP is computed as follows. We first initialize N = C = 0 . Then, for each partial held-out event (i.e., only t given in Amazon Review, t and some attributes given in GDELT), the model gives its top M predictions on the attribute of interest (e.g., type k on Amazon Review and 'object' on GDELT). There might be multiple ground-truth attributes for a held-out event since multiple events may be recorded at the same time (due to time quantization and recording errors). If any of the ground-truth is in the top M list, we update N and C : first, we have N += J where J ≤ M is the number of the ground-truth attributes that are covered in the top M list; then, for j th covered ground-truth, we let C += j/R where R is its rank in the top M list. In the end, MAP is defined as C/N . Each i/R is a pseudo-count for this event: it is in (0 , 1] since the rank R of the j th correct prediction will be surely ≥ j ; if it ranks at the top, i/R is close to 1 and it is counted as 'predicted correctly'; if it ranks at the bottom, i/R is close to 0, meaning that it is almost missed by the top M proposals.

MARis defined as follows. We first initialize N = C = 0 . For each partial held-out event, the model gives its top M proposals on the attribute of interest; for the j th correct prediction in the list, we let C += j/R where R is its rank in the top M list; in the end, MAR is defined as C/N where N is the number of actual events.

E.3 Implementation Details

All models are implemented using the PyTorch framework (Paszke et al., 2017).

For the implementation of NHP, AttNHP and energy functions, we used the code from the public Github repository at https://github.com/ant-research/EasyTemporalPointProcess (Xue et al., 2023) with Apache License 2.0.

For the implementation of Know-Evolve, we used the code from the public Github repository at https://github.com/rstriv/Know-Evolve (Trivedi et al., 2017) without any license.

For the implementation of DyRep, we used the code from the public Github repository at https: //github.com/uoguelph-mlrg/LDG (Trivedi et al., 2019) without any license.

For the implementation of MAR@M and MAP@M, we used the code from the public Github repository at https://github.com/statisticianinstilettos/recmetrics without any license.

For the implementation of Levenshtein distance, we used the code from the public Github repository at https://github.com/maxbachmann/Levenshtein.git with General Public License 2.0.

To compute the dense representations of text descriptions of events, we use the code from the public Github repository at https://github.com/UKPLab/sentence-transformers with Apache License 2.0.

E.4 Prompts

We show one example of the prompt structures used for GDELT dataset as below. The effect event consists of an event predicate, an event time, a subject name and an object name while the cause events consists of an event headline additionally.

I want you to do the reasoning over social events. I given you an effect event and you give me four or five cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event consists of an event headline, an event predicate, an event time, subject name and object name of describing the event.

The predicates of the effect and cause events are restricted to 20 options, with names (in capital) and the descriptions below.

APPEAL: appeal for material, economic, military, humanitarian aid or cooperation.

MAKE STATEMENT: decline comment, make comments.

EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic, military aid.

ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.

CONSULT: make a visit, host a visit.

ENGAGE IN MATERIAL COOPERATION: cooperate economically, militarily, judicially.

PROVIDE AID: provide economic, military, humanitarian aid.

INVESTIGATE: investigate crime, corruption, human rights abuses, military actions.

YIELD: ease admin or political sanctions or military blockade, return and release.

DEMAND: demand any type of cooperation, aid, reforms, rights, easing of sanctions.

REJECT: reject material, economic, military, judicial operations, requests or plans.

DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses, complain officially and make lawsuit against.

THREATEN: threaten to reduce aid, to boycott, to reduce or break relations, to impose sanctions.

EXHIBIT MILITARY POSTURE.

PROTEST: civilian demonstrations.

REDUCE RELATIONS: reduce or break any relations.

ASSAULT: use of unconventional forms of violence.

COERCE: seize or damage properties, impose administrative sanctions or restrictions.

FIGHT: uses of conventional force.

ENGAGE IN UNCONVENTIONAL MASS VIOLENCE.

Now I give you 10 examples of event reasoning. In each example, the first event is the effect event, the next three to five events are cause events that happen earlier.

We show one example of the prompt structures used for ICEWS dataset as below. The effect event consists of an event predicate, an event time, a subject name and an object name while the cause events consists of an event sub predicate additionally.

I want you to do the reasoning over social events. I given you an effect event and you give me four or five cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event consists of an event text, an event predicate, an event time, subject name and object name of describing the event.

The predicates of the effect and cause events are restricted to 20 options, with names (in capital) and the descriptions below.

APPEAL: appeal for material, economic, military, humanitarian aid or cooperation.

MAKE STATEMENT: decline comment, make comments.

EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic, military aid.

ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.

CONSULT: make a visit, host a visit.

ENGAGE IN MATERIAL COOPERATION: cooperate economically, militarily, judicially.

PROVIDE AID: provide economic, military, humanitarian aid.

INVESTIGATE: investigate crime, corruption, human rights abuses, military actions.

YIELD: ease admin or political sanctions or military blockade, return and release.

DEMAND: demand any type of cooperation, aid, reforms, rights, easing of sanctions.

REJECT: reject material, economic, military, judicial operations, requests or plans.

DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses, complain officially and make lawsuit against.

THREATEN: threaten to reduce aid, to boycott, to reduce or break relations, to impose sanctions.

EXHIBIT MILITARY POSTURE.

PROTEST: civilian demonstrations.

REDUCE RELATIONS: reduce or break any relations.

ASSAULT: use of unconventional forms of violence.

COERCE: seize or damage properties, impose administrative sanctions or restrictions.

FIGHT: uses of conventional force.

ENGAGE IN UNCONVENTIONAL MASS VIOLENCE.

Now I give you 10 examples of event reasoning. In each example, the first event is the effect event, the next three to five events are cause events that happen earlier.

## Example 1

predicate: MAKE STATEMENT

time: 2022-10-10

subject: CHINA

object: JAPAN

sub predicate: Make pessimistic comment.

The following is a prompt exemplar used for the Amazon Review dataset. Each effect event consists of the product category and event time while each cause event also includes the content of the product review.

I want you to do the reasoning over the events that are extracted from onlineshopping review data. I given you an effect event and you give me two to four cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event corresponds to an review submitted by the customer, which consists of an product category(event type), a product title, an event time, summary text and review text from the user that describes the feedback of the shopping event.

cause event 2

For the full set of prompts used in the paper, please see scripts/gdelt/gdelt_prompts and scripts/amazon/amazon_prompts in our code.

E.5 Does GPT remember Amazon Review data?

We have a conversion with GPT-3.5-turbo to verify that it does not remember the exact information of Amazon Review data.

input: We are using Amazon Review Data, published by Jianmo Ni, et al., 2018, which is firstly used in the paper "Justifying recommendations using distantlylabeled reviews and fined-grained aspects", EMNLP 2019.

The dataset includes millions of reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features) in the range May 1996 Oct 2018. Do you remember any of the review information if I give you a review id?

GPT-3.5 output: As an AI language model, I am not capable of remembering any review information unless I have been programmed to store and retrieve specific pieces of information. However, I can search for and retrieve review information from the dataset if given a review ID.

input: can you recall any information with reviewer ID A28GK1G2KDXHRP on date 04 17, 2014.

GPT-3.5 output: I apologize, but the Amazon Review dataset does not include information about the specific date when a review was posted. It only includes the year and month when the review was posted. Therefore, I cannot provide you with any information about reviews posted by reviewer ID A28GK1G2KDXHRP on April 17, 2014.

input: Reviewer A0148968UM59JS3Y8D1M bought Hanes Men's Classic Crew on 01 18, 2014, what review did he wrote?

GPT-3.5 output: I apologize, but as an AI language model, I do not have access to real-time or updated information about specific Amazon reviews or purchases made by individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of a specific review. It only includes metadata such as the reviewer ID, product ID, and overall rating. Therefore, I cannot provide you with the review written by the reviewer A0148968UM59JS3Y8D1M for the Hanes Men's Classic Crew on January 18, 2014 or any other date.

input: Reviewer A0148968UM59JS3Y8D1M bought Hanes Men's Classic Crew on 01 18, 2014, his review starts with 'they seem very well made..', can you recall the complete review he wrote?

GPT-3.5 output: I apologize, but as an AI language model, I do not have access to real-time or updated information about specific Amazon reviews or purchases made by individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of a specific review. It only includes metadata such as the reviewer ID, product ID, and overall rating. Therefore, I cannot provide you with the complete review written by the reviewer A0148968UM59JS3Y8D1M for the Hanes Men's Classic Crew on January 18, 2014, or any other date.

input: on which date did reviewer A0148968UM59JS3Y8D1M buy Hanes Men's 7-Pack ComfortSoft Briefs?

GPT-3.5 output: Sorry, as an AI language model, I do not have access to personal information or purchase history of individuals.

Listing 6: A conversation with GPT-3.5-turbo to test its memorization about Amazon Review data.

E.6 Training and Inference Details

In this section, we present the details about training the models and doing inference with them.

LLMs. We use GPT-3-davinci and GPT-3.5-turbo. We set temperature to be 0 to ensure that our results are reproducible.

Training Base Models. For each model, we did grid search and chose the hyperparameters based on their performance on the dev set; see Table 2 for the values of the hyperparameters. For KE, DyRep, ANHP, the main hyperparameters to tune are the entity embedding size D 1 , predicate embedding size D 2 , RNN hidden size D 3 used in the network and the number of layers L of the attention structure (DyRep and ANHP). In practice, the optimal D for a model was usually 4 , 8 , 16 , 32 ; the optimal L was usually 1 , 2 , 3 , 4 . In the experiment, to train the parameters of the base model, we performed early stopping based on log-likelihood on the held-out dev set.

Retrieval. For each event, we use few-shot prompting (see Appendix E.4 for examples) to generate a set of cause events. For each LLM-generated cause, we retrieve the most similar D actual events from the history, following the procedure in section 3.2.

For GDELT, we set D = 2 and the average number of retrieved events is 10 .

For ICEWS, we set D = 2 and the average number of retrieved events is 10 .

For Amazon Review, we set D = 4 and the average number of retrieved events is 10 .

Training Ranking Model. The energy function used in our ranking model is the same as what's proposed in Xue et al. (2022), which consists a continuous-time Transformer and an MLP. The hyperparameters are tuned within a range of values that make the score function to have a similar size as the base ANHP model. Training this ranking model needs negative samples. On GDELT and ICEWS, we use 5 negative samples forpredicate prediction and 20 for object and predicate-object prediction. On Amazon Review, we use 5 for both type and time prediction.

2

5

10

15

20

30

50

M

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Mean Reciprocal Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

10

25

50

75

100

M

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Mean Reciprocal Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

(a) GDELT.

10

25

50

75

100

M

0.0

0.1

0.2

0.3

0.4

0.5

Mean Reciprocal Rank

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

2

3

4

5

L

1.5

1.6

1.7

1.8

1.9

2.0

RMSE

×10 1

KE

DyRep

ANHP

KE-G3.5

DyRep-G3.5

ANHP-G3.5

(b) ICEWS.

2

3

4

5

M

0.4

0.6

0.8

1.0

1.2

Mean Reciprocal Rank

NHP

ANHP

NHP-G3.5

ANHP-G3.5

2

3

4

5

L

1.25

1.30

1.35

1.40

1.45

1.50

RMSE

×10 2

NHP

ANHP

NHP-G3.5

ANHP-G3.5

(c) Amazon Review.

Figure 9: Prediction perfor- mance measured by MRR of different methods on each dataset. The configura- tion of this figure is the same as that of Figure 2: on GDELT, we show ob- ject (upper) and predicate- object (lower) prediction; on ICEWS, we show object (upper) and time (lower) pre- diction; on Amazon Review, we show type (upper) and time (lower) prediction.

Computation Cost. All the experiments were conducted on a server with 256G RAM, a 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and one NVIDIA A100 GPU for acceleration. The training batch size is 8. For GDELT and ICEWS, the wall-clock training time is: 4.9ms per sequence for KE; 6.7ms per sequence for DyRep; 28.7ms per sequence for ANHP. For Amazon Review, the wall-clock time is: 3.6ms per sequence for NHP; 5.2ms per sequence for ANHP. For each of the datasets, the wall-clock time of training the ranking model is 9ms per training sample.

F Additional Results and Analysis

In this section, we present additional results and analysis that complement section 4.

F.1 Main MRR results.

Our main results in section 4 are the MR results. In this section, we show the corresponding MRR results, which are known to be more robust to bad predictions. The MRR results are in Figure 9.

F.2 Analysis About Precision and Recall

Figure 10 shows the mean average precision (MAP) and mean average recall (MAR) of each base model for a range of M . These metrics complement MR and MRR for evaluating the base models: MAP measures the fraction of the predictions that are correct; MAR measures the fraction of the

1

5

10

15

20

M

0.30

0.35

0.40

0.45

0.50

MAP

KE

DyRep

ANHP

(a) MAP on predicate pre- diction.

1

5

10

15

20

M

0.15

0.20

0.25

0.30

0.35

0.40

0.45

MAR

KE

DyRep

ANHP

(b) MAR on predicate pre- diction.

1

25

50

75 100125150175200

M

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

MAP

KE

DyRep

ANHP

(c) MAP on object predic- tion.

1

25

50

75 100125150175200

M

0.01

0.02

0.03

0.04

0.05

0.06

0.07

MAR

KE

DyRep

ANHP

actual events that are covered by the predictions; higher is better. Their precise definitions (and formulas) for our setting can be found in Appendix E.2.

As we can see in Figure 10, when M increases, MAR monotonically increases but the change of MAP may not be monotonic (which is consistent with how MAP and MAR behave in other retrieval tasks). For predicate prediction, we found that M = 5 yields a good balance between precision and recall as well as a low financial cost (i.e., GPT API calls). For object prediction, M = 50 seems to give the best precision-recall balance but we chose M = 20 due to the GPT API budget. For object prediction, both MAP and MAR are lower than 0.1 (but still significantly better than a random guess of 0.0015), indicating that this data is very difficult to model.

F.3 Analysis About Generalization

The LLM in LAMP is instructed by a few demonstrations. On GDELT, these demonstrations cover

17 out of 20 (85%) predicates;

30 out of 2279 (1.31%) subjects;

28 out of 2279 (1.22%) objects.

However, the LLM is able generalize beyond the demonstrations, suggesting cause events that involve novel predicates and entities. Precisely, the LLM-generated cause events involve

4699 distinct predicates, which cover all the 20 ground-truth predicates;

60552 distinct subjects, which cover 30 (of 30) subjects mentioned in demonstrations, as well as 787 not-mentioned ground-truth subjects.

65554 distinct objects, which cover 28 (of 28) objects mentioned in demonstrations, as well as 788 not-mentioned ground-truth objects.

The retrieved cause events cover:

all the 20 ground-truth predicates.

30 (of 30) subjects in demonstrations, as well as 186 not-mentioned ground-truth subjects.

28 (of 28) objects in demonstrations, as well as 216 not-mentioned ground-truth objects.

Note that retrievals involve fewer distinct subjects and objects than the LLM-generated causes. It is because the LLM generation is often diverse and creative, proposing many novel subject-predicateobject combinations that involve ground-truth entities but haven't actually happened in real history. Such causes will eventually be grounded to actual previous events that exhibit less diversity and creativity, thus ending up with fewer distinct subjects and objects in the retrievals. Benefiting from the strong generalization capability of LLMs, our LAMP framework has a significant potential for broad applications.

F.4 Additional Results About LLMs

Now we present our additional results of comparing different LLMs. They include the MR results of comparing GPT-3 and GPT-3.5 on GDELT (Figures 11 and 12), MR results of comparing GPT-3 and GPT-3.5 on Amazon Review (Figure 13), and MRR results corresponding to Figure 3 (Figure 14).

Mean Rank

Mean Reciprocal Rank

2.75

2.50

2.25

2.00

1.75

1.50

2

M

KE

KE-G3.0

KE-G3.5

3

4

5

2

5

10

15

20

M

2

4

6

8

Mean Rank

KE

KE-G3.0

KE-G3.5

2

3

4

5

M

1.00

1.25

1.50

1.75

2.00

2.25

Mean Rank

NHP

NHP-G3.0

NHP-G3.5

2

3

4

5

M

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Mean Rank

ANHP

ANHP-G3.0

ANHP-G3.5

F.5 Additional Results About Prompt Design

Now we present additional results about the prompt design analysis in section 4. Figure 15 and Figure 16 show the MRR results corresponding to Figure 5 and Figure 6, respectively.

0

4

10

Num of Shots

0.54

0.55

0.56

0.57

0.58

Mean Reciprocal Rank

ANHP

ANHP-G3.0

ANHP-G3.5

0

4

10

Num of Shots

0.10

0.15

0.20

0.25

Mean Reciprocal Rank

ANHP

ANHP-G3.0

ANHP-G3.5

F.6 Additional Results About Retrievals

Now we present our additional analysis results about retrievals. Figure 17 presents the MRR results with different numbers of retrieved evidence events on GDELT.

0.58

0.56

0.54

0.52

5

10

15

20

Num of Retrieved Evidence Events

0.18

0.20

0.22

0.24

0.26

Mean Reciprocal Rank

KE-G3.5

DyRep-G3.5

ANHP-G3.5

KE-G3.5

DyRep-G3.5

ANHP-G3.5

15

Num of Retrieved Evidence Events

5

10

20

2

3

4

5

M

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Mean Rank

KE

KE-G3.5

KE-G3.5-edit

5

10

15

20

M

2

4

6

8

Mean Rank

KE

KE-G3.5

KE-G3.5-edit

2.25

2.00

1.75

1.50

1.25

1.00

NHP

NHP-G3.5

NHP-G3.5-edit

3

M

ANHP

ANHP-G3.5

ANHP-G3.5-edit

3

M

2

4

5

2

4

5

2

3

4

5

M

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Mean Rank

DyRep

DyRep-G3.5

DyRep-G3.5-edit

5

10

15

20

M

2

4

6

8

Mean Rank

DyRep

DyRep-G3.5

DyRep-G3.5-edit

2.2

2.0

1.8

1.6

1.4

1.2

1.0

ANHP

G3.0-P1

G3.0-P2

G3.5-P1

G3.5-P2

0.50

0.52

0.54

0.56

0.58

0.60

Mean Reciprocal Rank

ANHP

G3.0-P1

G3.0-P2

G3.5-P1

G3.5-P2

0.10

0.15

0.20

0.25

Mean Reciprocal Rank

Mean Rank

Mean Rank

2.75

2.50

2.25

2.00

1.75

1.50

2

M

M

DyRep

DyRep-G3.0

DyRep-G3.5

3

4

5

2

DyRep

DyRep-G3.0

DyRep-G3.5

5

10

15

20

2

3

4

5

M

0.55

0.60

0.65

0.70

0.75

0.80

Mean Reciprocal Rank

ANHP

ANHP-G3.0

ANHP-G3.5

2

5

10

15

20

M

0.1

0.2

0.3

0.4

0.5

Mean Reciprocal Rank

ANHP

ANHP-G3.0

ANHP-G3.5

Mean Rank

Mean Rank

8

6

4

2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies using large language models (LLMs) to assist event prediction. The authors proposed a pipeline of procedures, where 1) a traditional event sequence model is first applied to predict a set of possible events along with their time stamps, 2) then for each event candidate, an LLM is used to reason about their cause events, based on which a set of relevant events are retrieved from the history, and 3) finally the retrieved relevant events are used by a reranking model to score each event candidate. This framework is tested on two datasets, GDELT and Amazon Review, and the results showed that the LLM-based reranking approach outperforms the naive event sequence model.\n\nSTRENGTHS:\n1. This work studies leveraging LLMs for temporal modeling. This is an important topic as it can be used to support many applications (e.g., political event forecasting). Using LLMs for reranking also seems to be novel.\n2. The authors have conducted experiments on two datasets, both showing an advantage of the LLM-based reranking. \n3. This paper also comes with analyses of the proposed framework, such as how tuning the size of event candidates (M) can affect the overall performance and whether the LLM is sensitive to different variants.\n\nWEAKNESSES:\n1. The paper needs significant justifications for the experiment design. \n- While the authors formulated the event prediction task into two subtasks, the majority of the experiments have focused on only one of them (i.e., event type prediction), whereas the discussion on the time stamp prediction is rather brief. If the authors regard the time prediction task as an important subtask, then they may want to consider using a dataset that can facilitate more careful discussions other than GDELT.\n- The event prediction on GDELT is not end-to-end. Instead, the current experiments always assume one or two (ground-truth) components from each <subject, predicate, object> triplet and aim to predict the remaining. This could show a discrepancy between the evaluation and the realistic application.\n- More clarifications about each dataset are needed. For example, it is unclear how many events are provided as history in each prediction. \n\n2. Results in Figure 2 and several other figures showed worse performance as M increases. This could say that as the LLM-based reranker is exposed to more candidates, it ranks the truth even worse. Looking into the description of Mean Rank in lines 250-256, it can be imagined that as M increases, there will be more test cases (whose ground truth is included in top-M) get counted into \"N\". In other words, when M has different values, the Ns have been different, so the results of the same approach across different Ms are not really comparable. However, this is not clarified in the paper and the current plots read confusing.\n\n3. The proposed framework heavily relies on the event sequence model. Based on Figure 3, it seems that the poor performance of the event sequence model is the real bottleneck (<0.1 MAR in Fig 3(b)). In other words, while introducing LLMs does lead to improvement in event reranking (under the current experimental setting), it is not solving the real bottleneck. Therefore, I'm not very convinced by the contribution of this framework. Instead of reranking, a more worth-studying problem is how to use LLMs to more accurately predict future events in the first place. \n\n4. More explorations on the use of LLMs in reranking can provide more insight to the community. It is noticed that, while the contribution of this paper is about using LLMs for event reranking, none of the baselines or variants are following the same reranking idea. Is there a better way to use LLMs for event reranking? For example, can one use SentenceBERT or BM25 to retrieve relevant events from the history, and then directly use LLMs to score the candidate event?\n\n5. Some of the writing can be more self-contained. For example, the paragraph in line 51 introduced several math formulations but there is not a single reference, and the intuition of these formulas is not explained clearly.\n\nQUESTIONS:\nPlease try to address my comments in Weaknesses.\n\nLIMITATIONS:\nThe authors discussed the limitations and potential negative impact in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of predicting future world events based on the past and proposes an approach that combines the existing event sequence models with the powerful (abductive) reasoning ability of large language models (LLMs) that results in better performance both for predicting the actual event as well as predicting the time of the event.\n\nSTRENGTHS:\n* Predicting future world events based on the past is an important problem.\n* The idea of leveraging LLMs for future event prediction is interesting and can spark further research in this direction.\n* Experimental results are strong, albeit on a small set of datasets.\n\nWEAKNESSES:\n* The LLM prompts presented on Page 4 are a bit worrisome as they hint at a possible leakage: The time of the first demonstrating example is 2022-03-08 and so is the time of the queried effect. Could you please clarify how you selected the fewshots, and whether there might be any leakages?\n* I’m a bit confused about the metrics. From what I understand, the performance of the base Event Sequence model is computed on all the data, but the performance of the LLM-augmented models is only computed on the examples where the correct answer was among the top answers of the base model. Is this correct or did I misunderstand something?\n* Some of the presented results on the two datasets are strong, but given the simplicity of the proposed solution, it would have been more convincing if results on more datasets were reported, especially because the mean rank results on the Amazon dataset is not that glaring. (To be clear, I like the simplicity of the approach)\n* Given that the LLM module is the main novelty of the work, I would expect to see some experiments where the LLM is replaced with 1) a random selection module that selects 10 pieces of evidence at random, 2) a heuristic-based selection module that, e.g., selects the most recent 10 pieces of evidence with the same subject.\n* Some qualitative examples are needed to show how the LLM module actually helps make better predictions.\n* [Minor] The font in Figure 1 is a bit small when printed. Maybe make the size of box 3 smaller (rotate the text if needed) so that the other parts become slightly bigger.\n\nQUESTIONS:\n* The abductive reasoning module is quite similar to the backward reasoning works such as [1, 2, 3]. Could the authors say a few words about whether it is possible to bring more of backward reasoning into their framework? As an example, can we predict multiple hops of causes instead of one (e.g., for an event E, predicting a cause C and then predicting a cause C’ for C, …)?\n* Any idea how all three modules can be trained end-to-end instead of training separate modules?\n* Are the Event Sequence Model and Reranking Model both trained on the same subset of the data?\n* [Minor] Why not use Mean Reciprocal Rank instead of Mean Rank? The former has been advocated in many previous works.\n* [Minor] LLMs can hallucinate causes and it seems like all causes will be mapped to an event. I wonder whether it helps if the authors ignore the causes whose similarity to existing events is below some threshold.\n* [Minor] In line 150, how is time represented so that the model can know about the distance between the events?\n\n[1] LAMBADA: Backward Chaining for Automated Reasoning in Natural Language\n\n[2] Neural story planning\n\n[3] Language Models with Rationality (I understand that this paper was published too close to the NeurIPS deadline)\n\nLIMITATIONS:\n* It is typical in real-world problems to have both temporal and non-temporal facts/events. Can the approach be extended to this case?\n* It is typical in real-world problems to have temporal events that have a timestamp and temporal events that have a time interval. Can the approach be extended to work with intervals?\n* Can the model predict that no event will happen?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a large language model (LLM-) based approach to enhancing event prediction methods. Instructed by a few annotated demonstrations, a large language model is used to suggest possible causes for a proposal to be predicted. Then a search module is used to find out previous events that match the suggested causes. Finally, a scoring function is exploited to evaluate whether the retrieved events could actually cause the proposal. The above three steps can be used to enhance an existing event prediction method. Experimental results on two benchmark datasets (Amazon Review and GDELT) demonstrate that the performance improvement is significant by comparing enhanced methods with baseline ones.\n\nSTRENGTHS:\nSummary: \n(1) A new idea for using LLMs to enhance event prediction is proposed.\n(2) There are some empirical results for justifying the proposed idea.\n\nWEAKNESSES:\nSummary:\n(1) The effectiveness of the proposed approach, when being applied to other datasets or different LLMs that are trained by other corpora, is doubt.\n(2) The detail about how the proposed approach is used to enhance a baseline method for event prediction is clear.\n(3) The description of the proposed methodology lacks of rigor, with some important notations undefined.\n(4) The proposed approach has some potential limitations not mentioned in the submission (see below in the Limitation part).\n\nSoundness:\nThe prediction performance of the proposed approach depends on the quality of the suggested causes by LLMs. If the suggested causes differ a lot from any events in the experimental dataset, the proposed approach may not help. Thus, there is a doubt whether the prediction performance improvement gained by the proposed approach still exists for other datasets or different LLMs that are trained by other corpora. In other words, the proposed approach seems not always sound in terms of improving performance for event prediction.\n\nPresentation:\nAlthough the basic idea is presented clearly in the submission, the details are confusing in some important aspects. First of all, it is unclear how to enhance an existing event prediction method by the three steps (see comments in the Summary part). Should the existing event prediction method be treated as the scoring function in the last step? In other words, does an enhanced method only differs from the baseline one in retrieving relevant events to conduct a prediction? Secondly, the template for constructing prompts to an LLM is unclear. There are two templates (Listing 1 and Listing 2 in page 4) presented in the submission, which one is used to construct a prompt? Finally, some descriptions of the proposed approach are unclear. In Lines 89-90, what does the superscript (1), …, (M) mean? How is M defined? In Line 113, what does the superscript (m) mean? In Line 117, what does the superscript (m,M’) mean? How is M’ defined? In Line 170, what does the superscript (m) mean? How is M in Equation (5) defined?\n\nContribution:\nThe submission presents a new idea for utilizing LLMs to enhance event prediction. However, the soundness of this idea is questionable (see comments in the Soundness part). It calls for further theoretical analysis and empirical study to confirm the soundness.\n\nQUESTIONS:\nThe questions have been raised in the weakness comments for Presentation.\n\nLIMITATIONS:\nThe authors have not adequately addressed all limitations in the proposed approach. Some of them may not be addressed by the state-of-the-art techniques, but authors need to explicitly mention them in the submission. Firstly, the problem setting studied in the work restricts all considering event types to be in a predefined finite set. If the template for constructing a prompt to LLMs is something like Listing 1 in Page 4, this predefined set of event types should be rather small, otherwise the constructed prompt could be too long to feed into the LLM. Secondly, the names of event types should be meaningful, since they need to be fed into an LLM for generating their embeddings. Finally, the effectiveness of the proposed approach depends on whether the utilized LLM involves knowledge about the considering event types.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper investigates the use of large language models (LLMs) to improve the accuracy of event sequence models. Specifically, the authors propose an abductive reasoning framework based on an LLM. First, the event model produces some proposals of predictions. Then, the LLM suggests some causes for each proposal based on annotated demonstrations. At this stage, a search module retrieves some events that match the causes and finally a scoring function gives the likelihood that the retrieved events could actually cause the proposal.

STRENGTHS:
* The proposed method is sound and applicable in a wide range of practical settings
* The experimental evaluation is solid, with several analyses and ablations that help the reader understand the contribution of different design choices
* The paper is well written and easy to follow

WEAKNESSES:
* The results are not surprising, as the baselines do not rely on the same setting
* The paper should mention the limitation that it requires textual data for the LLM

QUESTIONS:
None

LIMITATIONS:
Limitations are discussed in the Appendix. The paper should mention the limitation that it requires textual data for the LLM.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\nThis paper investigates the effectiveness of large language models in reasoning and predicting event sequences. The authors propose a general framework that combines event sequence models with large language models for the task of event prediction. In this framework, an event sequence model proposes predictions on future events given the past, while a large language model assists by generating possible causes for each proposal. Experiments on two challenging real-world datasets (Amazon Review and GDELT) have shown that this framework significantly outperforms state-of-the-art event sequence models.\n\nContributions:\n  1. Proposes a domain-agnostic framework that combines event sequence models with large language models for the task of event prediction\n  2. Demonstrates the framework's ability to significantly outperform state-of-the-art event sequence models on two challenging real-world datasets (Amazon Review and GDELT).\n  3. Conducts extensive ablation studies to test the framework's robustness to prompt design and hyperparameters.\n\nSTRENGTHS:\n1. The idea of using abductive reasoning to improve event prediction is really a natural fit, interesting, and novel.  The authors have made such a nice connection between two NLP problems. The framework is also pretty general.\n  2. Performance: The experimental results show that the proposed framework outperforms state-of-the-art event sequence models on two datasets, and the amount of improvement seems large from the chart.\n  3. The paper is well written and easy to understand.\n\nWEAKNESSES:\n* The literature review is a bit limited.  For example, the paper claims \"We are the first—to the best of our knowledge—to integrate large language models into temporal modeling.\"  But this is not true, as there are already works that apply LLMs to temporal reasoning.  For example, please see paper \"Generic Temporal Reasoning with Differential Analysis and Explanation (https://arxiv.org/pdf/2212.10467.pdf)\" and other papers from Dan Roth's group.\n\nQUESTIONS:\n1. The result presentation is a bit hard to read, can tables be presented instead of charts?  It's really hard to read especially at the regions where methods are close to each other.\n  2. In figure 4, the shapes are really too close to each other that it becomes difficult for the readers to read the figures.  Either make it bigger or differentiate them in other ways.\n  3. Its really surprising that gpt3.5 is so much better than davinci-003, given the general belief of reasoning abilities of these models are gpt4 > text-davinci-003 > gpt3.5.  Do the authors have an interpretation?  Was the prompt tuning done on the gpt3.5 and later used for davinci?\n  4. Can the authors do some qualitative analysis on the generated event causes?  What happens if the generated event causes for all event predictions are not included in past events?\n\nLIMITATIONS:\nThe authors did not discuss limitations of this work.  I mainly see two limitations:\n  1. The use of closed-source model limits the applicability of this approach.\n  2. It requires manually writing prompts for specific tasks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}