Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning
Authors: Xiaoming Shi
Siqiao Xue
Kangrui Wang
Fan Zhou
James Y. Zhang
JUN ZHOU
Chenhao Tan
Hongyuan Mei
Venue: NeurIPS 2023 poster (2023)
Keywords: event sequences
irregular time series
event prediction
large language model
reasoning
few-shot prompting
TL;DR: We develop LAMP, a framework that utilizes the reasoning ability of large language models to help improve the prediction accuracy of event sequence models.

Abstract: Large language models have shown astonishing performance on a wide range of reasoning tasks. In this paper, we investigate whether they could reason about real-world events and help improve the prediction performance of event sequence models. We design LAMP, a framework that integrates a large language model in event prediction. Particularly, the language model performs abductive reasoning to assist an event sequence model: the event model proposes predictions on future events given the past; instructed by a few expert-annotated demonstrations, the language model learns to suggest possible causes for each proposal; a search module finds out the previous events that match the causes; a scoring function learns to examine whether the retrieved events could actually cause the proposal. Through extensive experiments on several challenging real-world datasets, we demonstrate that our framework---thanks to the reasoning capabilities of large language models---could significantly outperform the state-of-the-art event sequence models.

Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning
Xiaoming Shi!
Siqiao Xue
Kangrui Wang
Fan Zhou'
James Zhang'
Jun Zhou
Chenhao Tan
Hongyuan Mei
Ant Group
UChicago TTICC
(peter.sxm, sigiao xsq, ian 2f, james 2, jun /housunnaaanggrruuucooooo Bantgroup.com chenhaoRuchicago Ikangrui, hongyuan @ttic.et edd
Abstract
Large language models have shown astonishing performance on a wide range of reasoning tasks. In this paper, investigate whether they could reason about real-world events and help improve the prediction performance of event sequence models. We design LAMP, framework that integrates large language model in event prediction. Particularly, the language model performs abductive reasoning to assist event sequence model: the event model proposes predictions on future events given the past; instructed by few expert-annotatee demonstrations, the language model learns to suggest possible causes for each proposal; search module finds out the previous events that match the causes; scoring function learns examine whether the retrieved events could actually cause the proposal. Through extensive experiments on several challenging real-world datasets, we demonstrate that our framework-than to the reasoning capabilities large language models- could significantly outperform the tate-offthhe-rt event sequence models.
Introduction
Prompting large language models (LLMss) such as GPT-3.5 has recently become a standard approach perform text-based reasoning tasks. In this paper, investigate their capabilities in reasoning about real-world events and improving event prediction. Particularly, we focus on the problem of modeling sequences of time-stamped events and predicting future events given the past. For example, in the healthcare domain, we would like to model patients' sequences of time-stamped hospital visits and predict their future symptoms given their past diagnosis and treatments. It has been long-standing and important problem in machine learning. Large language models are potentially useful for advancing solutions this problem because event sequences are often accompanied with rich text information which large language models excel at handling. For example.
Healthcare. Each hospital visit will have a doctor note summarizing this visit, including the department that the patient visits, the clinical measurements and treatments, and any future medical plans. By reading such textual information, large language model may be elicited to recall the medical knowledge that it has read during pretraining and then reason about the future hospital visits such as what symptoms or treatments that the patient may have.
Political. Each political event may generate a series of news articles describing the political agents involved in it and discussing its possible influences. A language model reading these articles may recall its knowledge which acquired from pretraining about these agents, their relations, and fundamental principles in politics such that it could reason about future political events.
Similar scenarios arise in commercial, dialogue, etc.
In this paper, propose LAMP, framework that integrates large language model in event prediction. The overview of our framework is illustrated in Figure 1. Given a history of previous
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sequunnea Model
Pomptin
Large I language Mocwl
Mari Kareeu
Tibuu CONGUT
END Prscias ARTIAA
Ranking Modwl
KAMl CMN
Pediusm RBLEC
COOPCE-ATE
5000-
VILLD
5152-3445
Farthev rolewrrs suands
COMPANY
Girglyy
COOPERATE
Protuue FRGHT
Plaze alud AG4
An overview of our framework that leverages a large language model to reason about events. Firstly, an event sequence model proposes predictions: in this example, we predict the predicate of the structured event type given its time, subject, and object. Secondly, a language model suggests cause events, which will patter--math against actual previous events and retrieve the most relevant. In the end, neural model learns to assign high scores to the proposed predictions that are strongly supported by the retrieved evidence.
Figure 1:
events, we use pretrained event sequence model to propose predictions on the future events, which are then examined with the assistance of an LLM. The LLM learns to perform abductive reasoning: is instructed by a few expert-annotated demonstrations, and generates possible causes that may explain the possible occurrence of each proposal. Each generated cause serves as a query to search for similar or relevant events that have actually happened. Then another neural model learns to embed these retrievals and examine whether they could really justify the corresponding proposal.
We are the first-to to the best of our knowledge to integrate large language models into event sequence modeling. Our modeling and prediction framework is general: it can incorporate all kinds of event sequence models and large language models. We experimented with a range of model choices and demonstrate that large language models could indeed help improve the prediction performance of event sequence models. On several challenging real-world datasets, our framework significantly outperforms the current state-of-the-art event sequence models.
2 Problem Formulation and Technical Background
Now give formal introduction to our problem setting and review the background knowledge.
Event sequence modeling. The problem is to model event sequences (t1, k1), (t2, k2).... where 0 t < f2 < ... are times of occurrence and each k K. is a discrete event type. The goal is to predict the next event for given history of events H4 (t1, ..)..... (f:-1, R;-1). Precisely, it consists of two subtasks: the first to predict the time t; of the next event; the second is to predict the type k; of the next event with the knowledge of its time t,.
The standard approach is to build probabilistic model over the sequences. Such models typically define an intensity function As: the intensity value A is the instantaneous rate that an event of type & occurs at time t. Given the function Aς. one could obtain the minimum Bayes risk (MBR) prediction of the next event given the history. Particularly, the MBR time prediction f is
1A(1) exp (-("__ X(s)ds) dt where x(t) >>EE( (x
i
(1)
kEK
and it could approximated by averaging samples given by the thinning algorithm (Lewis Shedler, 1979; Liniger, 2009). The MBR type prediction k; given time &; is
k = arginax Au (4:)
The intensity function \ is typically learned by maximizing the log- likeliild of the model. For time period (0, 7) that contains observed events (11, ki), ..., (1r, ky), the log-likelihood is
log AR. (t+) - -/T (ttt))tt
1= KEE:
Rich text information. In roal-world data, each type & may be represented as text-based identifier: in the example of Figure 1, each is one of the possible interactions between the political entities (organizations and individuals) in the G20 countries, which can be represented with structured name

---

such as Tesla-cooperate- Australia. additional information about the event: in Figure 1, each m is news headline about the event (e.g.. "EV battery prices go up"). For notation simplicity, we will only mention the mark m of an event when necessary. While reading such text information, a human may recall their relevant domain knowledge (e.g., influence of battery prices on Tesla and Australia) and increase their estimate on the probability that an event of Tesll=cooperatc-Austtall happens in the near future. An important way that humans learn such knowledge is reading text such as textbooks, research publications, and news articles. But event sequence models can not directly leverage this kind of information.
-usttaaa In addition, each event may have text mark m that contains
Large language models. Language models learn by reading text. Over the past years, large language models that have read nearly the entire internet have shown astonishing performance on many challenging tasks such as arithmetic reasoning and multi-turn dialogue (Wei al., 2022b; OpenAl, 2023). So it seems tempting to pair large language model with an event sequence model to improve its prediction performance: the language model has consumed a tremendous amount of information that the event model may not have seen but should be able to benefit from.
3 LAMP: Large Language Model in Event Prediction
Now we present our LAMP framework, in which an LLM is leveraged to enhance the prediction process of an event sequence model. As shown in Figure 1, LAMP has three key components:
A base event sequence model. This model is pretrained and we use it to propose candidate predictions. Section 3.1 is the discussion of this phase.
A large language model. Its duty in the framework is to perform abductive reasoning, a form of logical inference seeking the most plausible explanations for given observation (Russell & Norvig, 2010). Particularly, the language model reads each proposed prediction and suggests possible cause events for it. Then we pattern-match each LLM--geerrted cause against the actual previous events in the history, and retrieve those which are most similar. Section 3.2 discusses this phase.
A ranking model. The ranking model learns to examine each combination of the candidate prediction and its retrieved events-or, in other words, its evidence-and and assign high scores to the candidates that are strongly supported by the retrieved evidence. Section 3.3 discusses this phase.
3.1 Phase-I: Proposing Predictions
Given a history of previous events H. = (4,,1), kz)...... "t))....,,,1,)) (1:-1, 1)) the base event sequence model is used proposer to generate candidate predictions on the time and type of the next event.
((L.) from the base model via the thinning algorithm. If we were to only use this base model but not our LLM enhannndd framework, the final
For time prediction, we draw L i..dd samples {(1) ....... :
MBR time prediction would be the average of the samples, i.e., i, + E/_ ((4) However, the MBR prediction may not be accurate since the base model is imperfect. Therefore, our LAMP candidates, and utilize the LLM and ranking model to score them in later phases. If any of the L draws is actually better prediction than the MBR estimate, our framework has chance rank it higher.
framework treats all the L + 1 samples- wwth & f(L+1) denoting the MBR prediction
As we'll show shortly in sections 3.2 and 3.3, the LLM and ranking model work on full events. So we
find the most probable M full events { , R/ 2 JM for each time proposoal ( f where R (f..)
())
the event type that has the m-th highest intensity at time ((())
For type prediction given the ground ground-truth truth time t, we find M event types k(2)) R (af) where R (m) has the m-th highest intensity at time t, under the base model. If we were only use this base model for prediction, the MBR type prediction would be the event type (1)) with the highest model intensity, However, our full framework will use the LLM and ranking model to examine each of the top M full events {tt,, ) JMM at time t; in order to make a more informed prediction.
In practice, the event types often have structures and we may be interested in predicting an attribute of the structured type. Figure shows an example in which we are trying to predict the predicate of the structured event type given its time, subject, and object. In such cases, we just need to select as proposals the most probable M event types whose other attributes are the same as the known information (c.g., Tesla and Australia in Figure l).
3

---

I want you do the reasoning social events. given you an effect event and you give me four or five cause events. An effect event is an event that happens. cause event is believed to be have triggerred the effect event to happen. Each event consists of a time, type (that includes subject, predicate, object), and news headline describing the event.
Example
effect
predicate: APPEAL
time: 2022-04-23
of the
that
subject: GERMANY
object: GREEN PROJECT
reasoning:
cause event 1
predicate: REDUCE RELATIONS
The predicates are restricted to the 20 options below.
time: 2022-04-21
subject: EUROPE
1. MAKE STATEMENT
object: RUSSIA
headline: Europe determined to
// Full list are in Appendix E.4. 20. ENGAGE IN MASS VIOLENCE
Russian energy exports.
cause event 2
Now I give you 10 examples. each example, the first event is the effect and the next several events the causes that happened earlier.
predicate: DISAPPROVE
time: 2022-03-16
subject: EUROPE
object: RUSSIAN
headline: Europe endure painful transition live without Russian oil.
// Examples in Listing 2.
Now please generate possible causes for
effect predicate: CONSULT
// Other causes are Appendix E.4.
time: 2022-07-05
## Example
subject: CHINA
object: YELLEN
// Other examples in Appendix E.4.
Listing 1: Format of our LLM prompt.
Listing 2: Few-shot examples in prompt.
The L and M are hyperparameters. Ideally, would like analyze all the possible candidates (i.e., = oo and M = (C)). which is intractable for time prediction and expensive for type prediction. In practice, we focus the most plausible candidates maintain low computation cost. In our experiments, our framework already performs remarkably well with small and M.
3.2 Phase-II: Prompting LLM to Perform Abductive Reasoning
For each proposed event (t, k), our framework selects set of previous events from its full history as its supporting evidence e(1, k). The selection guided by an LLM (e.g., GPT-3.5). Technically, we prompt the LLM to imagine some possible cause events that-under the LLM's belief-would be able to explain the occurrence this proposal. The imaginary cause events may not exactly match any actual event in the history, but we could use them queries to search for the most similar ones.
Prompting is a widely used technique extract knowledge from an LLM. A prompt is a concise statement designed to elicit response from the LLM. It typically includes the task description and few demonstrations. Our prompt follows the format in Listings and
Each LLM- LeM-ggenrrtted cause event used as a query to search for most similar events in the history, where union of the previous events retrieved by the LLM-generated causes. Retrieval is vector-based: we construct query embedding Vg for the query event, and key embedding vx for each actual previous
is hyperparameter. The overall evidence e(t, k) then defined to be the
v.
event; the similarity measured by cosine between Vy and Vı. i.e., |[ IiMv We use pretrained SentenceBERT (SBERT) (Reimers & Gurevych, 2019) as our embedding model. The model takes as input text string concatenating the event time I, the text identifier (e.g., phbject-rrdiicat--ojjeed of the event type h; and-if any-the textual mark m of the event. It returns an embedding for each token, we take the event embedding v to the average of the SBERT embeddings.

---

3.3 Phase-III:: Ranking Proposals
In this phase, our framework scores each proposed event (t, k) based on the compatibility with its retrieved evidence e(t, k). Precisely, the score is defined to be
Sevent (t, k) def exp (ct,, k), k)))
The function takes as input the proposed event (t, k) as well as its evidence e(t, k), and returns scalar R. A high value of e means that this proposal is strongly supported by its retrieved evidence. and thus is more likely to be an actual event at time f; a low value means that this proposal has no strong evidence even after we have tried our best to search from the history.
Given the most probable . events {(t, ) the overall belief of our framework in an event occurring at time t. That is,
m= at time we sum their Sevent scores to measure
M
Stime (1)
def
Sevent (L, k: (m!)))
(5)
Intuitively, this score is high when any of the top-ranked event types this time be strongly supported by the retrieved evidence. Otherwise, even the top-ranked event types have no strong evidence in the history, which implies that the next event is unlikely to occur at this time.
For time prediction, each proposed time (00 (l 1, Em= Sevent (f(0), A.(..))) and our final prediction is the proposal with the highest score.
L++) has a score Stime (7 (mo) precisely,
For type prediction given time I, each proposed type (mm) (m .....: M) has score Sevent (1, k(m)), and our framework takes the final prediction to be the type with the highest score.
Model architecture. Our function e is an energy function with a continuous-time Transformer architecture (Xue et al., 2022). It reads the proposal (t, k) followed by its evidence events in the chronological order, and returns a compatibility score E IR.
We choose this architecture because its continuous-time attention is suitable for our setting. First, the attention mechanism may learn to disregard any of the retrieved events that are not really relevant and focus on those which can actually support the proposal. Second, its sophisticated handling of time may capture how the time of evidence event may influence its relevance to the proposal (e.g.. recent evidence event may be more important than an ancient event).
Training. We train the ranking model by maximizing the objective J del Jactual + 3J where 3 2 is a hyperparameter. The first term Jacual is defined to be
M Σ
Jactual
def log Sevent (ti, ki) - log Sevent k:)) +
Σ
Sevent (ti, k (m)))
(6)
where (t1, ky).... (tz, k2) is sequence of events over the time interval (0. and each k. (17) is the event type with the m-th highest intensity under the base model at time t,. By maximizing J Jacuall the function e learns to increase the scores of the events that have actually happened, but suppress the scores of the non-events times I1, .......,y Ly. The second term Jnn is defined to be
N
J
def
log Stime (4a)
Σ
log Σ
Sevent (b k((m))
where each I, is time point uniformly sampled from (0, T) and each (m) is the event type with the m-th highest intensity at time fq. By maximizing Jno. the function e learns to decrease the scores of the non-events at times over (0, other than f,...., .t..
Although not explicitly mentioned in equations (6) and (7), computing Jacuua and Jna involves searching for the evidence of the actual and proposed events.
Experiments
Our code is at https: / /tt.ttb.tbuuu.g.o//g wwwitttuu..ooiiaaaaeaeeee www/iiaaaaard/aaaa..aamoaa This repository includes CSV files con- taining numerical results of our experiments. also includes qualitative results such as L.LM- generated cause events. Experiment details (c.g., hyperparameters) are in Appendix E.
5

---

4.1 Experimental Setup
We conducted experiments on three real-world datasets (see Appendix E.1 for dataset details).
GDELT (Leetaru & Schrodt, 2013). The GDELT Project monitors events all over the world, with live datasets updated every 15 minutes. We only focused on the political events that happened in 2022-07-31, ending up with a corpus of 109000 time-stamped event tokens. This choice of time range guarantees that our data was not included in the training data of the most recent GPT. The event type k of each token has structured name of the format subject-predcaat--bjjee. Each predicate is one of the twenty CAMEO codes such as CONSULT and INVESTIGATE (see Appendix E.4 for a full list); each subject or object is one of the 2279 political entities (individuals, groups, and states) such as Tesla and Australia. So there are about 104M event types in total, making this dataset extremely challenging. Each event token has a news headline that concisely describes the event. We split the dataset into disjoint train, dev, and test sets based on their dates: the 83100 events that happened before 2022-07-05 are training data; the 16650 events after 2022-07-19 are test data; the 9250 events between these dates are development data.
G20 countries from 2022-01-01
ICEWS (Boschee et al., 2015). Similar to GDELT, this dataset logs interactions between social- political entities. We collected 79410 event tokens from 2022-10-11 I to 2023-02-28. Its event types have the same structure as GDELT: each predicate is one of the twenty CAMEO codes; each subject or object one of the 2981 political entities. We split the dataset into disjoint train, dev, and test sets based on their dates: the 4 41600 events that happened before 2023-01-16 166 are training data; the 22030 events after 2023-02-01 are test data; the 15780 events between these dates are development data.
Amazon Review (Jianmo Ni, 2019). This dataset contains user reviews on Amazon shopping website from 2014-01-04 to 2016-10-02. We focused on the most active 2500 users and each user has a sequence of product review events. The type le is the category of the product: we selected the most frequently-reviewed 23 categories and grouped all the others into a special OTHER category, ending up with 24 categories in total. Each review event also has a mark m which is the actual content of the review. Each of the 2500 sequences is cut into three segments: the events that happened before 2015-08-01 are training data; those after 2016-02-01 are test data; the events between these dates are dev data. Then we have 49,680 training tokens, 7,020 dev tokens, and 13,090 test tokens.
We experimented with four state-of -hhte-art event sequence models: NHP (Mei Eisner, 2017a), Know-Fxovle (KE) (Trivedi et al., 2017), DyRep (Trivedi et al., 2019), and ANHP (Yang et al., 2022). For each of them, we evaluated it as a baseline method as well as integrated it into our LAMP framework. KE and DyRep require domain-specifie knowledge to configure their structural-sparse architectures: we evaluated them on GDELT since their GDELT-specific architectures are available in the original papers; we didn't 't evaluate them on Amazon Review since we do not have such knowledge on this data. ANHP can take domain knowledge into its architecture but it is optional, so we evaluated it on both GDELT and Amazon data: GDELT, we adapt the knowledge used in KE and DyRep into its structure; on Amazon Review, we use the generic architecture. On Amazon Review, we also experimented with NHP since it doesn' 't require any -omaii-specific structure knowledge.
We experimented with three strong I.MMs: GPT-3-davinci (Brown et al., 2020) which we also denote as G3.0; GPT-3.5-turbo (Brown et al., 2020; Stiennon et al., 2020; Gao et al., 2022) which we also denote as G3.5, and --ca-t with 13B parameters (Touvron et al., 2023) which we also denote as llama. For GDELT and ICEWS data, used 10-shot prompts; for Amazon Review data, we used 8-shot prompts. Each "shot" is a demonstration that contains an effect event followed by one or more expert-annotated cause events. Prompt examples can be found in Appendix E.4.
4.2
Main Results on Type and Time Prediction
Our main results are displayed in Figure 2. Figure 2a shows the result of each method on GDELT data. GDELT data is updated every fifteen minutes so the time intervals are regular and thus it is not interesting to predict them. For type prediction, we focus on predicting certain attributes given the others, which is more practical than full type prediction. In practice, predicting "which of the hundreds of millions of events is the most probable" is too difficult and existing models will all perform disastrously. But answering questions like "what will A do to B" and "to whom A will do this" is usually useful enough for real applications. Note that attribute prediction is still very challenging: e.g., there are 45580 distinct predicate-object combinations in GDELT data.

---

220
Figure 2: Prediction perfor- mance of different methods each dataset. On GDELT, the upper figure is for ob- ject prediction, and the lower figure is for predicate- object joint prediction. On ICEWS, the upper figure is for object prediction, and the lower figure is for time prediction. On Amazon Re- view, the upper figure is for type prediction, and the lower figure is for time pre- diction.
1.0
d- AAP
AMP--22
15
60
30
60
129
20
1,0
10
2400
40
1:30
AMP
144
21.40
1:5
10
1.10
125
(a) GDELT.
(b) ICEWS.
(c) Amazon Review.
We evaluate each model by the quality of its top-ranked predictions. For each baseline model (KE, DyRep, or ANHP), the list of top predictions contains the top M event types (with known attributes filled, if any) that have the highest intensities; see section 3.1. Our LLM-enhanced framework takes the list given by its base model, and sorts it based on the Sevent scores of the proposals. Our primary evaluation metric is the mean rank (MR). This metric has straightforwan interpretation: given sorted list of proposed predictions, it measures the average rank the ground-truth type in the list; a smaller MR means higher rank, and thus a better result (e.g., MR = 1 means "ranked at the top on average"). We also used the mean reciprocal rank (MRR), which is less interpretable but more robust to bad predictions than MR. Appendix E.2 includes full procedures for computing these metrics. In our experiments, MR and MRR results yield the same conclusions. So we present MR results in this section for its straightforward interpretation, but leave MRR results to Appendix F.
On ICEWS and Amazon Review, we evaluate each model on time prediction (in addition to type prediction), which is measured by the root of mean squared error (RMSE). For each held-out token, each base model proposes a list of scored predictions; see section 3.1. Our LAMP framework reranks the list given by its base model. In either case, the final prediction is the highest-ranked proposal.
In each evaluation setting shown in Figure 2, our LLM-enhanced framework substantially and consistently outperforms the corresponding baseline model across range of L and M values. All the results throughout the paper have 95% bootstrap confidence intervals but they are too tight to be visually obvious in most cases, implying that our improvements are significant. When we draw more proposals from the base model (i.c., L and M are larger), our framework tends to enjoy a larger improvement over the base model. For predicate-object joint prediction GDELT, the ground-trute -tuth type ranks at around 40 in the lists given by the base models, but is moved to around 20 by our LLM-based adjustment. For object prediction around 90 to about 20. Note that it is not fair to compare the same method across L or M.
ICEWS, our method improved the MR results from
The MR and MRR are only evaluated on the held-out tokens whose ground-truth event types fall into the top proposals. But how many are there such tokens? This solely depends on how well the base event sequence model works. A detailed analysis can be found in Appendix F.2.
4.3 Analysis
Now we present our analysis on GDELT. On GDELT, focus on the predicate and object prediction. It is less expensive than the time and predicate-objed prediction (which requires an order of magnitude more GPT hours and GPT API calls) but the results are well correlated in our pilot experiments. More analysis can be found in Appendix F, including analysis on the other datasets.
Analysis-I: About LLMs. Section 4.2 only shows the results of the GPT-3.5 version of our framework. Figure 3 shows the results of the other versions with ANHP as the base model. These LLMs all help improve the prediction performance but GPT-3.5 and Llama work significantly better than GPT-3. Interestingly, the Llama version performs competitive to the GPT-3.5 version. Its strong performance demonstrates that our framework doesn't have to depend on black-box LLMss; indeed, it can excel with open-source L.MMs like Ilama. Note that Ilama has considerably fewer parameters than GPTs. An important reason for its success may be the reinforcement lcarning from human
7

---

ATTE-C34
XTOP GEs
CAMP CSL-
Gr-23
2.00
1.75
150
125
hum of sced
Dvarts
Rechloed Dridenee
Figure MR of using different LLMs on predicate (left) and object (right) prediction on GDELT.
Figure Effect of the number of evidenee events on predicate (left) and object (right) prediction on GDELT.
26
AMM-
AAHP
AMMP--53 O
2.5
22
AMM
03..
610
015
Nom Shots
Share
Figure 5: Effect of the number of shots: MR of predi- cate (left) and object (right) prediction GDELT.
Figure 6: Effect of the shot choices: MR of predicate (left) and object (right) prediction GDELT.
feedback (RLHF) technique, which adopted by GPT-3.5 and Llama-2-chat, but not by GPT-3. Further investigation may be interesting direction for future work.
Analysis-II: How many evidence events do we need? In our GDELT experiments, we varied the number of retrieved evidence events for each proposed prediction. As shown in Figure 4, the results improve when the number of retrievals increases from 5 to 10, but become worse when the number further goes up. Apparently, if we retrieve too many "evidence" events, then most of them might not be actually helpful so the performance will degrade due to the decreased signal-to-noie also tried another retrieval criteria: instead of retrieving fixed number of previous events, we only retrieve event if its similarity score exceeds prespecified threshold. It turns out that this criteria only had negligible impact on the results.
ratio. We
Prompt design. How will the design of the prompt influence the performance of our LAMP framework? As shown in Figure 5, LAMP performs better when the prompt includes a larger number of demonstrations, and the GPT-3.5 version consistently outperforms the GPT-3 version. Interestingly, even O-shot prompting yields better result than the baseline ANHP model, emphasizing that the I.LMS are indeed very helpful.
Analysis-III:
Figure 6 shows how sensitive our framework is to the choice of demonstrations. We use P1 to denote the set of demonstrations used throughout our main experiments. Then P2 refers new set of demonstrations obtained by randomly replacing half of the demonstrations in Pl. As we can see, changing the demonstrations has only slight effect on performance, and LAMP consistently outperforms ANHP, whether using P1 or P2.
We designed the prompt templates without prior experimentation or tuning any dataset. In post analysis, we tested several different templates on a small subset of GDELT data, and found only minimal variations in results, long as the task description that we tested doesn't 't include the full list of possible predicates (see Listing 1), yet the results with this template closely match those of our original version.
clear. Interestingly, new template
Analysis-IV: About generalization. The LLM in LAMP is instructed by few demonstrations. On GDELT, these demonstrations cover 17 out of 20 (85%) predicates and 30 out of 2279 (1.31%) entities. However, GPT-3.5 is able generalize beyond the demonstrations, suggesting cause events that involve novel predicates and entities. Precisely, the LLM --Meeeeedtedd cause events involve 4699 distinct predicates and 65554 distinet entities, covering all the 20 ground-truth predicates and more than 800 ground-truth entities. Appendix F.3 includes more details about this analysis.
LLMs are known for strong in-context learning capability. In specific problem setting, an LLM may have already understood the meanings of the predicates and been familiar with many entities
For Figures 4 6, tixed M 5 for predicate prediction and M 20 for object prediction.

---

3.66
30
2.00
Von
1.15
15
Figure 7: Predicate (left) and object (right) prediction of different retrieval methods on GDELT.
Figure 8: Effect of similarity metrie: MR of predicate (left) and object (right) prediction on GDELT.
through its pretraining and fine-tuning processes. Consequently, when presented with a proposal, it can follow demonstrations and draw upon its internal knowledge to suggest plausible causes. Benefiting from the strong generalization capability of LLMs, our LAMP framework has significant potential for broad applications.
Analysis-V. V: About retrieval methods. To further investigate the usefulness of L.MMs, we tested several versions of our framework that do not involve the LLM in retrieval: rnd randomly samples 10 previous events from the history for each proposal: rec uses the most recent 10 past events; rec-pre retrieves the most recent 10 events that share the same predicate with the proposal; rec-obj retrieves the most recent 10 events that share the same object; bert uses SBERT to embed the text-baser identifier of each event and retrieve the most similar 10 events based on the cosine similarity of SBERT embeddings. As shown in Figure 7, all these retrieval methods perform significantly worse than our LLM- method. Noticeably, the "most recent" retrievals work poorly. It suggests that "recent" is not good inductive bias for our problem setting. On dev data, found that the cause events retrieved based on LLMM-genrrted clues are often not "recent".
We also tested using the edit distance as the similarity score in our LLM-based retrieval method. For a pair of events, we compute the character-level edit distance (Jurafsky & Martin, 2009) between their text strings that are otherwise fed into the SBERT embedding model (section 3.2), and define the similarity to the reciprocal of this distance. As shown in Figure 8, using this metric yields a better result than the baseline method, but performs worse than original framework.
Analysis- VI: About data leakage. GPT models are trained on online text data up to 2021. This dataset doesn't 't include the GDELT or ICEWS data used in our experiments. Our demonstrations span a wide time range in training data, and there is a large temporal gap between the training and test data. Precisely, the most recent demonstration is about event on 2022-06-07, while the earliest test event occurred on 2022-07-05 Below are the percentiles for the time differences (in days) between the test events and the latest demonstration:
percentage 1%
5%
25%
50%
75%
95%
99%
percentile
41
44
55
55
The GPT training data may have covered the Amazon Review data. Therefore, we had a conversation with GPT-3.5, checking whether it could recall information about specific users specific reviews. GPT-3.5 could say nothing specific about the users or reviews, indicating that we do not have an issue of data leakage. Our conversation can be found in Appendix E.5.
5 Discussion and Related Work
Event reasoning and prediction is an important problem that arises in various real-world applications. A great number of event sequence models have been proposed and advanced this field, including the classical graphical models (Hawkes, 1971; Du et al., 2015a), recurrent neural models (Du et al., 2016; Mei & Eisner, 2017b; Xiao et al., 2017; Omi et al., 2019; Shchur et al., 2020; Boyd et al., 2020), and Transformer-basee models (7uo et al., 2021; Zhang et al., 2020; Enguehard et al., 2020; Sharma et al., 2021; Zhu et al., 2021; Yang et al., 2022; Xue et al., 2022; Liu et al., 2022). These models have been applied to a wide range of applications such as network analysis (Choi et al., 2015; Etesami
-ervv3.--avvinc''s training data goes up to October 2019, while GPT-3.5- --trubb''ss training data goes up to September 2021 (source: htips:/ /ptttttaatteettt form. openai con/ does models/ Ls/ /gppt/ptttte

---

et al., 2016), recommendation systems (Du et al., 2015b), social analysis (Guo et al., 2015; Lukasik et al., 2016; Zhang et al.. 2022), and healtheare (Hua al.. 2022; Zhang al.. 2023b).
Recently, there has been a growing body of research that has directed its focus towards the textual features of the real-world events, such the reports and news articles about the events. This line of work includes Deng et al. (2020, 2021, 2022); Han & Ning (2022). Their methods all treat text as auxiliary features but do not consider reasoning about that text information. Our work is significantly different from this line of research since we focus on reasoning. Our framework reasons about the events and their text information (e.g., text identifiers of the event types, associated documents) by prompting a large language model. The large language model has consumed a massive amount of text during pretraining and aware of diverse world knowledge, thus able perform the kind of reasoning beyond the capacities of the aforementioned models.
By leveraging large language models, our framework induces domain-specific knowledge into the deployment of event sequence models. It is related to previous work of configuring domain-specific knowledge into model architectures (Trivedi al., 2017, 2019; Mei et al., 2020; Yang al., 2022). But our work is significantly different: it doesn't It rely on human domain experts to write down a full set of knowledge; instead, we extract knowledge from language models via few-shot prompting. Prompting has become a standard way of applying large language models to reasoning tasks. There has been a recent surge of work that develops novel prompting techniques for a better performance (Wei et al., b022a,b; Zhou et al., 2023). Our framework is general and can adopt any prompting methods.
This paper is closely related research in logical reasoning, which primarily focuses on proving goals from known facts and rules. A major approach for this problem is backward chaining: it works backward from the goal, chaining through rules to find known facts that support the proof (Russell & Norvig, 2010). This approach has been applied to various application problems, including reasoning and planning in natural language (Ye et al., 2022; Weir & Van Durme, 2022; Kazemi et al., 2023; Kassner et al., 2023). In our setting, each proposed event is treated as goal, and previous events serve as known facts. Like backward chaining, our method identifies previous events that support the proposal. But our method only performs a single step of reasoning, assuming complete data where all events are observable. A second step of reasoning unnecessary since direct causes are observable and more temporally recent than indirect causes. In cases of incomplete data, multiple reasoning steps may be required to identify indirect causes. Handling incomplete data will be a non-trivial extension of our current framework, which we leave to future research.
This paper is also closely related to research in event-centric natural language processing (NLP) (Chen et al., 2021). Over the past decades, there has been a great amount of progress in this area, including many datasets and benchmarks (Ning al., 2020; Li et al., 2020; Han et al., 2021: Wen al., 2021: Feng et al., 2022) as well as a diversity of methods for key problems such as event detection and extraction (Ji & Grishman, 2008; Li et al., 2013; Feng et al., 2016: Lyu et al., 2021; Wang et al., 2021). relation extraction and prediction (Chan & Roth, 2011: Ning et al.. 2018b: 8b; Wang et al., 2020; Wen & Ji, 2021: Li et al., 2022), event induction and summarization (Do et al., 2012: Saravanakumar et al.. 2021; Li et al., 2021b,a; Jin et al., 2022), and temporal reasoning (Ning et al., 2018a; Ballesteros et al., 2020; Zhou et al., 2020, 2021). Recently, there has been research in leveraging LLMS to solve these problems (Dror et al., 2023; Li et al., 2023; Zhang et al., 2023a). Our work complements this line of research, focusing on the problem of event sequence modeling, which has been out of the scope of the classical event-centric NLP.
Conclusion
In this paper, present LAMP, general modeling and prediction framework that leverages the abductive reasoning ability of large language models to help improve the prediction performance of event sequence models. Empirical studies demonstrate that our LLM-enhanced framework can significantly outperform the state-of-the--n event sequence models. Our findings have significant implications for future research in the field of event sequence modeling. In principle, an event sequence model should benefit from a range of reasoning abilities of large language models such as deductive reasoning, inductive reasoning, commonsense reasoning, and arithmetic reasoning. Further exploration in this area may lead to impactful innovations.
10

---

Acknowledgments
This work was supported by research gift the last author by Adobe Research. We thank the anonymous NeurlPS reviewers and meta-reviewer for their constructive feedback. We also thank our colleagues at Ant Group, UChicago, and TTIC for helpful discussion.
References
Ballesteros, M., Anubhai, R., Wang. S., Pourdamghani, N., Vyas, Y., Ma, Bhatia, P., McKeown, K., and Al-Onaizan, Y. Severing the edge between before and after: Neural architectures for temporal ordering of events. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Boschee, E., Lautenschlager, J., O'Brien, S., Shellman, S., Starz, J., and Ward, M. ICEWS Coded Event Data, 2015.
Boyd, A., Bamler, R., Mandt, S., and Smyth, P. User-dependent neural sequence models for continuous-ting event data. Advances Neural Information Processing Systems (NeurIPS), 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P.. Sastry, G., Askell, A., al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chan, Y. and Roth, D. Exploiting syntactico-semanne structures for relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2011.
Chen, M., Zhang, H., Ning, Q., Li, M., Ji, H., McKeown, K., and Roth, D. Event-centric natural language processing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Choi, E., Du, N., Chen, R., Song, L., and Sun, J. Constructing disease network and temporal progression model via context-sensitive Hawkes process. In IEEE International Conference on Data Mining (ICDM), 2015.
Deng. S., Rangwala, H., and Ning, Y. Dynamic knowledge graph based multi-event forecasting, In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
Deng, S., Rangwala, H., and Ning, Y. Understanding event predictions via contextualized multilevel feature learning. Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pp. 342-351, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469,
Deng, S., Rangwala, H. and Ning, Y. Causality enhanced societal event forecasting with het- erogeneous graph learning. 2022 IEEE International Conference on Data Mining (ICDM), 2022.
Do, Q., LU, W., and Roth, D. Joint inference for event timeline construction. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), 2012.
Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. In Findings of the Conference of the European Chapter the Association for Computational Linguistics (Findings of EACL), 2023.
Du, N., Farajtabar, M., Ahmed, A., Smola, A. J., and Song, L. Dirichlet-Hawkes processes with applications to clustering continuous-th document streams. In Proceedingy of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015a.
Du, N., Wang, Y., He, N., Sun, J., and Song. I.. Time-sensitive recommendation from recurrent user activities. In Advances in Neural Information Processing Systems (NeurIPS), 2015b.

---

Du, N., Dai. H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
Enguehard. J., Busbridge, D., Bozson, A., Woodcock, C., and Hammerla, N. Neural temporal point processes [for] modelling electronic health records. In Proceedings of Machine Learning Research, volume 136. pp. 15-113, 2020. NeurIPS 2020 Workshop on Machine Learning for Health (ML4H).
Etesami, J., Kiyavash, N., Zhang, K., and Singhal, K. Learning network of multivariate Hawkes processes: A time series approach. arXiv preprint urXiv:1603.0433 2016.
Feng, X., Huang, L., Tang, D., Ji, H., Qin, B., and Liu, T. A language-indeependet neural network for event detection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
Feng, Y., Zhou, B., Wang. H., Jin, H., and Roth, D. Generic temporal reasoning with differential anal- ysis and explanation. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL), 2022.
Gao, L.,, Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210. 10760, 2022.
Guo, F., Blundell, C., Wallach, H., and Heller, K. The Bayesian echo chamber: Modeling social influence via linguistic accommodation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2015.
Han, R., Hsu, I.-H,, Sun, J., Baylon, J., Ning, Q. Roth, D., and Peng, N. Ester: A machine reading comprehension dataset for reasoning about event semantic relations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Han, X. and Ning. Y. Text-enhanced multi coaliiigamuulriity temporal graph learning for event prediction. In Zhu, X., Ranka, S., Thai, M. T., Washio, T., and Wu, X. (eds.), IEEE International Conference on Data Mining, ICDM 2022, Orlando, FL, USA, November 28- Dec. 1, 2022, pp. 171-180. IEEE, 2022.
Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971.
Hua, W., Mei, H., Zohar, S., Giral, M., and Xu, Personalized dynamic treatment regimes in continuous time: A Bayesian joint model for optimizing clinical decisions with timing. Bayesian Analysis, 2022.
Ji, H. and Grishman, R. Refining event extraction through cross-document inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL),, 2008
Jianmo Ni, Jiacheng Li, J. M. Justifying recommendations using distantly-labeled reviews and find-grainee aspects. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
Jin, X., Li, M. and Ji, H. Event schema induction with double graph autoencoders. In Proceedings of the Conference the North American Chapter of the Association for Computational Linguisties (NAACL), 2022.
Jurafsky, D. and Martin, J. H. Speech and language processing introduction to natural language processing. computational linguistics, and speech recognition. 2009.
Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schutze, H., and Clark, P. Language models with rationality. arXiv preprint arXiv: 14250, 2023.
Kazemi. S. M., Kim, N., Bhatia, D., Xu, X., and Ramachandran. D. Lambada: Backward chaining for automated reasoning in natural language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
Leetaru and Schrodt. Gdelt: Global data on events, location, and tone, 1979-2012, ISA annual convention, 2, 04 2013.
12

---

Lewis, P. A. and Shedler, G. S. Simulation of nonhomogeneous Poisson processes by thinning. Naval Research Logistics Quarterly, 1979.
Li, M., Zareian, A., Zeng, Q. Whitehead, S., Lu, D., Ji, H., and Chang, S.-F. Cross-media structured common space for multimedia event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., and Voss, C. Future not one- dimensional: Graph modeling based complex event schema induction for event prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021a.
Li, M., Ma, T., Yu, M., Wu, L., Gao, T., Ji, H., and McKeown, K. Timeline summarization based on event graph compression via time-aware optimal transport. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021b.
Li, Q. Ji, H., and Huang, L. Joint event extraction via structured prediction with global features. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2013.
Li, L., Ji, H., and Han, J. Open relation and event type discovery with type abstraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
Li, S., Zhao, R., Li, M., Ji, H., Callison-Buree C., and Han, J. Opendomain hierarchical event schema induction by incremental prompting and verification. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL), 2023.
Liniger, T. J. Multivariute Hawkes processes. Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 18403, 2009.
Liu, S., Yu, H., Liao, C., Li, Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low- compcexpyy pyramidal attention for long-range time series modeling and forecasting. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Lukasik, M., Srijith, P. K., Vu, D., Bontcheva, K., Zubiaga, A., and Cohn, T. Hawkes processes for continuous time sequence classification: An application to rumour stance classification in Twitter. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
Lyu, Q. Zhang. H., Sulem, E., and Roth, D. Zero-shot event extraction via transfer learning: Chal- lenges and insights. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Mci, H. and Eisner, J. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems (NeurIPS), 2017a
Mei, H. and Eisner, J. The neural hawkes process: A neurally self-modulating multivariate point process. In Proceedings of the International Conference on Machine Learning (ICML), 2017b.
Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Ning, Q., Feng, Z., Wu, H., and Roth, D. Joint reasoning for temporal and causal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018a.
Ning, Q., Wu, H., and Roth, D. A multi-axis annotation scheme for event temporal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018b.
Ning, Q. Wu, H., Han, R., Peng, N., Gardner, M., and Roth, D. Torque: A reading comprehension dataset of temporal ordering questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
13

---

OpenAl. Gpt-4 technical report. arXiv preprint 2:x3::2303.8874,
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. 2017.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach. Prentice Hall, 2010.
Saravanakumar, K. K., Ballesteros, M., Chandrasekaran, M. K., and McKeown, K. Event-driven news stream clustering using entity-aware contextual embeddings. In Proceedings of the Conference of the European Chapter the Association for Computational Linguistics (EACL), 2021.
Sharma, K., Zhang. Y., Ferrara, E., and Liu, Y. Identifying coordinated accounts social media through hidden influence and group behaviours. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2021.
Shchur, O., BiloK, M., and Günneeann, S. Intensity-free learning of temporal point processes. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Stiennon, N., Ouyang, L., Wu, Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint S.Iiv:2307.092888 2023.
Trivedi, R., Dai, H., Wang. Y., and Song, L. Know-Evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Trivedi, R., Farajtabar, M., Biswal, P., and Zha, H. DyRep: Learning representations over dynamic graphs. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Wang, H., Chen, M., Zhang, H., and Roth, D. Joint constrained learning for event-event relation ex- traction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Wang, H., Zhang, H., Chen, M., and Roth, D. Learning constraints and descriptive segmentation for subevent detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D.,, Bosma, M., Zhou, D.. Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O.,, Liang. P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions of Machine Learning Research (TMLR), 2022a.
Wei, J.. Wang, X., Schuurmans, D., Bosma, M., Chi, H. Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Weir, N. and Van Durme, B. Dynamic generation of interpretable inference rules in a neuro-symbolic expert system. arXiv preprint arXiv:2209.07662. 2022.
Wen, H. and Ji, H. Utilizing relative event time to enhance event-event temporal relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Wen, H., Qu, Y., Ji, H., Ning, Q. Han, J., Sil, A., Tong, H., and Roth, D. Event time extraction and propagation graph attention networks. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
14

---

Xiao, S., Yan, J., Farajtabar, M., Song, L.. Yang, X., and Zha, H. Joint modeling of event sequence and time series with attentional twin recurrent neural networks, arXiv preprint arXiv: 0ee.17733....44 2017.
Xue, S., Shi, X., Zhang, Y. J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long- nong-horizon prediction of event sequences. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Xue, S., Shi, X., Chu, Z., Wang, Y., Zhou, F., Hao, H., Jiang, C., Pan, C., Xu, Y., Zhang, J. Y., et al. Easytpp: Towards open benchmarking the temporal point processes. arXiv preprint arXiv:2307.08097, 2023.
Yang, C., Mei, H., and Eisner, Transformer embeddings of irregularly spaced events and their participants. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Ye, A., Cui, C., Shi, T., and Riedl, M. O. Neural story planning. arXiv preprint arXiv:2212.087/8, 2022.
Zhang. Q., Lipani, A., Kimap, O., and Yilmaz, E. Self-attentive hawkes processes. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Zhang, T., Tham, I., Hou, Z., Ren, J., Zhou, L., Xu, H., Zhang, L., Martin, L. Dror, R., S., et al. fluman-in-the-lotee schema induction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023a.
Zhang, Y., Cao, D., and Liu, Y. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Zhang, Z., Mei, H., and Xu, Y. Continuous-time decision transformer for healthcare applications. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2023b.
Zhou, B., Ning, Q., Khashabi, D., and Roth, D. Temmporl common sense acquisition with minimal supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Zhou, B., Richardson, K., Ning. Q., Khot, T., Sabharwal, A., and Roth, D. Temporal reasoning on implicit events from distant supervision. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
Zhou, D., Schärli, N., Hou, I.., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Zhu, S., Zhang, M., Ding, R., and Xie, Y. Deep fourier kernel for self elf-attennivee point processes. In Proceedings of the International Conference Artificial Intelligence and Statistics (AISTATS), 2021.
Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
15

---

Appendices
A
Societal Impacts
Our paper develops a novel method to integrate large language models into temporal modeling. By describing the model and releasing code, we hope to facilitate the modeling of continuous-time sequential data in many domains. However, like many other machine learning models, our method may be applied unethical ends. For example, its abilities of better fitting data and making more accurate predictions could potentially be used for unwanted tracking individual behavior, e.g. for surveillance.
B Limitations
Our framework utilizes LLMS such GPT-3.5 and thus inherits the issues of those models such as hallucination and biased content. Therefore, the cause events generated by LLMs may be irrelevant, incorrect, or misleading, thus harming the overall performance of our framework. Also because of using LLMs, our approach requires human-craftin prompts, and the events need to have textual information (e.g., event types have textual identifiers or marks).
Our framework needs a pretrained event sequence model, and its overall capability is partially determined by this base model. When the base model can handle events with duration (i.e., events that last for a period of time, such as holding a tool), our framework will inherit this merit. But our framework will also inherit some of the technical limitations of its base event sequence model. If the base model can only handle a finite set of event types, then our framework will inherit this limitation. If the base model misspecified or poorly trained, our framework may not work well.
Our compatibility function is Transformer-base neural network which is known be data-hungnn Though it worked well in our experiments, it might struggle when starved of data.
C Possible Extensions
Our framework currently trains each component separately. A natural improvement will be to jointly train its three components. In principle, we can iteratively
refine the base event sequence model with the feedback from the LLM and ranking model:
learn prompt the LLM that generates cause events of higher quality;
train the ranking model with proposals and evidence given by the improved base model and LLM.
Another interesting extension of our framework is to apply its propose-justify-rank pipeline to other problems. At the core of our framework is the LLM, which is able examine each proposed prediction drawn from the base model, and suggest clues to justify its validity. This idea seems to be a natural fit wide range of reasoning problems. Question answering is an example: given a question, an existing model proposes multiple plausible answers; an LLM reads each proposed answer and reasons about its preconditions; search module finds out evidence (from local knowledge bases or the open internet) which may match the LLM-suggested preconditions; a ranking model learns to score each combination of answer and evidence.
D Method Details
D.1
ANHP
On Amazon data, the ANHP model used is directly from Yang al. (2022). On GDELT and ICEWS data, we modified the definition of event embedding so that it easier to scale up to hundreds of millions of event types. Particularly, ANHP defines dynamic event embedding [t] for the event type k at time t. On GDELT and ICEWS, each event type is tuple of sabjjet-predicate-obiieee we redefine the event embedding be
[t) concat (IK subject (t)). [(rrecicate]))). [bojeect](B))
(8)
and keep everything clse the same in the original ANHP.
16

---

This modified version is more scalable since it needs to maintain a significantly smaller number of dynamic embeddings: there are hundreds of millions possible event types, but there are only thousands of possible subjects and objects.
E
Experimental Details
E.1
Dataset Details
Table 1 shows statistics about each dataset in our experiments.
DATASET
OF
# OF
# OF EVENT TOKENS
ENTITIES
PREDICATES
TRAIN
DEV
TEST
GDELT ICEWS
2279 2981
20
83100
9250
16650
20
41600
15780
22030
AMAZON
21
49680
7020
13090
Table Statistics of datasets.
E.2 Evaluation Metric Details
Now we give the formal definitions of MR and MRR. The primary evaluation metric is the mean rank (MR): for each method, we initialize N = R O; for each held-out token, the method provides a list of MM proposed predictions sorted in the decreasing order of their scores; if the ground-trute truth type is among the proposals, we update N += 1 and R + where is the rank of the ground-true truth type in this list; the mean rank is R/N and the lower is the better.
The mean reciprocal rank (MRR) is as follows: for each method, we initialize N= held-out token, the method provides list of M proposals sorted in the decreasing order of their scores; if the ground-truth type is among the proposals, we update N += and It += 1/r where the rank of the ground-truth type this list; the MRR R/N and the higher is the better.
= R = O; for each
Note that MR and MRR are not affected by the fact that there might be multiple ground-truth attributes. That is because the contribution of jth ground truth (among the multiple) to the final number is only dependent on its rank in the list but not its j index.
Next we give the precise definitions of MAP and MAR.
MAP is computed follows. We first initialize N C = 0. Then, for each partial held-out event (i.e., only i given in Amazon Review, and some attributes given in GDELT), the model gives its top M predictions on the attribute of interest (e.g., type k Amazon Review and "object" on GDELT). There might be multiple ground-truth attributes for a held-out event since multiple events may be recorded the same time (due time quantization and recording errors). If any of the ground-truth is in the top M list, we update N and C: first, we have N += J where J S M is the number of the ground-truth attributes that are covered in the top M list; then, for 5th covered bround-truth, C' C'+= j/RR where R is its rank in the top M list. In the end, MAP is defined as C/N. Each i/ R is pseudo-count for this event: it in (0, 1] since the rank of the jth correct prediction will surely 2 j: if it ranks at the top, i/ R is close to and it is counted as "predicted correctly"; if it ranks at the bottom, i/ is close to O, meaning that is almost missed by the top M proposals.
let
MAR is defined as follows. We first initialize N = C = O. For each partial held-out event, the model gives its top M proposals on the attribute of interest; for the jth correct prediction in the list, let C' + j/ R where A is its rank in the top M list; in the end, MAR is defined as C/N where N is the number of actual events.
E.3
Implementation Details
All models are implemented using the PyTorch framework (Paszke et al., 2017).
For the implementation of NHP, AuNHP and energy functions, we used the code from the pub- lie Github repository at https: htrp::://iiie //gittubb../ com/ ant weesearrch/taayyeemprr.aa (Xue et al., 2023) with Apache License 2.0.
17

---

For the implementation of Know-Evolve, we used the code from the public Github repository at attps://gitthnn com/ wstriv/Know-Eveeee
(Trivedi et al., 2017) without any license.
For the implementation of DyRep, we used the code from the public Github repository at https: rg/L.GG (Trivedi al., 2019) without any license.
For the implementation of MAR@M and MAP@M, we used the code from the public Github reposi- wttte/errcccccce/eeecceececceeec without any license.
tory at
For the implementation of Levenshtein distance, we used the code from the public Github repository at httpp:::.....
http:///.......aaacccca.aaaaooaa itw.eccccnannn.eeeeeesssteiiiii git with General Public License 2.0.
To compute the dense representations of text descriptions of events, the code from the public Github repository at https: //gi thub. www/c//m///////c sent teeeeteeeeeeeeeeeeeeeee forners with Apache License 2.0.
E.4 Prompts
We show one example of the prompt structures used for GDELT dataset as below. The effect event consists of an event predicate, event time, subject name and object name while the cause events consists of an event headline additionally.
I want you do the reasoning and you give four or five cause events. effect event happens. A cause event is an event that is believed to one of the causes that trigger an effect event to happen. Each event consists of an event headline, an event predicate, an event time, subject name and object name of describing the
social events. given you effect event event that
The predicates of the effect and cause events are restricted to 20 options, with names (in capital) and the descriptions below.
1. MAKE STATEMENT: decline comment, make comments.
APPEAL: appeal for material, economic, military, humanitarian aid cooperation.
3. EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic,
military aid.
4. CONSULT: make visit, host visit.
ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.
6. ENGAGE IN MATERIAL COOPERATION: cooperate economically, militarily, judicially.
7. PROVIDE AID: provide economic, military, humanitarian aid.
YIELD: ease admin or political sanctions or military blockade, return release.
9. INVESTIGATE:
inveatigate crime, corruption, human rights abuses, military
actions.
10. DEMAND: demand any type of cooperation, aid, reforms, rights, easing of sanctions.
11. DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses,
complain officially and make larsuit against.
12. REJECT: reject material, economic, military, judicial operations, requests plans.
13. THREATEN: threaten to reduce aid, to boycott, to reduce or break relations,
to impose sanctions.
14. PROTEST: civilian demonstrations.
15. EXHIBIT MILITARY POSTURE.
REDUCE RELATIONS: reduce or break any relations.
17. COERCE: seize or damage properties, impose administrative sanctions or restrictions.
18. ASSAULT: use of unconventional forms of violence.
FIGHT: of conventional force. VIOLENCE.
ENGAGE UNCONVENTIONAL
Now I give you 10 examples of event reasoning. In each example, the first event is the effect event, the next three to five events are cause events that happen earlier.
18

---

## Example 1
effect
predicate: APPEAL
time: 2022-04-23
subject: GERMANY
object: GREEN PROJECT
reasoning: cause event
predicate: REDUCE RELATIONS
time: 2022-04-21
subject: EUROPE
object: RUSSIA
headline: Europe determined ban Russian energy exports.
cause event
predicate: DISAPPROVE
time: 2022-03-16
subject: EUROPE
object: RUSSIAN
headline: Europe endure painful transition to live without Russian oil.
cause event
predicate: ENGAGE IN DIPLOMATIC COOPERATION
time: 2022-03-05
subject: BUSINESS
object: GOVERNMENT
headline: 0il prices surge to multi-year highs as Ukraine conflict causes ripple effect in global oil supplies.
cause event
predicate: REJECT
time: 2022-03-04
aubject: SENATOR
object: RUSSIA
headline: Marshall, Moran seek ban Russian oil importa, urge expansion domestic production.
cause event
predicate: DISAPPROVE
time: 2022-03-03
subject: TRADER
object: RUSSIA
headline: Energy markets in turmoil European gas climbs 60%
Listing prompt example used GDELT.
We show one example of the prompt structures used for ICEWS dataset below. The effect event consists of an event predicate, an event time, subject name and an object name while the cause events consists of an event sub predicate additionally.
I want you to do the reasoning over social events. I given you effect event and you give four five cause events. An effect event is event that happens. cause event is event that believed to one of causes that trigger an effect event to happen. Each event consists an event text, an event predicate, an event time, aubject name and object name of describing the event.
The predicates of the effect and names (in capital)
reatricted 20 options, with descriptions below.
1. MAKE STATEMENT: decline comment, make comments.
2. APPEAL: appeal for material, economic, military, humanitarian aid or cooperation.
19

---

3. EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic,
military aid.
CONSULT: make a visit, host visit.
ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.
6. ENGAGE IN MATERIAL COOPERATION: cooperate economically; militarily, judicially.
PROVIDE AID: provide economic, military, humanitarian aid.
8. YIELD: ease admin or political sanctions or military blockade, return
release.
9. INVESTICATE: investigate crime, corruption, human rights abuses, military
actions.
DEMAND: demand any type cooperation, aid, reforms, rights, easing of sanctions.
11. DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses, complain officially and make lausuit against.
12. REJECT: reject material, economic, military, judicial operations, requesta plans.
13. THREATEN: threaten to reduce aid, to boycott, to reduce or break relations,
to impose sanctions.
14. PROTEST: civilian demonstrations.
EXHIBIT MILITARY POSTURE.
16. REDUCE RELATIONS: reduce or break any relations.
17. COERCE: seize or damage properties, impose administrative sanctions or
restrictions.
18. ASSAULT:
of unconventional forms of violence.
19. FIGHT: uses of conventional force.
20. ENGAGE IN UNCONVENTIONAL MASS VIOLENCE.
Now give you 10 examples of event reasoning. In each example, the first event is the effect event, next three five events are cause that happen earlier.
## Example 1
effect
predicate: DISAPPROVE
time: 2022-10-28
subject: CHINA
object: JAPAN
reasoning:
cause event
predicate: ENGAGE IN DIPLOMATIC COOPERATION
time: 2022-10-18
subject: Japan
object: USA
predicate: Rally support on behalf
cause event 2
predicate: ENGAGE IN MATERIAL COOPERATION
time: 2022-10-16
subject: SOUTH KOREA
object: UNITED NATIONS sub predicate: Cooperate militarily.
cause event
predicate: ENGAGE IN DIPLOMATIC COOPERATION
time: 2022-10-13
subject: SOUTH KOREA
object: JAPAN
sub predicate: Praise or endorse.
cause event
20

---

predicate: MAKE STATEMENT
time: 2022-10-10
subject: CHINA
object: JAPAN
sub predicate: Make pessimistic comment.
Listing prompt example used for ICEWS.
The following is prompt exemplar used for the Amazon Review dataset. Each effect event consists of the product category and event time while each cause event also includes the content of the product review.
I want you to do the reasoning over the events that are extracted from online- shopping review data. given you effect event and you give two to four cause events. An effect event that is believed to one the causes that trigger effect event to happen. Each event corresponds to an review submitted by the customer, which consists of an product category (event type), a product title, event time, summary text and review text from the user that describes the feedback of the shopping event.
that happens.
event event
The product categories are restricted the following set:
1. Women Shoes,
2. Men Shoes,
Men Clothing,
Women Clothing,
Novelty & More,
6. Men Uniforms, Work Safety,
7. Women Jewelry, Costumes Accessories, Men Accessories,
10. Luggage Travel Cear,
11. Men Watches,
12. Women Accessories,
Children Shoes,
Children Clothing
15. Shoe, Jewelry Watch Accessories,
16. Women Watches,
17. Women Uniforms, Work & Safety,
Men Surf, Skate Street,
19. Women Handbags & Wallets
20. Men Jewelry
21. Children Accessories
Women Maternity
Women General
24. Others
Example
effect event
product category: Luggage & Travel Gear event time: 2013-10-19
reasoning:
cause event
product category: Novelty & More
product title: Sports Katz Peace Out Socks
event time: 2013-09-24
summary text: Peace Out Socks
review text: We ordered these for soccer for my daughter and they worked out well. quite thick, which is why only gave them 4 stars. They will work better for fall season; would not recommend them for summer soccer, if soccer is your intended use.
They very cute and have held up. They
cause event
21

---

product category: Luggage & Travel Gear
product title: Wildkin Pink Retro Kickstart Messenger Bag event time: 2013-09-09
summary text: Wildkin Kickstart Messenger Bag
review text: My daughter, going to 4th grade, wanted a messenger type bag this year for school. I searched and searched of the adult bags seemed huge (like they would definately look funny with her carrying them). Thank goodness found this bag. is the perfect size for her, and her favorite color!.
Amazon for the perfect bag, but all
event
product category: Children Clothing
product title: Levi's Girls' 715 Thick Stitch Bootcut Jean
event time: 2013-09-01
summary text: Thick Stitch Bootcut Jeans
review text: I love Levis and this style is cute! Ny daughter likes them! The price was amazing these, as we found similar Levis at Dillard's for 36. The waist of these Levi in particular run amall compared to other Levis of the same size. During the Levis sale on Amazon, we purchased these, the Sweetie Skimmers, and the Scooter. All of the others the waist. worry larger size would huge on her, I think will try the larger size, as they must comfortable on her or she will not wear them.
almost half inch larger in
cause event
product category: Children Shoes
product title: New Balance KJ750 Running Running Shoe (Lttle Kid/Big Kid)
event time: 2013-02-02
summary text: Cute and durable!
review text: I ordered these for my 8 year old daughter. She usually destroys shoes within months, these have held up!
Listing 5: A prompt example used for the Amazon data.
For the full of prompts used in the paper, please see scripts/gaelt/gdes p and scripts/ amazen/ amazon _aazpn__proppts s in our code.
E.5 Does GPT remember Amazon Review data?
We have conversion with GPT-3.5-turo verify that it does not remember the exact information of Amazon Review data.
input: are using Amazon Review Data, published by Jianmo Ni, al., 2018, which is firstly used the paper "Justifying recommendations using distantly- labeled reviews and fined-grained aspects", EMNLP 2019.
The dataset includes millions of reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features) in the range May 1996 Oct 2018. Do you remember any of the review information if I give you a review id?
GPT-3.5 output:
capable of remembering any review information unless have been programmed store and retrieve specific pieces of information. However, I can search for and retrieve review information from the dataset if given review ID.
language model,
input: can you recall any information with reviewer ID A28GK1C2KDXHRP on 17, 2014.
CPT-3.5 output: apologize, but the Amazon Review dataset does not include information about the specific date when review poated. It only includes the year month when the review was posted. Therefore, cannot provide you with any information about reviews posted by reviewer ID A28GK1G2KDXHRP on April 17, 2014.
22

---

input: Reviewer A0149966UU69JJS3Y9011 bought Hanes Men's Classic Crew on 01 18, 2014, what review did he wrote?
GPT-3.5 output: apologize, but as an AI language model, do not have access to real-time or updated information about specific Amazon reviews or purchases made individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of apecific review. It only includes metadata auch as the reviewer ID, product ID, and overall rating.
Therefore, I cannot provide you with the review written by the reviewer 10148968UM5SJ33Y0D11 for the Hanes Men's Classic Crew on January 18, 2014 any other date.
input: Reviewer A0148968UM59JS3Y80011 bought Hanes Men's Classic Crew on 01 18, 2014, his review starts with 'they seem very well made..', can you recall the complete review he wrote?
apologize, but as an AI language model, do not have access to real-time or updated information about apecific Amazon reviews or purchases made by individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of specific review. It only includes metadata such as the reviewer ID, product ID, Therefore, I cannot provide you with the complete review written by the reviewer 001489680MM90133Y8D1 for the Hanes Men's Classic Crew on January 18, 2014, or any other date.
GPT-3.5 output:
overall rating.
input: on which date did reviewer 40148969UX69133YYDDXX buy Hanes Men's 7-Pack ComfortSoft Briefs?
Sorry, information or purchase history individuals.
GPT-3.5 output:
AI language model, I do not have access to personal
Listing 6: A conversation with GPT-3.5-turbo to test its memorization about Amazon Review data.
E.6 Training and Inference Details
In this section, we present the details about training the models and doing inference with them.
LLMs. We use GPT-3-davinci and GPT-3.5-turbo, We set temperature to be 0 to ensure that our results are reproducible.
Training Base Models. For each model, did grid search and chose the hyperparametees based on their performance on the dev set; see Table 2 for the values of the hyperparameters. For KE, DyRep, ANHP, the main hyperparameters to tune are the entity embedding size D1, predicate embedding size D2, RNN hidden size D3 used in the network and the number of layers L of the attention structure RDyRep and ANHP). In practice, the optimal D for model was usually 4, 8, 16, 32; the optimal was usually 2, 3, 4. In the experiment, to train the parameters of the base model, we performed early stopping based on logllikelihood on the held-out dev set.
Retrieval. For each event, we use few-shot prompting (see Appendix E.4 for examples) to generate set of cause events. For each L.M--genertedd cause, we retrieve the most similar /) actual events from the history, following the procedure section 3.2.
For GDELT,
D = and the average number of retrieved events is 10.
For ICEWS, we D) 2 and the average number of retrieved events 10.
For Amazon Review, we set 4 and the average number of retrieved events 10.
Training Ranking Model. The energy function used in our ranking model is the same as what's proposed in Xue et al. (2022), which consists continuous time Transformer and an MLP. The hyperparameters are tuned within a range of values that make the score function to have a similar size as the base ANHP model. Training this ranking model needs negative samples. On GDELT and ICEWS, we use 5 negative samples forpredicate prediction and 20 for object and predicate-object prediction. On Amazon Review, use 5 for both type and time prediction.
23

---

MODEL
DESCRIPTION
VALUE USED
GDELT
ICEWS
AMAZON
KNOW-EVOLVE
ENTITY EMBEDDING SIZE
16
16
NA
PREDICATE EMBEDDING SIZE
4
4
NA
RNN SIZE
16
16
NA
DYREP
ENTITY EMBEDDING SIZE
16
16
NA
PREDICATE EMBEDDING SIZE
4
4
NA
RNN HIDDEN SIZE
16
16
NA
ATTENTION LAYERS NUMBER
1
1
NA
NHP
RNN HIDDEN SIZE
NA
NA
36
ENTITY EMBEDDING SIZE
NA
NA
16
PREDICATE EMBEDDING SIZE
NA
NA
8
ANHP
ENTITY EMBEDDING SIZE
32
PREDICATE EMBEDDING SIZE
4
4
32
HEADS NUMBER
2
1
1
ATTENTION LAYERS NUMBER
1
1
2
RANKING MODEL.
TEMPORAL AL EMBEDDING SIZE
40
40
24
HIDDEN SIZE
60
60
56
HEADS NUMBER
4
3
16
ATTENTION LAYERS NUMBER
3
3
3
Table Values of hyperparameters used for models trained on three datasets.
ET
J UP 4 uPea
Figure 9: Prediction perfor- mance measured by MRR of different methods each datasct. The configura- tion of this figure is the same as that of Figure GDELT, we show ob- ject (upper) and predicate- object (lower) prediction; on ICEWS, we show object (upper) and time (lower) pre- diction; on Amazon Review, we show type (upper) and time (lower) prediction.
3.00
0.0
2.0
<122
1.30
AMP
AMB
615
21.40
6.15
125
(a) GDELT.
(b) ICEWS.
(c) Amazon Review.
Computation Cost. All the experiments were conducted server with 256G RAM. 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and one NVIDIA A100 GPU for acceleration. The training batch size is 8. For GDELT and ICEWS, the wall-clock training time is: 4.9ms per sequence for KE; 6.7ms per sequence for DyRep: 28.7ms per sequence for ANHP. For Amazon Review, the wall-clock time is: 3.6ms per sequence for NHP; 5.2ms per sequence for ANHP. For each the datasets, the wall-clock time of training the ranking model is 9ms per training sample.
F Additional Results and Analysis
In this section, we present additional results and analysis that complement section
F.1
Main MRR results.
Our main results in section 4 are the MR results. In this section, we show the corresponding MRR results, which are known be more robust to bad predictions. The MRR results in Figure 9.
F.2
Analysis About Precision and Recall
Figure 10 shows the mean average precision (MAP) and mean average recall (MAR) of each base model for range of M. These metrics complement MR and MRR for evaluating the base models: MAP measures fraction of the predictions that are correct; MAR measures the fraction of
24

---

042
60h
3.00
[ 0.00
6.4
602
+ E
0.69
0.02
0.50
20
035
10
50 10
(a) MAP on predicate pre- (b) MAR on predicate pre-
(c) on object predic-
(d) MAR on object predic- tion.
diction.
diction.
tion.
Figure 10: MAP@M and MAR @ GDELT dataset.
actual events that covered by the predictions; higher is better. Their precise definitions (and formulas) for our setting can be found Appendix E.2.
As we can see in Figure 10, when M increases, MAR monotonically increases but the change of MAP may not be monotonic (which is consistent with how MAP and MAR behave other retrieval tasks). For predicate prediction, we found that M 5 yields good balance between precision and recall as well as low financial cost (i.e., GPT API calls). For object prediction, M = 50 seems to give the best precision-rectt balance but chose M 20 due to the GPT API budget. For object prediction, both MAP and MAR are lower than 0.1 (but still significantly better than a random guess of 0.0015), indicating that this data is very difficult to model.
E33
Analysis About Generalization
The LLM in LAMP is instructed by a few demonstrations. On GDELT, these demonstrations cover
17 out of 20 (85%) predicates;
30 out 2279 (1.31 () subjects;
28 out of 2279 (1.22%) objects.
However, the LLM is able generalize beyond the demonstrations, suggesting cause events that involve novel predicates and entities. Precisely, the LLM-ggnnrraed cause events involve
4699 distinct predicates, which cover all the 20 ground-truth predicates;
60552 distinct subjects, which cover 30 (of 30) subjects mentioned in demonstrations, as well as 787 not-mentioned ground-truth subjects.
65554 distinct objects, which cover 28 (of 28) objects mentioned in demonstrations, as well 788 not-mentioned ground-truth objects.
The retrieved cause events cover:
all the 20 ground-truth predicates.
30 (of 30) subjects in demonstrations, as well as 186 not-mentioned ground-truth subjects.
28 (of 28) objects in demonstrations, as well as 216 not-mentioned ground-truth objects.
Note that retrievals involve fewer distinct subjects and objects than the LLM-generated causes. because the LLM generation is often diverse and creative, proposing many novel subject-predicate- object combinations that involve ground-truth entities but haven't actually happened real history. Such causes will eventually be grounded to actual previous events that exhibit less diversity and creativity, thus ending up with fewer distinct subjects and objects in the retrievals. Benefiting from the strong generalization capability of LLMs, our LAMP framework has significant potential for broad applications.
F.4 Additional Results About LLMs
Now we present our additional results of comparing different LLMss. They include the MR results of comparing GPT-3 and GPT-3.5 on GDELT (Figures and 12), MR results of comparing GPT-3 and GPT-3.5 on Amazon Review (Figure 13), and MRR results corresponding to Figure 3 (Figure 14).
25

---

235
6563
(8-30
250
200
Figure 11: Different LLMs on predicate (left) and ob- ject (right) prediction on GDELT dataset, with KE,
Figure 12: Different LLMs on predicate (left) and ob-
ject (right) prediction GDELT dataset, with DyRep
222
MP MIR-AS
che CAM
0.80
B CLM
2.0
clee Dt..
$1.75
70.00
1.00
0.55
3.3
Figure 13: Different LLMs type prediction Ama- zon Review.
Figure 14: MRR results of different LLMs predicate (left) and object (right) prediction on GDELT.
F.5
Additional Results About Prompt Design
Now present additional results about the prompt design analysis in section 4. Figure 15 and Figure 16 show the MRR results corresponding to Figure and Figure 6, respectively.
2.25
0.56
Tedarcee
20
Plunt
0.54
0.30
13.8
Figure 15: MRR results corresponding to Figure 5: Figure 16: MRR results corresponding Figure 6:
predicate (left) and object (right) prediction GDELT. predicate (left) and object (right) prediction GDELT.
F.6 Additional Results About Retrievals
Now we present our additional analysis results about retrievals. Figure 17 presents the MRR results with different numbers of retrieved evidence events on GDELT.
2.25
ASIE
20
200
0.24
#1
0.5
1.00
Salt eved Exidince Events
Babfeved Ev dence Events
Figure 17: MRR results on GDELT corresponding to
Figure 18: MR results of type prediction with different similarity metrics on Amazon Review.
Figure 4: predicate (left) and object (right) prediction.
Zyha CaNe-GS.Sett
4543
3223
32.00
3.00
1.75
Figure About similarity metric: predicate (left) and object (right) prediction on GDELT, with KE.
Figure 20: About similarity metrie: predicate (left) and object (right) prediction on GDELT, with DyRep.
26


Reviews:
[{"review_id": "vXfdOpy9f4", "reviewer": "Reviewer_E9bj", "summary": "Summary:\nThis paper investigates the effectiveness of large language models in reasoning and predicting event sequences. The authors propose a general framework that combines event sequence models with large language models for the task of event prediction. In this framework, an event sequence model proposes predictions on future events given the past, while a large language model assists by generating possible causes for each proposal. Experiments on two challenging real-world datasets (Amazon Review and GDELT) have shown that this framework significantly outperforms state-of-the-art event sequence models.\n\nContributions:\n  1. Proposes a domain-agnostic framework that combines event sequence models with large language models for the task of event prediction\n  2. Demonstrates the framework's ability to significantly outperform state-of-the-art event sequence models on two challenging real-world datasets (Amazon Review and GDELT).\n  3. Conducts extensive ablation studies to test the framework's robustness to prompt design and hyperparameters.", "questions": "  1. The result presentation is a bit hard to read, can tables be presented instead of charts?  It's really hard to read especially at the regions where methods are close to each other.\n  2. In figure 4, the shapes are really too close to each other that it becomes difficult for the readers to read the figures.  Either make it bigger or differentiate them in other ways.\n  3. Its really surprising that gpt3.5 is so much better than davinci-003, given the general belief of reasoning abilities of these models are gpt4 > text-davinci-003 > gpt3.5.  Do the authors have an interpretation?  Was the prompt tuning done on the gpt3.5 and later used for davinci?\n  4. Can the authors do some qualitative analysis on the generated event causes?  What happens if the generated event causes for all event predictions are not included in past events?", "limitations": "The authors did not discuss limitations of this work.  I mainly see two limitations:\n  1. The use of closed-source model limits the applicability of this approach.\n  2. It requires manually writing prompts for specific tasks.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "  1. The idea of using abductive reasoning to improve event prediction is really a natural fit, interesting, and novel.  The authors have made such a nice connection between two NLP problems. The framework is also pretty general.\n  2. Performance: The experimental results show that the proposed framework outperforms state-of-the-art event sequence models on two datasets, and the amount of improvement seems large from the chart.\n  3. The paper is well written and easy to understand.\n\n", "weaknesses": "  * The literature review is a bit limited.  For example, the paper claims \"We are the first—to the best of our knowledge—to integrate large language models into temporal modeling.\"  But this is not true, as there are already works that apply LLMs to temporal reasoning.  For example, please see paper \"Generic Temporal Reasoning with Differential Analysis and Explanation (https://arxiv.org/pdf/2212.10467.pdf)\" and other papers from Dan Roth's group.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and being supportive! We have added new results to resolve your concerns; please see [New Baselines] and [New Dataset]. Now we'll answer your remaining questions. \n\n> The literature review is a bit limited… There are already works that apply LLMs to temporal reasoning… \n\nThanks for reference! We apologize for our misleading claim. What we meant is: we are the first to apply LLMs to the area of event sequence modeling (specifically, the community discussed in [Contribution, Significance, and Impact], which I meant by \"temporal modeling\"); or more precisely, we propose the first framework that leverages LLMs to enhance event sequence models. \n\nWe are well aware of the line of \"temporal reasoning\" work in NLP. We apologize for not discussing it in the submitted version. In final version, we will add a paragraph discussing the similarities and differences between our work and the \"temporal reasoning\" NLP work. This line of NLP work includes many papers (e.g., Feng et al. 2022 as you suggest) from NLP groups led by Dan Roth, Heng Ji, Kathleen McKeown, Muhao Chen, Qiang Ning, etc. \n\n> can tables be presented instead of charts? ... In figure 4, the shapes are really too close… \n\nDefinitely! We'll keep figures in the main paper, but add the actual numbers in appendices. We will also make the curves and markers more distinguishable, by using larger sizes and different shapes.  \n\n> really surprising that gpt3.5 is so much better than davinci-003… Do the authors have an interpretation? Was the prompt tuning done on the gpt3.5 and later used for davinci? \n\nWe found it surprising, too. Based on our new results ([New Baselines]), it seems that RLHF is important for this problem setting: when using LLAMA2-chat-13B, our framework works competitive to the GPT-3.5 version, and significantly better than the GPT-3 version. It is unclear how RLHF has helped this particular problem, which seems to be a good future research topic. \n\nMoreover, we didn't tune prompts: we designed the prompt templates before running any experiments, and haven't changed it since then. \nOver the past week, we tried a few different prompt templates for each dataset, and found that it doesn't change the ranking of the methods (i.e., GPT-3.5 > LLAMA2-chat-13B > GPT-3). We'll add this new analysis in the final version. \n\n> What happens if the generated event causes for all event predictions are not included in past events?\n\nFor each generated cause, the most relevant (measured by sBERT embedding similarity) will be retrieved; see Line-133. It is the duty of the ranking module to analyze each (proposal, retrieved evidence) combination and assign low scores to implausible combinations. \n\nIn [New Baselines], we show results of \"only retrieve events if its retrieval score is higher than threshold\", but its performance is not very different from \"always retrieve 10 highest-scored events, no matter how low they are\". \n\n> ​​did not discuss limitations… the use of closed-source model limits the applicability… requires manually writing prompts \n\nLimitations section is in appendices. And we will add your points in the final version. \n\nIn response to your point of \"close-source LLMs\", we add new results using open-source LLAMA2-chat-13B; please see [New Baselines].  \n\n> Can the authors do some qualitative analysis \n\nWe'll add them in the final version. See [Qualitative Results]. "}, {"review_id": "moJVtcYRaf", "reviewer": "Reviewer_HaQW", "summary": "The paper investigates the use of large language models (LLMs) to improve the accuracy of event sequence models. Specifically, the authors propose an abductive reasoning framework based on an LLM. First, the event model produces some proposals of predictions. Then, the LLM suggests some causes for each proposal based on annotated demonstrations. At this stage, a search module retrieves some events that match the causes and finally a scoring function gives the likelihood that the retrieved events could actually cause the proposal.", "questions": "None", "limitations": "Limitations are discussed in the Appendix. The paper should mention the limitation that it requires textual data for the LLM.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method is sound and applicable in a wide range of practical settings\n* The experimental evaluation is solid, with several analyses and ablations that help the reader understand the contribution of different design choices\n* The paper is well written and easy to follow", "weaknesses": "* The results are not surprising, as the baselines do not rely on the same setting\n* The paper should mention the limitation that it requires textual data for the LLM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Title: Response to Reviewer HaQW\n\nThank you for your constructive feedback. We have added new results for you; please see [New Baselines] and [New Dataset]. Now we'll answer your remaining questions. \n\n> The results are not surprising, as the baselines do not rely on the same setting\n\nDo you mean that the baselines are only event sequence models, without any cause-retrieving and reranking mechanism? If yes, you may like our new results presented in [New Baselines]. Some new baseline methods we implemented follow the general propose-retrieve-rerank pipeline of our proposed framework, and they differ in how to retrieve causes. Can they be considered to \"rely on the same setting\"? As you can see, our full framework outperforms all the new baseline methods. \n\nIn addition, we'd like to point out that our original baseline event sequence models are strong and difficult to outperform. Moreover, from an NLP perspective, whether LLM could help predict the future is an open question. Therefore, we honestly didn't know the improvements for sure before we actually carried out the experiments. \n\nIf you still think that they are \"not surprising\" after reading this rebuttal, we would really appreciate that you elaborate it so we could kickstart a more in-depth discussion. \n\nAlso, if we misunderstand what you mean by \"same setting\", please advise so that we could give a more informed response to this concern and further improve paper. \n\n> should mention the limitation that it requires textual data for the LLM.\n\nSure. We will include it in the final version. \n"}, {"review_id": "ipTNbGhP6E", "reviewer": "Reviewer_Hwvw", "summary": "This submission proposes a large language model (LLM-) based approach to enhancing event prediction methods. Instructed by a few annotated demonstrations, a large language model is used to suggest possible causes for a proposal to be predicted. Then a search module is used to find out previous events that match the suggested causes. Finally, a scoring function is exploited to evaluate whether the retrieved events could actually cause the proposal. The above three steps can be used to enhance an existing event prediction method. Experimental results on two benchmark datasets (Amazon Review and GDELT) demonstrate that the performance improvement is significant by comparing enhanced methods with baseline ones.", "questions": "The questions have been raised in the weakness comments for Presentation.", "limitations": "The authors have not adequately addressed all limitations in the proposed approach. Some of them may not be addressed by the state-of-the-art techniques, but authors need to explicitly mention them in the submission. Firstly, the problem setting studied in the work restricts all considering event types to be in a predefined finite set. If the template for constructing a prompt to LLMs is something like Listing 1 in Page 4, this predefined set of event types should be rather small, otherwise the constructed prompt could be too long to feed into the LLM. Secondly, the names of event types should be meaningful, since they need to be fed into an LLM for generating their embeddings. Finally, the effectiveness of the proposed approach depends on whether the utilized LLM involves knowledge about the considering event types.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Summary: \n(1) A new idea for using LLMs to enhance event prediction is proposed.\n(2) There are some empirical results for justifying the proposed idea.", "weaknesses": "Summary:\n(1) The effectiveness of the proposed approach, when being applied to other datasets or different LLMs that are trained by other corpora, is doubt.\n(2) The detail about how the proposed approach is used to enhance a baseline method for event prediction is clear.\n(3) The description of the proposed methodology lacks of rigor, with some important notations undefined.\n(4) The proposed approach has some potential limitations not mentioned in the submission (see below in the Limitation part).\n\nSoundness:\nThe prediction performance of the proposed approach depends on the quality of the suggested causes by LLMs. If the suggested causes differ a lot from any events in the experimental dataset, the proposed approach may not help. Thus, there is a doubt whether the prediction performance improvement gained by the proposed approach still exists for other datasets or different LLMs that are trained by other corpora. In other words, the proposed approach seems not always sound in terms of improving performance for event prediction.\n\nPresentation:\nAlthough the basic idea is presented clearly in the submission, the details are confusing in some important aspects. First of all, it is unclear how to enhance an existing event prediction method by the three steps (see comments in the Summary part). Should the existing event prediction method be treated as the scoring function in the last step? In other words, does an enhanced method only differs from the baseline one in retrieving relevant events to conduct a prediction? Secondly, the template for constructing prompts to an LLM is unclear. There are two templates (Listing 1 and Listing 2 in page 4) presented in the submission, which one is used to construct a prompt? Finally, some descriptions of the proposed approach are unclear. In Lines 89-90, what does the superscript (1), …, (M) mean? How is M defined? In Line 113, what does the superscript (m) mean? In Line 117, what does the superscript (m,M’) mean? How is M’ defined? In Line 170, what does the superscript (m) mean? How is M in Equation (5) defined?\n\nContribution:\nThe submission presents a new idea for utilizing LLMs to enhance event prediction. However, the soundness of this idea is questionable (see comments in the Soundness part). It calls for further theoretical analysis and empirical study to confirm the soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We have added new results for you; please see [New Baselines] and [New Dataset]. Now we'll answer your remaining questions. \n\n> the soundness of this idea is questionable (see comments in the Soundness part). It calls for further theoretical analysis and empirical study to confirm the soundness. \n> effectiveness of the proposed approach, when being applied to other datasets or different LLMs that are trained by other corpora, is in doubt. \n> If the suggested causes differ a lot from any events in the experimental dataset, the proposed approach may not help… the proposed approach seems not always sound in terms of improving performance for event prediction.\n\nWe reply to the \"soundness\" concern in [Scientific Rigor] of our to-all message. \n\nAs for \"further empirical study\", we have managed to add new results for you and hope that you like them! Please see [New Baselines] and [New Dataset]. First, we experimented with open-source LLAMA2-chat-13B, which was very different from OpenAI's GPTs. Second, we evaluated a new ICEWS dataset, on which we tried both time and predicate prediction. At least for the new model and new data, our proposed framework still exhibits superior performance compared to baseline methods. \n\nMoreover, the LLMs used in our work are all trained with general online data, which has a broad coverage and does not specialize in the particular kinds of data/tasks that this paper is about. Therefore, it seems reasonable to expect that the success can generalize to other test datasets. \n\n> The proposed approach has some potential limitations not mentioned in the submission\n> the problem setting restricts all considering event types to be in a predefined finite set \n> the names of event types should be meaningful, since they need to be fed into an LLM \n> effectiveness of the proposed approach depends on whether the utilized LLM involves knowledge about the considering event types\n\nThanks. We will include these limitations in the final version. \n\nHowever, we'd like to point out that the \"finite\" limitation comes from the base model part of our framework, but not our new LLM and ranking mechanism: nearly all the event sequence models assume a finite set of event types, so our framework still has a broad applicability. Generalization to infinite event types could be a very interesting future direction. \n\n> First of all, it is unclear how to enhance an existing event prediction method by the three steps (see comments in the Summary part). Should the existing event prediction method be treated as the scoring function in the last step? In other words, does an enhanced method only differs from the baseline one in retrieving relevant events to conduct a prediction? \n\nThe 3-step framework is shown in Figure-1 of paper, and it is qualitatively described in line-33 to 41 in Introduction. In the final version, we will add a high-level technical introduction at the beginning of Section 3 to describe how the 3 steps work together, and clarify where the event sequence model is used: \n- 1st step: an even sequence model is used as a proposer to propose predictions on the future (time, type, or attribute of type). This is where the existing event prediction model is used. This is also why we call an event sequence model \"base model\": if we do not do step-2 and 3, this framework falls back to an ordinary event sequence model. \n- 2nd step: LLM reads each proposed prediction, and generates possible causes as if it actually happened. Then sBERT-based retriever finds out events that are similar/relevant to the generated causes from the history. See Section-3.1 and 3.2 in paper. \n- 3rd step: a reranking module examines the combos of (proposal, retrieved evidence) and reranks them. See Section-3.3 in paper. This module has to score sequences of time-stamped events, but it doesn't need to propose events. Therefore, its architecture is like an event sequence model on the input side (which may have confused you? we can clarify it in the final version!), but different on the output side. Technically, it is an energy function just like the ranking modules in non-autoregressive sequence models (e.g., HYPRO by Xue et al. 2022). \n\n> the template… is unclear… two templates (Listing 1 and Listing 2 in page 4)... which one is used to construct a prompt? \n\nListing 1 is the general template that includes instructions and demonstrations. Listing 2 is an example of demonstration. Listing 2 is used in Listing 1. (In Listing 1, we wrote \"// Examples are in Listing 2.\")\n\n> description of the proposed methodology lacks of rigor, with some important notations undefined. \n> In Lines 89-90, what does the superscript (1), …, (M) mean? … In Line 113, what does the superscript (m) mean? In Line 117, what does the superscript (m,M’) mean? How is M’ defined? In Line 170, what does the superscript (m) mean? How is M in Equation (5) defined?\n\nWe apologize for the confusion. We'll clarify the notation details in the final version.  \n\nThroughout the paper, we use little letters (e.g., m) as indices and use capitalized letters (e.g., M) as the upper bounds of the indices. \nSpecifically, M is the # of proposals and the variable m indexes the m-th proposal. \nDouble index (m,m') shows up because each of the M time proposals corresponds to M' different type proposals at that time: i.e., (m,m') indexes the m'-th type proposal at the m-th time proposal. \nLine-50 to 64 in Section 2 explains the relation between time and type prediction in the setting of classical event sequence modeling. \n"}, {"review_id": "KxwtHgPppA", "reviewer": "Reviewer_pKDi", "summary": "This paper studies the problem of predicting future world events based on the past and proposes an approach that combines the existing event sequence models with the powerful (abductive) reasoning ability of large language models (LLMs) that results in better performance both for predicting the actual event as well as predicting the time of the event.", "questions": "* The abductive reasoning module is quite similar to the backward reasoning works such as [1, 2, 3]. Could the authors say a few words about whether it is possible to bring more of backward reasoning into their framework? As an example, can we predict multiple hops of causes instead of one (e.g., for an event E, predicting a cause C and then predicting a cause C’ for C, …)?\n* Any idea how all three modules can be trained end-to-end instead of training separate modules?\n* Are the Event Sequence Model and Reranking Model both trained on the same subset of the data?\n* [Minor] Why not use Mean Reciprocal Rank instead of Mean Rank? The former has been advocated in many previous works.\n* [Minor] LLMs can hallucinate causes and it seems like all causes will be mapped to an event. I wonder whether it helps if the authors ignore the causes whose similarity to existing events is below some threshold.\n* [Minor] In line 150, how is time represented so that the model can know about the distance between the events?\n\n[1] LAMBADA: Backward Chaining for Automated Reasoning in Natural Language\n\n[2] Neural story planning\n\n[3] Language Models with Rationality (I understand that this paper was published too close to the NeurIPS deadline)", "limitations": "* It is typical in real-world problems to have both temporal and non-temporal facts/events. Can the approach be extended to this case?\n* It is typical in real-world problems to have temporal events that have a timestamp and temporal events that have a time interval. Can the approach be extended to work with intervals?\n* Can the model predict that no event will happen?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Predicting future world events based on the past is an important problem.\n* The idea of leveraging LLMs for future event prediction is interesting and can spark further research in this direction.\n* Experimental results are strong, albeit on a small set of datasets.", "weaknesses": "* The LLM prompts presented on Page 4 are a bit worrisome as they hint at a possible leakage: The time of the first demonstrating example is 2022-03-08 and so is the time of the queried effect. Could you please clarify how you selected the fewshots, and whether there might be any leakages?\n* I’m a bit confused about the metrics. From what I understand, the performance of the base Event Sequence model is computed on all the data, but the performance of the LLM-augmented models is only computed on the examples where the correct answer was among the top answers of the base model. Is this correct or did I misunderstand something?\n* Some of the presented results on the two datasets are strong, but given the simplicity of the proposed solution, it would have been more convincing if results on more datasets were reported, especially because the mean rank results on the Amazon dataset is not that glaring. (To be clear, I like the simplicity of the approach)\n* Given that the LLM module is the main novelty of the work, I would expect to see some experiments where the LLM is replaced with 1) a random selection module that selects 10 pieces of evidence at random, 2) a heuristic-based selection module that, e.g., selects the most recent 10 pieces of evidence with the same subject.\n* Some qualitative examples are needed to show how the LLM module actually helps make better predictions.\n* [Minor] The font in Figure 1 is a bit small when printed. Maybe make the size of box 3 smaller (rotate the text if needed) so that the other parts become slightly bigger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and being supportive! We added new results for you; please see [New Baselines] and [New Dataset]. Now we'll answer your remaining questions. \n\n> The time of the first example is 2022-03-08 and so is the time of the queried effect… clarify how you selected the fewshots, … any leakages? \n\nWe randomly sampled training events and manually found evidence/cause events for each of them. \n\nRaw data has HH-MM-SS information so we know the temporal order of same-day events. For presentation simplicity, we omit HH-MM-SS in examples. We'll clarify it (or use a better example) in final version. \n\nWe have taken efforts to ensure that there is no leakage in data. For GDELT and ICEWS, we use the data that is surely not in GPT training data. App C.5 in paper explains why GPTs do not memorize Amazon Review data. For each dataset, we split train/dev/test based on time-stamps of events (see Line 214). \n\n> I’m a bit confused about the metrics… \n\nAll methods are evaluated where the correct answers were among the top answers of the base model. That is why we feel necessary to show MAP and MAR of base models (see Figure-3): MAP and MAR measures precision and coverage of each base model; mean rank and RMSE measure how much better our framework could do given that precision and coverage of base model. \n\n> it would have been more convincing if results on more datasets were reported \n\nSure! Please see our new results in [New Dataset], and we hope you like them!  \n\n> 1) selects 10 pieces of evidence at random, 2) selects the most recent 10 pieces of evidence with the same subject… \n\nWe added these new baselines (and more), and they all underperform our framework; see [New Baselines]. \n\n> font in Figure 1 is a bit small… make the size of box 3 smaller (rotate the text if needed)... \n\nGreat suggestion! We'll do it in the final version. \n\n> say a few words about whether it is possible to bring more of backward reasoning into their framework? \n> predict multiple hops of causes instead of one (e.g., for an event E, predicting a cause C and then predicting a cause C’ for C, …)?\n\nWe drafted a long and in-depth discussion for these questions, but we couldn't show it in this message due to 6000-char limit. Here we only give TLDR, but we can post full discussion via an Official Comment in Discussion phase, if you are interested. We will add full discussion in final version. \n\nAbductive reasoning is a type of reasoning ability, in parallel with deductive reasoning and inductive reasoning. Backward chaining is a general method for reasoning, in parallel with forward chaining and resolution. \n\nIn this work, we leverage abductive reasoning abilities of LLMs, and our proposed method resembles backward chaining (although it only does 1-step reasoning). \n\nWhen data is complete (i.e., full observed, no missing events), 1-step seems to be good enough: both C' and C are in history, and retrieving C seems more proper since C is more recent. \n\nWhen data is incomplete, multi-step reasoning seems to be necessary: maybe C is not observed in history but C' is, so we have to do a 2nd step reasoning and retrieve C'. This is an extension of our current method. \n\n> Any idea how all three modules can be trained end-to-end\n\nYes. Conceptually, one can do: \n- use retrieved causes to regularize (e.g., attention mechanism of) event sequence models\n- prompt LLM so that it generates cause events that tend to have high reranking scores \nThese can be future directions and we will discuss them in the final version. \n\n> … both trained on the same subset of the data? \n\nThe use same set of training data. But reranking uses negative samples drawn from event model. They are both early-stopped based on dev results. \n\n> Why not use Mean Reciprocal Rank \n\nWe thought Mean Rank is the norm of the IR community. We can add Mean Reciprocal Rank in the final version. \n\n> I wonder whether it helps if the authors ignore the causes whose similarity to existing events is below some threshold. \n\nWe tried this idea and showed new results in [New Baselines]. There is a small difference (thresholding worse), but a heavier tuning of threshold may close the performance gap. \n\n> In line 150, how is time represented… model can know about the distance between the events? \n\nIn the continuous-time Transformer of Yang et al. 2022, time is represented via a temporal embedding mechanism (a generalization of positional embedding) so high-layer representations are aware of the temporal distances between events. \n\n> It is typical in real-world problems to have both temporal and non-temporal facts/events. Can the approach be extended to this case?\n\nYes, we believe that the general propose-LLM-retrieve-rerank pipeline can apply to other settings, although they may need non-trivial technical extensions. Here is how it may apply to language-based logical reasoning: given a theory of multiple statements, a base model can propose possible conclusions; LLM proposes possible explanations for each proposed conclusion; retriever finds out most relevant statements from the theory; reranking module examines if the retrieved statements can actually prove the proposed conclusion. \n\nWe will add this discussion in our final version. \n\n> Can the approach be extended to work with intervals?\n\nIf the base event sequence model is capable of handling intervals, then it seems straightforward for our framework to handle intervals as well (through the base model). \n\n> Can the model predict that no event will happen?\n\nThat is the duty of base event sequence model. If the most recent event is at time t0, and the base model predicts the next event at time t1, then any time in (t0, t1) is regarded to be \"no event happens here\". The base models we use in this paper are all probabilistic point processes, so \"no event will happen ever from now\" has zero probability under the base model. \n\n> qualitative examples needed \n\nWe'll add them in final version. See [Qualitative Results]. "}, {"review_id": "3oYaslAV3K", "reviewer": "Reviewer_mHgh", "summary": "This paper studies using large language models (LLMs) to assist event prediction. The authors proposed a pipeline of procedures, where 1) a traditional event sequence model is first applied to predict a set of possible events along with their time stamps, 2) then for each event candidate, an LLM is used to reason about their cause events, based on which a set of relevant events are retrieved from the history, and 3) finally the retrieved relevant events are used by a reranking model to score each event candidate. This framework is tested on two datasets, GDELT and Amazon Review, and the results showed that the LLM-based reranking approach outperforms the naive event sequence model.", "questions": "Please try to address my comments in Weaknesses. ", "limitations": "The authors discussed the limitations and potential negative impact in Appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work studies leveraging LLMs for temporal modeling. This is an important topic as it can be used to support many applications (e.g., political event forecasting). Using LLMs for reranking also seems to be novel.\n2. The authors have conducted experiments on two datasets, both showing an advantage of the LLM-based reranking. \n3. This paper also comes with analyses of the proposed framework, such as how tuning the size of event candidates (M) can affect the overall performance and whether the LLM is sensitive to different variants.", "weaknesses": "1. The paper needs significant justifications for the experiment design. \n- While the authors formulated the event prediction task into two subtasks, the majority of the experiments have focused on only one of them (i.e., event type prediction), whereas the discussion on the time stamp prediction is rather brief. If the authors regard the time prediction task as an important subtask, then they may want to consider using a dataset that can facilitate more careful discussions other than GDELT.\n- The event prediction on GDELT is not end-to-end. Instead, the current experiments always assume one or two (ground-truth) components from each <subject, predicate, object> triplet and aim to predict the remaining. This could show a discrepancy between the evaluation and the realistic application.\n- More clarifications about each dataset are needed. For example, it is unclear how many events are provided as history in each prediction. \n\n2. Results in Figure 2 and several other figures showed worse performance as M increases. This could say that as the LLM-based reranker is exposed to more candidates, it ranks the truth even worse. Looking into the description of Mean Rank in lines 250-256, it can be imagined that as M increases, there will be more test cases (whose ground truth is included in top-M) get counted into \"N\". In other words, when M has different values, the Ns have been different, so the results of the same approach across different Ms are not really comparable. However, this is not clarified in the paper and the current plots read confusing.\n\n3. The proposed framework heavily relies on the event sequence model. Based on Figure 3, it seems that the poor performance of the event sequence model is the real bottleneck (<0.1 MAR in Fig 3(b)). In other words, while introducing LLMs does lead to improvement in event reranking (under the current experimental setting), it is not solving the real bottleneck. Therefore, I'm not very convinced by the contribution of this framework. Instead of reranking, a more worth-studying problem is how to use LLMs to more accurately predict future events in the first place. \n\n4. More explorations on the use of LLMs in reranking can provide more insight to the community. It is noticed that, while the contribution of this paper is about using LLMs for event reranking, none of the baselines or variants are following the same reranking idea. Is there a better way to use LLMs for event reranking? For example, can one use SentenceBERT or BM25 to retrieve relevant events from the history, and then directly use LLMs to score the candidate event?\n\n5. Some of the writing can be more self-contained. For example, the paragraph in line 51 introduced several math formulations but there is not a single reference, and the intuition of these formulas is not explained clearly. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! \n\n> consider using a dataset that can facilitate more careful discussions other than GDELT.\n\nWe added new results on a new dataset, ICEWS. It is similar to GDELT but less dense in time so it is meaningful to predict time on this new dataset. Please see [New Dataset] for details. \n\nIn the final version, we will include a full set of experiments on ICEWS (all baselines, all analysis) just as we did for GDELT. \n\n> assume one or two (ground-truth) components from each <subject, predicate, object> triplet and aim to predict the remaining. This could show a discrepancy between the evaluation and the realistic application.\n\nOur evaluation setup matches real-world applications. Please see [About Experiment Design]. \n\n> More clarifications about each dataset are needed. For example, it is unclear how many events are provided as history in each prediction.\n\nWe will add this detail to the final version: \"history\" includes all the previous events in the data. \n\n> when M has different values, the Ns have been different, so the results of the same approach across different Ms are not really comparable. However, this is not clarified in the paper and the current plots read confusing. \n\nSharp catch! Yes, our current text in Line-264 to 266 is misleading. We didn't mean to compare the same method across M; instead, we meant to compare different methods for the same M. We apologize and will definitely correct our presentation. \n\n> proposed framework heavily relies on the event sequence model… it is not solving the real bottleneck… a more worth-studying problem is how to use LLMs… \n\nWhich's real bottleneck seems to be a subjective call? Please see [Contribution, Significance, and Impact] for our justification of the problem setting and our technical approach. \n\n> none of the baselines or variants are following the same reranking idea. Is there a better way to use LLMs for event reranking?\n> can one use SentenceBERT or BM25 to retrieve relevant events from the history, and then directly use LLMs to score the candidate event? \n\nIn [New Baselines], we added several new baseline methods that follow the same reranking idea of our framework. \nOne of them retrieves with sBERT and reranks with our method, but it works poorly. That is because sBERT tends to retrieve events that are similar to the proposal (and BM25 will as well), but the real causes may look very different. Fortunately, LLMs may generate real causes that look very different. \n\nWe suspect that using LLMs as scoring function will only make the performance worse because \n- LLMs do not seem to be good at numbers as they are good at words; \n- LLMs are pretrained but our proposed reranking module has been trained on the task (so it is more specialized). \n\n> Some of the writing can be more self-contained. For example, the paragraph in line 51 introduced several math formulations but there is not a single reference, and the intuition of these formulas is not explained clearly. \n\nWe are sorry that we made this background paragraph too compact due to the page limit. In the final version, we will appropriately label these formulas, refer to them when needed, and clarify the intuition. "}]

Author rebuttal:

We thank reviewers for constructive feedback! In this to-all message, we clarify our technical contributions and present new results. We will address other concerns in responses to individual reviews. 

Due to 6000-char limit, we have to keep this response concise. If anyone asks for more details, we'd be happy to elaborate them in Discussion phase.  

[Contribution, Significance, and Impact] 

Event sequence modeling (i.e., learning to predict future events given the past) is an important area in machine learning. In recent years, over a thousand new papers on this topic have been posted on arXiv. Our paper makes a significant contribution to this area: we propose a simple modeling and inference framework, which leverages recent advances in NLP---LLMs and in-context learning---to improve prediction accuracies of traditional event sequence models. Our framework is compatible with any event sequence models. Our experiments show that it significantly and consistently improves over a range of strong and influential event sequence models. This finding is certainly of great interest to both event sequence modeling and NLP (particularly, LLM) research communities, which is a broad audience. In addition, we will release code to assist researchers in reproducing our results and conducting their future research. 

R.mHgh thinks that "a more worth-studying problem is how to use LLMs to more accurately predict future events in the first place". We agree it is "worth-studying", but it is orthogonal to our contribution. Moreover, "which is more worth-studying" is subjective: in history of science, many problems were regarded as "not important", but some of them turned out to be extremely important later in time. 

[Scientific Rigor]

R.Hwvw is concerned with scientific rigor and soundness of this paper, and asks for more "theoretical analysis and empirical study". Please see [New Baselines] and [New Dataset] for new results and analysis. 

As for "theoretical analysis", we'd like to emphasize: some papers are theoretically rigorous; some are empirically rigorous. This paper is empirically rigorous: we carefully thought through the framework design, evaluated against strong and proper baselines, and conducted extensive empirical studies. Our method has no theoretical guarantees, but this limitation is shared by many great empirical papers and should not be a reason for rejection. 

Surely, there could be settings where our framework is no better than baselines: e.g., LLMs provide irrelevant, incorrect, or misleading information. We'll discuss such cases in the Limitations section. 

[About Experiment Design] 

R.mHgh is concerned with our experiment design. For a fair comparison with previous work, this paper follows standard experiment setup: 
- evaluate on predicting time and type of future events, given the history;  
- only predict an attribute of type (e.g., predicate) when there are too many types. 

GDELT has over 20 millions of event types (in format of subject-predicate-object). First, full type prediction is extremely hard and all methods will end up with indistinguishably low accuracies. Second, in real applications, users usually care about "what will happen between entity A and entity B" (predicate prediction) and "what will A do to B" (predicate-object joint prediction), instead of "which of 20M possible events is most probable" (full type prediction). Therefore, previous work using GDELT also evaluates on attribute prediction.  

[New Baselines]

We ran experiments with 7 new baselines. They include: 3 methods that follow the pipeline of our framework but differ in how to retrieve evidence; a new version of ours that uses LLAMA2-chat-13B; a new event sequence model that uses text information; a method that directly uses LLMs for event prediction; a new version of ours that retrieves a past event only if its retrieval score is higher than a threshold. 
Results are in Fig-16 and 17 of submitted pdf. Takeaways are: 
- our GPT-3.5-based framework outperforms all the new baselines.  
- LLAMA2-chat-13B version of our framework is better than GPT-3 (175B) version, indicating that RLHF may be important for our problem setting. This finding is interesting and worth studying in the future. 
- directly using LLMs for future prediction performs worse than base ANHP model. 
- there is no much difference between "retrieve 10 highest-scored events" and "retrieve if it passes threshold". 

[New Dataset]

We ran experiments on ICEWS, a dataset similar to GDELT but less dense in time (so that time prediction is more meaningful). 
Results are in Fig-18 of submitted pdf: like on GDELT and Amazon Review, our LLM-enhanced framework significantly outperforms baselines on ICEWS. 

[Qualitative Results] 

LLM-generated causes look reasonable and meaningful. We'll show them in the final version. 

[Limitations]

Our Limitations section is in appendices. In final version, we'll discuss the additional limitations pointed out by reviewers. We already wrote a new version, but couldn't share it here due to char limit. If needed, we can show it in Discussion phase. 

However, we kindly request reviewers to be mindful of our existing contributions and significance. After all, what hasn't been done should not undermine the value and significance of what has already been done. 

[Presentation Improvements]

We will include new results in final version. In addition, we have planned a list of presentation improvements, including ways to fix issues pointed out by reviewers (e.g., notation definition, formula references). With extra page allowed by NeurIPS camera-ready, the fixes are easy to execute. Considering that we have managed to deliver many new results within a week, we hope that reviewers feel assured of our promise and commitment to complete the proposed improvements. If needed, we'd like to show our full list of presentation improvements in Discussion phase, for reviewers to check and approve. 

Meta-review: This paper demonstrate that language models can be employed to assist event sequence models for even prediction. The framework proposes using an event sequence model to propose potential events, then use a language model to retrieve past events that could help in predicting which event is more likely in the future and outputting likely events and finally using a scoring mechanism to find the most likely event. The paper is interesting, well-written and easy to follow. The contributions are significant as they open up the opportunity for more LLM-related research in this area. All reviewers voted for acceptance. Therefore, I recommend acceptance.
Decision: Accept (poster)