SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our framework utilizes LLMs such as GPT-3.5 and thus inherits the issues of those models such as hallucination and biased content. Therefore, the cause events generated by LLMs may be irrelevant, incorrect, or misleading, thus harming the overall performance of our framework. Also because of using LLMs, our approach requires human-crafted prompts, and the events need to have textual information (e.g., event types have textual identifiers or marks).
Our framework needs a pretrained event sequence model, and its overall capability is partially determined by this base model. When the base model can handle events with duration (i.e., events that last for a period of time, such as holding a tool), our framework will inherit this merit. But our framework will also inherit some of the technical limitations of its base event sequence model. If the base model can only handle a finite set of event types, then our framework will inherit this limitation. If the base model is misspecified or poorly trained, our framework may not work well.
Our compatibility function  $c$  is a Transformer-based neural network which is known to be data-hungry. Though it worked well in our experiments, it might struggle when starved of data.
C Possible Extensions
Our framework currently trains each component separately. A natural improvement will be to jointly train its three components. In principle, we can iteratively
- refine the base event sequence model with the feedback from the LLM and ranking model;
- learn to prompt the LLM so that it generates cause events of higher quality;
- train the ranking model with proposals and evidence given by the improved base model and LLM.
Another interesting extension of our framework is to apply its propose-justify-rank pipeline to other problems. At the core of our framework is the LLM, which is able to examine each proposed prediction drawn from the base model, and suggest clues to justify its validity. This idea seems to be a natural fit to a wide range of reasoning problems. Question answering is an example: given a question, an existing model proposes multiple plausible answers; an LLM reads each proposed answer and reasons about its preconditions; a search module finds out evidence (from local knowledge bases or the open internet) which may match the LLM-suggested preconditions; a ranking model learns to score each combination of answer and evidence.
D.1 ANHP
On Amazon data, the ANHP model we used is directly from Yang et al. (2022). On GDELT and ICEWS data, we modified the definition of event embedding so that it is easier to scale up to hundreds of millions of event types. Particularly, ANHP defines a dynamic event embedding  $\llbracket k\rrbracket (t)$  for the event type  $k$  at time  $t$ . On GDELT and ICEWS, each event type is a tuple of subject-predicate-object, so we redefine the event embedding to be
$$
[   [ k ]   ] (t) \stackrel {\text {d e f}} {=} \operatorname {c o n c a t} \left([   [ k ^ {\text {s u b j e c t}} ]   ] (t), [   [ k ^ {\text {p r e d i c a t e}} ]   ] (t), [   [ k ^ {\text {o b j e c t}} ]   ] (t)\right) \tag {8}
$$
and keep everything else the same as in the original ANHP.
This modified version is more scalable since it needs to maintain a significantly smaller number of dynamic embeddings: there are hundreds of millions of possible event types, but there are only thousands of possible subjects and objects.
E.1 Dataset Details
Table 1 shows statistics about each dataset in our experiments.
DATASET	# OF
ENTITIES	# OF
PREDICATES	# OF EVENT TAXENS
TRAIN	DEV	TEST
GDELT	2279	20	83100	9250	16650
ICEWS	2981	20	41600	15780	22030
AMAZON	1	24	49680	7020	13090
Table 1: Statistics of datasets.
E.2 Evaluation Metric Details
Now we give the formal definitions of MR and MRR. The primary evaluation metric is the mean rank (MR): for each method, we initialize  $N = R = 0$ ; for each held-out token, the method provides a list of  $M$  proposed predictions sorted in the decreasing order of their scores; if the ground-truth type is among the proposals, we update  $N += 1$  and  $R += r$  where  $r$  is the rank of the ground-truth type in this list; the mean rank is  $R / N$  and the lower is the better.
The mean reciprocal rank (MRR) is as follows: for each method, we initialize  $N = R = 0$ ; for each held-out token, the method provides a list of  $M$  proposals sorted in the decreasing order of their scores; if the ground-truth type is among the proposals, we update  $N += 1$  and  $R += 1 / r$  where  $r$  is the rank of the ground-truth type in this list; the MRR is  $R / N$  and the higher is the better.
Note that MR and MRR are not affected by the fact that there might be multiple ground-truth attributes. That is because the contribution of  $j^{\mathrm{th}}$  ground-truth (among the multiple) to the final number is only dependent on its rank in the list but not its  $j$  index.
Next we give the precise definitions of MAP and MAR.
MAP is computed as follows. We first initialize  $N = C = 0$ . Then, for each partial held-out event (i.e., only  $t$  given in Amazon Review,  $t$  and some attributes given in GDELT), the model gives its top  $M$  predictions on the attribute of interest (e.g., type  $k$  on Amazon Review and "object" on GDELT). There might be multiple ground-truth attributes for a held-out event since multiple events may be recorded at the same time (due to time quantization and recording errors). If any of the ground-truth is in the top  $M$  list, we update  $N$  and  $C$ : first, we have  $N += J$  where  $J \leq M$  is the number of the ground-truth attributes that are covered in the top  $M$  list; then, for  $j^{\text{th}}$  covered ground-truth, we let  $C += j / R$  where  $R$  is its rank in the top  $M$  list. In the end, MAP is defined as  $C / N$ . Each  $i / R$  is a pseudo-count for this event: it is in (0,1] since the rank  $R$  of the  $j^{\text{th}}$  correct prediction will be surely  $\geq j$ ; if it ranks at the top,  $i / R$  is close to 1 and it is counted as "predicted correctly"; if it ranks at the bottom,  $i / R$  is close to 0, meaning that it is almost missed by the top  $M$  proposals.
MAR is defined as follows. We first initialize  $N = C = 0$ . For each partial held-out event, the model gives its top  $M$  proposals on the attribute of interest; for the  $j^{\text{th}}$  correct prediction in the list, we let  $C += j / R$  where  $R$  is its rank in the top  $M$  list; in the end, MAR is defined as  $C / N$  where  $N$  is the number of actual events.
E.3 Implementation Details
All models are implemented using the PyTorch framework (Paszke et al., 2017).
For the implementation of NHP, AttNHP and energy functions, we used the code from the public Github repository at https://github.com/ant-research/EasyTemporalPointProcess (Xue et al., 2023) with Apache License 2.0.
For the implementation of Know-Evolve, we used the code from the public Github repository at https://github.com/rstriv/Know-Evolve (Trivedi et al., 2017) without any license.
For the implementation of DyRep, we used the code from the public Github repository at https://github.com/uoguelph-mlrg/LDG (Trivedi et al., 2019) without any license.
For the implementation of MAR@M and MAP@M, we used the code from the public Github repository at https://github.com/statisticianinstilettos/recmetrics without any license.
For the implementation of Levenshtein distance, we used the code from the public Github repository at https://github.com/maxbachmann/Levenshtein.git with General Public License 2.0.
To compute the dense representations of text descriptions of events, we use the code from the public Github repository at https://github.com/UKPLab/sentence-transformers with Apache License 2.0.
E.4 Prompts
We show one example of the prompt structures used for GDELT dataset as below. The effect event consists of an event predicate, an event time, a subject name and an object name while the cause events consists of an event headline additionally.
I want you to do the reasoning over social events. I given you an effect event and you give me four or five cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event consists of an event headline, an event predicate, an event time, subject name and object name of describing the event.
The predicates of the effect and cause events are restricted to 20 options, with names (in capital) and the descriptions below.
1. MAKE STATEMENT: decline comment, make comments.
2. APPEAL: appeal for material, economic, military, humanitarian aid or cooperation.
3. EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic, military aid.
4. CONSULT: make a visit, host a visit.
5. ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.
6. ENGAGE IN MATERIAL COOPERATION: cooperate economically, militarily, judicially.
7. PROVIDE AID: provide economic, military, humanitarian aid.
8. YIELD: ease admin or political sanctions or military blockade, return and release.
9. INVESTIGATE: investigate crime, corruption, human rights abuses, military actions.
10. DEMAND: demand any type of cooperation, aid, reforms, rights, easing of sanctions.
11. DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses, complain officially and make lawsuit against.
12. REJECT: reject material, economic, military, judicial operations, requests or plans.
13. THREATEN: threaten to reduce aid, to boycott, to reduce or break relations, to impose sanctions.
14. PROTEST: civilian demonstrations.
15. EXHIBIT MILITARY POSTURE.
16. REDUCE RELATIONS: reduce or break any relations.
17. COERCE: seize or damage properties, impose administrative sanctions or restrictions.
18. ASSAULT: use of unconventional forms of violence.
19. FIGHT: uses of conventional force.
20. ENGAGE IN UNCONVENTIONAL MASS VIOLENCE.
Now I give you 10 examples of event reasoning. In each example, the first event is the effect event, the next three to five events are cause events that happen earlier.
headline: Europe can endure painful transition to live without Russian oil.
cause event 3
predicate: ENGAGE IN DIPLOMATIC COOPERATION
time: 2022-03-05
subject: BUSINESS
object: GOVERNMENT
headline: Oil prices surge to multi-year highs as Ukraine conflict causes ripple
effect in global oil supplies.
cause event 4
predicate: REJECT
time: 2022-03-04
subject: SENATOR
headline: Marshall, Moran seek ban on Russian oil imports, urge expansion of
domestic production.
cause event 5
time: 2022-03-03
subject:TRADER
headline: Energy markets in turmoil as European gas climbs  $60\%$
Listing 3: A prompt example used for GDELT.
We show one example of the prompt structures used for ICEWS dataset as below. The effect event consists of an event predicate, an event time, a subject name and an object name while the cause events consists of an event sub predicate additionally.
I want you to do the reasoning over social events. I given you an effect event and you give me four or five cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event consists of an event text, an event predicate, an event time, subject name and object name of describing the event.
1. MAKE STATEMENT: decline comment, make comments.
2. APPEAL: appeal for material, economic, military, humanitarian aid or cooperation.
3. EXPRESS INTENT TO COOPERATE: Express intent to engage in material, diplomatic, military aid.
4. CONSULT: make a visit, host a visit.
5. ENGAGE IN DIPLOMATIC COOPERATION: praise or endorse, defend verbally.
6. ENGAGE IN MATERIAL COOPERATION: cooperate economically, militarily, judicially.
7. PROVIDE AID: provide economic, military, humanitarian aid.
8. YIELD: ease admin or political sanctions or military blockade, return and release.
9. INVESTIGATE: investigate crime, corruption, human rights abuses, military actions.
10. DEMAND: demand any type of cooperation, aid, reforms, rights, easing of sanctions.
11. DISAPPROVE: criticize or denounce, accuse of crime, human rights abuses, complain officially and make lawsuit against.
12. REJECT: reject material, economic, military, judicial operations, requests or plans.
13. THREATEN: threaten to reduce aid, to boycott, to reduce or break relations, to impose sanctions.
14. PROTEST: civilian demonstrations.
15. EXHIBIT MILITARY POSTURE.
16. REDUCE RELATIONS: reduce or break any relations.
17. COERCE: seize or damage properties, impose administrative sanctions or restrictions.
18. ASSAULT: use of unconventional forms of violence.
19. FIGHT: uses of conventional force.
20. ENGAGE IN UNCONVENTIONAL MASS VIOLENCE.
time: 2022-10-28
subject: CHINA
object:JAPAN
time: 2022-10-18
subject: Japan
object: USA
sub predicate: Rally support on behalf of.
predicate: ENGAGE IN MATERIAL COOPERATION
time: 2022-10-16
subject: SOUTH KOREA
object: UNITED NATIONS
sub predicate: Cooperate militarily.
time: 2022-10-13
sub predicate: Praise or endorse.
Listing 4: A prompt example used for ICEWS.
The following is a prompt exemplar used for the Amazon Review dataset. Each effect event consists of the product category and event time while each cause event also includes the content of the product review.
I want you to do the reasoning over the events that are extracted from online-shopping review data. I given you an effect event and you give me two to four cause events. An effect event is an event that happens. A cause event is an event that is believed to be one of the causes that trigger an effect event to happen. Each event corresponds to an review submitted by the customer, which consists of an product category(event type), a product title, an event time, summary text and review text from the user that describes the feedback of the shopping event.
The product categories are restricted to the following set:
1. Women Shoes,
2. Men Shoes,
3. Men Clothing,
4. Women Clothing,
5. Novelty & More,
6. Men Uniforms, Work & Safety,
7. Women Jewelry,
8. Costumes & Accessories,
9. Men Accessories,
10. Luggage & Travel Gear,
11. Men Watches,
12. Women Accessories,
13. Children Shoes,
14. Children Clothing
15. Shoe, Jewelry & Watch Accessories,
16. Women Watches,
17. Women Uniforms, Work & Safety,
18. Men Surf, Skate & Street,
19. Women Handbags & Wallets
20. Men Jewelry
21. Children Accessories
22. Women Maternity
23. Women General
24. Others
effect event
product category: Luggage & Travel Gear
event time: 2013-10-19
product category: Novelty & More
product title: Sports Katz Peace Out Socks
event time: 2013-09-24
summary text: Peace Out Socks
review text: We ordered these for soccer for my daughter and they worked out well. They are very cute and have held up. They are a quite thick, which is why I only gave them 4 stars. They will work better for fall season; I would not recommend them for summer soccer, if soccer is your intended use.
predicate: MAKE STATEMENT  
time: 2022-10-10  
subject: CHINA  
object: JAPAN  
sub predicate: Make pessimistic comment.
product category: Luggage & Travel Gear   
product title: Wildkin Pink Retro Kickstart Messenger Bag   
event time: 2013-09-09   
summary text: Wildkin Kickstart Messenger Bag   
review text: My daughter, going to 4th grade, wanted a messenger type bag this year for school. I searched and searched on Amazon for the perfect bag, but all of the adult bags seemed huge (like they would definally look funny with her carrying them). Thank goodness I found this bag. It is the perfect size for her, and her favorite color!.   
cause event 3   
product category: Children Clothing   
product title: Levi's Girls' 715 Thick Stitch Bootcut Jean   
event time: 2013-09-01   
summary text: Thick Stitch Bootcut Jeans   
review text: I love Levis and this style is sooo cute! My daughter even likes them! The price was amazing in these, as we found similar Levis at Dillard's for 36. The waist of these Levi in particular run small compared to other Levis of the same size. During the Levis sale on Amazon, we purchased these, the Sweetie Skimmers, and the Scooter. All of the others ran almost half an inch larger in the waist. I worry the larger size would be huge on her, but I think I will try the larger size, as they must be comfortable on her or she will not wear them.   
cause event 4   
product category: Children Shoes   
product title: New Balance KJ750 Running Running Shoe (Little Kid/Big Kid)   
event time: 2013-02-02   
summary text: Nice and durable!   
review text: I ordered these for my 8 year old daughter. She usually destroys shoes within months, but these have held up!
Listing 5: A prompt example used for the Amazon data.
For the full set of prompts used in the paper, please see scripts/gdelt/gdelt_prompts and scripts.amazon/amazon_prompts in our code.
E.5 Does GPT remember Amazon Review data?
We have a conversion with GPT-3.5-turbo to verify that it does not remember the exact information of Amazon Review data.
input: We are using Amazon Review Data, published by Jianmo Ni, et al., 2018, which is firstly used in the paper "Justifying recommendations using distantly-labeled reviews and fined-grained aspects", EMNLP 2019.
The dataset includes millions of reviews (ratings, text, helpfulness votes), product metadata (descriptions, category information, price, brand, and image features) in the range May 1996 - Oct 2018. Do you remember any of the review information if I give you a review id?
GPT-3.5 output: As an AI language model, I am not capable of remembering any review information unless I have been programmed to store and retrieve specific pieces of information. However, I can search for and retrieve review information from the dataset if given a review ID.
input: can you recall any information with reviewer ID A28GK1G2KDXHRP on date 04 17, 2014.
GPT-3.5 output: I apologize, but the Amazon Review dataset does not include information about the specific date when a review was posted. It only includes the year and month when the review was posted. Therefore, I cannot provide you with any information about reviews posted by reviewer ID A28GK1G2KDXHRP on April 17, 2014.
input: Reviewer A0148968UM59JS3Y8D1M bought Hanes Men's Classic Crew on 01 18, 2014, what review did he wrote?
GPT-3.5 output: I apologize, but as an AI language model, I do not have access to real-time or updated information about specific Amazon reviews or purchases made by individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of a specific review. It only includes metadata such as the reviewer ID, product ID, and overall rating. Therefore, I cannot provide you with the review written by the reviewer A0148968UM59JS3Y8D1M for the Hanes Men's Classic Crew on January 18, 2014 or any other date.
input: Reviewer A0148968UM59JS3Y8D1M bought Hanes Men's Classic Crew on 01 18, 2014, his review starts with 'they seem very well made..', can you recall the complete review he wrote?
GPT-3.5 output: I apologize, but as an AI language model, I do not have access to real-time or updated information about specific Amazon reviews or purchases made by individual reviewers. Additionally, the Amazon Review dataset does not include information about the specific content of a specific review. It only includes metadata such as the reviewer ID, product ID, and overall rating. Therefore, I cannot provide you with the complete review written by the reviewer A0148968UM59JS3Y8D1M for the Hanes Men's Classic Crew on January 18, 2014, or any other date.
input: on which date did reviewer A0148968UM59JS3Y8D1M buy Hanes Men's 7-Pack ComfortSoft Briefs?
GPT-3.5 output: Sorry, as an AI language model, I do not have access to personal information or purchase history of individuals.
Listing 6: A conversation with GPT-3.5-turbo to test its memorization about Amazon Review data.
E.6 Training and Inference Details
In this section, we present the details about training the models and doing inference with them.
LLMs. We use GPT-3-davinci and GPT-3.5-turbo. We set temperature to be 0 to ensure that our results are reproducible.
Training Base Models. For each model, we did grid search and chose the hyperparameters based on their performance on the dev set; see Table 2 for the values of the hyperparameters. For KE, DyRep, ANHP, the main hyperparameters to tune are the entity embedding size  $D_{1}$ , predicate embedding size  $D_{2}$ , RNN hidden size  $D_{3}$  used in the network and the number of layers  $L$  of the attention structure (DyRep and ANHP). In practice, the optimal  $D$  for a model was usually 4, 8, 16, 32; the optimal  $L$  was usually 1, 2, 3, 4. In the experiment, to train the parameters of the base model, we performed early stopping based on log-likelihood on the held-out dev set.
Retrieval. For each event, we use few-shot prompting (see Appendix E.4 for examples) to generate a set of cause events. For each LLM-generated cause, we retrieve the most similar  $D$  actual events from the history, following the procedure in section 3.2.
- For GDELT, we set  $D = 2$  and the average number of retrieved events is 10.
- For ICEWS, we set  $D = 2$  and the average number of retrieved events is 10.
- For Amazon Review, we set  $D = 4$  and the average number of retrieved events is 10.
Training Ranking Model. The energy function used in our ranking model is the same as what's proposed in Xue et al. (2022), which consists a continuous-time Transformer and an MLP. The hyperparameters are tuned within a range of values that make the score function to have a similar size as the base ANHP model. Training this ranking model needs negative samples. On GDELT and ICEWS, we use 5 negative samples for predicate prediction and 20 for object and predicate-object prediction. On Amazon Review, we use 5 for both type and time prediction.
MODEL	DESCRIPTION	VALUE USED
GDELT	ICEWS	AMAZON
KNOW-EVOLVE	ENTITY EMBEDDING SIZE	16	16	NA
PREDICATE EMBEDDING SIZE	4	4	NA
RNN HIDDEN SIZE	16	16	NA
DYREP	ENTITY EMBEDDING SIZE	16	16	NA
PREDICATE EMBEDDING SIZE	4	4	NA
RNN HIDDEN SIZE	16	16	NA
ATTENTION LAYERS NUMBER	1	1	NA
NHP	RNN HIDDEN SIZE	NA	NA	36
ENTITY EMBEDDING SIZE	NA	NA	16
PREDICATE EMBEDDING SIZE	NA	NA	8
ANHP	ENTITY EMBEDDING SIZE	16	16	32
PREDICATE EMBEDDING SIZE	4	4	32
HEADS NUMBER	2	1	1
ATTENTION LAYERS NUMBER	1	1	2
RANKING MODEL	TEMPORAL EMBEDDING SIZE	40	40	24
HIDDEN SIZE	60	60	56
HEADS NUMBER	4	3	16
ATTENTION LAYERS NUMBER	3	3	3
Table 2: Values of hyperparameters used for models trained on the three datasets.
Figure 9: Prediction performance measured by MRR of different methods on each dataset. The configuration of this figure is the same as that of Figure 2: on GDELT, we show object (upper) and predicate-object (lower) prediction; on ICEWS, we show object (upper) and time (lower) prediction; on Amazon Review, we show type (upper) and time (lower) prediction.
Computation Cost. All the experiments were conducted on a server with 256G RAM, a 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and one NVIDIA A100 GPU for acceleration. The training batch size is 8. For GDELT and ICEWS, the wall-clock training time is: 4.9ms per sequence for KE; 6.7ms per sequence for DyRep; 28.7ms per sequence for ANHP. For Amazon Review, the wall-clock time is: 3.6ms per sequence for NHP; 5.2ms per sequence for ANHP. For each of the datasets, the wall-clock time of training the ranking model is 9ms per training sample.
F Additional Results and Analysis
In this section, we present additional results and analysis that complement section 4.
F.1 Main MRR results.
Our main results in section 4 are the MR results. In this section, we show the corresponding MRR results, which are known to be more robust to bad predictions. The MRR results are in Figure 9.
F.2 Analysis About Precision and Recall
Figure 10 shows the mean average precision (MAP) and mean average recall (MAR) of each base model for a range of  $M$ . These metrics complement MR and MRR for evaluating the base models: MAP measures the fraction of the predictions that are correct; MAR measures the fraction of the
(a) MAP on predicate prediction.
(b) MAR on predicate prediction.
Figure 10: MAP@M and MAR@M on GDELT dataset.
(c) MAP on object prediction.
(d) MAR on object prediction.