SlowGuess/NeurIPS_2023_OCR huggingface dataset:
actual events that are covered by the predictions; higher is better. Their precise definitions (and formulas) for our setting can be found in Appendix E.2.
As we can see in Figure 10, when  $M$  increases, MAR monotonically increases but the change of MAP may not be monotonic (which is consistent with how MAP and MAR behave in other retrieval tasks). For predicate prediction, we found that  $M = 5$  yields a good balance between precision and recall as well as a low financial cost (i.e., GPT API calls). For object prediction,  $M = 50$  seems to give the best precision-recall balance but we chose  $M = 20$  due to the GPT API budget. For object prediction, both MAP and MAR are lower than 0.1 (but still significantly better than a random guess of 0.0015), indicating that this data is very difficult to model.
F.3 Analysis About Generalization
The LLM in LAMP is instructed by a few demonstrations. On GDELT, these demonstrations cover
17 out of 20  $(85\%)$  predicates;
- 30 out of 2279 (1.31%) subjects;
- 28 out of 2279 (1.22%) objects.
However, the LLM is able generalize beyond the demonstrations, suggesting cause events that involve novel predicates and entities. Precisely, the LLM-generated cause events involve
- 4699 distinct predicates, which cover all the 20 ground-truth predicates;
- 60552 distinct subjects, which cover 30 (of 30) subjects mentioned in demonstrations, as well as 787 not-mentioned ground-truth subjects.
- 65554 distinct objects, which cover 28 (of 28) objects mentioned in demonstrations, as well as 788 not-mentioned ground-truth objects.
The retrieved cause events cover:
- all the 20 ground-truth predicates.
- 30 (of 30) subjects in demonstrations, as well as 186 not-mentioned ground-truth subjects.
- 28 (of 28) objects in demonstrations, as well as 216 not-mentioned ground-truth objects.
Note that retrievals involve fewer distinct subjects and objects than the LLM-generated causes. It is because the LLM generation is often diverse and creative, proposing many novel subject-predicate-object combinations that involve ground-truth entities but haven't actually happened in real history. Such causes will eventually be grounded to actual previous events that exhibit less diversity and creativity, thus ending up with fewer distinct subjects and objects in the retrievals. Benefiting from the strong generalization capability of LLMs, our LAMP framework has a significant potential for broad applications.
F.4 Additional Results About LLMs
Now we present our additional results of comparing different LLMs. They include the MR results of comparing GPT-3 and GPT-3.5 on GDELT (Figures 11 and 12), MR results of comparing GPT-3 and GPT-3.5 on Amazon Review (Figure 13), and MRR results corresponding to Figure 3 (Figure 14).
Figure 11: Different LLMs on predicate (left) and object (right) prediction on GDELT dataset, with KE.
Figure 12: Different LLMs on predicate (left) and object (right) prediction on GDELT dataset, with DyRep.
Figure 13: Different LLMs on type prediction on Amazon Review.
Figure 14: MRR results of different LLMs on predicate (left) and object (right) prediction on GDELT.
F.5 Additional Results About Prompt Design
Now we present additional results about the prompt design analysis in section 4. Figure 15 and Figure 16 show the MRR results corresponding to Figure 5 and Figure 6, respectively.
Figure 15: MRR results corresponding to Figure 5: predicate (left) and object (right) prediction on GDELT.
Figure 16: MRR results corresponding to Figure 6: predicate (left) and object (right) prediction on GDELT.
F.6 Additional Results About Retrievals
Now we present our additional analysis results about retrievals. Figure 17 presents the MRR results with different numbers of retrieved evidence events on GDELT.
Figure 17: MRR results on GDELT corresponding to Figure 4: predicate (left) and object (right) prediction.
Figure 18: MR results of type prediction with different similarity metrics on Amazon Review.
Figure 19: About similarity metric: predicate (left) and object (right) prediction on GDELT, with KE.
Figure 20: About similarity metric: predicate (left) and object (right) prediction on GDELT, with DyRep.