SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\alpha$  provides a trade-off between two loss terms. Like many penalty constraint optimization approaches, when  $\alpha \rightarrow \infty$ , the optimal solution tends to satisfy the pairwise causal knowledge constraint; when  $\alpha \rightarrow 0$ , it is domain-knowledge-free. An advantage of such an incompatibility framework is that it is linearly additive, and therefore more statements can be easily incorporated as additional loss terms.
4.3 Outcome Model with Window  $w > 1$
While the outcome model for the probability of occurrence of  $y$  in the very next position ( $w = 1$ ) can be easily computed for an autoregressive model, the task of computing this probability for  $w > 1$  windows becomes combinatorial in nature. To illustrate this, note that the quantity  $p(I(y)_i^w | h_{i-1}, l_i)$  or  $1 - p(l_{i+1} \neq y, l_{i+2} \neq y, \dots, l_{i+w} \neq y | h_i)$  can be computed (sequentially) as  $1 - \prod_{k=1}^{w} p(l_{i+k} \neq y | h_{i+k-1})$  for  $w = 1$ . Some efficient estimation techniques are available from recent work for probabilistic querying [50] for larger  $w$ . Consider the selection of window  $w$  for the outcome in an event sequence of length  $n$ . The problem in this case is relevant to the following probabilistic query – the probability of not observing any event  $Y$  within the next  $w$  events. Thus the (worst case) computational cost is  $O((|L| - 1)^{n-1})$  for the largest window  $w = n-1$  in our setting, according to Definition 1. In this paper, we assume that  $w$  is relative small compared to sequence length for the sake of practicality. ILE and SLE with respect to the statement 'Z reduces the occurrence of  $Y$ ' for a sequence  $k$  can be defined similarly for a window  $w > 1$  and can be expressed as  $p(I(y)_i^w | h_{i-1}, l_i = z) - p(I(y)_i^w | h_{i-1}, l_i \neq z)$  and  $\frac{1}{n-w} \sum_{i=1}^{n-w} (p(I(y)_i^w | h_{i-1}, l_i = z) - p(I(y)_i^w | h_{i-1}, l_i \neq z))$ , respectively.
Correspondingly, for a statement such as  $Z$  inhibits  $Y$  and  $Z$  amplifies  $Y$  for a sequence  $k$ , our proposed combined loss for  $w = 2$  can be expressed respectively as
$$
\begin{array}{l} L _ {t o t} = - \sum_ {i = 1} ^ {n _ {k}} p ^ {*} (l _ {i}) + \alpha m a x (\frac {1}{n - 2} \sum_ {i = 1} ^ {n - 2} (\frac {\mathbb {1} (l _ {i} = z) (1 - p ^ {*} (l _ {i + 1} \neq y) p ^ {*} (l _ {i + 2} \neq y))}{p ^ {*} (l _ {i} = z)}) \\ - \frac {\mathbb {1} (l _ {i} \neq z) (1 - p ^ {*} (l _ {i + 1} \neq y) p ^ {*} (l _ {i + 2} \neq y))}{1 - p ^ {*} (l _ {i} = z)}, 0) \tag {5} \\ \end{array}
$$
$$
\begin{array}{l} L _ {t o t} = - \sum_ {i = 1} ^ {n _ {k}} p ^ {*} (l _ {i}) + \alpha m a x (\frac {1}{n - 2} \sum_ {i = 1} ^ {n - 2} (\frac {\mathbb {1} (l _ {i} \neq z) (1 - p ^ {*} (l _ {i + 1} \neq y) p ^ {*} (l _ {i + 2} \neq y))}{1 - p ^ {*} (l _ {i} = z)}) \\ - \frac {\mathbb {1} \left(l _ {i} = z\right) \left(1 - p ^ {*} \left(l _ {i + 1} \neq y\right) p ^ {*} \left(l _ {i + 2} \neq y\right)\right)}{p ^ {*} \left(l _ {i} = z\right)}, 0) \tag {6} \\ \end{array}
$$
We perform some ablation experiments on the choice of  $w$  (i.e.  $w = 2$ ) in (sub)section 5.4.
We implement our approach using a transformer architecture [51] and test the performance of our model on synthetic datasets as well as real-world datasets on the task of event prediction. We also explore a unique application where event concepts are generated from a large language model. Further details around implementation and training can be found in the Appendix.
Datasets. We run experiments on 4 generated synthetic datasets to verify the learning capabilities and validity of our approach. We simulate event sequence data using a binary summary Markov model (BSuMM), which assumes that the probability of an event label's occurrence at any position in a sequence depends on whether or not labels from its influencing set have occurred in some recent look-back period [23]. Here we consider simple event sequences involving BSuMM dynamics over 5 event labels, denoted A, B, C, D and E. For each experiment, we generated 5 samples of datasets where each dataset was split into train/dev/test sets  $(60/20/20)\%$ . We first used BSuMM to simulate 50 event sequences over the 5 event labels, each of length 100, where only two interactions occur:  $B$  excites  $A$  and  $B$  inhibits itself. We incorporated the ground truth causal pair as  $(B, A)$ , with  $B \nearrow A$ . This makes up experiments for Synth-1 and Synth-2 where the look-backs (windows) used for generation are 2 and 4 respectively. Similarly, we generated another 2 sets of 50 sequences over the 5 event labels, each of length 100, with interactions as shown in Figure 3; these are the Synth-3 and Synth-4 datasets. Here we only consider the causal pair  $(C, B)$  with ground truth  $C \nearrow B$  as injected knowledge. In the Appendix, we show results for when other causal pairs are injected (one at a time) as knowledge.
Baselines and Metric. We consider 5 baselines to compare against our model. BSuMM and OSuMM are two types of parametric summary Markov models that have been used for event prediction and influencing set identification [23]. Another common parametric model for event sequences is the k-th order Markov chain (kMC); for this baseline, we show the best performing version over  $k \in \{1,2,3,4\}$ . We also consider a recent state-of-the-art transformer model - Probabilistic Attention to influence (PAIN) - that uses attention-to-influence techniques for modeling temporal event sequences [34]. Lastly, we include a vanilla transformer for event sequences (TES) [51] as a direct comparison to our proposed PC-TES with  $\alpha \in \{100,10,1,0.1,0.01,0.001\}$ . We evaluate model performance using the metric of loglikelihood (LL) on the test subset, which is equivalent to a logarithmic scoring rule for the task of predicting the next event label given the history [52]. Further details such as hyper-parameters chosen based on the validation set are discussed in the Appendix.
Results. Results of the synthetic experiments are shown in Table 1. Our model PC-TES outperforms others by incorporating one pairwise statement. It is worth noting that the neural models considered here predict better than non-neural models by a significant margin, even with relatively small datasets.
Table 1: Next event prediction loglikelihood on 4 synthetic datasets.
Dataset	BSuMM	OSuMM	kMC	PAIN	TES	PC-TES
Synth-1	-382.82(4.67)	-373.85(6.37)	-364.60(6.47)	-141.06(5.11)	-107.81(2.85)	-107.19(3.01)
Synth-2	-371.13(6.36)	-370.66(5.12)	-350.37(7.45)	-137.94(8.42)	-114.01(2.31)	-111.89(1.52)
Synth-3	-358.25(7.23)	-359.678(12.30)	-378.97(5.64)	-131.20(12.79)	-119.99(2.29)	-113.58(3.81)
Synth-4	-363.36(6.93)	-361.82(5.45)	-371.75(6.35)	-134.09(13.39)	-113.95(3.05)	-113.15(3.66)
5.2 Real Applications
Datasets, Baselines and Metric. We perform experiments on the following 4 real-world applications from various domains shown in Table 2, referring the reader to dataset details in the Appendix. Diabetes contains meals, exercise activity, insulin dosage, and changes in blood glucose measurements for a group of 65 diabetes patients [53]. We incorporated the single statement 'insulin intake decreases blood glucose' into our objective, which agreed with the assessments mentioned in [47]. Defi contains user trading events including borrow, repay, deposit, redeem, liquidation and swap from the Aave website, aave.com. The sole qualitative statement in this study is: 'borrowing increases the probability of liquidation' [33]. LinkedIn contains user-level employment-related events for LinkedIn users. We included all 10 event types as self-exciting, according to prior work [54, 23], which provided 10 qualitative statements. Beigebooks consists of a series of economic topics derived from documents issued by the Federal Reserve Board regarding the economic state of affairs in the US. We incorporated 8 causal statements about economic topics, such as 'vehicle sale robust' increases 'dealer inventory increase'. All datasets were split into train-dev-test sets (60/20/20)% for fair evaluation, where hyper-parameters were chosen using train-dev sets and evaluation was performed on the test sets. We consider the same baselines and evaluation metric.
Table 2: Dataset summary: # of event labels  $(M)$  ,# of sequences  $(K)$  and # of events  $(N)$
Dataset	M	K	N
BeigeBooks	15	260	2370
Diabetes	13	65	20210
LinkedIn	10	1000	2212
Defi	6	500	17258
LLM-Generated Event Sequences	50	243	1398
Table 3: Next event prediction loglikelihood on 4 real-world applications.
Dataset	BSuMM	OSuMM	kMC	PAIN	TES	PC-TES
Diabetes	-658.61	-577.76	-656.19	-466.36	-373.27	-370.64
Defi	-1172.49	-1152.84	-1155.60	-129.49	-126.56	-120.54
LinkedIn	-114.14	-115.05	-111.39	-25.32	-22.48	-21.71
Beigebooks	-40.21	-40.82	-46.09	-21.077	-10.27	-10.04
Results. Results on the 4 real-world benchmarks are shown in Table 3. Overall, our model PC-TES outperforms others by incorporating pairwise statements. We highlight again that results of neural models are substantially better than their non-neural counterparts. The highest performance boost by PC-TES occurs on Defi where the relative gain is around  $5\%$  compared to a vanilla transformer model TES, validating the practicality of our approach for incorporating domain knowledge.
5.3 LLM-Generated Event Sequences
Motivation. We explore a use case of event prediction where we have partial causal knowledge about the events, along with powerful generative large language models (LLMs) to generate event sequences. It is challenging to use LLMs to generate event sequences, since depending on the choice of parameters (e.g., temperature which controls the stochasticity of the generation process), one is able to either generate a large number of very noisy sequences or a small number of high-quality sequences. A noisy collection will typically result in much lower precision in a downstream application, while a small collection will result in a loss of recall. Here we explore a solution that leverages a small but
Figure 2: Causal knowledge graph of Wikidata event concepts.
Figure 3: Examples of LLM-generated event concept sequences.
[demonstration, regime_change, coup_d'état, looting, civil_disorder]  
[impeachment, statutory_law, regime_change, coup_d'état, civil_disorder]  
[earthquake, tsunami, disaster, looting, conflict]  
[arson, explosion, mass_murder, crime]  
[school_shooting, civil_disorder, riot, social-issue]  
[attack, mass_murder, riot, civil_disorder, war, regime_change]  
[energy_crisis, shortage, economic_crisis, civil_disorder, protest]
high-quality set of generated sequences along with an incomplete source of knowledge to produce a high-accuracy prediction model.
Dataset. For this application, we use an LLM (f1an-t5-xx1(11B)) to create sequences of event concepts from Wikidata [55]. These are high-level concepts (classes) in Wikidata that represent newsworthy event types. We gather these types by querying Wikidata for concepts that had links to Wikinews articles and belonged to a subclass of the occurrence class, indicating that they are newsworthy event classes. We compile the common top-level classes of these concepts, which yielded 50 event-related Wikidata classes. Some of these have causal relations (e.g., has_cause or has-effect) either directly (e.g., earthquake causes tsunami) or through their instances (an earthquake event causing a tsunami event). Using these relations, we build a causal knowledge graph, depicted in Figure 2. The pairs in this network are used for knowledge injection.
To generate event sequences, we first create a prompt that restricted the output to the 50 event concept labels and asked for the next event concept in a sequence in the form of a question: "What usually happens after X?" for an input event concept X. For subsequent events, we remove the already-seen concepts from the vocabulary and use the generated event concept to ask the next question in a similar way. We repeat the questions until we reach a pre-defined maximum sequence length, or if the LLM fails to generate an in-vocabulary response in  $k$  number of attempts. For the experiments in this paper, we use a random number between 2 and 10 as the maximum sequence length and  $k = 10$  for the number of LLM generation attempts. We repeat this sequence generation procedure 5 times for each of our 50 event sequences, which resulted in 243 sequences (with 7 failed generation attempts).
Figure 3 shows examples of the generated event sequences.
Results. Table 4 demonstrates how 3 neural models perform prediction on the LLM generated dataset, with and without 5 representative pairwise causal relations in different domains, from as natural disaster to social activities. We demonstrate that with our approach, without any pretraining, we can boost predictive performance by  $3 - 4\%$  as compared to a transformer event model. This shows the effectiveness of our approach in building event models with limited or no event sequences, using LLMs in conjunction with knowledge injection.
5.4 Ablation on Window for Outcome
We perform ablation experiments with window size in Table 5 ( $w = 2$ ) on the synthetic simulations generated by the two different graphs from section 5.1 and Diabetes dataset. Overall, results from  $w = 2$  do not differ significantly from  $w = 1$ , yet consistent improvements over TES are still observed.
Table 4: Next event prediction on LLM generated event sequences.
Causal Pair	LL
None (PAIN)	-33.10
None (TES)	-28.29
impeachment /protest (PC-TES)	-28.19
tsunami /earthquake (PC-TES)	-27.48
mass_shooting /protest (PC-TES)	-28.10
infectious_disease /epidemic (PC-TES)	-28.21
murder /riot (PC-TES)	-27.81
All pairs above (PC-TES)	-27.37
Table 5: Next event prediction on synthetic datasets with window  $w = 2$
Dataset	LL
Synth-1	-115.04(2.06) ↑
Synth-2	-113.27(2.50) ↓
Synth-3	-107.65(3.33) ↓
Synth-4	-111.38(2.78) ↑
Diabetes	-370.60 ↑
In this paper, we addressed the under-explored area of incorporating pairwise causal relations into deep learning models for temporal event sequences. We proposed a novel approach that injects qualitative causal knowledge into transformer-based models and demonstrated a significant enhancement in performance. Our established framework for causal inference in temporal event sequences provides a theoretical justification that ensures unbiased estimation of the proposed measure. Through extensive experimentation, we validate the superior performance of our proposed approach over several state-of-the-art models on a probabilistic prediction task. By effectively leveraging knowledge about causal pairs, our approach showcases its ability to capture the intricate relationships between events and make enhanced predictions in multivariate event sequences. Furthermore, we introduce a unique application where we leverage a large language model to generate sequences of societal events. By demonstrating how a causal knowledge graph can be used for event prediction within such sequences, we highlight the practical implications of our framework. In summary, our work offers a promising direction for improving the performance of transformer-based models in multivariate event sequences by explicitly incorporating pairwise causal information.
Our approach is centered around event sequences that may span a considerable length of time and involve potentially repeated instances of all event types. We operate under the assumption that the influence of past events are confounding, while ensuring robustness against confounding factors through strong ignorability. However, we advise caution when employing causal pairs, as incorrect causal knowledge may potential lead to incorrect conclusions. We highlight that the availability of knowledge that is consistent with the ground truth in general is crucial to performance in models that involve knowledge augmentation.
This work was supported by the Rensselaer-IBM AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network (http://ibm.biz/AIHorizons). We express our gratitude to the anonymous reviewers and meta-reviewer for their valuable feedback, which contributed to improving the quality of our work.
[49] Frank Wittig and Anthony Jameson. Exploiting qualitative knowledge in the learning of conditional probabilities of Bayesian networks. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2000.
[50] Alex Boyd, Sam Showalter, Stephan Mandt, and Padhraic Smyth. Predictive querying for autoregressive neural sequence models. arXiv preprint arXiv:2210.06464, 2022.
[51] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[52] Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[53] Andrew Frank and Arthur Asuncion. UCI machine learning repository, 2010.
[54] Hongteng Xu, Dixin Luo, and Hongyuan Zha. Learning Hawkes processes from short doubly-censored event sequences. In Proceedings of the International Conference on Machine Learning (ICML), pages 3831-3840, 2017.
[55] Denny Vrandecic and Markus Krötzsch. Wikidata: A free collaborative knowledge base. Communications of ACM, 57(10):78-85, 2014.