SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Amazon (Ni, 2018). This dataset includes time-stamped user product review behavior from January 2008 to October 2018. Each user has a sequence of produce review events where each event containing the timestamp and category of the reviewed product, with each category corresponding to an event type. We work on a subset of 5200 most active users with an average sequence length of 70 event tokens and then end up with  $K = 16$  event types.
StackOverflow (Leskovec & Krevl, 2014). This dataset has two years of user awards on a question-answering website: each user received a sequence of badges and there are  $K = 22$  different kinds of badges in total. We work on a subset of xx most active users with an average sequence length of xx event tokens.
For the Taobao dataset, each task includes approximately 1 day of time; for the Amazon dataset, each task includes approximately 2 years of time; for the StackOverflow dataset, each task includes approximately 5 months of time. Table 1 shows statistics about each dataset mentioned above.
Algorithm 2 PromptTPP at test time of the  $\mathcal{T}$ -th task.
Output: Sampled next event  $\widehat{e}_{I + 1} @ \widehat{t}_{I + 1}$ .
1: procedure DRAWNEXTEVENT  $(s_{[0,T]},f_{\phi_{enc}},f_{\phi_{dec}})$
Input: An event sequence  $s_{[0,T]} = \{e_i@t_i\}_{i=1}^I$ . Trained base model with a encoder  $f_{\phi_{enc}}$  and a decoder  $f_{\phi_{dec}}$ ; trained CtRetroPromptPool  $(\mathcal{K},\mathcal{V}) = \{(k_i,P_i)\}_{i=1}^M$  and the score function  $\varphi$ .
2:  $t_0\gets T;\mathcal{H}\gets s_{[0,T]}$
3:  $\triangleright$  Compute sampling intensity
4:  $\{\lambda_{e}(t_{j} \mid \mathcal{H})\}_{j=1}^{N} \gets \text{SAMPLEINTENSITY}(s_{[0,T]}, f_{\phi_{enc}}, f_{\phi_{dec}}$ ,  $\{(k_{i}, P_{i})\}_{i=1}^{M}$ ) for all  $t_{j} \in (t_{0}, \infty)$
5:  $\triangleright$  Compute the upper bound  $\lambda^{*}$
6:  $\triangleright$  Technical details can be found in Mei & Eisner (2017)
7: find upper bound  $\lambda^{*} \geq \sum_{e=1}^{E} \lambda_{e}(t_{j} \mid \mathcal{H})$  for all  $t_{j} \in (t_{0}, \infty)$
8: repeat
9: draw  $\Delta \sim \mathrm{Exp}(\lambda^{*}); t_{0} += \Delta$  ▷time of next proposed event  $\widehat{t}_{I+1}$
10:  $u\sim \mathrm{Unif}(0,1)$
11: until  $u\lambda^{*} \leq \sum_{e=1}^{E} \lambda_{e}(t_{0} \mid \mathcal{H})$
12: draw  $\widehat{e}_{I + 1}\in \{1,\ldots ,E\}$  where probability of  $e$  is  $\propto \lambda_{e}(t_{0}\mid \mathcal{H})$
13: return  $\widehat{e}_{I + 1}@\widehat{t}_{I + 1}$
14: procedure SAMPLEINTENSITY  $(s_{[0,T]},f_{\phi_{enc}},f_{\phi_{dec}},\{(k_i,P_i)\}_{i = 1}^M)$
15: Assume the last event in  $s_{[0,T]}$  is  $e@t$
16: Generate a list of sample times  $\{t_j\}_{j = 1}^N,t_j\geq T$
17: Compute the intensity at sample times  $\lambda_{e}t_{j}$  ← CALCINTEN-SITY(s[0,t], e@t,  $f_{\phi_{enc}}$ ,  $f_{\phi_{dec}}$ , {( $\pmb{k}_i, \pmb{P}_i$ )}M{i=1})
18: return  $\{\lambda_{e}(t_{j} \mid \mathcal{H})\}_{j=1}^{N}$
DATASET	K	# EVT TOKENS TOTAL	Avg # EVT TOKENS PER SEQ	Avg # EVT TOKENS PER TASK	Avg # SEQ PER TASK
TAOBAO	20	720,000	150	80,000	32
AMAZON	16	360,000	70	42,000	10
STACKOVERFLOW	22	240,000	60	43,000	12
Table 1: Statistics of each dataset.
D.2 3S statistics and Experiment on Distribution Shift
We use the 3S (sum-of-squared-spacings) statistics proposed by Shchur et al. (2021) to depict the distribution of an event sequence in continuous time. Compared to the classical KS statistics (Lewis, 1965), it uniformly captures multiple properties of event sequence, such as total event count and distribution of inter-event times. Empirically, replacing the KS score with the 3S statistic consistently leads to a better separation between distributions generated by different TPPs. Please refer to the original paper (Shchur et al., 2021) for a detailed discussion.
For exploring the distribution shift in the Taobao dataset, we randomly sampled a thousand sequences of events and split them into 10 subsets by timestamps: each subset has approximately equal time horizon and is notated sequentially from 0-th to the 9-th subset. Then we follow the procedure in (Shchur et al., 2021) to compute the 3S statistics for each subset and illustrate the results in Figure 8a.
D.3 Evaluation Setup
To set up the training and evaluation process, we partition Taobao and Amazon datasets into 10 consecutively rolling slides (namely 10 tasks) and partition the StackOverflow dataset into 6 rolling slides (namely 6 tasks). For the Taobao dataset, each slide covers approximately 1 day of time; for the Amazon dataset, each slide covers 2 years of time; for the StackOverflow dataset, each slide covers approximately 5 months time. The subset in each task is split into training, validation, and test sets with a  $70\%$ ,  $10\%$ ,  $20\%$  ratio by chronological order. Each task has no overlap in the test set. In such a setting, the total test set covers approximately  $70\%$  of data.
MODEL	# PARAMETERS
TAOBAOAmazon	SO
PRE-NHP	23.3K	23.4K	23.3K
PRE-ANHP	25.4K	25.6K	25.4K
RE-NHP	23.3K	23.4K	23.3K
RE-ANHP	25.4K	25.6K	25.4K
O-TPP	<1K	<1K	<1K
CL-NHP	27.6K	27.7K	27.6K
CL-ANHP	29.5K	29.6K	29.5K
PT-NHP	26.2K	26.3K	26.2K
PT-ANHP	27.8K	27.0K	27.8K
Table 2: Total number of parameters for models trained on the three datasets.
We train and evaluate each task sequentially. Our evaluation setup is close to that used in real applications: train the model using a fixed length of historical data and evaluate the model using the following window of the data.
Figure 9: A demonstration of the sliding window validation method.
For the implementation of NHP, AttNHP, and thinning algorithm, we used the code from the public GitHub repository at https://github.com/yangalan123/anhp-andtt (Yang et al., 2022) with MIT License.
For O-TPP, as the authors Yang et al. (2017) have not published the code, we implement it using the tick (Bacry et al., 2017) library.
For CL-NHP and CL-ANHP, without the public code, by following the main idea of the authors Dubey et al. (2022), we develop the hypernetwork with an MLP layer and add apply regularizer to the hypernetwork parameters while learning a new event sequence, which prevents adaptation of the hypernetworks parameters completely to the new event sequence. Note that, the base models used are NHP and ANHP, respectively, instead of FullyRNN (Omi et al., 2019) applied in the original paper.
We implemented our methods with PyTorch (Paszke et al., 2017) and published the code at https://github.com/yanyanSann/PromptTPP.
D.5 Training and Testing Details
D.5.1 Training and Hyperparameters Selection
Training base TPP model. To train the parameters for a given neural TPP, we performed early stopping based on log-likelihood on the held-out dev set.
- For NHP, the main hyperparameters to tune are the hidden dimension  $D$  of the neural network. In practice, the optimal  $D$  for a model was usually 32, 64, 128, and we search for the optimal value among them for different datasets.
MODEL	DESCRIPTION	VALUE USED
TAOBAO	AMAZON	SO
PRE-NHP	RNN HIDDEN SIZE	76	76	76
TEMPORAL EMBEDDING	64	32	64
PRE-ANHP	HIDDEN SIZE	64	64	64
LAYER NUMBER	3	3	3
RE-NHP	RNN HIDDEN SIZE	76	76	76
TEMPORAL EMBEDDING	64	32	64
RE-ANHP	HIDDEN SIZE	64	64	64
LAYER NUMBER	3	3	3
O-TPP	KERNEL SIZE	20 × 20	16 × 16	22 × 22
CL-NHP	RNN HIDDEN SIZE	64	64	64
TEMPORAL EMBEDDING	64	32	64
CL-ANHP	HIDDEN SIZE	64	64	64
LAYER NUMBER	3	3	3
RNN HIDDEN SIZE	64	64	64
M(RETRYVAL PROMPT POOL SIZE)	10	10	10
Pt-NHP	N(TOP-N SELECTED)	4	4	4
Lp(PROMPT LENGTH)	10	10	10
C(ASYCHRONOUS REFRESH FREQUENCY)	2	2	2
TEMPORAL EMBEDDING	64	32	64
HIDDEN SIZE	64	64	64
LAYER NUMBER	2	2	2
Pt-ANHP	M(RETRYVAL PROMPT POOL SIZE)	10	10	10
N(TOP-N SELECTED)	4	4	4
Lp(PROMPT LENGTH)	10	10	10
C(ASYCHRONOUS REFRESH FREQUENCY)	2	2	2
Table 3: Descriptions and values of hyperparameters used for models trained on the three datasets.
- For AttNHP, in spite of  $D$ , another important hyperparameter to tune is the number of layers  $L$  of the attention structure. In practice, the optimal  $L$  was usually 1, 2, 3, 4. In the experiment, we choose the hyperparameter based on the held-out dev set while keeping AttNHP to have a similar size to that of NHP.
Training PromptTPP. We find  $\alpha$  in equation 11 is not sensitive and works well in a large range, so we set  $\alpha = 0.1$  consistently for both datasets. For the prompts, we set  $M = 10$ ,  $N = 4$ ,  $\bar{L}_p = 10$  for both datasets. For the asynchronous training parameter  $C$ , we choose  $C = 2$  for Taobao and Amazon datasets by default.
Figure 10: Statistical significance of the improvements from a temporal prompt on Taobao and Amazon datasets.
Chosen Optimizer and Hyperparameters. All models are optimized using Adam (Kingma & Ba, 2015). Table 3 contains descriptions that list all of the hyperparameters set throughout our experiments.
Testing. As described in Mei & Eisner (2017), we minimized the Bayes risk via the thinning algorithm to determine decisions for what a predicted next event time  $\widehat{t}_{i+1}$  and type  $\widehat{e}_{i+1}$  would be after conditioning on a portion of a sequence  $s_{[0,t_i]} = [e_1@t_1, \ldots, e_i@t_i]$ . All experimental results are averaged over 5 runs, and the corresponding standard deviation is reported as well. In the experiment, we report the metrics on the test set for each task as well as the average metrics over all the tasks.
Integral Approximations. During training and testing, there are a number of integrals (e.g., log-likelihood in equation 11) that need to be computed, which are not feasible in closed form. Thus, we must approximate them. All integrals and expectations are approximated via Monte Carlo (MC) estimates with certain amounts of samples used.  $\mathcal{L}_{\text {non\_event }}$  in equation 1 uses  $100\mathrm{MC}$  samples during training and testing. When evaluating the integrals used for next event predictions in thinning algorithm, we used 100 samples where the sample points were shared across integrals for a single set of predictions in order to save on computation. The exact approximation procedure for the log-likelihood can be found in Mei & Eisner (2017).
Environment. All the experiments were conducted on a server with 256G RAM, a 64 logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz) and one NVIDIA Tesla P100 GPU for acceleration.
D.6 Analysis II Details: Statistical Significance
We performed the paired permutation test to validate the significance of our proposed temporal prompt. Particularly, for each model variant (Pt-NHP and Pt-ANHP), we split the test data into ten folds and collected the paired test results with temporal prompt and with the standard prompt, respectively, for each fold. Then we performed the test and computed the p-value following the recipe at https://axon.cs.byu.edu/Dan/478/assignments/permutation_test.php.
The results are in Figure 10. It turns out that, on both datasets, the performance differences are strongly significant for the error rate metric (p-value  $< 0.05$ ) and weakly significant for the RMSE metric (p-value  $\approx 0.08$ ).
D.7 More Result: Computational Complexity of PromptTPP
The computational complexity comes from two parts: the TPP model and prompt pool.
Take ANHP as the base model. Assume the input sequence length is  $K$ , event embedding size  $d_{e}$ . Recall the prompt length  $L_{p}$ , selection size  $N$  and prompt pool size  $M$ , key size  $D$ .
ANHP is attention-based, so its original complexity is  $\mathcal{O}(K^2 d_e)$ . By considering the attached prompt, the prompt-augmented ANHP's complexity becomes  $\mathcal{O}((K + NL_p)^2 d_2)$ . Retrieval's complexity is  $\mathcal{O}(MD^2)$ . So the total complexity is  $\mathcal{O}((K + NL_p)^2 d_2 + MD^2)$ .
(a) Evaluation results on Task 9.
(b) Evaluation results on Task 10.
Figure 11: OTD distance comparison of generated sequence of all models on Amazon dataset.
D.8 More Result: Generation Ability Comparison
We investigated the generative ability of the models empirically on Amazon dataset. Given a trained model, we fixed the prefix event sequence and performed the 1-event sampling and 3-event sampling (autoregressively) on the test set of task 9 and task 10. We followed (Xue et al., 2022) to compute the average optimal transport distance (OTD) to measure the distance between the generated sequence and the ground truth. Seen from Figure 11, our proposed models Pt-NHP and Pt-AttNHP achieves the best results. This is consistent with the findings in Main Results in the paper.