SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siqiao Xue*, Yan Wang*, Zhixuan Chu*, Xiaoming Shi, Caigao Jiang, Hongyan Hao, Gangwei Jiang, Xiaoyun Feng, James Y. Zhang, Jun Zhou
{siqiao.xsq, luli.wy, chuzhixuan.czx}@alibaba-inc.com
Neural Temporal Point Processes (TPPs) are the prevalent paradigm for modeling continuous-time event sequences, such as user activities on the web and financial transactions. In real-world applications, event data is typically received in a streaming manner, where the distribution of patterns may shift over time. Additionally, privacy and memory constraints are commonly observed in practical scenarios, further compounding the challenges. Therefore, the continuous monitoring of a TPP to learn the streaming event sequence is an important yet under-explored problem. Our work paper addresses this challenge by adopting Continual Learning (CL), which makes the model capable of continuously learning a sequence of tasks without catastrophic forgetting under realistic constraints. Correspondingly, we propose a simple yet effective framework, PromptTPP<sup>1</sup>, by integrating the base TPP with a continuous-time retrieval prompt pool. The prompts, small learnable parameters, are stored in a memory space and jointly optimized with the base TPP, ensuring that the model learns event streams sequentially without buffering past examples or task-specific attributes. We present a novel and realistic experimental setup for modeling event streams, where PromptTPP consistently achieves state-of-the-art performance across three real user behavior datasets.
Event sequences are ubiquitous in a wide range of applications, such as healthcare, finance, social media, and so on. Neural TPPs (Mei & Eisner, 2017; Shchur et al., 2020; Zuo et al., 2020; Zhang et al., 2020; Yang et al., 2022) have emerged as the dominant paradigm for modeling such data, thanks to their ability to leverage the rich representation power of neural networks. However, most existing works assume a static setting, where the TPP model is trained on the entire data, and parameters remain fixed after training. In contrast, real-world event data usually arrives in a streaming manner, rendering it impractical to store all data and retrain the model from scratch at each time step due to computational and storage costs. Shown in Figure 1, a common approach is to use sliding windows to frame the data for model training and prediction. Traditional schemes include pretraining a TPP, which is used for all the following test periods, retraining TPP on the data of each slide of windows and online TPPs. However, they either may fail to adapt to new data or suffer from catastrophic forgetting (see Appendix A for an empirical analysis).
In our work, we approach the problem by adopting Continual Learning (CL) (Hadsell et al., 2020; Hao et al., 2023; Chu & Li, 2023; Chu et al., 2023b), a relevant area studying how systems learn sequentially from a continuous stream of correlated data. Yet, classical CL models are not fully
$\spadesuit$  Corresponding author.
$^{1}$ Our code is available at https://github.com/yanyanSann/PromptTPP
applicable to our problem. A major line of CL methods (Cha et al., 2021; Buzzega et al., 2020) rely on a rehearsal buffer to retrain a portion of past examples. However, they become ineffective when a rehearsal buffer is not allowed – for example, in real-world scenarios where data privacy matters (Shokri & Shmatikov, 2015) or there are resource constraints. Another branch of works (Ke et al., 2020) bypass the forgetting issue by assuming known task identity at test time, but knowing task identity at test time restricts practical usage. Furthermore, the problem of sequential tasks of event sequence in continuous time have barely been studied.
To develop a CL algorithm for such data in real-world scenarios with applicability and generality, we draw inspiration from recent advances in prompt-augmented learning (Liu et al., 2022a; Varshney et al., 2022; Cho et al., 2022; Li et al., 2023; Chu et al., 2023a; Wang et al., 2023). Prompt-augmented learning is a form of machine learning that involves adding additional information or prompts to the training data in order to further improve the performance of the model. This can include adding labels or annotations to the data, providing additional context to help the model better understand the data, or incorporating feedback from human experts to guide the learning process. By incorporating these prompts, the model is able to learn more effectively and make more accurate predictions. Prompt-augmented learning has been used successfully in a variety of applications, including natural language processing, computer vision,
and speech recognition. Intuitively, prompt-augmented learning reformulates learning downstream tasks from directly adapting model weights to designing prompts that "instruct" the model to perform tasks conditionally while maintaining model plasticity. Thus, it is promising to leverage prompts to sequentially learn knowledge and further store learned knowledge of event sequence in the CL context. While prompt learning (Wang et al., 2022b,a) already demonstrates its effectiveness on multiple CL benchmarks in language modeling, we wish to extend their success to the models of neural TPPs.
To this end, we propose PromptTPP, a novel CL framework whose basis is a continuous-time retrieval prompt pool for modeling streaming event sequences. Specifically, we develop a module of temporal prompt that learns knowledge and further store the learned knowledge for event sequences in continuous time. To improve the applicability, building upon prior works (Wang et al., 2022b), we structure the prompts in a key-value shared memory space called the retrieval prompt pool, and design a retrieval mechanism to dynamically lookup a subset of task-relevant prompts based on the instance-wise input of event sequences. The retrieval prompt pool, which is optimized jointly with the generative loss, ensures that shared (unselected) prompts encode shared knowledge for knowledge transfer, and unshared (selected) prompts encode task-specific knowledge that helps maintain model plasticity. PromptTPP has two distinctive characteristics: (i) applicability: despite the effectiveness in augmenting TPP with CL, the prompt pool and the event retrieval mechanism removes the necessity of a rehearsal buffer and knowing the task identity, making the method applicable to modeling the event streams in a more realistic CL setting, i.e., memory efficient and task agnostic. (ii) generality: our approach is general-purpose in the sense that it can be integrated with any neural TPPs. In summary, our main contributions are:
- We introduce PromptTPP, a novel prompt-augmented CL framework for neural TPPs. It represents a new approach to address the challenges of modeling streaming event sequences by learning a pool of continuous-time retrieval prompts. These prompts serve as parameterized instructions for base TPP models to learn tasks sequentially, thus enhancing the performance of the model.
- We formalize an experimental setup for evaluating the streaming event sequence in the context of CL and demonstrate the effectiveness of our proposed method across three real user datasets.
- By connecting the fields of TPP, CL, and prompt learning, our method provides a different perspective for solving frontier challenges in neural TPPs.
Figure 1: Overview of the classical schemes and PromptTPP framework for streaming event sequences.
Generative Modeling of Event Sequences. Suppose we observe  $I$  events at a fixed time interval  $[0, T]$ . Each event is denoted mnemonically as  $e@t$  (i.e., "type  $\in$  at time  $t$ " and the sequence is denoted as  $s_{[0, T]} = [e_1@t_1, \ldots, e_I@t_I]$  where  $0 < t_1 < \ldots < t_I \leq T$  and  $e_i \in \{1, \ldots, E\}$  is a discrete event type. Note that representations in terms of time  $t_i$  and the corresponding inter-event time  $\tau_i = t_i - t_{i-1}$  are isomorphic, we use them interchangeably.
Generative models of event sequences are TPPs. Specifically, TPPs define functions  $\lambda_{e}$  that determine a finite intensity  $\lambda_{e}(t\mid s_{[0,t)})\geq 0$  for each event type  $e$  at each time  $t > 0$  such that  $p_e(t\mid s_{[0,t)}) = \lambda_e(t\mid s_{[0,t)})dt$ . Then the log-likelihood of a TPP given the entire event sequence  $s_{[0,T]}$  is
$$
\mathcal {L} _ {l l} = \sum_ {i = 1} ^ {I} \log \lambda_ {e _ {i}} \left(t _ {i} \mid s _ {[ 0, t _ {i})}\right) - \int_ {t = 0} ^ {T} \sum_ {e = 1} ^ {E} \lambda_ {e} (t \mid s _ {[ 0, t)}) d t, \tag {1}
$$
Instead of posing strong parametric assumptions on the intensity function, neural TPPs (Du et al., 2016; Mei & Eisner, 2017; Zhang et al., 2020; Zuo et al., 2020; Yang et al., 2022) use expressive representations for the intensity function via neural networks and maximize the associated log-likelihood equation 1 via stochastic gradient methods.
CL Problem Formulation for Streaming Event Sequences. The typical CL problem is defined as training models on a continuum of data from a sequence of tasks. Given a sequence  $s_{[0,T]}$ , we split it based on a sliding window approach shown in Figure 1 and form a sequence of tasks over the time  $\{\mathcal{D}_0, \dots, \mathcal{D}_N\}$ , where the  $T$ -th task  $\mathcal{D}_{\mathcal{T}} = (s_{train}^T, s_{test}^T)$  contains a tuple of train and test set of event sequences and the two sets have no overlap in time. Data from the previous tasks are not available when training for future tasks. We use the widely-adopted assumption that the task boundaries are clear and the task switch is sudden at training time (Pham et al., 2021). Our goal is to continually learn the sequences while avoiding catastrophic forgetting from the previous tasks.
Prompt Learning. Prompt learning methods propose to simply condition frozen language models (LMs) to perform down-stream tasks by learning prompt parameters that are prepended to the input tokens to instruct the model prediction. Compared with ordinary fine-tuning, literature shows In our context, a naive application of prompt learning is to preprocess learnable parameters  $P_{s} \in \mathbb{R}^{L_{p} \times D}$ , called a prompt, to the event embedding  $h = [P_{s}||x]$ , where  $x \in \mathbb{R}^{D}$  denotes the output of a TPP's embedding layer of an event, and then feed it to the model function  $g(h)$ , i.e., a decoder, to perform downstream tasks. Instead of the native application, in our proposed method, we design a novel prompt learning mechanism to properly model the event streams (see section 3.3).
3 Prompt-augmented TPP
We introduce a simple and general prompt-augmented CL framework for neural TPPs, named PromptTPP. As shown in Figure 2, PromptTPP consists of three components: a base TPP model, a pool of continuous-time retrieval prompts and a prompt-event interaction layer. In this section, we omit the task index  $\mathcal{T}$  in our notation as our method is general enough to the task-agnostic setting.
3.1 Base TPP
A neural TPP model autoregressively generates events one after another via neural networks. For the  $i$ -th event  $e_i@t_i$ , it computes the embedding of the event  $\pmb{x}_i \in \mathbb{R}^D$  via an embedding layer, which takes the concatenation of the type and temporal embedding  $\pmb{x}_i = [\pmb{x}_i^{\mathrm{TYPE}}||\pmb{x}_i^{\mathrm{TEMP}}]$  where  $||$  denotes concatenation operation and  $\pmb{x}_i^{\mathrm{TYPE}} \in \mathbb{R}^{D_1}$ ,  $\pmb{x}_i^{\mathrm{TEMP}} \in \mathbb{R}^{D_2}$ ,  $D = D_1 + D_2$ . Then one can draw the next event conditioned on the hidden states that encode history information sequentially:
$$
t _ {i + 1}, e _ {i + 1} \sim \mathbb {P} _ {\theta} \left(t _ {i + 1}, e _ {i + 1} \mid \boldsymbol {h} _ {i}\right), \quad \boldsymbol {h} _ {i} = f _ {r} \left(\boldsymbol {h} _ {i - 1}, \boldsymbol {x} _ {i}\right), \tag {2}
$$
where  $f_{r}$  could be either RNN (Du et al., 2016; Mei & Eisner, 2017) or more expressive attention-based recursion layer (Zhang et al., 2020; Zuo et al., 2020; Yang et al., 2022). For the simplicity of notation, we denote the embedding layer and recursion layer together as the encoder  $f_{\phi_{enc}}$  parameterized by  $\phi_{enc}$ . Our proposed PromptTPP is general-purpose in the sense that it is straightforward to incorporate any version of neural TPP into the framework.
3.2 Continuous-time Retrieval Prompt Pool
The motivations for introducing Continuous-time Retrieval Prompt Pool (CtRetroPromptPool) are two-fold. First, existing prompt-learning works focus on classification tasks in NLP or CV domains,
2The sum operation is also used in some literature. In this paper, we apply concatenation for event embedding.
Figure 2: Overview of PromptTPP. Up: At training time, PromptTPP selects a subset of temporal prompts from a key-value paired CtRetroPromptPool based on our proposed retrieval mechanism; then it prepends the selected prompts to the event representations; finally it feeds the extended event representations into the prompt-event interaction and intensity layer, and optimizes the CtRetroPromptPool through the loss defined in equation 11. Down Left: Illustration of how to parameterize a temporal prompt. Down Right: Illustration of prompt tuning in the prompt-event interaction layer.
whose methods are not directly applicable for sequential tasks of learning event streams in continuous time (see section 4.2). Second, the practical setup for modeling event streams closes to the task-agnostic CL setting, where we do not know task identity at test time so that training task-dependent prompt is not feasible. Even if we use extra sources to memorize the task identity, naive usage of prompts (Liu et al., 2022b, 2021; Tam et al., 2022) are still found to result in catastrophic forgetting.
For the first motivation, we construct temporal prompt that properly encodes the knowledge of temporal dynamics of event sequence. To address the second, we build a store of prompts in a key-value shared space to transfer knowledge sequentially from one task to another without distinguishing between the common features among tasks versus the features that are unique to each task.
Temporal Prompt. In contrast to the standard prompt, the temporal prompt is a time-varying learnable matrix that encodes not only the structural but also the temporal knowledge of the event sequence. We define the temporal prompt  $\pmb{P} = [P_s; P_t] \in \mathbb{R}^{L_p \times D}$ , where  $L_p$  is the prompt length and  $P_s \in \mathbb{R}^{L_p \times D_1}, P_t \in \mathbb{R}^{L_p \times D_2}$  denotes the structural component and temporal component. While  $P_s$  is a learnable submatrix, the temporal component  $P_t$  is set to be continuous-time positional encodings of the estimated conditional time so as to consider the timing. More concretely, given  $i$ -th event, we estimate the arithmetic mean of inter-event times up to  $t_{i-1}$ , denoted by  $\mathcal{E}_i = \mathbb{E}[\{\tau_j\}_{j < i}]$  and add this estimated inter-event time to  $t_{i-1}$  to get the estimated conditional time  $t_p := \widehat{t_i} = t_{i-1} + \mathcal{E}_i$ . Inline with Yang et al. (2022), we compute the temporal embedding  $\mathrm{TE}(t_p) \in \mathbb{R}^{D_2}$  by
$$
\mathrm {T E} (t) = \cos \left(\frac {t}{n _ {t e}} \cdot \left(\frac {5 N _ {t e}}{n _ {t e}}\right) ^ {\frac {d - 1}{D _ {2}}}\right) \text {i f} d \text {i s o d d , T E} (t) = \sin \left(\frac {t}{n _ {t e}} \cdot \left(\frac {5 N _ {t e}}{n _ {t e}}\right) ^ {\frac {d}{D _ {2}}}\right) \text {i f} d \text {i s e v e n} \tag {3}
$$
where  $\{N_{te}, n_{te} \in \mathbb{N}\}$  are hyperparameters selected according to the time scales in different periods. As  $\mathrm{TE}(\mathcal{E}_i)$  is a vector, we concatenate it repeatedly to form  $P_t$ , i.e.,  $P_t = [\mathrm{TE}(t_p)|\ldots,||\mathrm{TE}(t_p)] \in \mathbb{R}^{L_p \times D_2}$ . Note that the structural component  $P_s$  is learnable while the temporal component  $P_t$  is computed deterministically.
An important consideration of employing such a mechanism is that the mean characterizes the most important property (the long-run average) of the inter-event time distribution, and the computation is straightforward. By taking the temporal embedding of the estimated average conditional time, the prompt efficiently encodes the time-varying knowledge up to the current event, which facilitates learning prediction tasks. We verify the effectiveness of temporal prompt in section 4.2.
From Prompt to Prompt Pool. Ideally, one would learn a model that is able to share knowledge when tasks are similar while maintaining knowledge independently otherwise. Thus, instead of
applying a single prompt, we introduce a pool of temporal prompts to store encoded knowledge, which can be flexibly grouped as an input to the model. The pool is defined as
$$
\boldsymbol {P} = \left[ \boldsymbol {P} _ {1}, \dots , \boldsymbol {P} _ {M} \right], \tag {4}
$$
where  $M$  denotes the total number of prompts and  $\pmb{P}_i\in \mathbb{R}^{L_p\times D}$  is a single temporal prompt. Following the notation in section 3.1, recall  $\pmb{h}_i\in \mathbb{R}^D$  denotes the hidden representation of the  $i$ -th event in the sequence which encodes the event history up to  $t_i$  via the recursion by equation 2 and let  $\{P_{r_j},j = 1,\dots,N\}$  be a subset of  $N$  selected prompts, we then incorporate them into the event sequences as in-context augmentation as follows:
$$
\left[ \boldsymbol {P} _ {r _ {1}} \right| |, \dots , \left| \left| \boldsymbol {P} _ {r _ {N}} \right| \right| \boldsymbol {h} _ {i} ], \tag {5}
$$
Prompts are free to compose, so they can jointly encode knowledge for the model to process, which provides flexibility and generality in the sense that a more fine-grained knowledge sharing scheme can be achieved via prompt retrieval mechanism. Under this mechanism, a combination of prompts is selected for each task - similar inputs tend to share more common prompts, and vice versa.
Retrieval Prompt Pool. The retrieval prompt pool shares some design principles with methods in other fields, such as RETRO (Borgeaud et al., 2022). Specifically, the prompt pool is augmented to be a key-value store  $(\mathcal{K},\mathcal{V})$ , defined as the set of learnable keys  $\pmb{k}\in \mathbb{R}^{D}$  and values - temporal prompts  $\pmb{P}$  in equation 4:
$$
(\mathcal {K}, \mathcal {V}) = \left\{\left(\boldsymbol {k} _ {i}, \boldsymbol {P} _ {i}\right) \right\} _ {i = 1} ^ {M} \tag {6}
$$
The retrieval prompt pool may be flexible to edit and can be asynchronously updated during the training procedure. The input sequence itself can decide which prompts to choose through query-key matching. Let  $\varphi : \mathbb{R}^D \times \mathbb{R}^D$  be the cosine distance function to score the match between the query and prompt key. Given a query  $h_i$ , the encoded event vector, we search for the closest keys over  $\mathcal{K}$  via maximum inner product search (MIPS). The subset of top-N selected keys is denoted as:
$$
\mathrm {K} _ {\text {t o p} - N} = \underset {\{r _ {j} \} _ {j = 1} ^ {N}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {N} \varphi \left(\boldsymbol {h} _ {i}, \boldsymbol {k} _ {r _ {j}}\right) \tag {7}
$$
Importantly, the design of this strategy brings two benefits: (i) it decouples the query learning and prompt learning processes, which has been empirically shown to be critical (see section 4.2); (ii) the retrieval is performed in an instance-wise fashion, which makes the framework become task agnostic, meaning the method works without needing to store extra information about the task identity at test time. This corresponds to a realistic setting for modeling event streams in real applications.
3.3 Prompt-Event Interaction
The interaction operation controls the way we combine prompts with the encoded event states, which directly affects how the high-level instructions in prompts interact with low-level representations. Thus, we believe a well-designed prompting function is also vital for the overall CL performance. The interaction mechanism is also called prompting function in the NLP community. We apply the multi-head self-attention mechanism (Vaswani et al., 2017) (MHSA) for modeling the interactions and adopt the mainstream realization of prompting function - Prefix Tuning (Pre-T) (Li & Liang, 2021). Denote the input query, key, and values as  $z_{Q}, z_{K}, z_{V}$  and the MHSA layer is constructed as:
$$
\operatorname {M H S A} \left(\boldsymbol {z} _ {Q}, \boldsymbol {z} _ {K}, \boldsymbol {z} _ {V}\right) = \left[ \boldsymbol {z} _ {1} \right| |, \dots , \left| \left| \boldsymbol {z} _ {m} \right] \boldsymbol {W} ^ {O}, \right. \tag {8}
$$
where  $\boldsymbol{z}_i = \mathrm{ATTN}(\boldsymbol{z}_Q\boldsymbol{W}_i^Q, \boldsymbol{z}_K\boldsymbol{W}_i^K, \boldsymbol{z}_V\boldsymbol{W}_i^V), \boldsymbol{W}^O, \boldsymbol{W}_i^Q, \boldsymbol{W}_i^K, \boldsymbol{W}_i^V$  are projection matrix. In our context, let  $\{\boldsymbol{P}_{r_i}\}_{i=1}^N$  be the retrieved prompts from the pool, we set  $h_i$  to be the query, split each prompt  $\boldsymbol{P}_{r_i}$  into  $\boldsymbol{P}_{r_i}^K, \boldsymbol{P}_{r_i}^V \in \mathbb{R}^{L_p/2 \times D}$  and prepend them to keys and values, respectively, while keeping the query as-is:
$$
\boldsymbol {h} _ {i} ^ {\text {P r e - T}} = \operatorname {M H S A} \left(\boldsymbol {h} _ {i}, \left[ \boldsymbol {P} ^ {K} \| \boldsymbol {h} _ {i} \right], \left[ \boldsymbol {P} ^ {V} \| \boldsymbol {h} _ {i} \right]\right), \tag {9}
$$
where  $\pmb{P}^{K} = [\pmb{P}_{r_{1}}^{K}||, \dots, ||\pmb{P}_{r_{N}}^{K}], \pmb{P}^{V} = [\pmb{P}_{r_{1}}^{V}||, \dots, ||\pmb{P}_{r_{N}}^{V}]$ . Apparently, the key and value  $z_{K}, z_{V} \in \mathbb{R}^{\left(\frac{L_{p^{*}N}}{2} + 1\right) \times D}$  and the output  $\pmb{h}_{i}^{Pre - T} \in \mathbb{R}^{D}$ . Noted that there exist other prompting methods, such as Prompt Tuning (Pro-T), where all the prompts concurrently prepend to the query, key and values:
$$
\boldsymbol {h} _ {i} ^ {\text {P r o - T}} = \operatorname {M H S A} \left(\left[ \boldsymbol {P} ^ {Q} \right| \left| \boldsymbol {h} _ {i} \right], \left[ \boldsymbol {P} ^ {K} \right| \left| \boldsymbol {h} _ {i} \right], \left[ \boldsymbol {P} ^ {V} \right| \left| \boldsymbol {h} _ {i} \right]\right), \tag {10}
$$
<sup>3</sup>As  $D_{1} + D_{2} = D$ , we use  $D$  and  $(D_{1} + D_{2})$  interchangeable throughout the paper.
where  $P^Q = P^K = P^V = [P_{r_1}||, \dots, ||P_{r_N}]$ . As a result, the query, key, value and output  $z_Q, z_K, z_V, h_i^{Pro - T} \in \mathbb{R}^{(L_p*N + 1)\times D}$ . Despite being less efficient in computation, we empirically demonstrate that Pre-T brings better performance. See Analysis III in section 4.2.
The output of the MHSA is then passed into an intensity layer (an MLP with softmax activation) to generate the intensity  $\lambda_{e}(t_{i}), e \in \{1, \dots, E\}$ . For simplicity, we denote the prompt-event interaction and intensity layer together as the decoder  $f_{\phi_{dec}}$  parameterized by  $\phi_{dec}$ .
3.4 Model Optimization
The full picture of PromptTPP at training and test time is described in Algorithm 1 and Algorithm 2 in Appendix C.1. At every training step, each event  $e_i@t_i$  is recursively fed into the encoder  $f_{\phi_{enc}}$ , after selecting  $N$  prompts following the aforementioned retrieval strategy, the intensity  $\lambda(t_i)$  is computed by the decoder  $f_{\phi_{dec}}$ . Overall, we seek to minimize the end-to-end loss function:
$$
\min  _ {\boldsymbol {P}, \phi_ {e n c}, \phi_ {d e c}, \mathcal {K}} \mathcal {L} _ {n l l} (\boldsymbol {P}, f _ {\phi_ {e n c}}, f _ {\phi_ {d e c}}) + \alpha \sum_ {i} \sum_ {\mathrm {K} _ {t o p - N}} \varphi \left(f _ {\phi_ {e n c}} \left(e _ {i} @ t _ {i}\right), \boldsymbol {k} _ {r _ {j}}\right), \tag {11}
$$