Prompt-augmented Temporal Point Process for Streaming Event Sequence
Authors: Siqiao Xue
Yan Wang
Zhixuan Chu
Xiaoming Shi
Caigao JIANG
Hongyan Hao
Gangwei Jiang
Xiaoyun Feng
James Y. Zhang
JUN ZHOU
Venue: NeurIPS 2023 poster (2023)
Keywords: prompt
point process
event sequence
continual learning.
TL;DR: We propose a prompt-augmented temporal point process to continuously learn the streaming event sequence. We formalize a novel and realistic experimental setup, where our method sets state-of-the-art performance across two real user behavior datasets.

Abstract: Neural Temporal Point Processes (TPPs)  are the prevalent paradigm for modeling continuous-time event sequences, such as user activities on the web and financial transactions. In real world applications, the event data typically comes in a streaming manner, where the distribution of the patterns may shift over time. Under the privacy and memory constraints commonly seen in real scenarios, how to continuously monitor a TPP to learn the streaming event sequence is an important yet under-investigated problem. In this work, we approach this problem by adopting Continual Learning (CL), which aims to enable a model to continuously learn a sequence of tasks without catastrophic forgetting. While CL for event sequence is less well studied, we present a simple yet effective framework, PromptTPP, by integrating the base TPP with a continuous-time retrieval prompt pool. In our proposed framework, prompts are small learnable parameters, maintained in a memory space and jointly optimized with the base TPP so that the model is properly instructed to learn event streams arriving sequentially without buffering past examples or task-specific attributes. We formalize a novel and realistic experimental setup for modeling event streams, where PromptTPP consistently sets state-of-the-art performance across two real user behavior datasets.

Prompt-augmented Temporal Point Process for Streaming Event Sequence
Siqiao Xue", Yan Wang", Zhixuan Chu Xiaoming Shi, Caigao Jiang, Hongyan Hao
Gangwei Jiang, Xiaoyun Feng, James Y. Zhang, Jun Zhou
Ant Group
Hangzhou, China
(sigiao. xxsq, wy, chuzhixuan czx @alibaba -inaainnncc.o com
Abstract
Neural Temporal Point Processes (TPPs) are the prevalent paradigm for modeling continuous-time event sequences, such as user activities on the web and financial transactions. In real-world applications, event data typically received in stream- ing manner, where the distribution of patterns may shift over time. Additionally, privacy and memory constraints are commonly observed in practical scenarios, further compounding the challenges. Therefore, the continuous monitoring of TPP to learn the streaming event sequence is an important yet under-explored problem. Our work paper addresses this challenge by adopting Continual Learning (CL), which makes the model capable of continuously learning sequence of tasks without catastrophic forgetting under realistic constraints. Correspondingly, we propose a simple yet effective framework, PrompíTPP', by integrating the base TPP with continuous-tim retrieval prompt pool. The prompts, small learnable parameters, are stored in a memory space and jointly optimized with the base TPP, ensuring that the model learns event streams sequentially without buffering past examples or task-specific attributes. We present a novel and realistic experi- mental setup for modeling event streams, where PromprTPP consistently achieves state-of-ther of-the-art performance across three real user behavior datasets.
Introduction
Event sequences are ubiquitous wide range of applications, such healthcare, finance, social media, and so on. Neural TPPs (Mei & Eisner, 2017; Shchur et al., 2020; Zuo et al., 2020; Zhang al., 2020; Yang et al., 2022) have emerged as the dominant paradigm for modeling such data, thanks to their ability to leverage the rich representation power of neural networks. However, most existing works assume static setting, where the TPP model trained on the entire data, and parameters remain fixed after training. In contrast, real-world event data usually arrives in streaming manner, rendering impractical to store all data and retrain the model from scratch at each time step due to computational and storage costs. Shown in Figure 1, common approach is to use sliding windows to frame the data for model training and prediction. Traditional schemes include pretraining a TPP, which is used for all the following test periods, retraining TPP on the data of each slide of windows and online TPPs. However, they either may fail to adapt to new data or suffer from catastrophic forgetting (see Appendix A for empirical analysis).
In our work, we approach the problem by adopting Continual Learning (CL) (Hadsell al., 2020; Hao et al., 2023; Chu & Li, 2023; Chu et al., 2023b), relevant area studying how systems learn sequentially from continuous stream of correlated data. Yet, classical CL models are not fully
These authors contributed equally to this work.
Conresponding author.
OOur code is available at http:://gihbbb.oomyyyaaaaaaa//pp
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

applicable to our problem. A major line of CL methods (Cha et al., 2021; Buzzega et al., 2020) rely on rehearsal buffer to retrain a portion of past examples. However, they become ineffective when rehearsal buffer is not allowed for example, in real-world scenarios where data privacy matters (Shokri & Shmatikov, 2015) or there are resource constraints. Another branch of works (Ke et al., 2020) bypass the forgetting issue by assuming known task identity at test time, but knowing task identity at test time restricts practical usage. Furthermore, the problem of sequential tasks of event sequence in continuous time have barely been studied.
To develop a CL algorithm for such data in real- world scenarios with applicability and general- ity, we draw inspiration from recent advances in prompt-augmented learning (Liu et al., 2022a;
Stresrinn Evert Sequeence
Train #1
Train 92
Test 12
Varshney et al., 2022; Cho et al., 2022; I.i et al., 2023; Chu et al., 2023a; Wang et al., 2023). Prompt-augmentt
Pretrained
Train
Test
learning is a form of ma- chine learning that involves adding additional information or prompts to the training data in order to further improve the performance of the model. This can include adding labels or amno- tations to the data, providing additional context to help the model better understand the data, or incorporating feedback from human experts to guide the learning process. By incorporating these prompts, the model is able to learn more effectively and make more accurate predictions. Prompt-augmented learning has been used suc- cessfully in a variety of applications, including natural language processing, computer vision, and speech recognition. Intuitively, tasks from directly adapting model weights to designing prompts that "instruct" the model to perform tasks conditionally while maintaining model plasticity. Thus, it is promising to leverage prompts to sequentially learn knowledge and further store learned knowledge of event sequence in the CL context. While prompt learning (Wang et al., 2022b,a) already demonstrates its effectiveness on multiple CL benchmarks in language modeling, we wish extend their success to the models of neural TPPs.
Retrained
Train Mocial #1
Text $1
Triin Modhi #0
Trxi AP
Onlinee
Online Model
Test
Online Model
Tost 92
Ours
Coet-tiee
Train Mooal #1
Text #1
Petro
Tran Model AN
Text inu
Instret
tranable
Figure 1: Overview of the classical schemes and PromptTPP framework for streaming event sequences.
learning reformulates learning downstream
To this end, we propose PromptTPP, a novel CL framework whose basis is a continuous-time retrieval prompt pool for modeling streaming event sequences. Specifically, we develop a module of temporal prompt that learns knowledge and further store the learned knowledge for event sequences in continuous time. To improve the applicability. building upon prior works (Wang et al.. 2022b), we structure the prompts in key-value shared memory space called the retrieval prompt pool, and design retrieval mechanism to dynamically lookup subset of task-relevant prompts based on the instance-wist input of event sequences, The retrieval prompt pool, which optimized jointly with the generative loss, ensures that shared (unselected) prompts encode shared knowledge for knowledge transfer, and unshared (selected) prompts encode task-specific knowledge that helps maintain model plasticity. Promp(TPP has two distinctive characteristics: (i) applicability: despite the effectiveness in augmenting TPP with CL, the prompt pool and the event retrieval mechanism removes the necessity of rehearsal buffer and knowing the task identity, making the method applicable to modeling the event streams in a more realistic CL setting, i.., memory efficient and task agnostic. (ii) generality: approach is general-pe -urpose in the sense that it can be integrated with any neural TPPs. In summary, our main contributions are:
We introduce PromptTPP, novel prompt-augmented CL framework for neural TPPs. It represents a new approach to address the challenges of modeling streaming event sequences by learning pool of continuous time retrieval prompts. These prompts serve as parameterized instructions for base TPP models to learn tasks sequentially, thus enhancing the performance of the model.
We formalize an experimental setup for evaluating the streaming event sequence in the context of CL and demonstrate the effectiveness of our proposed method across three real user datasets.
By connecting the fields of TPP, CL, and prompt learning, our method provides different perspective for solving frontier challenges in neural TPPs.

---

2 Preliminaries
Generative Modeling of Event Sequences. Suppose we observe events at a fixed time interval T). Each event is denoted mnemonically as e@t (i.e., "type e at time t²) and the sequence is denoted as T|o.T]
[er @t1,......_.._...... e, [tt}] where t < ... t I T and e; E fl..... E}} is a discrete event type. Note that representations in terms of time 4 and the corresponding inter-event time T; = 4 - 1--1 are isomorphic, we use them interchangeably.
Generative models of event sequences are TPPs. Specifically, TPPs define functions Re that determine a finite intensity Xe (& $1o.))) 2 0 for each event type e at each time > such that pe s(o,t)
(t S(o.e)ddt. )dt. Then the log- likelihood of TPP given the entire event sequence $[o.7] is
L
Lu Σ log de, ((o...))
E de 8s0.))
(I)
1=0
Instead of posing strong parametric assumptions on the intensity function, neural TPPs (Du et al., 2016; Mei & Fisner, 2017: Zhang et al., 2020; Zuo et al., 2020; Yang et al., 2022) use expressive representations for the intensity function via neural networks and maximize the associated log- likelihood equation 1 via stochastic gradient methods.
CL Problem Formulation for Streaming Event Sequences. The typical CL problem is defined as training models on a continuum of data from a sequence of tasks. Given sequence $|o,2]. we split based on sliding window approach shown in Figure and form a sequence of tasks over the time (Do.... D where the T-th task Dy (sTrain sestt) contains a tuple of train and test set of event sequences and the two sets have no overlap in time. Data from the previous tasks are not available when training for future tasks. We use the widely-adopter assumption that the task boundaries are clear and the task switch is sudden at training time (Pham et al., 2021). ). Our goal is to continually learn the sequences while avoiding catastrophic forgetting from the previous tasks.
Prompt Learning. Prompt learning methods propose to simply condition frozen language models (LMs) perform down-stream tasks by learning prompt parameters that are prepended to the input tokens to instruct the model prediction. Compared with ordinary fine-tuning, literature shows In our context, naive application of prompt learning to prepend learnable parameters P. E RL,xD called prompt. to the event embedding h = [P&l æz), where I E R denotes the output of a TPP's embedding layer of an event, and then feed the model function g(h). ) i.e., decoder, to perform downstream tasks. Instead of the native application, in our proposed method, we design novel prompt learning mechanism to properly model the event streams (see section 3.3).
3 Prompt-augmented TPP
We introduce simple and general prompt-augmented CL framework for neural TPPs, named PromptTPP. As shown in Figure 2, PromptTPP consists of three components: a base TPP model, a pool of continuous- time retrieval prompts and a prompt-event interaction layer. In this section, omit the task index T in our notation as our method is general enough to the task-agnostic setting.
3.1 Base TPP
A neural TPP model autoregressively generates events after another via neural networks. For the i-th event e, @t,, it computes the embedding of the event æ; E R via an embedding layer, which takes the concatenation' of the type and temporal embedding æ; [xx TYPE la. [[MP) where || denotes concatenation operation and xV: E IR . _EMP E R2 D D1 + D2. Then one can draw the next event conditioned on the hidden states that encode history information sequentially:
4,41, C++1 Po 1:41 (h,), h; fr (h:-1: .z)),
(2)
where fr could be either RNN (Du et al., 2016; Mei Eisner, 2017) or more expressive attention- based recursion layer (Zhang et al., 2020; Zuo et al., 2020: Yang et al.. 2022). For the simplicity of notation, we denote the embedding layer and recursion layer together as the encoder fa.. parame- terized by dene- Our proposed PromptTPP is general-purpont in the sense that it is straightforward to incorporate any version of neural TPP into the framework.
3.2 Continuous-time Retrieval Prompt Pool
The motivations for introducing Continuous-ting time Retrieval Prompt Pool (CtRetroPromptPPo) are two-fold. First, existing prompt-learning works focus on classification tasks in NLP or CV domains,
"The sum operation is also used in some literature. In this paper, we apply concatenation for event embedding.
3

---

50 e
4 1
5,6
TTput Soquenie
Prompt-TPP
Base TPP Ensocer
Quary
Retrieval
h4
O.0.o!
Sakwa
Prompt-Ever Interaction
Intensily Layer
Pronpt
aomiin Retroval Phompt Pool
PrompiTroer
Temporal Prompt
Prompt-E Event Interaction
Tap-i
Erandingg
Pronge
P
Secumiic
Matrivood Sell oftintionnnnn
Figure 2: Overview of PromptTPP, Up: At training time, PromptTPP selects subset of temporal prompts from a key-value paired CiRetroPromptPool based on our proposed retrieval mechanism; then prepends the selected prompts to the event representations; finally feeds the extended event representations into the prompt-event interaction and intensity layer, and optimizes the CiRetroPromptPPol through the loss defined in equation 11. Down Left: Illustration of how parameterize temporal prompt. Down Right: Illustration of prompt tuning in the prompt-event interaction layer.
whose methods not directly applicable for sequential tasks of learning event streams in continuous time (see section 4.2). Second, the practical setup for modeling event streams closes to the task- agnostic CL setting, where we do not know task identity at test time so that training task-dependent prompt is not feasible. Even if we use extra sources to memorize the task identity, naive usage of prompts (Liu al., 2022b, 2021; Tam al., 2022) are still found to result in catastrophic forgetting.
For the first motivation, we construct temporal prompt that properly encodes the knowledge of temporal dynamics of event sequence. To address the second, we build a store of prompts in key- value shared space to transfer knowledge sequentially from one task to another without distinguishing between the common features among tasks versus the features that are unique each task.
Temporal Prompt. In contrast to the standard prompt, the temporal prompt time-van unr-ingying learnable matrix that encodes not only the structural but also the temporal knowledge of the event sequence. We define the temporal prompt P = PP; P.] E RL,xD,, where Lp is the prompt length and P. E RD, P. E R denotes the structural component and temporal component. While P. is a learnable submatrix, the temporal component P. is set to be continuous-time positional encodings of the estimated conditional time so as to consider the timing. More concretely, given i-th event, we estimate the arithmetic mean of inter-event times up to 44-1, denoted by & [B(TT]b< and add this estimated inter-event time 4-1 get the estimated conditional time 4, == 4 4i-1 & Inline with Yang et al. (2022), we compute the temporal embedding TE(t,) E Pa by
t Tite
5N Riee
if d is odd, TE(t) = sin ( ft_
5N d
TE(t) = COS
t3
if d is even
(3)
Tl
7se
where {Neey Thee E N) are hyperparameters selected according to the time scales in different periods.
As TE(E) is a vector, we concatenate repeatedly to form P,. i.e. P. [TE(tp]ll- ..., [ET L, x D2, Note that the structural component P, is learnable while the temporal component P. is computed deterministically.
An important consideration of employing such mechanism is that the mean characterizes the most important property (the long-run average) of the inter-event time distribution, and the computation is straightforward. By taking the temporal embedding of the estimated average conditional time, the prompt efficiently encodes the time-varying knowledge up to the current event, which facilitates learning prediction tasks. We verify the effectiveness of temporal prompt in section 4.2.
From Prompt to Prompt Pool. Ideally, one would learn a model that able to share knowledge when tasks are similar while maintaining knowledge independently otherwise. Thus, instead of

---

applying single prompt, we introduce pool of temporal prompts to store encoded knowledge, which can be flexibly grouped as an input to the model. The pool is defined
P = [PI,... Pm]:
where M denotes the total number of prompts and P. E R is single temporal prompt. Following the notation in section 3.1, recall h; E RD denotes the hidden representation of the i-th event in the sequence which encodes the event history up to &; via the recursion by equation 2 and let (Pry. = =I...,N be a subset of N selected prompts, we then incorporate them into the event sequences in-context augmentation follows:
(5) Prompts are free to compose, so they can jointly encode knowledge for the model to process, which provides flexibility and generality in the sense that a more fine-grained knowledge sharing scheme can be achieved via prompt retrieval mechanism. Under this mechanism, combination of prompts is selected for each task similar inputs tend to share more common prompts, and vice versa.
PP .... .... PP [bg],
Retrieval Prompt Pool. The retrieval prompt pool shares some design principles with methods in other fields, such as RETRO (Borgeaud et al., 2022). Specifically, the prompt pool is augmented to be key-value store (K, V), defined as the set of learnable keys & E R2d and values temporal prompts P in equation 4:
(K, - (As,, P.)
The retrieval prompt pool may be flexible to edit and can be asynchronously updated during the training procedure. The input sequence itself can decide which prompts to choose through query-key matching, Let op R P x IP" be the cosine distance function to score the match between the query and prompt key. Given a query h, the encoded event vector, we search for the closest keys over IC via maximum inner product search (MIPS). The subset of top-N selected keys is denoted as:
N
Ktop-N = argmin
Σ co Kry))
fry
Importantly, the design of this strategy brings two benefits: (i) it decouples the query learning and prompt learning processes, which has been empirically shown to be critical (see section 4.2); (ii) the retrieval is performed in an instance- wise fashion, which makes the framework become task agnostic, meaning the method works without needing to store extra information about the task identity at test time. This corresponds to a realistic setting for modeling event streams in real applications.
3.3 Prompt-Event Interaction
The interaction operation controls the way we combine prompts with the encoded event states, which directly affects how the high-level instructions in prompts interact with low-level representations. Thus, we believe a well-designed prompting function is also vital for the overall CL performance. The interaction mechanism is also called prompting function in the NLP community. We apply the multi-hean self-attention mechanism (Vaswani et al., 2017) (MHSA) for modeling the interactions and adopt the mainstream realization of prompting function Prefix Tuning (Pre-T) (Li & Liang, 2021). Denote the input query, key, and values as ZQ, kK, Zv and the MHSA layer is constructed as:
(=Q &K, zV) [ ..... Imm] W
(8)
where z; ATTN(= W zR W.K zv W.Y. Wo, w W.K, W.V are projection matrix. In our context, let (Pr. 1 be the retrieved prompts from the pool, we set h; to be the query, split each prompt P., into pk PV E g4./2XD and prepend them keys and values, respectively, while keeping the query as-is:
h
Pre-T MHSA(h; [ [1.),, [P' [h:)),
where PK = [PK I....,
Pk I = [PX [PK]] Apparently, the key and value =K, ZV E Py
R N +1JxD and the output h
Pre-T E RD. Noted that there exist other prompting methods, such as Prompt Tuning (Pro-T), where all the prompts concurrently prepend the query, key and values:
h Pro-T = MHSA([PP) [|/,], [PX [[4:], [PV]]] [b,)),
(10)
As D1 + D2 D. use D and (Di D2) interchangeable throughout the paper.
5

---

where po = -Pk = pV- = [P.l)....IIPr. As a result, the query, key, value and output ZQ, 2K. 2v, h Pro-T E RRL,,N+1)xD Despite being less efficient in computation, we empirically demonstrate that Pre-T brings better performance. See Analysis III in section 4.2.
The output of the MHSA is then passed into an intensity layer (an MLP with softplus activation) to generate the intensity (e(4,), e E {I,..., E). For simplicity, we denote the prompt-event interaction and intensity layer together as the the decoder Seuse parameterized by &dec-
3.4 Model Optimization
The full picture of PromptTPPP at training and test time is described in Algorithm I and Algorithm 2 in Appendix C.I. At every training step, each event e, is recursively fed into the encoder foom.. after selecting N prompts following the aforementioned retrieval strategy, the intensity A(t,) is computed by the decoder Soace Overall, we seek to minimize the end-to-end loss function:
min
CPu fosne) + a: E Σ KLop-N
(( (e,0t,),
(11)
P Pe
where the first term the negative loglikelihood of the event sequence (Lnt equals to - Lu defined in equation 1) and the second term refers to a surrogate loss to pull selected keys closer to corresponding query in the retrieval process. a is a scalar to control the importance of the surrogate loss. Given the learned parameters, we may wish to make minimum Bayes risk prediction about the next event via the thinning algorithm (Mei & Eisner, 2017; Yang et al., 2022).
A Asynchronous Refresh of Prompt Pool. The prompts may lead to the variable contextual representa- tion of the event as the parameters of the based model are continually updated. To accelerate training. we propose to asynchronously update all embeddings in the prompt pool every C' training epochs.
4
Experiments
4.1 Experimental setup
Datasets and Evaluation Setup We conduct our real-world experiments three sequential user- behavior datasets. In each dataset, sequence is defined as the records pertaining to a single individual. The Taobao (Alibaba, 2018) dataset contains stmempt user click behaviors Taobao shopping pages with the category of the item involved noted as the event type. The Amazon (Ni, 2018) dataset contains timm-st records of user- user-geneet generated reviews of clothing, shoes, and jewelry with the category of the reviewed product defined as the event type. The StackOverflow (Leskovec & Krevl, 2014) dataset contains two years of user awards question-answeriin website: each user received a sequence of badges with the category of the badges defined as the event type. See Appendix D.1 for dataset details.
We partition Taobao and Amazon datasets into 10 consecutively rolling slides (namely 10 tasks) and partition the SteckOverflow dataset into 6 rolling slides (namely 6 tasks). For the Taobao dataset, each slide covers approximately day of time; for the Amazon dataset, each slide covers years of time; for the StackOverflow dataset, each slide covers approximately 5 months time. The subset in each task is split into training, validation, and test sets with a 70%. 10%. 20% ratio by chronological order. Each task has overlap in the test set. For a detailed discussion, demonstration of the evaluation process is provided in Figure 9 in Appendix D.3.
Metrics. Following the common next-event prediction task in TPPs (Du et al., 2016; Mei & Eisner, 2017), each model attempts to predict every held-out event (t,, kú) from its history H.. We evaluate the prediction & with the error rate and evaluate the prediction i, with the RMSE.
Base models. While our proposed methods are amenable to neural TPPs of arbitrary structure, we choose two strong neural TPPs as our base models: NHP (Mei & Eisner, 2017) and AttNHP (Yang et al., 2022), an attention based TPP whose performance is comparable to better than that of the NHP as well as other attention-based based models (Zuo et al., 2020; Zhang et al., 2020).
Competitors. With NHP and AuNHP as base models, we trained PromptNHP (PI-NHP) and PromptAnNNH (Pt-ANHP) the proposed prompt-augmenter setup and compared with 7 baselines.
PretrainedTPP. PretrainedNHP (Pre-NHP) and PretrainedAuNHP (Pre-ANHP) represent NHP and AttHHP learned at the first task (time step) and not trained any longer.

---

0.80
Aeeayy no Rane %
Ancogg Iime Ave
0.75
lime
2.00
155K 1D avertine
TaX overtime
PRAAWIP
.e- Ro-MiP
Coip
G_AhiP
PLANI
Taobao dataset
0.72
0.70
Arisse Tmc RRSE
Aecouge Error Rate
tima 1.5
Tasc 1D overtime
Text D overtime
-e- T6-Me
Codre
FLANI
(b) Amazon datasct
546
Avc Sg Eror haae
1.10
1.b
Liz
Duert
Phe
(c) StackOverflow dataset
Figure 3: Performance methods Taobao (up), Amazon (middle) and StackOverflow (down). each figure, the subfigures from left to right are the evolution of type error rate and the time RMSE of each task, the average error rate, and the average time RMSE of all the tasks.
RetrainedTPP. RetrainedNHP (Re-NHP) and RetrainedAnNHP (Re-ANHP) refer to TPPs re- trained at every sliding widow.
OnlineTPP. As there is prior work online neural TPPs, OnlineMHP (O-TPP) (Yang et al., 2017), trained in an online manner without any consideration for knowledge consolidation.
online Hawkes process
CLTPP. The concurrent work (Dubey et al., 2022), to the best of our knowledge, is the only neural TPP with CL abilities proposed so far. Based on their work 4, we implement CL-NHP (CL--NH)) and CLAuNHP (CL-ANHP) two variants of the hypernetwork-basee CLTPPs.
Implementation and Training Details. For fair comparison, they (except O-TPP which is classical TPP model) are of similar model size (see Table 2 in Appendix D.4). For Pt-NHP and Pt-ANHN, set M 10, N 4, T.p 10 for both datasets. During training, set C by default and explore the effect of asynchronous training in Analysis IV of section 4.2. More details of the implementation and training of all the methods are in Appendix D.5.
4.2 Results and Analysis
The main results shown in Figure 3. Pre-NHP and Pre-ANHP work the worst in most cases because of inadequate ability to handle the distribution shift the event sequence. Besides. O-TPP has similarly poor performance because of two reasons: first it is classical (non-neural) TPP with weaker representation power of modeling event sequence compared to its neural counterparts; second traditional online learning method, it easily loses memory of previously encountered data and suffers from catastrophic forgetting. Retraining at every task (Re-NHP and Re-ANHP) achieves
'Thyy have not published the code yet.

---

Tacbap
Amazonn
Tacbap
Amazzn
Recryp Broi 5200 4
73.
Average Time BAGE
72.8
PL418E and
955
Mastipp add Atopp
4.2
Fre.
(a) Effect of the temporal prompt: performances of Pt- std and Pt-ANHP on Taobao (the two on the left) and Amazon (the two the right) datasets. Figure S: Effect of temporal prompt and prompting function of PromptTPP.
(b) Effect prompting functions: performances of
N-P, Pro-T, and Pre-T on Taobao (the two on the left) and Amazon (the two on the right) datasets.
2-
SUMR2
$3224
B3723
11/.0
Average Error Rate %
0533
Prompt length
Avara RMSE
69:23
$0.00
6,5782
85732
6572L
35726
-03
Promot
69:03
$2.99
65327
3322
<ec
15
25
25
0322
Selection size
Selection size
Prompt Paol S2e
(a) Average performance of Pt-ANHP w.r.t. prompt length Lp and prompt selection size N, given prompt pool size 15.
(b) Average performance of Pt-ANHP w.r.t.
prompt pool size M, given Lp = 10, N Left: Average Type error rate (%); Right: Average time RMSE. 4.
Figure Effect hyperparameters of PromptTPP on Amazon datasct.
moderate results but it also causes calastrophic forgelting. Not surprisingly, CL-NHP and CL-ANHP perform better than retraining, by applying regularized hypernetwork to avoid forgetting. However, the hypernetwork relies on task descriptors built upon rich meta data, which limits its applicability and performance in our setup (and in real applications as well!). Lastly, our methods (both Pt-NHP and Pt-ANHP) work significantly better than all these baselines across the three datasets: they substantially beat the non-CL methods; they also consistently outperform CL-NHP and CL-ANHP by a relative 4% 6% margin on both metrics, thanks to our novel design of the CiRetroPromptPPol which successfully reduces catastrophic forgetting (see Analysis 0).
forggtttt
Analysis O: How models perform on previous tasks after learning new events? We aim to validate that the improvement in performances indeed due the alleviation in catastrophic forgetting instead of simply a better fit the current task. We use ANHP trained on task 9 and Pt-ANHP contin- uously trained on task 9, re-evaluate them previous tasks and see how metries changed. Specifically, on Figure 4,
Rata
RR-ANHP
Abs Chenge on Time AMSE
Re Re-ANHP CLANHP
CLANHP
Pt-ANHP
Pt-ANHP
0.00
Task time
Task time
Figure 4: The performance drop when re-evaluating the 0-8-th tasks using model trained on the 9-th task on Amazon datasct.
(i) Each number on the curves of Re- ANHP and CL-ANHP corresponds to the performance difference on the test
set of task i, < 9 using ANHP trained on task 9 vs ANHP trained on task i.
(ii) Each number the curves of Pt-ANHP corresponds to the performance difference the test set of task i, á < using Pt-ANHP trained until (including) task 9 vs Pt-ANHP trained until task
See from Figure 4, on both metrics, we see the drop in performance (i.e., error rate RMSE increases) of Pt-ANHP much less significant than ANHP, indicating Pt-ANHP stores well the knowledge of previous tasks, which largely alleviates catastrophic forgetting.
8

---

108
Lox Hlethood
Log HEMIEROOO
win Erer
PRANIIC-4) PLAMIPOOO)
PHANIDICHO
Br-A0RP
calcouor
Nam Epoch
Spocks
(a) Learning curves of Re-ANHP and Pt-ANHP with varying
(b) Effect of prompt related components of PPP on Amazon dataset.
asynchronous refresh parameter C' on a randomly selected task. Figure 7: Effect of asynchronous refresh and prompt related components of PromptTPP on dataset.
Analysis I: Does stronger base TPP model naively improve CL? Our method builds upon backbone TPP and understanding this question is important for fair comparisons and future research. From Figure Re-ANHP makes no consistent improvement against Re-NHP on average CL perfor- mance, which indicates a stronger TPP is not a solution for CL without being appropriately leveraged. Besides, for the CL-based methods, CL-ANHP tied with CL-NHP Taobao and makes limited advancement against CL-NHP on Amazon, while Pt-NHP and Pt-ANHP perform closely on both datasets. Therefore, we can conclude that, although AttNHP common non attention-based TPP, i.c., NHP, it is not necessarily translated to CL performance.
more robust base model than
Analysis II: Temporal prompt standard prompt. For a fair comparison, initialize pool of standard prompts without time-varyine varying parameters by fixing their temporal components Pa to be an all-ones matrix and incorporate it into the base model AttNHP. This method is named Pt-ANHP-std With other components fixed, we compare Pt-ANHP-std with Pt-ANHP to validate the effectiveness of the temporal prompt introduced in section 3.2.
Figure 5a shows that Pt-NHHP achieves better performance on both datasets: the introduction of temporal prompts slightly improves the RMSE metric and reduces the error rate with larger margin. We did the paired permutation test to verify the statistical significance of the improvements. See Appendix D.6 for details. Overall, on both datasets, we find that the performance improvements by using the temporal prompts are enormously significant on error rate (p-value < 0.05 ) and weakly significant RMSE (p- (p-value z 0.08).
Analysis III: How to better attach prompts? We explore how to attach prompts and enhance their
influences on overall performance. We compare three types of prompting: I Naive Prompting (N-P), where the retrieval and prompting with æ, in equation 7, prepend the selected prompts to æ; and pass the rest structure of TPP. Prompi Taning (Pro-T): which concurrently prepend the prompts to query, key, and value, introduced at the end of section 3.3. 3 Prefix-Tuning (Pre-T), proposed in the main body of section 3.3, which the prompting method used in PromptTPP.
performed after the event embedding layer: we replace h;
In Figure 5b, we observe that Pre-T leads to better performance on both datasets compared to those two variants. Despite its empirically better performance, the architecture of Pre-T is actually more scalable and efficient when attached to multiple layers since it results in unchanged output size: h Pre-T E TR D remains the same size as the input while h Pro-T e R( increases the size along the prompt length dimension.
Analysis IV: Efficiency of our method. We examine the efficiency of our method two steps:
Firstly, from Table 2, on both datasets, Pt-NHP Pt-ANHP leads to 12% / 8% total parameter increase to the base model, which in fact causes a marginal impact on training speed: Figure 7a shows that learning curves of Re-ANHP and Pt-ANHP(C converge at almost the same speed to achieve competitive log-likelihood, respectively.
Furthermore, to accelerate the training (especially when introducing large size prompts), we intro- duce the asynchronous refresh mechanism (see section 3.4) with prompts updated in a frequency C' > 1 (refresh the prompt pool less frequently). We observe in Figure 7a that Taobao training with = 2 has a comparable performance with = while Amazon training with C' improves the convergence notably. C = 4 leads to no advancement.

---

Overall, PromptTPP only adds small number of parameters so that it generally has the same convergence rate as the base model. The asynchronous prompt optimization scheme with C' improves the convergence more remarkably on the Amazon dataset. In addition, we indeed provide complexity analysis. See Appendix D.7.
Analysis V: Efffec of prompt related components of our method. Firstly we completely remove the CiRetroPromptPPol design (w/o CtRroPP in Figure 7b) and use single temporal prompt train tasks sequentially. The performance declines with a notable drop, indicating that single prompt suffers severe catastrophic forgetting between tasks, while our design of CiRetroPrommpIPol encodes task-invariant and task-spech task-specific knowledge well. Secondly, we remove the learnable key associated with prompts (w/o k-v in Figure 7b) and directly use the mean prompts as keys. This strategy causes moderate drop in performance. To conclude, learnable keys decouple the query and prompt learning processes and markedly contribute to the performance.
Analysis VI: Effect of hyperparameters of our method. We evaluate how the performance of Promp(TPP changes as we vary three key hyperparameters: (i) prompt length Ly, (ii) selection size N, and (iii) prompt pool size M. Theoretically, Lp determines the capacity of a single prompt (which jointly encodes certain knowledge), Lp N is the total size used prepend the event vector, while M sets the up limit of the capacity of learnable prompts.
Prompt length Lp and selection size N. From the results in Figure 6a, a too small Lp negatively affects results as single prompt has given an optimal I.p. an overly large makes the total prompts excessively oversized, leading to underfitting and negatively impacting the results. We conclude that a reasonably large 4, and N enable the model properly encode the shared knowledge between the tasks of event sequences and substantially improve the predictive performance.
limited ability to encode the knowledge. Besides,
Prompt pool size M. Figure 6b illustrates that M positively contributes the performance. This is because the larger pool size means the larger capacity of the prompts.
Conclusion
In summary, this paper has proposed a groundbreaking framework, known as Promp(TPP, for modeling streaming event sequences. By incorporating continuous-time retrieval prompt pool, the framework effectively facilitates the learning of event streams without requiring rehearsal or task identification. Our experiments have shown that Promp(TPP performs exceptionally well compared to other competitors, even under challenging and realistic conditions.
Limitations and Societal Impacts
Linitations. Our method neural networks, which typically data-hungry. Although it worked well in our experiments, it might still suffer compared to non-neural models if starved of data.
Societal Impacts. By describing the model and releasing code, hope to facilitate probabilistic modeling of continuous-time sequential data many domains. However, our model may be applied to unethical ends. For example, it may be used for unwanted tracking of individual behavior.
10

---

References
Alibaba. User behavior data from taobao for recommendation, 2018.
Bacry, E., Bompaire, M., Gaiffas, S., and Poulsen, tick: Python library for statistical learning.
with particular emphasis on time-dependent modeling. ArXiv e-prints, 2017.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J.-B., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring. R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., and Sifre, Improving language models by retrieving from trillions of tokens, 2022.
Boyd, A., Bamler, R., Mandt, S., and Smyth, P. User-dependent neural sequence models for continuous-timg event data. 2020.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learning: strong, simple baseline. In NeurlPS, 2020.
Cha, H., Lee, J., and Shin, J. Co2l: Contrastive continual learning. In ICCV, 2021.
Cho, H., Kim, J., Kim, J., Lee, S.-W., Lee, S.-g.. Yoo, K. M., and Kim, T. Prompt-augmented linear probing: Scaling beyond the limit of few-shot in-context learners, arXiv preprint 112.:22222.21773 2022.
Chu, Z. and Li, S. Continual treatment effect estimation: Challenges and opportunities. AAAI Bridge Program Continual Causality, pp. 11-17, 2023.
Chu, Z. Hao, H., Ouyang, X., Wang, S., Wang. Y., Shen, Y., Gu, J., Cui, Q. Li, L., Xue, S., et al. Leveraging large language models for pre-trained recommender systems. arXiv preprint arXiv:2308. 10837, 2023a.
Chu, Z., Li, R., Rathbun, S., and Li, S. Continual causal inference with incremental observational data, arXiv preprint arXiv:2303.017 2023b.
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, Recurrent marked temporal point processes: Embedding event history to vector. 2016.
Dubey, M., Srijith, P. K., and Desarkar, M. S. Hyperhawkes: Hypernetwork based neural temporal point process. CoRR, abs/2210.00213, 2022.
Hadsell, R., Rao, D., Rusu, A., and Pascanu, Embracing change: Continual learning in
deep neural networks. Trends in cognitive sciences, 24(12): 1040, December 2020. ISSN 1364-6613.
Hall, E. and Willett, R. M. Tracking dynamic point processes on networks. IEEE Transactions on Information Theory, 62(7):432-44366, 2016.
Hao, H., Chu, Z., Zhu, S., Jiang, G., Wang, Y., Jiang, C., Zhang, J., Jiang, W., Xue, S., and Zhou, J. Continual learning in predictive autoscaling. In CIKM, 2023.
Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971.
Hoi, S. C., Sahoo, D., Lu, J., and Zhao, P. Online learning: A comprehensive survey. Neurocomput., 2021. ISSN 0925-2312.
459(C):249-289
Ke, Z.,, Liu, B., and Huang, X. Continual learning of a mixed sequence of similar and dissimilar tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20. 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829566.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. 2015.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska--Brwwinkee A., Overcoming catastrophic forgetting neural networks. PNAS. 114(13)3321133266 2017.

---

Leskovec, J. and Krevi, A. SNAP Datasets: Stanford large network dataset collection, 2014.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.086/2 2021.
Lewis, P. A. W. Some results tests for poisson processes. Biometrika, 52(1/2), 1965.
Li, B., Dou, L., Hou, Y., Feng, Y., Mu, H., and Che, W. Mixpro: Simple yet effective data augmentation for prompt-based learning. arXiv preprint arXiv2230.009402, 2023.
Li, L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint TrXiv:2101.00/99 2021.
Liu, P., Yuan, W., Fu, J., Jiang. Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Comput. Surv., sep 2022a. ISSN 0360-0300. Just Accepted.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. Gpt understands, too. arXiv preprint ar arxiv:2103. 103385, 2021.
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang. Z., and Tang, J. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022b.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, 2018.
McCloskey, M. and Cohen, J. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989.
Mei, H. and Eisner, J. The neural Hawkes process: A neurally self- eeffmodulating multivariate point process. Advances in Neural Information Processing Systems, 2017.
Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. 2020.
Ni, J. Amazon review data, 2018.
Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems, 2019.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., Vetoo 7.,, Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. 2017.
Pham, Q., Liu, C., and Hoi, S. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 2021.
Qu, C., Tan, X., Xue, S., Shi, X., Zhang, J., and Mei, H. Bellman meets hawkes: Model-based reinforcement learning via temporal point processes. In AAAI 2023. AAAI Press, 2023.
Rusu, A. A.. Rabinowitz. N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.0466 04671, 2016.
Shehur, O., Biloš, M., and Günnemann, S. Intensity-free learning of temporal point processes. 2020.
Shehur, O., Turkmen, A. C., Januschowski, T., Gasthaus, Jnd and Günemann, Detecting anomalous event sequences with temporal point processes. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Shi, X., Xue, S., Wang, K., Zhou, F. Zhang, J. Y., Zhou, J., Tan, C., and Mei, H. Language models can improve event prediction by few-shot abductive reasoning. Advances Neural Information Processing Systems, 2023.
12

---

Shokri, R. and Shmatikov, V. Privacy- peeeee--reeevingggg deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS '15, pp. 1310-1321, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781 1881450338325
Snoek, J., Ovadia, Y., Fertig, E.,, Lakshminarayannn, B., Nowozin, S., Sculley, D., Dillon, J. V., Ren, J., and Nado, Z. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13969-13980, 2019.
Tam, W. L., Liu, X., Ji, K., Xue, L.,, Zhang, X., Dong, Y., Liu, J., Hu, M., and Tang, J. Parameter- efficient prompt tuning makes generalized and calibrated neural text retrievers. arXiv preprint arXiv:2207.07087 2022.
Varshney, V., Patidar, M., Kumar, R., Vig, L.., and Shroff, G. Prompt augmented generative replay via supervised contrastive learning for lifelong intent detection. In Findings of the Association for Computational Linguistics: NAACI. 2022, pp. 1113-1127, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is all you need. 2017.
Wang. J., Song, G., Wu, Y., and Wang, L. Streaming graph neural networks via continual learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, pp. 1515-1524, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145688 7781450368599
Wang Y., Chu, Z., Ouyang, X., Wang, S., Hao, H., Shen, Y., Gu. J., Xue, S., Zhang, Y., Cui, Q. et al. Enhancing recommender systems with large language model reasoning graphs arXiv preprint arXiv:2308. 10835, 2023.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal-fri continual learning. European Conference on Computer Vision, 2022a.
Wang. Z., Zhang, 7., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEF/CVÉ Conference on Computer Vision and Puttern Recognition, pp. 139-149, 20222b.......
Xiao, S., Yan, J., Yang, X., Zha, H., and Chu, S. Modeling the intensity function of point process via recurrent neural networks. In AAAI, 2017.
Xue, S., Shi, X., Hao, H., Ma, L., Zhang, J.. Wang, S., and Wang, S. A graph regularized point process model for event propagation sequence. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-7, 2021.
Xue, S., Shi, X., Zhang, Y. J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. In Advances in Neural Information Processing Systems, 2022.
Xue, S., Shi, X., Chu, Z., Wang, Y., Zhou, F., Hao, H., Jiang, C., Pan, C., Xu, Y., Zhang, J. Y., Wen, Q. Zhou, J., and Mei, H. Easytpp: Towards open benchmarking the temporal point processes. 2023.
Yang, C., Mei, H., and Eisner, J. Transformer embeddings of irregularly spaced events and their participants. 2022.
Yang, Y., Etesami, J., He, N., and Kiyavash, N. Online learning for multivariate hawkes processes. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In ICML, 2017.
Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, E. Self-attentive Hawkes process. 2020.
Zuo, S., Jiang, H., Li, 7.,, Zhao, T., and Zha, H. Transformer Hawkes process. In International Conference Machine Learning, pp. 11692-11702. PMLR, 2020.
13

---

Appendices
A Discussion on Classical Schemes for Modeling Event Sequence
As shown Figure I, a common approach is to use sliding windows to frame the data for model training and prediction. In this setup, we discuss three classical schemes.
Pretrained TPP. A straightforward solution is to pretrain TPP in the first train set and use it for all the following test periods. Such an approach faces the problem of distribution shift, i.e., the new data is systematically different from the data the model was trained Snoek al. (2019). Take the Taobao dataset (Alibaba, 2018) for example, which contains time-stamped user click behaviors 5. We split the dataset by timestamps into 10 periods sequentially and compute the 3S statistics (Shchur et al., 2021) as measure of the distribution of event sequences for each period. Figure 8a shows that the 3S statistics differentiate between periods while Figure 8b illustrates an increasing KL divergence of the 3S statistics between the first and later period, implying a pattern shift over time. As result, this approach may fail to adapt to new data and produce unsatisfactory predictions.
Retrained TPP. Another classical solution is to train new TPP on the data of sliding windows over again. The TPP can quickly adapt to new data but may suffer from catastrophic forgetting (McCloskey & Cohen, 1989): adaptation usually implies that the model loses memory of previously encountered data that may be relevant to future predictions. For example, Figure 8a shows a large overlap in distributions of different periods on the Taobao dataset, indicating the necessity of maintaining the knowledge of existing patterns to improve generalization (Snock et al., 2019; Wang et al., 2020).
Online TPP. A better solution is online approach: discretize the time axis into small intervals and then incrementally update the TPP at the end of each interval using an online algorithm. However, online models are generally more difficult to maintain and may also cause catastrophic forgetting forgeett (Hoi et al., 2021). Besides, to the best of our knowledge, apart from online classical TPPs ( (Yang al., 2017; Hall & Willett, 2016), the field of online neural TPPs is much less well-studied.
B Related Work Details
Here we draw connections and discuss differences between our method related works.
Temporal Point Process. A large variety of Neural TPPs have been proposed over recent decades, aimed at modeling event sequences with varying sorts of properties. Many of them are built on recurrent neural networks (Du et al., 2016; Mci & Eisner, 2017; Xiao et al., 2017; Omi et al., 2019; Shchur et al., 2020; Mei et al., 2020; Boyd et al., 2020). Models of this kind enjoy continuous state spaces and flexible transition functions, thus achieving superior performance on many real-world datasets, compared to classical models such as the Hawkes process (Hawkes, 1971). To properly capture the long-range long-range dependency in the sequence, the attention mechanism (Vaswani et al., 2017) has been adapted to TPPs (Zuo et al., 2020; Zhang et al., 2020; Xue et al., 2021; Qu et al., 2023; ?: Wang et al., 2023; Shi al., 2023) enhance the predictive performance. However, learning the event sequence under the stream setting is largely unexplored. To the best of our knowledge, there exist two prior works (Yang et al., 2017; Hall & Willett, 2016) that propose online learning algorithms for classical TPPs while that for neural TPP have rarely been studied. We show our method works better than classical online TPPs in practice (see section 4.2).
Continual Learning. There is also rich existing literature CL: the models can be categorized into regularization based methods (Kirkpatrick et al., 2017; Zenke et al., 2017), which regularize important parameters for learned tasks, architecture-based methods (Rusu al., 2016; Mallya & Lazebnik, 2018) which assign isolated parameters for each task and rehearsal-base methods (Cha al., 2021; Buzzega al., 2020) which save data from learned tasks in rehearsal buffer train with
'Please see Appendix D.I for the details of the Taobao dataset and Appendix D.2 for the explanations on 3S statistics and the procedure of the experiment on distribution shift.
14

---

0.08
0.30
Period 0
KLD Statisties
0.06
Period 5
Probability
Period
0.25
0.04
0.20
0.02
0.00
0.15
3
5
7
9
1st
3rd
5th
7th
9th
35 statistic
Between 0-th and the following period
(a) Distributions of 3S statistics for event sequence data in three randomly selected periods.
(b) KL divergence of distributions of 3S statistics between the O-th and the following period.
Figure Analysis distribution shift on Taobao dataset.
the current task. In retrospect, we realize a concurrent work (Dubey al., 2022) which also augments TPP with CL abilities. Important distinctions of their work from ours include: 1. setup: they use standard TPP train/valid setting while we formalize more realistic streaming setting to train/validate the models; 2. methodology: they model the event streams with hypernettwrkk-based regularizer while we use a trainable prompt pool with more flexibility and generality in CL. 3. task agnostic: they rely task descriptor built from meta attributes of events while we do not method is task agnostic. As their source code is not available yet, we independently implement it and our method still outperforms it (see section 4.2).
Prompt Learning. Prompt-based learning (or prompting), as an emerging transfer learning tech- nique in NLP, applies fixed function to condition the model so that the language model gets additional instructions to perform the downstream task. Continuous prompts have also been proposed (Lester et al., 2021; Li & Liang, 2021) to reduce prompt engineering, which directly appends series of learnable embeddings as prompts into the input sequence, achieving outstanding performance on transfer learning. Wang al. (2022a.b) connect prompting and CL., which attaches prompts the pretrained backbone to learn task-invariant and task-specific instructions. Note that it is non-trivial apply prompting neural TPPs (See Analysis II and III in section 4.2), and proposed novel framework reveals its values to event sequence modeling.
C
Method Details
C.1
PromptTPP at Training and Test Time
The training and test time Algorithms for PromptTPP are illustrated Algorithm and Algorithm 2, respectively.
For simplicity of notations, in test time, we show how to sample the next event given one historical event sequence via the thinning algorithm (Mei & Eisner, 2017), which can be easily extended to batch-wise inference (see the implementation in EasyTPP (Xue et al., 2023)).
D Experimental Details
D.1 Dataset Details
We evaluate our methods on three industrial user behavior datasets. We provide details on the preparation and utilization of each below. For both datasets, users associated with anonymous aliases to remove personally identifiable information (PII).
Taobao (Alibaba, 2018). This dataset contains time-stamped user click behaviors on Taobao shopping pages from November 25 to December 03, 2017. Each user has sequence of item click events where each event contains the timestamp and the category of the item. Following the previous work (Xue al., 2022), the categories of all items are first ranked by frequencies, and the top 19 are kept while the rest merged into one category, with each category corresponding to event type. We work
15

---

Algorithm PrompiTPP at training time of the T-th task.
Input: Train set {Strain ). CiRetroPrompttPol (K, {(k, P.) (inherited from the previous
task). score function (,, loss weight and asynchronous refresh frequency C.
Output: Trained base model with a encoder foanc and decoder foanc trained CtRetroPromptPPol
(K, V) = {K,, P.)
1: procedure TRAIN((
(k; P.)
2: for epoch_io __ddidddh in votal_epochs.
3: Draw a mini batch
4: For illustration purposes, use batch size= I here. The computation here can be easily extended to
batch size 2 1.
5:
for S(0,7] in B:
6: Update the loss +- CALCLOSS($00.7) foenet f a)
7:
Update former fozee by backpropagation. if epoch_id %C == 0 Update (k; P.) AM by backpropagation.
8:
9: procedure CALCLOSS($1o.r) forne S faaee a)
10:
C
11:
Recursively compute the loss.
12:
for eot in 8(0,T)
13:
Compute the likelihood loss; the technical details can be found in Mei Eisner (2017) and Yang et al. (2022).
14:
Levent + Take a sum of log intensity at the event time by calling CALCINTEN- STTY() e Sohne Souceh (ki, P. P.)
15:
Enon_event &- Integrate log CALCINTENSITY( e@t) over inter-event time interval. Cull t- Enon_event Levent Compute the matching loss.
17:
Ematching < EKoop x P(forn (e@t),
19:
L+ Lull + aLmmachhng
20:
return L.
21:
procedure CALCINTENSITY(HIL.)) e@t, forne foace (ks, P.)
22:
$[0,4] Append e@t to history S(u,()-
23:
Encode $[o.4] by fōene to generate the hidden state ht.
24:
N Matching the index rjj=1 based on equation 7.
25:
Select Top-N prompts (Pr.)
26:
Prepend (P,, J.N to h, and pass to the decode Seuc to generate the intensity de(t),
(1...,, F).
27:
return (t) (1,..., E}
on subset of 4800 most active users with an average sequence length of 150 and then end up with K 20 event types.
Amazon (Ni, 2018). This dataset includes time-stamped user product review behavior from January 2008 October 2018. Each user has sequence of produce review events where each event containing the timestamp and category of the reviewed product, with each category corresponding to an event type. We work on subset of 5200 most active users with average sequence length of 70 event tokens and then end with K = 16 event types.
StackOverflow (Leskovec & Krevl, 2014). This dataset has two years of user awards question- answering website: each user received a sequence of badges and there are = 22 different kinds of badges in total. We work on subset of most active users with average sequence length of xx event tokens.
For the Taobao dataset, each task includes approximately day of time:for the Amazon dataset, each task includes approximately 2 years of time; for the StackOverflow dataset, each task includes approximately 5 months of time. Table shows statistics about each dataset mentioned above.
16

---

Algorithm/ 2 PromprTPP at test time of the T-th task.
Input: An event sequence $[o.T] {ez (e, Trained base model with encoder Sō.... and decoder feu trained CiRetroPromptPPol (K, V) {(k,, Pi) M and the score function yp.
Output: Sampled next event 1f1+1
1: procedure DRAWNEXTEVENTICS föaet
fôree)
2: to <- -T; H.H c $10,T1
3: Compute sampling intensity
4: (Ae(t) H01 < SAMPLEINTENSITY Joono Soan {(k; P.)) for all t, E
(l0, oc)
5:
Compute the upper bound X"
6: & Technical details can be found in Mei & Fisner (2017)
7: find upper bound X 2 Eest do(tz H) for all tj E 00)
8: repeat
9: draw A ~ Exp(A"); to +=
b time of next propused event [
(n,
11: until uλ" S Ee de H)
12:
draw 6141 (1,.... E} where probability of is Ae 7
return ē₁- 11
14:
procedure SAMPLEINTENSITYYUUU.. foener Soace ((k,, Pi)
15:
Assume the last event in Stu,r] is e Generate list of sample times {1,))=1,4 2 T.
17:
at sample times (S eat, forncs Soan : {ki, P.)
Compute the intensity
dely
- CALCINTEN-
18:
return (tx H}}
DATASET
K
# EvT TOKENS
AVG # EvT TOKENS PER SEQ
AvG # EVT TOKENS
AVG SEQ
TOTAL
PER TASK
PER TASK
TAOBAO
20
720,000
150
80,000
32
AMAZON
16
360,000
70
42,000
10
STACKOVERFLOW
22
240,000
43,000
Table 1: Statistics of each dataset.
D.2 3S statistics and Experiment on Distribution Shift
statistics proposed by Shehur et al. (2021) depict the distribution of an event sequence continuous time. Compared to the classical KS statistics (Lewis, 1965), it uniformly captures multiple properties of event sequence, such total event count and distribution of inter-event times. Empirically, replacing the KS score with the statistic consistently leads better separation between distributions generated by different TPPs. Please refer to the original paper (Shchur et al., 2021) for a detailed discussion.
We use the 3S Ssum-of-squrred-specce
For exploring the distribution shift in the Taobao dataset, we randomly sampled a thousand sequences of events and split them into 10 subsets by timestamps: each subset has approximately equal time horizon and is notated sequentially from 0-th to the 9-th subset. Then we follow the procedure in (Shchur al., 2021) to compute the 3S statistics for each subset and illustrate the results in Figure Sa.
D.3 Evaluation Setup
To set up the training and evaluation process, we partition Taobao and Amazon datasets into 10 consecutively rolling slides (namely tasks) and partition the StackOverflow dataset into 6 rolling slides (namely 6 tasks). For the Taobao dataset, each slide covers approximately day of time; for Amazon dataset, each slide covers years of time; for the StackOverflow dataset, each slide covers approximately 5 months time. The subset in each task is split into training, validation, and test sets with 70%, 10%, 20% ratio by chronological order. Fach task has no overlap in the test set. In such setting, the total test set covers approximately 70% of data.
17

---

MODEL
PARAMETERS
TAOBAOAAZZON
SO
PRE-NHP
23.3K
23.4K
23.3K
PRE-ANHIP
25.4K
25.6K
25.4K
RE-NHP
23.3K
23.4K
23.3K
RE-ANHP
25.4K
25.6K
25.4K
O-TPP
<IK
<IK
<IK
CL-NHP
27.6K
27.7K
27.6K
CL-ANHP
29.5K
29.6K
29.5K
PT-NHP
26.2K
26.3K
26.2K
PT-ANHP
27.8K
27.0K
27.8K
Table 2: Total number parameters models trained on the three datasets.
We train and evaluate each task sequentially. Our evaluation setup is close to that used in real applications: train the model using fixed length of historical data and evaluate the model using the following window of the data.
Union of test
Time
Task
Train
Test
Task #2
Train
Val
Test
Task #3
Train
Val
Test
Task #4
Train
Val
Test
Figure 9: demonstration of the sliding window validation method.
D.4 Implementation Details
All models implemented using the PyTorch framework (Paszke al., 2017).
For the implementation of NHP, AttHH, and thinning algorithm, we used the code from the public
GitHub repository at https//gihhub..co/yaagaaaaa 133anhp (Yang al., 2022) with MIT License.
For O-TPP, as the authors Yang et al. (2017) have not published the code, we implement it using the tick (Bacry et al., 2017) library.
For CL-NHP and CL-ANHP, without the public code, by following the main idea of the authors Dubey et al. (2022), we develop the hypernetwor with an MLP layer and add apply regularizer to the hypernetwork parameters while learning new event sequence, which prevents adaptation of the hypernetworks parameters completely to the new event sequence. Note that, the base models used are NHP and ANHP, respectively, instead of FullyRNN (Omi al., 2019) applied in the original paper.
We implemented our methods with PyTorch (Paszke et al., 2017) and published the code at at
D.5 Training and Testing Details
D.5.1
Training and Hyperparameters Selection
Training base TPP model. To train the parameters for a given neural TPP, we performed early stopping based on log-likeli likelihood on the held-out dev set.
For NHP, the main hyperparameters to tune are the hidden dimension D of the neural network. In practice, the optimal D for model was usually 32, 64, 128, and we search for the optimal value among them for different datascts.
18

---

MODEL
DESCRIPTION
VALUE USED
TAOBAO
AMAZON
SO
PRE-NHP
RNN HIDDEN
76
76
76
SIZE
TEMPORAL
64
32
64
EMBEDDING
PRE-ANHP
HIDDEN SIZE
64
64
64
LAYER NUMBER
3
RE-NHP
RNN HIDDEN
76
76
76
SIZE
TEMPORAL
64
32
64
EMBEDDING
RE-ANHP
HIDDEN SIZE
64
64
64
LAYER NUMBER
3
O-TPP
KERNEL. SIZE
20 x 20
16 16
22
22
CL-NHP
RNN HIDDEN
64
64
64
SIZE
TEMPORAL
64
32
64
EMBEDDING
CL-ANHP
HIDDEN SIZE
64
64
64
LAYER NUMBER
3
3
RNN HIDDEN
64
64
64
SIZE
M(RETRIEVAL
10
10
10
PROMPT POOL
SIZE)
PT-NHP
N(Top-N
4
SELECTED)
(PROMPT
10
10
10
LENGTH)
CIASYNCHRONOUS
2
REFRESH
FREQUENCY)
TEMPORAL
64
32
64
EMBEDDING
HIDDEN SIZE
64
64
LAYER NUMBER
2
2
PT-ANHP
M(RETRIEVAL
10
10
10
PROMPT POOL
SIZE)
N(Tor-N
4
4
SELECTED)
(PROMPT
10
10
10
LENGTH)
C(ASYNCHRONOUS
2
REFRESH
FREQUENCY)
Table 3: Descriptions and values of hyperparameters used for models trained on the three datasets.
For AttHH,, in spite of D, another important hyperparameter tune is the number of layers L of the attention structure. In practice, the optimal L was usually 2,2,3, 4. In the experiment, we choose the hyperparameter based the held-out dev while keeping AttNHP have a similar size to that of NHP.
Training PromptTPP. We find equation is not sensitive and works well in large range, so we set a 0.1 consistently for both datasets. For the prompts, we set M 10, N = 4, L.p 10 for both datasets. For the asynchronous training parameter C, we choose C for Taobao and Amazon datasets by default.
19

---

Taobao
Amazon
p-value
e-value
0.04
0.04
0.02
0.02
0.00
0.00
Error Rate
RMSE
Error Rate
RMSE
Figure 10: Statistical significance the improvements from a temporal prompt on Taobao and Amazon datasets.
Chosen Optimizer and Hyperparameters. All models are optimized using Adam (Kingma & Ba, 2015). Table contains descriptions that list all of the hyperparameters set throughout our experiments.
Testing. As described in Mei & Eisner (2017), we minimized the Bayes risk via the thinning algorithm determine decisions for what a predicted next event time f and type 2,+1 would be after conditioning on portion of sequence 8|0,4,] [er @l,,.... e, W.... All experimental results are averaged over runs, and the corresponding standard deviation is reported well. In the experiment, we report the metrics on the test set for each task as well as the average metrics over all the tasks.
Integral Approximations. During training and testing, there are number of integrals (e.g.. log- likelihood in equation 11) that need be computed, which are not feasible closed form. Thus, we must approximate them. All integrals and expectations are approximated via Monte Carlo (MC) estimates with certain amounts of samples used. L during training and testing. When evaluating the integrals used for next event predictions in thinning algorithm, we used 100 samples where the sample points were shared across integrals for a single set of predictions in order to save computation. The exact approximation procedure for the log-likelihood can be found in Mei Eisner (2017).
equation uses 100 MC samples
Environment. All the experiments were conducted on server with 256G RAM, 64 logical cores CPU (Intel(R) Xcon(R) Platinum 8163 CPU 2.50GHz) and one NVIDIA Tesla P100 GPU for acceleration.
D.6 Analysis II Details: Statistical Significance
We performed the paired permutation test validate the significance of our proposed temporal prompt. Particularly, for each model variant (PtNHH and Pt-ANHP), split the test data into ten folds and collected the paired test results with temporal prompt and with the standard prompt, respectively, for each fold. 'Ten we performed the test and computed the p-value following the recipe https://axon.cs ww..oom.css.uuu... wwwp.eye/aan/////////// aeadinnnnnnaaaiiimemmee/eaaaaaon pesp
The results are in Figure 10. It turns out that, on both datasets, the performance differences are strongly significant for the error rate metric (p-value < 0.05 ) and weakly significant for the RMSE metric (p-value z 0.08) ).
D.7
More Result: Computational Complexity of PromptTPP
The computational complexity comes from two parts: the TPP model and prompt pool.
Take ANHP the base model. Assume the input sequence length K event embedding size de. Recall the prompt length L,, selection size N and prompt pool size M, key size D.
ANHP is attention- based, so its original complexity O(Kdde) By considering the attached prompt, the prompt-augmented ANHP's complexity becomes O((K NL,)) (d))) Retrieval's complexity is O OMMD)) So the total complexity is O((K N., d2 MD2).
20

---

J-vvert
CLatee
the ter weette Cutto
For
(a) Evaluation results Task 9.
(b) Evaluation results on Task 10.
Figure 11: OTD distance comparison of generated sequence of all models on Amazon dataset.
D.8
More Result: Generation Ability Comparison
We investigated the generative ability of the models empirically on Amazon dataset. Given a trained model, we fixed the prefix event sequence and performed the 1-event sampling and 3-event sampling (autoregressively) on the test set of task 9 and task 10. We followed (Xue et al., 2022) to compute the average optimal transport distance (OTD) to measure the distance between the generated sequence and the ground truth. Seen from Figure 11, our proposed models Pt-NHP and Pt-AttNHP achieves the best results. This is consistent with the findings in Main Results in the paper.
21


Reviews:
[{"review_id": "4yCeGlyx2X", "reviewer": "Reviewer_n7nj", "summary": "This paper aims to enhance the Neural Temporal Point Processes (TPPs) used in modeling continuous-time event sequences. Particularly in the real-world, the distribution of event data shifts over time, which existing TPPs fail to reflect. To address this, the authors propose PromptTPP, a continual learning framework that utilizes prompts. The effectiveness of this proposed method has been demonstrated through experiments with Taobao and Amazon datasets.", "questions": "1. In line 112, it appears that the latter part of the sentence is missing. (`Compared with ordinary fine-tuning, literature shows […] In our`)\n2. According to Figure 3, while the type error rate shows improvement (a decrease) for later tasks in the sequence, the time RMSE demonstrates the opposite trend, worsening (increasing) over time. What is the reason for this opposing trend between the two performance metrics?\n3. I am interested in understanding the specific degree of distribution variances across tasks. The authors illustrate the KL divergence of each subsequent period relative to the 0-th period in Supplementary Figure 7. However, a more comprehensive comparison across all task pairs, possibly utilizing a confusion matrix, might yield more insightful and useful information.", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is reader-friendly, with comprehensive equations and illustrative figures.\n2. The proposed method is well-structured and intuitively compelling.\n3. The experimental results convincingly demonstrate the effectiveness of the proposed approach.\n4. The availability of the code simplifies the process of reproduction, making it easily accessible for further study.", "weaknesses": "While I'm not deeply familiar with this field, there don't seem to be significant weaknesses. However, I do have a concern related to the experiments conducted. The authors refer to CLTPP (Dubey et al., 2022) as the only neural TPP with continual learning (CL) capabilities. However, they did not conduct experiments on the Yelp and Meme datasets used in this paper (Dubey et al., 2022). Is there any reason for it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors refer to CLTPP (Dubey et al., 2022) as the only neural TPP with continual learning (CL) capabilities. However, they did not conduct experiments on the Yelp and Meme datasets used in this paper (Dubey et al., 2022). Is there any reason for it?\n\nThe problem formulation and dataset processing in our paper are different from Dubey's.\n1. In our work, we focus on the streaming event setting: each sequence corresponds to a unique user's behavior over time. We truncate the dataset by time to make the tasks: each user's sequence and we study how a model can continuously learn to predict the behavior of the users.\n2. In Dubey's work, they focus on zero-shot prediction in the CL setting, e,g., predict crimes for new cities while retaining and transferring information from old cities. So they do not necessarily require each sequence to be that of the same user. \n\nWe don't use Yelp and Meme because they have too short average sequence length (both < 20)  user-wise. So we process two datasets by our own from the source.\n\nLastly, to ensure the applicability of our proposed method, we perform the experiments on a newly processed dataset. Due to page limit, we put the results in [New results - new dataset StackOverflow] in Author Rebuttal above globally.  The findings are consistent with those on Taobao and Amazon datasets.\n\n\n> In line 112, it appears that the latter part of the sentence is missing.\n\nWe want to say \"Compared with ordinary fine-tuning, literature shows that prompt-augmented learning results in a sequence-based model having higher capacity to learn features [1,2,3,4].\"  \n\nThanks for point this and we will fix it in the next version.\n\n[1] The Power of Scale for Parameter-Efficient Prompt Tuning，EMNLP 2021.\n\n[2] Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, ACM Computing Surveys 2022.\n\n[3] Learning to prompt for continual learning, CVPR 2022.\n\n[4] Prefix-tuning: Optimizing continuous prompts for generation, ACL 2021.\n\n>  According to Figure 3, while the type error rate shows improvement (a decrease) for later tasks in the sequence, the time RMSE demonstrates the opposite trend, worsening (increasing) over time. What is the reason for this opposing trend?\n\nIn TPP, the event type follows categorical distribution while the time follows a continuous distribution. The distribution shift also evolves with time as the tasks are defined sequentially by time. We believe the event time prediction is more susceptible to the effects of distribution shift over time and the error may accumulate. As a result, the time RMSE worsens in later tasks. \n\nHowever, for time RMSE, at each task our method Pt-ANHP performs better than baselines with a notable margin, and this margin becomes larger in later tasks, showing the effectiveness of our method. \n\nTo further validate our method on later tasks, we investigate the generative ability on the Taobao dataset. Given a trained model, we fixed the prefix event sequence and performed the 1-event sampling and 3-event sampling on the test set of task 8 and task 9. We followed [5] to compute the average OTD as the metrics to measure the distance between the generated sequence and the ground truth.  Taking ANHP as the base model, our proposed model Pt-ANHP achieves the best results. \n\nOTD of 1-event sampling (the lower the better)\n| Model | Task 8   |  Task 9  |\n|------|---------|----------|\n| Pre-ANHP     |   1.345      |  1.497        |\n| Re-ANHP     |   1.205      |    1.461     |\n| CL-ANHP     |   1.170      |    1.391      |\n| Pt-ANHP     |   1.142      |    1.357      |\n\n\nOTD of 3-event sampling (the lower the better)\n| Model | Task 8   |  Task 9  |\n|------|---------|----------|\n| Pre-ANHP     |   4.743     |  5.796        |\n| Re-ANHP     |   4.376      |    5.590     |\n| CL-ANHP     |   4.186      |    5.470      |\n| Pt-ANHP     |   4.090      |    5.362      |\n\n\n[5] HYPRO: a hybridly normalized probabilistic model for long-horizon prediction of event sequence, NeurIPS 2022.\n\n>  I am interested in understanding the specific degree of distribution variances across tasks. The authors illustrate the KL divergence of each subsequent period relative to the 0-th period in Supplementary Figure 7. However, a more comprehensive comparison across all task pairs, possibly utilizing a confusion matrix, might yield more insightful and useful information.\n\nWe have computed the confusion matrix of Taobao below. Not surprising, KL increases with the time distance between the tasks, e.g., KL (1st and 9th) = 0.2208 > KL(1st and 4th)=0.1156. This further indicates the distribution shift over the time.\n\n\n|\t  |Task 0 |\tTask 1\t|Task 2\t|Task 3\t|Task 4\t|Task 5\t|Task 6|\tTask 7\t|Task 8|\tTask 9|\n|-------|--------|-----------|------------|-----------|------------|---------|---------|--------------|-------|----------|\n|Task 0\t|0\t|  0.1543\t|0.1906\t| 0.1873 |\t 0.2228\t|0 .2406\t|0.2746\t|0.2571\t|0.2855|\t0.2961|\n|Task 1\t|\t| 0\t|0.0599\t|0.1094\t|0.1156\t|0.1300\t|0.1466|\t0.1344|\t0.1797|\t0.2208|\n|Task 2\t|\t|\t| 0 |\t0.0202\t|0.0619\t|0.0728\t|0.0901\t|0.1198\t|0.1311\t|0.1722|\n|Task 3\t|\t|\t|\t| 0\t| 0.0379\t|0.0581\t|0.0817|\t0.0633|\t0.1217\t|0.1617|\n|Task 4\t|   |\t|\t|\t|\t0\t| 0.0314\t|0.4166|\t0.7303|\t0.0993\t|0.1138|\n|Task 5\t|\t|\t|\t|\t|\t|0\t|0.0291\t|0.0881|\t0.6222\t|0.1114|\n|Task 6\t|\t|\t|\t|\t|\t|\t|0\t|0.0363\t|0.4210|\t0.8474|\n|Task 7\t|\t|\t|\t|\t|\t|\t|\t|0\t|0.0391\t|0.4726|\n|Task 8\t|\t|\t|\t|\t|\t|\t|\t|\t|0\t| 0.0163|\n|Task 9\t|\t|\t|\t|\t|\t|\t|\t|\t|\t |      0|"}, {"review_id": "vifKHuNgsq", "reviewer": "Reviewer_26jC", "summary": "The paper proposes a novel prompt-augmented continual learning (CL) framework for modeling streaming event sequences, called PromptTPP. The framework incorporates a continuous-time retrieval prompt pool and a retrieval mechanism to dynamically lookup a subset of task-relevant prompts based on the instance-wise input of event sequences. The main contributions of the paper are (i) the introduction of PromptTPP, a novel prompt-augmented CL framework for neural TPPs, and (ii) the applicability and generality of the approach.", "questions": "1. Can the authors provide more details on the retrieval mechanism used in the proposed framework? Specifically, how does the retrieval mechanism select a subset of task-relevant prompts based on the instance-wise input of event sequences?\n2.  How can the proposed approach be used for unwanted tracking of individual behaviour, and what steps can be taken to mitigate this risk?\n3.  It would greatly enhance the understanding of the proposed framework if the authors could provide more in-depth information regarding the generative loss and its optimization. Specifically, how are the generative loss and retrieval prompt pool jointly optimized?", "limitations": "See the weaknesses above", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The use of a continuous-time retrieval prompt pool and a retrieval mechanism to dynamically look up a subset of task-relevant prompts based on the instance-wise input of event sequences is a novel approach to CL.\n2. The paper is well-organized and easy to follow. The authors provide clear explanations of the proposed framework and the experiments. The figures and tables are also well-designed and easy to understand.\n3. The PromptTPP addresses modelling streaming event sequences in a continual learning setting. ", "weaknesses": "1. The paper does not provide a detailed analysis of the computational complexity of the proposed framework, which is an important consideration when working with large datasets and complex models. \n2. The paper does not provide a detailed comparison with other state-of-the-art methods for modeling streaming event sequences in a continual learning setting, which would help researchers understand the strengths and weaknesses of PromptTPP in comparison to other approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper does not provide a detailed analysis of the computational complexity of the proposed framework.\n\nThe computational complexity comes from two parts: the TPP model and prompt pool. \n\nTake ANHP as the base model. Assume the input sequence length is $K$, event embedding size $d_e$. Recall the prompt length $L_p$, selection size $N$ and prompt pool size $M$, key size $D$. \n- ANHP is attention-based, so its original complexity is $O(K^2 d_e)$. By considering the attached prompt, the prompt-augmented ANHP's complexity becomes $O((K+NL_p)^2 d_e)$.\n- Retrieval's complexity is $O(M D*D)$.\n\nSo the total complexity is $O((K+NL_p)^2 d_e + M D^2)$. \n\nIn the paper, we indeed provide a detailed analysis on the efficiency our method by looking at the convergency of learning curves (see Analysis IV in section 4.2), which is more common in TPP works [1,2,3]. The conclusion is that PromptTPP only adds a small number of parameters so that it generally has the same convergence rate as the base model. The asynchronous prompt optimization scheme improves the convergence (more remarkably on the Amazon dataset).\n\n[1] Neural Hawkes Process, NeurIPS 2017.\n\n[2] Transformer Hawkes Process, ICML 2020.\n\n[3] Noise-Contrastive Estimation for Multivariate Point Processes, NeurIPS 2020.\n\n> The paper does not provide a detailed comparison with other state-of-the-art methods for modeling streaming event sequences in a continual learning setting, which would help researchers understand the strengths and weaknesses of PromptTPP in comparison to other approaches.\n\nFirstly, in our experiments, we use two SOTA TPP as base models, one RNN-based and one attention-based, so that researchers could see how different base model affects the CL performance and choose appropriate base TPP for their own CL tasks.\n\nSecondly, so far the Dubey's work [4] is the only available neural TPP in CL setting. We introduce it in sec 4.1 and implement it as a baseline (notated as CL-NHP, CL-ANHP). In main results (section 4.2), we analyze the comparison between them and our method Pt-NHP/Pt-ANHP. Our methods outperform by a notable margin due to the novel design of CtRetroPromptPool.\n\nCompared to Dubey's work,\n- the strength of our method is, in the streaming setting, we have stronger ability to reduce catastrophic forgetting while adapting to new knowledge; our method has stronger applicability (the prompt pool and the event retrieval mechanism removes the necessity of a rehearsal buffer and knowing the task identity) and generality (our approach is general-purpose that it can be integrated with any TPPs);\n- the weakness is our method does not address zero shot event problem in CL setting, which his work focuses on (e.g., predict crimes for new cities while retaining and transferring information from old cities).  \n\n[4] HyperHawkes: Hypernetwork based Neural Temporal Point Process, (updated version is accepted by KDD 2023).\n\n> Can the authors provide more details on the retrieval mechanism used in the proposed framework? Specifically, how does the retrieval mechanism select a subset of task-relevant prompts based on the instance-wise input of event sequences?\n\nHow to select a subset of prompts based on the instance-wise input of event sequences? The input sequence decides which prompts to choose through query-key matching. Given a query $h_i$, the encoded event vector, we search for the closest keys over $K$ via MIPS.\n\nHow to guarantee that a similar key value represents a similar event feature ？ See the loss function Equation 11, during the optimization, the second term pulls selected keys closer to corresponding query event in the retrieval process, so similar keys will represent similar events.\n\nHow to validate the effectiveness of the key-value mechanism? See Analysis V in section 4.2. We remove the learnable key associated with prompts (w/o k-v in Figure 6b) and directly use the mean of prompts as keys. This strategy causes a moderate drop in performance.\n\n> How can the proposed approach be used for unwanted tracking of individual behavior, and what steps can be taken to mitigate this risk?\n\nExamples of event streams with potential social impact include a person’s detailed food/medical event log, their social media interactions, etc. Because our model is able to continually making more accurate predictions for user behavior, it could potentially be used for surveillance. This is a common societal impacts discussed in TPP [3,5]\n\nTo mitigate this risk, rehearsal-free methods are preferred to use, as no meta data (user identification) of events needs to be stored in the buffer. This line of methods help prevent the leakage of private data. Other measures may include stricter privacy protection, which is out of the scope of our paper.\n\n[5] HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequences, NeurIPS 2022.\n\n> It would greatly enhance the understanding of the proposed framework if the authors could provide more in-depth information regarding the generative loss and its optimization. Specifically, how are the generative loss and retrieval prompt pool jointly optimized?\n\nIn equation 11, the loss functions has two terms: the first term is the typical generative loss of the event sequence and the second term refers to a surrogate loss. During optimization, the generative term fits the joint distribution of events while the surrogate term pulls selected keys closer to corresponding query event in the retrieval process, so similar keys will represent similar events. Besides, to accelerate training, we adopt the techniques in [6] to asynchronously update all embeddings in the prompt pool every $C$ training epochs. So in every epoch we update the parameters of TPP encoder/decoder by backpropagation; when epoch_id %C ==0, we update the prompt pool by backpropagation. See Algorithm 1 in appendix for full details.\n\n[6] Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning, NeurIPS 2022."}, {"review_id": "LJvVevhk3R", "reviewer": "Reviewer_nDRf", "summary": "This paper addresses the problem of Temporal Point Processes (TPP), which involves modeling continuous-time events. Specifically, the authors focus on developing a model for streaming data, where the model needs to be improved in a Continual Learning (CL) setting. To tackle this challenge, the authors propose a TPP model that combines the concepts of Prompt (which is modified for continuous time) and CL. The proposed method has demonstrated superior performance compared to alternative approaches on two datasets.", "questions": "1. Can the authors provide justification on using only two datasets in the experiment section?\n2. In the abstract, the authors talk about privacy constraints associated with the TPP problem. Can authors provide more discussion on this with different TPP methods?\n", "limitations": "The paper includes a section discussing the limitations and potential societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to read and is well-structured. The experiments conducted effectively demonstrate how each idea (Prompt and CL) contributes to the improvement over the alternative design.", "weaknesses": "I am not familiar with the related works on Neural TPP. However, I noticed that the experiment in this paper only utilizes two datasets, which might be considered somewhat limited. It is unclear if this is a common approach for papers working on this specific problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Can the authors provide justification on using only two datasets in the experiment section?\n\nFirstly, using two benchmark datasets for a certain task is common in TPP papers. For example, [1] use MIMIC and StackOverflow to test their transformer structure on event sequences; [2] use Retweet and StockOverflow for interactive event modeling. our baseline [3] use Yelp/Meme for the zero shot event modeling.\n\nSecondly, a large number of datasets frequently used in TPP are not suitable for our problem. For example, we did not use Yelp/Meme because their average length is too short; Retweet is not suitable for our problem either because each sequence is not associated with an user ID or absolute timestamps so we can hardly make each sequence pertaining to a single individual. In our work, we processed two real datasets from the source,  which both have enough sequence lengths and clear user identifiers, therefore suitable for the CL problem in our context. We have performed extensive analysis on predictive performance, convergency, hyperparameters, ablation study etc.\n\nLastly, also proposed by other reviewer, to further validate the effectiveness of our method, we perform the test on a new dataset. Please see [New results - new dataset StackOverflow] in Author Rebuttal above as the reply-to-all message. The results are consistent with our main results in the paper.\n\n\n[1] Transformer Embeddings of Irregularly Spaced Events and Their Participants, ICLR'22.\n\n[2] Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processes, AAAI'23.\n\n[3] HyperHawkes: Hypernetwork based Neural Temporal Point Process, (updated version is accepted by KDD 2023).\n\n\n> In the abstract, the authors talk about privacy constraints associated with the TPP problem. Can authors provide more discussion on this with different TPP methods?\n\nFirstly, we clarify that the privacy constraints [4] mainly refers to common industrial cases where user data (detailed food/exercise/sleep/medical event log, their social media interactions, identification data) can not be stored in a buffer. In this context, rehearsal-based CL methods are not applicable because they rely on a rehearsal buffer to re-train a portion of past examples. Simply buffering past data and re-train the model does not work either. \n\nOur proposed method introduced a pool of continuous-time prompts to learn and store the learned knowledge for event sequences. We structure the prompts as a  retrieval prompt pool, and design a retrieval mechanism to dynamically lookup a subset of task-relevant prompts based on the instance-wise input of event sequences. The retrieval prompt pool, which is optimized jointly with the generative loss, ensures that shared (unselected) prompts encode shared knowledge for knowledge transfer, and unshared (selected) prompts encode task-specific knowledge that helps maintain model plasticity. This mechanism removes the necessity of a rehearsal buffer. Therefore our method is applicable in a more realistic CL setting, i.e., where privacy constraints exist.\n\n[4] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proc SIGSAC conference on computer and communications security, 2015"}, {"review_id": "wNobMAPmDS", "reviewer": "Reviewer_GihP", "summary": "The author introduces PromptTPP, a method that integrates classical neural temporal point processes with a continuous-time retrieval prompt pool. This approach enables the model to learn event streams sequentially, eliminating the need to buffer previous examples or task-specific features in the context of continual learning. ", "questions": "see above", "limitations": "see above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's motivation for introducing this method is clear, and the experiments conducted to validate its effectiveness are relatively comprehensive.", "weaknesses": "The method description is not clear. The author needs to present the prompt-learning-based task in more detail. For example, the prompt length Lp, selection size N, and prompt pool size M are hyperparameters, but under common sense, the buffer of continual learning is dynamically changing. You can show the case study of the Taobao dataset and the Amazon dataset.\n\nThe combination of PPT, CL, and PL is somewhat unnatural. For example, I hardly see the idea of CL in the method.\n\nExplain how the prompt learning method can help CL in terms of theoretical or technical details, rather than just experimental accuracy. Instead of a decline in catastrophic forgetting, it's possible that the decrease in average error is caused by an improvement in accuracy at each time point. Thus, the author also needs to show how the model performs on previous events after learning new events. If it is a part of the experiment, kindly explain it in detail.\n\nDo the prompt settings need to be related to downstream tasks? If I understand correctly, your setting only concerns time, and it does not change depending on the process's intended goal. But in classical prompt learning, the prompt is customized, in NLP, adapting language model using downstream tasks. As you show in experiments, the task of each dataset is known beforehand, so will custom methods improve the results more than general methods? Can we investigate temporal prompt + event prompt?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The method description is not clear.... the buffer of continual learning is dynamically changing. You can show the case study of the Taobao dataset and the Amazon dataset.\n\nIn our experiment we found the performance w.r.t. hyperparameters is relatively stable over time. See below table. For each task,  increasing N from 1 to 2 brings ~0.8% improvement on error rate constantly; increasing N from 2 to 4 brings the change within [-0.02%, 0.06%] mostly.\n\nAmazon dataset, given $L_p=10, M=10$, sensitivity analysis of error rate w.r.t. selection size $N$ across all tasks\n\n|\t|Task 0 \t|Task 1\t|Task 2\t|Task 3\t|Task 4\t|Task 5|\tTask 6\t|Task 7\t|Task 8 \t|Task 9|\tAvg|\n|-----|-----------|-----------|-----------|-----------|-----------|--------|-------------|-----------|-----------|-------|-------|\n|N=1\t|75.04\t|75.82|\t73.29|\t74.15|\t68.51\t|67.49|\t67.83|\t63.19|\t63.65|\t61.35|\t69.03|\nN=2  \t|74.41\t|75.20\t|72.79\t|73.5|\t67.89\t|66.91|\t67.30\t |62.81|\t63.18|\t60.779|\t68.47|\n|improve rel%  | 0.847% |\t0.824%|\t0.687%|\t0.884%|\t0.913%|\t0.867%|\t0.788%\t|0.605%\t|0.744%\t|0.939%\t|0.811% |\n\n|\t|Task 0 \t|Task 1\t|Task 2\t|Task 3\t|Task 4\t|Task 5|\tTask 6\t|Task 7\t|Task 8 \t|Task 9|\tAvg|\n|-----|-----------|-----------|-----------|-----------|-----------|--------|-------------|-----------|-----------|-------|-------|\n|N=2|\t74.41\t|75.20\t|72.79\t|73.50\t|67.89\t|66.91|\t67.3\t|62.81|\t63.18\t|60.779\t|68.47|\n|N=4|\t74.39\t|75.16\t|72.78\t|73.45\t|67.84\t|66.87\t|67.31\t|62.77\t|63.15|\t60.74|\t68.44|\n|improve rel %  |\t0.027%|\t0.053%\t|0.014%\t|0.068%\t|0.074%\t|0.060%|\t-0.015%|\t0.064%|\t0.048%|\t0.064%\t| 0.045%|\n\nTherefore to better measure the impact of hyperparameters, we use the average performance as the metrics, same as in [1].  More details is in Analysis VI in section 4.2. \n\n[1] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning. ECCV 2022.\n\n> The combination of PPT, CL, and PL is somewhat unnatural. For example, I hardly see the idea of CL in the method.\n\nWe consider the streaming setting for modeling event sequence. Through an empirical analysis, we validated that pretraining a model will fail to adapt to new data due to distribution shift in the data while retraining a model will suffer from catastrophic forgetting. See the analysis in Appendix A. \n\nSo it is natural to approach the problem by adopting CL, an area studying how systems learn sequentially from a continuous stream of data. In the paper, we aim to build a CL model for such data in real scenarios with applicability and generality. Unfortunately, rehearsal-based methods are not applicable when there are resource constraints. Therefore we draw inspiration from PL, where we leverage prompts to sequentially learn knowledge and further store learned knowledge of event sequence in CL context.\n\nThe idea of CL in our method mainly lies in our aim to make the model capable of continuously learning event sequences without catastrophic forgetting. This is done by integrating the TPP with a continuous-time retrieval prompt pool. The prompts are stored in a memory space and jointly optimized with the base TPP, ensuring the model learns event streams sequentially without buffering past examples or task-specific attributes.\n\n> Explain how the prompt learning method can help CL in terms of theoretical or technical details, ... Thus, the author also needs to show how the model performs on previous events after learning new events. \n\nOriginated in NLP, PL reformulates learning downstream tasks from directly adapting model weights to designing prompts that “instruct” the model to perform tasks conditionally. A prompt encodes task-specific knowledge and has the ability to utilize pretrained models more effectively than ordinary finetuning [2]. \n\nIn our problem, we develop a module of temporal prompt that learns knowledge and further store the learned knowledge for event sequence in continuous time. We further structure the prompts in a key-value shared memory space called the retrieval prompt pool, and design a retrieval mechanism to dynamically lookup a subset of task-relevant prompts based on the instance-wise input of event sequences.  The optimized prompt pool ensures that shared (unselected) prompts encode shared knowledge for knowledge transfer, and unshared (selected) prompts encode task-specific knowledge that helps maintain model plasticity. \n\n[2] The power of scale for parameter-efficient prompt tuning. ACL 2021.\n\n> how the model performs on previous events after learning new events? \n\nDue to page limit,  we put it in [New results - how the model performs on previous events after learning new events] above in Author Rebuttal. \n\n> Do the prompt settings need to be related to downstream tasks? ... so will custom methods improve the results more than general methods? Can we investigate temporal prompt + event prompt?\n\nIn most TPPs we do not consider downstream tasks. However, an extension of TPP is the recommender system when they consider continuous time information. In those cases, customized prompt may be appropriate to use because there are classical downstream tasks such as user identification etc. However, this is out of the scope of our paper.\n\nFor the event prompt, we define it by\n- compute a histogram vector based on the normalized frequency of history event types.\n- add a MLP to map it and concatenate to form a prompt $P_{evt} \\in \\mathbb{R}^{L_p \\times D}$ and concatenate $P_{evt}$ to one temporal prompt, and repeat the process to form a pool. Some resize layer is needed.\n\nHowever, on Amazon data, the improvement on most tasks is insignificant (p-value ~ 0.15) based on the permutation test. The possible reason is the learnable structure component in the temporal prompt, already learns to encode the information of events other than time, so adding an 'event prompt' has limited improvement on encoding the knowledge of the sequence. Given the shortness of the rebuttal period, we have not found a better custom method but we agree this is a direction that worths exploring."}]

Author rebuttal:
We thank all the reviewers for their constructive feedback. In this response, we will first use a few reply-to-all messages to clarify our dataset settings and present some new experiment results. Then we will provide a point-to-point response to each individual review. We can surely consolidate all the paper improvements (including clarified technical details, an improvement on writing, experiment results on new dataset, etc) in our final version.

###  Dataset settings

Reviewer n7nj and nDRf are concerned about the datasets we used. Here we clarify in a general way and also reply in more details in point-to-point responses, respectively.

Generally speaking, use two datasets for validating a certain model is common in TPP works[1,2,3]. In our works, we carefully preprocess two industrial and open sourced user behavior datasets and have done extensive analysis to support our conclusion. 

We did not use the same datasets (Yelp and Meme) in Dubey's [4] work because they are not suitable for our problem: the sequence length is too short user-wise. In our problem, we aim to continuously learn the user behavior and formulate each sequence to be one unique user's timestamped behaviors and truncate by time into several tasks, so each sequence needs to be long enough. (Dubey's datasets do not have such requirement because he focuses on zero-shot event modeling in the CL setting, e.g., predict crimes for new cities while retaining and transferring information from old cities.  So their dataset is constructed category-wise rather than user-wise over time. 

Lastly, we indeed process one more dataset and perform the experiment on it to show our method works across several real datasets. Please see [New results - new dataset StackOverflow].


[1] Transformer Embeddings of Irregularly Spaced Events and Their Participants, ICLR 2022.

[2] Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processes, AAAI 2023.

[3] Neural Jump Stochastic Differential Equations, NeurIPS 2019.

[4] HyperHawkes: Hypernetwork based Neural Temporal Point Process, Dubey et al, (updated version is accepted by KDD 2023).

### New results - new dataset StackOverflow

Preprocess: StackOverflow [5] dataset has two years of user awards on a question answering website. Each user received a sequence of 22 kinds of badges in total. Following the processing recipe of previous work[1,6], we work on a subset of 4000 most active users with an average sequence length of 60 and then end up with K = 22 event types. We truncate the sequences by time into 6 tasks sequentially. Each task coverages approximately 5 months time . 


Setup: we take ANHP as the base model, build Pt-ANHP, compare with three classical model Pre-ANHP, Re-ANHP, O-TPP and one CL-based model CL-ANHP. 

Result: our method Pt-ANHP continuously outperforms others: the margin over the non-CL baselines (e.g., Re-ANHP) becomes larger over times (tasks) ; it is also outperforms CL-ANHP due to our design of CtRetroPromptPool which reduces catastrophic forgetting. The findings are consistent with those for Taobao and Amazon datasets.


error rate on each task (the lower the better)

| Model| Task 0| Task 1|Task 2|Task 3|Task 4|Task 5|Avg |
| ---- |---- |---- |---- |---- |---- |---- |---- |
| Pre-ANHP |50.99|  52.97	| 52.23| 	52.20| 	53.18| 	53.91| 	52.58| 
|Re-ANHP| 	50.99| 	52.62	| 51.90	| 51.77	| 51.32| 	51.12| 	51.62| 
|O-TPP	| 50.67	| 52.12	| 52.23	| 52.98	| 53.01	| 52.76| 	52.29| 
|CL-ANHP| 50.76	| 51.93	| 51.23	| 51.16	| 50.36	| 50.27	| 50.95| 
|Pt-ANHP| 50.72	| 51.30	| 50.73	| 50.08	| 49.44	| 49.01	| 50.21| 


Time RMSE on each task (the lower the better)

| Model| Task 0| Task 1|Task 2|Task 3|Task 4|Task 5|Avg |
| ---- |---- |---- |---- |---- |---- |---- |---- |
|Pre-ANHP	|1.1109	| 1.1423	|1.1362	|1.1703	|1.1933	|1.2096	|1.1604|
|Re-ANHP|	1.1109	| 1.1365	|1.1321	|1.1455	|1.1562	|1.1690	|1.1417|
|O-TPP	| 1.1270	| 1.1461	|1.1456	|1.1653	|1.1772	|1.1969	|1.1596|
|CL-ANHP|	1.1201	| 1.1322	|1.1253	|1.1425	|1.1420	|1.1552	|1.1362|
|Pt-ANHP|	1.1204	| 1.1302	|1.1231	|1.1334	|1.1352	|1.1398	|1.1303|


[5] Leskovec, J. and Krevl, A. SNAP Datasets: Stanford large network dataset collection, 2014.

[6] HYPRO: a hybridly normalized probabilistic model for long-horizon prediction of event sequence, NeurIPS 2022.


### New results - how the model performs on previous events after learning new events

Request by reviewer GihP, we perform a test on Amazon dataset. We use ANHP trained on task 9 and Pt-ANHP continuously trained on task 9, evaluate them on previous tasks and see how the metrics changed. Specifically,
-  For ANHP,  evaluation on test set of task i (i<9):  ANHP trained on task 9 vs ANHP trained on task i.
-  For Pt-ANHP,  evaluation on test set of task i (i<9): Pt-ANHP trained until (including) task 9 vs Pt-ANHP trained until task i.

See from below tables, on both metrics, we see the drop in performance (error rate/rmse becomes higher) of Pt-ANHP is much less significant than ANHP: indicating our method stores well the knowledge of previous tasks,  which largely alleviates catastrophic forgetting.

Abs change on error rate: 
|Model|	Task 0	|Task 1	|Task 2|	Task 3|	Task 4|	Task 5|	Task 6|	Task 7|	Task 8|
|-------| -------|-------|-------|-------|-------|-------|-------|-------|-------| 
|ANHP|	+2.46	|+2.06|	+1.67|	+1.34|	+0.92|	+1.1|	+0.81|	+0.69|	+0.62|
|Pt-ANHP|	+0.63|	+0.5|	+0.64|	+0.42|	+0.28|	+0.48|	+0.2	|+0.07	|-0.04|

Abs change on time RMSE
|Model|	Task 0	|Task 1	|Task 2|	Task 3|	Task 4|	Task 5|	Task 6|	Task 7|	Task 8|
|-------| -------|-------|-------|-------|-------|-------|-------|-------|-------| 
|ANHP	|+0.0213	|+0.0201	|+0.017|	+0.0094|	+0.0109|	+0.009|	+0.0061|	+0.0055|	+0.0031|
|Pt-ANHP | +0.0061	 |+0.006	 |+0.0053|	+0.005	|+0.0023	|+0.0035	|+0.0031	|+0.0012|	+0.0025|

Meta-review: The paper aims to enhance the modeling of continuous-time event sequences using a novel framework called PromptTPP. The paper has received moderate-to-high ratings from the reviewers, with some concerns and questions regarding the method's clarity, computational complexity, dataset choices, and privacy considerations. The authors have addressed some of these concerns in their responses. Overall, the paper offers a valuable contribution to the field of neural temporal point processes.
Decision: Accept (poster)