TP$^2$DP$^2$: A Bayesian Mixture Model of Temporal Point Processes with Determinantal Point Process Prior

Authors: yiwei  dong
shaoxin  ye
yuwen  cao
qiyu  han
hongteng  xu
hanfang  yang
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-08-24 05:55:35

Abstract: Asynchronous event sequence clustering aims to group similar event sequences in an unsupervised manner. Mixture models of temporal point processes have been proposed to solve this problem, but they often suffer from overfitting, leading to excessive cluster generation with a lack of diversity. To overcome these limitations, we propose a Bayesian mixture model of Temporal Point Processes with Determinantal Point Process Prior (TP$^2$DP$^2$) and accordingly an efficient posterior inference algorithm based on conditional Gibbs sampling. Our work provides a flexible learning framework for event sequence clustering, enabling automatic identification of the potential number of clusters and accurate grouping of sequences with similar features. It is applicable to a wide range of parametric temporal point processes, including neural network-based models. Experimental results on both synthetic and real-world data suggest that our framework could produce moderately fewer yet more diverse mixture components, and achieve outstanding results across multiple evaluation metrics.

Main paper: 
 introduction: 
 As a powerful tool of asynchronous event sequence modeling, the temporal point process (TPP) plays a crucial role in many application scenarios [5, 7, 9, 28] . In practice, event sequences often demonstrate clustering characteristics, with certain sequences showcasing greater similarities when compared with others. For instance, event sequences of patient admissions may exhibit clustering patterns in response to specific medical treatments. Being able to accurately cluster event sequences can bring many benefits, including facilitating healthcare decision making. In recent years, researchers have built mixture models of TPPs to tackle the event sequence clustering problem [23, 22, 27] . However, these models often suffer from overfitting during training, leading to excessive cluster generation with a lack of diversity. Moreover, these methods require either manually setting the number of clusters in advance [22] or initializing a large number of clusters and gradually removing excessive clusters through hard thresholding [23, 27] . In addition, without imposing proper prior knowledge, the clusters obtained by these models may have limited diversity and cause the identifiability issue.In this study, we propose a novel Bayesian mixture model of temporal point processes named TP 2 DP 2 for event sequence clustering, imposing a determinantal point process prior to enhance the diversity of clusters and developing a universally applicable conditional Gibbs sampler-based algorithm for the model's posterior inference. As illustrated in Figure 1 , TP 2 DP 2 leverages the determinantal point process (DPP) as a repulsive prior for the parameters of cluster components, < l a t e x i t s h a 1 _ b a s e 6 4 = " m 9 b R s q 4 D S R B n y l J M 0 E b Q u K F l + v Y = " > A A A B 8 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I V J d F N y 4 r 2 A e 2 p W T S O 2 1 o J j M k G a E M / Q s 3 L h R x 6 9 + 4 8 2 / M t L P Q 1 g O B w z n 3 k n O P H w u u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 l G i G D Z Z J C L V 8 a l G w S U 2 D T c C O 7 F C G v o C 2 / 7 k N v P b T 6 g 0 j + S D m c b Y D + l I 8 o A z a q z 0 2 A u p G f t B y m a D c s W t u n O Q V e L l p A I 5 G o P y V 2 8 Y s S R E a Z i g W n c 9 N z b 9 l C r D m c B Z q Z d o j C m b 0 B F 2 L Z U 0 R N 1 P 5 4 l n 5 M w q Q x J E y j 5 p y F z 9 v Z H S U O t p 6 N v J L K F e 9 j L x P 6 + b m O C 6 n 3 I Z J w Y l W 3 w U J I K Y i G T n k y F X y I y Y W k K Z 4 j Y r Y W O q K D O 2 p J I t w V s + e Z W 0 L q p e r V q 7 v 6 z U b / I 6 i n A C p 3 A O H l x B H e 6 g A U 1 g I O E Z X u H N 0 c 6 L 8 + 5 8 L E Y L T r 5 z D H / g f P 4 A 4 O K R E g = = < / l a t e x i t > cPosterior Maximization < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 p K h / j f c k 7 B h 6 b + L J H 4 z + H J A R I 4 = " > A A A B + n i c b V D L S s N A F J 3 U V 6 2 v V J d u g k V w V R K R 6 r L o x m X F v q A J Y T K d t E M n k z B z o 5 T Y T 3 H j Q h G 3 f o k 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E s 4 U 2 P a 3 U V p b 3 9 j c K m 9 X d n b 3 9 g / M 6 m F X x a k k t E N i H s t + g B X l T N A O M O C 0 n 0 i K o 4 D T X j C 5 y f 3 e A 5 W K x a I N 0 4 R 6 E R 4 J F j K C Q U u + W X U j D O M g z N z 2 m A K e + f e + W b P r 9 h z W K n E K U k M F W r 7 5 5 Q 5 j k k Z U A O F Y q Y F j J + B l W A I j n M 4 q b q p o g s k E j + h A U 4 E j q r x s H n 1 m n W p l a I W x 1 E + A N V d / b 2 Q 4 U m o a B X o y D 6 q W v V z 8 z x u k E F 5 5 G R N J C l S Q x a E w 5 R b E V t 6 D N W S S E u B T T T C R T G e 1 y B h L T E C 3 V d E l O M t f X i X d 8 7 r T q D f u L m r N 6 6 K O M j p G J + g M O e g S N d E t a q E O I u g R P a N X 9 G Y 8 G S / G u / G x G C 0 Z x c 4 R + g P j 8 w d 8 l J Q q < / l a t e x i t > ⇥ S < l a t e x i t s h a 1 _ b a s e 6 4 = " D b d h Q 0 h 7 z d E X O 2 c / N j M o X g u R m u Q = " > A A A B + n i c b V D L S s N A F J 3 U V 6 2 v V J d u g k V w V R K R 6 r L o x m W V v q A J Y T K d t E M n k z B z o 5 T Y T 3 H j Q h G 3 f o k 7 / 8 Z J m 4 W 2 H h g 4 n H M v 9 8 w J E s 4 U 2 P a 3 U V p b 3 9 j c K m 9 X d n b 3 9 g / M 6 m F X x a k k t E N i H s t + g B X l T N A O M O C 0 n 0 i K o 4 D T X j C 5 y f 3 e A 5 W K x a I N 0 4 R 6 E R 4 J F j K C Q U u + W X U j D O M g z N z 2 m A K e + f e + W b P r 9 h z W K n E K U k M F W r 7 5 5 Q 5 j k k Z U A O F Y q Y F j J + B l W A I j n M 4 q b q p o g s k E j + h A U 4 E j q r x s H n 1 m n W p l a I W x 1 E + A N V d / b 2 Q 4 U m o a B X o y D 6 q W v V z 8 z x u k E F 5 5 G R N J C l S Q x a E w 5 R b E V t 6 D N W S S E u B T T T C R T G e 1 y B h L T E C 3 V d E l O M t f X i X d 8 7 r T q D f u L m r N 6 6 K O M j p G J + g M O e g S N d E t a q E O I u g R P a N X 9 G Y 8 G S / G u / G x G C 0 Z x c 4 R + g P j 8 w d 7 E J Q p < / l a t e x i t > ⇥ RCentralJ U = " > A A A B + X i c b V B N S 8 N A F H z x s 9 a v q E c v i 0 W o l 5 K I V I 9 F L x 4 r m L b Q x r L Z b t q l m 0 3 Y 3 R R K 6 D / x 4 k E R r / 4 T b / 4 b N 2 0 O 2 j q w M M y 8 x 5 u d I O F M a c f 5 t t b W N z a 3 t k s 7 5 d 2 9 / Y N D + + i 4 p e J U E u q R m M e y E 2 B F O R P U 0 0 x z 2 k k k x V H A a T s Y 3 + V + e 0 K l Y r F 4 1 N O E + h E e C h Y y g r W R + r b d i 7 A e B W H m z Z 6 y K r 6 Y 9 e 2 K U 3 P m Q K v E L U g F C j T 7 9 l d v E J M 0 o k I T j p X q u k 6 i / Q x L z Q i n s 3 I v V T T B Z I y H t G u o w B F V f j Z P P k P n R h m g M J b m C Y 3 m 6 u + N D E d K T a P A T O Y 5 1 b K X i / 9 5 3 V S H N 3 7 G R J J q K s j i U J h y p G O U 1 4 A G T F K i + d Q Q T C Q z W R E Z Y Y m J N m W V T Q n u 8 p d X S e u y 5 t Z r 9 Y e r S u O 2 q K M E p 3 A G V X D h G h p w D 0 3 w g M A E n u E V 3 q z M e r H e r Y / F 6 J p V 7 J z A H 1 i f P 1 O E k 3 k = < / l a t e x i t > U (a)< l a t e x i t s h a 1 _ b a s e 6 4 = " L r O t R N r s g R j S g Y T d 6 3 H m / 0 6 A T D 4 = " > A A A B + n i c b V B N S 8 N A F H z x s 9 a v V I 9 e g k W o l 5 K I V I 9 F L x 4 r m L b Q x r L Z b t q l m 0 3 Y 3 S g l 5 q d 4 8 a C I V 3+ J N / + N m z Y H b R 1 Y G G b e 4 8 2 O H z M q l W 1 / G y u r a + s b m 6 W t 8 v b O 7 t 6 + W T l o y y g R m L g 4 Y p H o + k g S R j l x F V W M d G N B U O g z 0 v E n 1 7 n f e S B C 0 o j f q W l M v B C N O A 0 o R k p L A 7 P S D 5 E a + 0 H q Z v d p j a P T b G B W 7 b o 9 g 7 V M n I J U o U B r Y H 7 1 h x F O Q s I V Z k j K n m P H y k u R U B Q z k p X 7 i S Q x w h M 0 I j 1 N O Q q J 9 N J Z 9 M w 6 0 c r Q C i K h H 1 f W T P 2 9 k a J Q y m n o 6 8 k 8 q F z 0 c v E / r 5 e o 4 N J L K Y 8 T R T i e H w o S Z q n I y n u w h l Q Q r N h U E 4 Q F 1 V k t P E Y C Y a X b K u s S n M U v L 5 P 2 W d 1 p 1 B u 3 5 9 X m V V F H C Y 7 g G G r g w A U 0 4 Q Z a 4 A K G R 3 i G V 3 g z n o w X 4 9 3 4 m I + u G M X O I f y B 8 f k D J d K T 8 Q = = < / l a t e x i t > U (na) < l a t e x i t s h a 1 _ b a s e 6 4 = " B Z 9 V a K w k D Q L U w J J B h G O m x v Q L 5 1 Y = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i Q i 1 W X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S a G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q G c x 9 U I 8 E i x g B G s j D W y 7 H 2 I 9 9 o O 0 k z 2 l V X y R D e y K U 3 P m Q K v E L U g F C j Q H 9 l d / G J E k p E I T j p X q u U 6 s v R R L z Q i n W b m f K B p j M s E j 2 j N U 4 J A q L 5 0 n z 9 C 5 U Y Y o i K R 5 Q q O 5 + n s j x a F S s 9 A 3 k 3 l O t e z l 4 n 9 e L 9 H B j Z c y E S e a C r I 4 F C Q c 6 Q j l N a A h k 5 R o P j M E E 8 l M V k T G W G K i T V l l U 4 K 7 / O V V 0 r 6 s u f V a / e G q 0 r g t 6 i j B K Z x B F V y 4 h g b c Q x N a Q G A K z / A K b 1 Z q v V j v 1 s d i d M 0 q d k 7 g D 6 z P H 1 a a k 3 s = < / l a t e x i t > W (a)< l a t e x i t s h a 1 _ b a s e 6 4 = " K g F 8 C G 8 y M a K x k 8 3 i 7 C h l P P s m which contributes to generating TPPs with diverse parameters. To make TP 2 DP 2 applicable for the TPPs with a large number of parameters, we apply Bayesian subnetwork inference [6] , employing Bayesian inference to partially selected parameters while utilizing maximum likelihood estimation for the remaining parameters. For selected parameters, we further categorize them into central and non-central parameters, in which the central parameters mainly determine the clustering structure and thus we apply DPP priors. We design an efficient conditional Gibbs sampler-based posterior inference algorithm, in which the stochastic gradient Langevin dynamics [21] is introduced into the updating process to facilitate convergence. To our knowledge, TP 2 DP 2 is the first work that explores event sequence clustering based on the TPP mixture model with DPP prior. It automatically identifies cluster numbers,with clustering results more reliable than existing variational inference methods [23, 27] .Z M A = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S L U T U l E q s u i G 5 c V 7 A P a W i b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O F 3 G m t O N 8 W y u r a + s b m 4 W t 4 v b O 7 t 6 + X T p o q T C W h D Z J y E P Z 8 b C i n A n a 1 E x z 2 o k k x Y H H a d u b X G d + + 4 F K x U J x p 6 c R 7 Q d 4 J J j P C N Z G G t i l X o D 1 2 P O T d n q f V A Q + T Q d 2 2 a k 6 M 6 B l 4 u a k D D k a A / u r N w x J H F C h C c d K d V 0 n 0 v 0 E S 8 0 I p 2 m x F y s a Y T L B I 9 o 1 V O C A q n 4 y i 5 6 i E 6 M M k R 9 K 8 4 R G M / X 3 R o I D p a a B Z y a z o G r R y 8 T / v G 6 s / c t + w k Q U a y r I / J A f c 6 R D l P W A h k x S o v n U E E w k M 1 k R G W O J i T Z t F U 0 J 7 u K X l 0 n r r O r W q r X b 8 3 L 9 K q + j A E d w D B V w 4 Q L q c A M N a A K B R 3 i G V 3 i z n q w X 6 9 3 6 m I + u W P n O I f y B 9 f k D K O q T 8 w = = < / l a t e x i t > W (na) 
preliminaries: 
 Temporal Point Processes TPP is a kind of stochastic process that characterizes the random occurrence of events in multiple dimensions, whose realizations can be represented as event sequences, i.e., {(t i , d i )} I i=1 , where t i ∈ [0, T ] are time stamps and d i ∈ D = {1, ..., D} are different dimensions (a.k.a. event types). Typically, we characterize a TPP by conditional intensity functions:EQUATIONHere, λ * d (t) is the conditional intensity function of the type-d event at time t, N d (t) denotes the number of the occurred type-d events prior to time t, and H t denotes the historical events happening before time t. Given an event sequence s = {(t i , d i )} I i=1 , the likelihood function of a TPP can be derived based on its conditional intensity functions:EQUATIONBy maximizing the likelihood in Eq. ( 2), we can learn the TPP model to fit the observed sequence.Mixture Models of TPPs Given multiple event sequences belonging to different clusters, i.e., {s n } N n=1 , we often leverage a mixture model of TPPs to describe their generative mechanism, leading to a hierarchical sampling process:EQUATIONwhere π = [π 1 , ..., π M ] ∈ ∆ M -1 indicates the distribution of clusters defined on the (M -1)-simplex, TPP(θ m ) is the TPP model of the m-th cluster, whose parameters are denoted as θ m .Determinantal Point Processes DPP [13] is a stochastic point process characterized by the unique property that its sample sets exhibit determinantal correlation. The structure of DPP is captured through a kernel function [14, 4] . Denote the kernel function by κ : X × X → R, where X represents a sample space. The density function for samples x 1 , ..., x M ∈ X in one realization of DPP is:EQUATIONwhereK(x 1 , ..., x M ) = [κ(x i , x j )] is a M × M Gram matrix corresponding to the samples. Given arbitrary two samples x i and x j , we have p(x i , x j ) = κ(x i , x i )κ(x j , x j ) -κ(x i , x j ) 2 = p(x i )p(x j ) -κ(x i , x j ) 2 ≤ p(x i )p(x j ). Therefore, DPP manifests the repulsion between x i and x j . As such, using DPP as the prior can help enhance the diversity of clustering results. EQUATIONwhereΘ = {θ m } M m=1 ∈ R P . p(M ), p(Θ | M ) and p(π | M) are prior distributions of M , Θ and π, respectively. By Bayes theorem, the posterior p(M, Θ, π, c | S) is proportional to Eq. ( 5).The exact sampling from p(M, Θ, π, c | S) is often intractable because the parameters of the TPPs in practice (especially those neural TPPs [15, 8, 26, 29, 16] ) are too many to perform full Bayesian posterior calculation. To overcome this issue, we conduct posterior inference only on a subset of model parameters [6, 18, 12] (i.e., the "subnetwork" of the whole model). In particular, we approximate the full posterior of the TPPs' parameters Θ asEQUATIONwhere we split the model parameters Θ into two parts, i.e., Θ S and Θ R , respectively. Θ S = {θ S,m } M m=1 corresponds to the subnetworks of the TPPs in the mixture model, while Θ R = {θ R,m } M m=1 denotes the remaining parameters. In Eq. ( 6), p(Θ | S) is decomposed into the posterior of the subnetworks p(Θ S | S) and a Dirac delta function on the remaining parameters Θ R , in which Θ R is estimated by their point estimation Θ R = { θ R,m } M m=1 , e.g., the maximum likelihood estimation achieved by stochastic gradient descent.Unlike existing work, in Eq. ( 6), we further decompose the parameters in the subnetworks into two parts, i.e., Θ S = {U , W }, where U = {µ m } M m=1 and W = {w m } M m=1 , respectively. For the m-th TPP in the mixture model, µ m corresponds to the "central" parameters of their conditional intensity functions, which significantly impacts the overall dynamics of event occurrence (e.g. the base intensity of Hawkes process [11] ). Accordingly, the other "non-central" parameters in each subnetwork are denoted as w m , which are contingent upon specific architectures of different models. Imposing the conditional independence on the central and non-central parameters, i.e.,EQUATIONwhere θ R,cn denotes the point estimates of the remaining parameters in the c n -th TPP. The DPP prior p(U | M ) is introduced to the central parameter U to mitigate the overfitting problem and diversify the cluster result. The computational method of DPP prior construction is introduced in Appendix. p(W | M ) = M m=1 p(w m ) is the prior of non-central parameters which can be Gaussian. For prior of π, instead of directly sampling {π m } M m=1 from its posterior distribution, we apply the ancillary variable method [3, 1] to make the posterior calculation tractable for the mixture weights {π m } M m=1 . Consider r = [r 1 , ..., r M ], which consists of i.i.d. positive continuous random variables following the Gamma distribution Γ(1, 1), each r m is independent of M and r is independent of {U , W }. Defining t = M m=1 r m and π = [r 1 /t, ..., r M /t], we establish a one-toone correspondence between π and (r, t). By introducing an extra random variable v ∼ Γ(N, 1), we define the ancillary variable u = v/t, with p(u) = u N -1 Γ(N ) ∞ 0 t N e -ut p(t)dt. Introducing u makes Algorithm 1 Conditional Gibbs Sampler for TP 2 DP 2 Input: Event sequences S, priors, initialization of the cluster number, maximum number of iteration T, number of burn-in, step sizes for each update, point estimates Θ R . Output: Posterior samples of variables in the model {M, U , r, W , c}.1: Initialize parameters and set j = 0. 2: while convergence not reached and j < T do 3:Sample non-allocated variables (U (na) , r (na) , W (na) ) using collapsed Gibbs sampler. The sampling for U (na) is given by: The r (na) and W (na) is given by:EQUATIONp(r (na) | • • • ) ∝ l m=1 p(r (na) m )e -ur (na) m , p(W (na) | • • • ) ∝ l m=1p(w (na) m ).Sample allocated variables (U (a) , r (a) , W (a) ).p(U (a) | • • • ) ∝ p(U (a) ∪ U (na) ) k m=1 i:ci=m L(s i | (µ (a) m , w (a) m , θ R,m ), p(r (a) | • • • ) ∝ k m=1 p(r (a) m )(r (a) m ) nm exp(-ur (a) m ). p(W (a) | • • • ) ∝ k m=1 p(w (a) m ) i:ci=m L(s i | (µ (a) m , w (a) m , θ R,m)).Sample cluster labels c using full conditional distribution:EQUATIONm , θ R,m ) for m = 1, ..., k, r(na) m L(s i | µ (na) m , w (na) m , θ R,m ) for m = k + 1, ..., k + l. 6:Update ancillary variable u by u ∼ Gamma(N, 1 t ).7:Increment j. 8: end while the posterior computation of π factorizable and gets rid of the sum-to-one constraint imposed on {π m } M m=1 , significantly simplifying the subsequent MCMC simulation process. In summary, the joint posterior density function becomesEQUATIONwhere p(U ) := p(M )p(U | M ) is the DPP prior.Since the number of clusters changes dynamically as these algorithms proceed, it is helpful to further partition model parameters into parameters of allocated clusters and those of non-allocated clusters when applying posterior sampling. In particular, we partition U into two sets according to cluster allocations c: one comprising cluster centers currently used for data allocation, denoted as U (a) = {µ c1 , . . . , µ cn }, and the other containing cluster centers not involved in the allocation, denoted as U (na) = U \U (a) . Note that the product measure dµ×dµ in Ω×Ω lifted by the map (x, y) → x∪y results in the measure dµ, so the prior density of (U (a) , U (na) ) is equivalent to p(U (a) , U (na) ) = p(U (a) ∪ U (na) ), which follows the DPP density. W and r are partitioned in the same way. As (U , π, W , c) and (U (a) , r (a) , W (a) , U (na) , r (na) , W (na) , c) are in a one-to-one correspondence, we can refer to Eq. ( 8) and obtain the posterior of (M, U (a) , r (a) , W (a) , c, U (na) , r (na) , W (na) , u): p(M, U (a) , r (a) , W (a) , c, U (na) , r (na) , W (na) , u|S)EQUATIONwhere n m is the number of sequences allocated to the m-th component, k denotes the cardinality of allocated clusters, and l denotes that of non-allocated ones. r (N -1)! e -ut t N . Our posterior inference algorithm follows the principle of conditional Gibbs sampler [17] . We split all parameters into three groups: an allocated block (U (a) , r (a) , W (a) ), a non-allocated block (U (na) , r (na) , W (na) ), and remaining parameters {c, u}, and update them in an alternating scheme. The posterior sampling procedure of TP 2 DP 2 is summarized in Algorithm 1. More detailed derivations are elaborated in Appendix. 
experiments: 
 Our model is compatible with various TPP backbones, which can detect clusters and fit event sequence data originating from a mixture of hybrid TPPs. To verify our claim, we generate a set of event sequences based on five different TPPs, including 1) Homogeneous Poisson process, 2) Inhomogeneous Poisson process, 3) Self-correcting process, 4) Hawkes process, and 5) Neural Hawkes process [15] . Based on the sequences, we construct three datasets with the number of mixture components ranging from three to five. For each dataset, we learn a mixture model of TPPs and set the backbone of the TPPs to be 1) the classic Hawkes process [11] , 2) the recurrent marked temporal point process (RMTPP) [8] , and 3) the Transformer Hawkes process (THP) [29] , respectively. The learning methods include the variational EM of Dirichlet mixture model (Dirichlet Mixture) [27] and our TP 2 DP 2 . The results in Table 1 show that our method achieves competitive performance. Especially when the backbone is Hawkes process, applying our method leads to notable improvements in purity [23] and ARI [20] , which means that our method is more robust to the model misspecification issue. In addition, learning RMTPP and THP by our method results in the best performance when K GT = 5, showcasing TP 2 DP 2 's adaptability to complex event sequences. More experiments are in Appendix.  
conclusion: 
 In this paper, we propose the Bayesian mixture model TP 2 DP 2 for event sequence clustering. It is shown that TP 2 DP 2 could flexibly integrate various parametric TPPs including the neural networkbased TPPs as components, achieve satisfying event sequence clustering results and produce more separated clusters. In the future, we plan to study the impact of alternative repulsive priors on event sequence clustering, and develop event sequence clustering methods in high-dimensional and spatio-temporal scenarios. 
 
 Appendix: In recent years, researchers have built mixture models of TPPs to tackle the event sequence clustering problem [23, 22, 27] . However, these models often suffer from overfitting during training, leading to excessive cluster generation with a lack of diversity. We illustrate this issue in the left panel of Figure 2 . [19] of RMTPP's event sequence embeddings [8] for a synthetic dataset with three clusters. [27] (left) produces four clusters wrongly, while our TP 2 DP 2 (right) leads to the clustering results matching well with the ground truth.DPP prior is introduced to the central parameter U to mitigate the overfitting problem and diversify the cluster result. We leverage the spectral density approach to approximate the DPP density. For a DPP shown in Eq. ( 4), its kernel function has a spectral representation κ(µ i , µ j ) = ∞ i=1 λ i ϕ i (µ i )ϕ i (µ j ), in which each eigenfunction could be approximated via the Fourier expansion and eigenvalues are specified by the spectral distribution, as defined in [14] . In this way, the DPP density approximation isp(U | M ) ≈ exp (|R| -D app ) det{ K}(µ 1 , • • • , µ M ), where κ(µ i , µ j ) = z∈Z q φ(z)e 2πiz•(µi-µj ) , D app = z∈Z q log(1+ φ(z)), {µ 1 , • • • , µ M } ⊂ R,|R| is the volume of the range of the parameter space, φ(z) = φ(z)/(1 -φ(z)), Z q is q-dimensional integer lattice, and φ is the spectral distribution.Our posterior inference algorithm follows the principle of conditional Gibbs sampler [17] . We split all parameters into three groups: an allocated block (U (a) , r (a) , W (a) ), a non-allocated block (U (na) , r (na) , W (na) ), and remaining parameters {c, u}, and update them in an alternating scheme.Sampling non-allocated variables: We begin with sampling U (na) from its conditional density:EQUATIONwhere "• • • " denotes variables excluding the target variable to be sampled, together with all the sample sequences S, and henceforth. p(U (a) ∪ U (na) ) is the DPP density. The second termEQUATIONm )]dr (na) , due to the fact thatEQUATIONApplying Birth-and-death Metropolis-Hastings algorithm [10] , we sample U (na) and determine the final number of clusters accordingly.We then sample r (na) and W (na) using the classical Metropolis-Hastings algorithm. The cardinality of non-allocated variables (i.e., l) is determined by the size of U (na) , so we haveEQUATIONSampling allocated variables: The allocated central parameter U (a) is sampled fromEQUATIONwhere the p(U (a) ∪ U (na) ) is again governed by the DPP. Subsequently, we sample r (a) from its full conditional using the Metropolis-Hastings algorithm:EQUATIONThe W (a) 's full conditional is:EQUATIONAs W (a) represents all the allocated parameters of the point process model to be inferred, excluding µ, it may still exhibit high dimensionality. To align with our framework and boost convergence, we leverage the stochastic gradient Langevin dynamics [21] when sampling W (a) . The proposed update for each wm is provided by:EQUATION)where j is the counting number of iterations, η j ∼ N (0, ϵ j ), ϵ j is the step size at the j-th iteration which is set to decay towards zero, and n * in the above equation is the number of selected sequences from each cluster to perform stochastic approximation.EQUATIONm , θ R,m ) is calculated through the automatic differentiation [2] .Sampling c and u: Each cluster label c i is sampled fromEQUATIONm , θ R,m ) for m = 1, ..., k, rEQUATIONNote that after this step, there is a positive probability that c i > k for certain indices i, indicating that some initially non-allocated components become allocated, and vice versasome initially allocated components become non-allocated. Consequently, a relabeling of (U (a) , r (a) , W (a) , U (na) , r (na) , W (na) ) and c is performed, ensuring that c takes values within the set {1, . . . , k} N . Thus, k may either increase or decrease or remain unchanged after the relabeling step. Finally, we sample u from a gamma distribution with a shape parameter of N and an inverse scale parameter of t.To comprehensively evaluate the effectiveness of our TP 2 DP 2 model and its inference algorithm, we test our method on both synthetic and real-world datasets and compare it with state-of-the-art event sequence clustering methods. For each method, we evaluate its clustering performance by clustering purity [23] and adjusted rand index (ARI) [20] and its data fitness by the expected log-likelihood per event (ELL) [24] . In addition, we report the expected posterior value of the number of clusters (M ) in real-world dataset, which reveals the inferred number of components given data. The code for TP 2 DP 2 is publicly available at https://anonymous.4open.science/r/TP2DP2/. We compare our TP 2 DP 2 with the Dirichlet mixture model of Hawkes processes (DMHP) learned by the variational EM algorithm [23] . For a fair comparison, we use the same Hawkes process backbone as in DMHP, ensuring identical parametrization, and we consider all model parameters in the Bayesian inference phase. For each method, we initialize the number of clusters randomly in the range [K GT -1, K GT + 1]. The averaged results in five trials are presented in Figure 3 .When the disparity in the true base intensity among different point processes is minimal, the inherent distinctions within event sequences are not apparent, as shown in Figure 4 . In this case, TP 2 DP 2 tends to categorize these event sequences into fewer groups than the ground truth, resulting in a relatively modest purity when δ is small. As δ increases, TP 2 DP 2 exhibits increasingly robust clustering capabilities, with means consistently outperforming DMHP when δ > 0.55.In addition, we examine the posterior distribution of base intensity parameters when both algorithms converge. At δ = 0.6, box plots in Figure 5 depict posterior estimations of base intensities for the first two clusters (the ground truth base intensities are 0.5 and 1.1). It is noteworthy that DMHP consistently underestimates the true values in all trials due to multiple times of approximations in its learning algorithm, and DMHP shows marginal disparity between clusters. In contrast, TP 2 DP 2 better captures the true base intensity values, meantime exhibiting greater dispersion between clusters compared with DMHP. Similar patterns are also observed in other datasets.In experiments on mixtures of hybrid TPPs, we further investigate the effect of incorporating DPP prior to different parameters in the models, and the results are shown in Table 2 . In this experiment, we aim to verify adding DPP priors to central parameters of TPPs would lead to superior performance. For Hawkes Process, the base intensity reflects the average level of event occurrence rate, and is considered the most crucial for analyzing the feature of corresponding event sequences [11] . For example, in the field of seismology, the base intensity specifically relates to background seismicity that needs special attention. Thus, for the event sequence clustering task, we also add DPP priors to the base intensity of the Hawkes process components, and find this yields the best clustering performance. For both RMTPP and THP, adding DPP prior to the output linear layer bias achieves the best purity and ARI scores, which are also consistently higher than the Dirichlet mixture frameworks or models without DPP priors. According to the architecture of these two neural point processes, the bias term of the last output layer has a direct impact on the estimated intensity function. This experimental result shows the effectiveness of applying DPP to the parameters that play a decisive role in intensity. To examine the performance of our method on real-world data, we use the following two benchmark datasets: 1) Amazon [25] . This dataset comprises time-stamped user product review events spanning from January 2008 to October 2018, with each event capturing the timestamp and the category of the reviewed product. Data is pre-processed according to the procedure in [24] . The final dataset consists of 5,200 most active users with 16 distinct event types, and the average sequence length is 70. 2) BookOrder2 . This book order dataset comprises 200 sequences, with two event types in each sequence. The sequence length varies from hundreds to thousands.To ensure fairness, hyperparameters of the Dirichlet mixture models are tuned first and we intentionally make each backbone TPP model in TP 2 DP 2 smaller or equivalent in scale compared to those of the Dirichlet mixture framework, which means that all hyperparameters related to the backbone structure, such as hidden size, number of layers, and number of heads within the TP 2 DP 2 framework are set to be less than or equal to their corresponding counterparts in the Dirichlet mixture framework. In this case, if TP 2 DP 2 model achieves a higher log-likelihood with fewer cluster numbers, it indicates that our method is better at capturing the characteristics of the data and provides a better fit. Table 3 summarizes the average results for different models in five trials.In the experiment on the real-world dataset, the Dirichlet Mixture framework performs generally worse than TP 2 DP 2 in both dataset, but the number of posterior cluster numbers inferred by the Dirichlet Mixture framework is generally larger. This reflects that TP 2 DP 2 moderately reduces the number of clusters to obtain more dispersed components without sacrificing much fitting capability.https://ant-research.github.io/EasyTemporalPointProcess/user_guide/dataset.html 



Reviews: review: title: title: BDU Review
main_review: main_review: # Overview

Thanks to the authors for their submission. While my response is fairly critical, it is given in a wholly constructive spirit.

The authors propose a mixture model for temporal point processes, including a "repulsive" prior designed to increase the diversity of clusters, and derive a conditional Gibbs sampling procedure to efficiently infer the posterior. While the paper makes some interesting original strides (the DPP priors, and deriving efficient updates), and could potentially be made improved with significant revisions, in its current form it has problems in terms of quality, significance, and clarity.

The most major problem in terms of quality is a lack of significant experimental results: the proposed model is compared against only one baseline (a Dirichlet Process Mixture Model), and although the comparisons slightly favor TP$^2$DP$^2$, it is overall not convincing: TP$^2$DP$^2$ does not consistently outperform the DPMM, and in some of the cases where it does outperform it does so only by a small margin. This could be forgiven if the authors demonstrate that TP$^2$DP$^2$ is competitive in other ways, but unfortunately does not support their claim that TP$^2$DP$^2$ produces "outstanding" results. The paper would also be improved by including more than one other baseline model (modulo backbones).

The presentation and writing of the paper could also be improved. Most importantly, if the authors make specific claims about experimental performance, then some of the experimental results (or a quantitative summary) should be included in the body of the paper. As written, all experimental results are included only in the appendix. The structure and wording of the body of the paper also need significant revision in order to be clear and persuasive.

## Pros
- Good motivation of why one might want to explore an alternative prior for TPPs (priors are important and sometimes overly-neglected!)
- The author's have clearly put significant thought into how to perform inference efficiently given limitations (e.g. in applying Bayesian subnetwork inference) which can lay the groundwork for further refinements and/or help inspire new approaches.

## Cons
- As mentioned above, experimental results are quite limited.
- Paper is unclear and needs significant restructuring and rewording.

# Suggestions

## Major
- TP$^2$DP$^2$ should be compared against more than one alternative baseline model.
- If the paper makes specific claims about the quality of experimental results, these should be backed up by at least a couple significant experimental results included in the body of the paper, or by a comprehensive and quantitative summary of results.
  - In order to make room for experimental results in the body of the paper, Algorithm 1 could be summarized by more-or-less removing all of the display-math. The full mathematical form can be given in the appendix.
  - Unfortunately, I do not think the results in the appendix are strong enough w.r.t. the baseline to qualify as "outstanding", and I am not persuaded that they are particularly significant without more evidence. 
- The body of the paper, in particular Section **3 Proposed TP$^2$DP$^2$ Model & Corresponding Posterior Inference Algorithm** requires significant restructuring for clarity. I suggest:
  - Paragraph break before "Unlike existing work, in Eq (6) ..."
  - Paragraph break before "Imposing the conditional independence on the central an non-central parameters ..."
  - Paragraph break before "For prior of $\pi$, instead of directly sampling ... "
  - Paragraph break before "In summary, the joint posterior density function becomes ..."
- In Section **D.1 Experiments on Mixtures of Hawkes Processes**, you say that TP$^2$DP$^2$ "consistently outperforms DMHP when $\delta > 0.55$", but your plots do _not_ show this: At best, they show that TP$^2$DP$^2$ generally outperforms DMHP when $\delta > 0.55$ (for $K_{GT} = 5$, at least): The mean of TP$^2$DP$^2$ may be slightly above DMHP, but there is significant overlap in their standard deviations. Since the mean and SD are only estimated with 5 trials, it is probably safer to say that "TP$^2$DP$^2$ is competitive with DMHP" when $\delta > 0.55$.
- In the "Broader Impacts" section of the checklist, you state that the paper does not have societal impacts. However, in the introduction, you motivate robust sequence clustering by claiming that it could facilitate healthcare decision making: what happens if your algorithm is used in such a setting but is not as robust or accurate as expected? While this may seem like nitpicking, I think these are exactly the kinds of issues that should be interrogated when considering broader societal impacts.

## Minor
- In Appendix Section *A*, the t-SNE plots aren't very informative (even as a general illustration) since the ground-truth clusters are not very evident (to my eye). Maybe you could find a better example, where the clusters are more distinct, so that NTPP-MIX is more clearly "wrong" and TP$^2$DP$^2$ more clearly "right"? Failing that, I would suggest at least plotting the ground-truth cluster assignments in a 3rd panel, so we can easily compare both to the GT.
- You should add error estimates to the ELL metric in Table 3: For example, TP$^2$DP$^2$ outperforms a Dirichlet Mixture with THP backbone by 1.631 to 1.629, but we might expect this is within the margin of error. Probably this means running more than 5 trials to get a good estimate.
- The qualitative comparison at the end of Appendix Section **D.3 Experiments on Real-World Datasets** is not very convincing: it's not a-priori clear that the bottom two sequences, although superficially similar, _should_ belong to the same cluster. Additionally, since these are just four arbitrary sequences from hundreds, what assurance do we have that they are representative of the clusters (and thereby the model performance): maybe these examples are just "lucky" for TP$^2$DP$^2$ and "unlucky" for the DPMM.
- Minor wording issues (although I fully understand these can be very hard to spot):
  - "$p( \mathbf{W} | M) = \prod_{m=1}^{M} p(\mathbf{w}_{m})$ is the prior of non-central parameters which can be Gaussian." Do you mean to say "which we choose to be Gaussian"?
  - "Introducing $u$ could make the posterior computation of $\mathbf{\pi}$ factorizable and get rid of the sum-to-one constraint imposed on ..." $\rightarrow$ "Introducing $u$ makes the posterior computation of $\mathbf{\pi}$ factorizable and gets rid of the sum-to-one constraint imposed on ..."
  - In general, I think the paper could use a thorough check of wording for clarity, and in some places for subject/verb agreement.
    - e.g. "**Determinantal Point Processes** DPP is a ..." $\rightarrow$ "**Determinantal Point Proccesses (DPPs)** are a ..."
title: title: Review
main_review: main_review: The paper proposes to incorporate the repulsive determinantal point process (DPP) prior into a Bayesian mixture model for temporal point processes. The authors develop a specialized conditional Gibbs posterior sampling algorithm to perform Bayesian inference on selected model parameters. The experiments show the proposed method can produce moderately fewer yet more diverse mixture components and perform competitively on evaluation metrics.

The paper is well-written, with a clear motivation and a solid technical foundation. I also appreciate the authors providing an anonymous code repository for reference, which enhances transparency and reproducibility.

A detailed runtime analysis of the posterior inference algorithm would strengthen the authors' claim regarding its computational efficiency. Including a comparison with existing methods in terms of runtime would provide a more complete picture. In particular, I imagine the number of selected parameters would have a huge impact on the performance.

Minor formatting issues in the reference list, e.g., reference [3], Mcmc -> MCMC, bayesian -> Bayesian

**Overall Assessment:** Although I have limited experience in the temporal point process domain and it's likely I did not fully understand some technical details of the paper, the paper seems to present a meaningful contribution to the field, and I lean toward acceptance.