Title: Is Score Matching Suitable for Estimating Point Processes?
Venue: NeurIPS 2024
Accepted: yes
Keywords: point processes

score matching

parameter estimation
Abstract: Score matching estimators for point processes have gained widespread attention in recent years because they do not require the calculation of intensity integrals, thereby effectively addressing the computational challenges in maximum likelihood estimation (MLE). Some existing works have proposed score matching estimators for point processes. However, this work demonstrates that the incompleteness of the estimators proposed in those works renders them applicable only to specific problems, and they fail for more general point processes. To address this issue, this work introduces the weighted score matching estimator to point processes. Theoretically, we prove the consistency of the estimator we propose. Experimental results indicate that our estimator accurately estimates model parameters on synthetic data and yields results consistent with MLE on real data. In contrast, existing score matching estimators fail to perform effectively. Codes are publicly available at \url{https://github.com/KenCao2007/WSM_TPP}.

Is Score Matching Suitable for Estimating Point Processes?

Abstract

Score matching estimators have gained widespread attention in recent years partly because they are free from calculating the integral of normalizing constant, thereby addressing the computational challenges in maximum likelihood estimation (MLE). Some existing works have proposed score matching estimators for point processes. However, this work demonstrates that the incompleteness of the estimators proposed in those works renders them applicable only to specific problems, and they fail for more general point processes. To address this issue, this work introduces the weighted score matching estimator to point processes. Theoretically, we prove the consistency of our estimator and establish its rate of convergence. Experimental results indicate that our estimator accurately estimates model parameters on synthetic data and yields results consistent with MLE on real data. In contrast, existing score matching estimators fail to perform effectively. Codes are publicly available at https://github.com/KenCao2007/WSM_TPP .

1 Introduction

Point processes are a class of statistical models used to characterize event occurrences. Typical models include Poisson processes [9] and Hawkes processes [3]. Their applications span various fields such as seismology [16; 17], finance [1; 4], criminology [15], and neuroscience [11; 26]. In the field of point processes, maximum likelihood estimation (MLE) has been a conventional estimator. However, MLE has an inherent limitation: it requires the computation of the normalizing constant, which corresponds to the intensity integral term in the likelihood. Except for simple cases, calculating the intensity integral analytically is generally infeasible. This necessitates the use of numerical integration methods like Monte Carlo or quadrature for approximating the computation. This introduces approximation errors, and more importantly, for high-dimensional problems, numerical integration encounters the curse of dimensionality, rendering training infeasible.

To address this issue, prior research has introduced the concept of score matching (SM) [5] to the field of point processes. For instance, [18] derived the application of SM to the estimation of traditional statistical Poisson processes. Furthermore, [24] extended the use of SM to the estimation of deep covariate spatio-temporal point processes. [10] also generalized the application of SM to Hawkes processes. These works have greatly advanced the utilization of SM for point processes. However, in practical applications, we have found that these estimators only work for specific point processes. For more general cases, these estimators cannot accurately estimate model parameters, even for some simple statistical point processes. One of the core contribution of this work is to theoretically demonstrate the incompleteness of the estimators proposed in the aforementioned studies.

The incompleteness of the estimators in the aforementioned studies stems from the transition from explicit SM to implicit SM. The explicit SM estimates model parameters by minimizing the expected distance between the gradient of the log-density of the model and the gradient of the log-density of the data. However, we cannot directly minimize the above objective function since it depends on the unknown data distribution. To facilitate solving, we need to convert the above explicit SM to implicit SM by using a trick of integration by parts, provided that some regularity conditions are satisfied [5]. In [18; 24; 10], they assume that the required regularity conditions are satisfied in their point process models and directly employ the implicit SM objective. However, as demonstrated in Section 3, the required regularity conditions cannot be met for general point processes. This implies that the concise implicit SM objectives (Equation (2) in [18], Equation (10) in [24], Equation (4) in [10]) are incomplete, and they cannot accurately estimate parameters for general point processes.

To address this issue, this work introduces a (autoregressive) weighted score matching (WSM) estimator that can be applied to more general point processes. WSM eliminates the intractable terms in SM objective by adding a weight function that takes zero at the boundary of the integration region. Compared to previous work on WSM [6; 22; 12], we are the first work to apply WSM on a stochastic process where the dimension N T is also random. This stochasticity in dimensionality poses greater challenges to the derivation, requiring special treatment to address this issue.

Specifically, we make following contributions: (1) We theoretically demonstrate that implicit (autoregressive) SM estimators in [18; 24; 10] are incomplete because the required regularity conditions cannot be satisfied for general point processes. (2) To address this issue, we propose a (autoregressive) WSM estimator that is applicable to general point processes. Theoretically, we establish its consistency and convergence rate. (3) In experiments, we confirm that on synthetic data, (autoregressive) WSM successfully recovers the ground-truth parameters; on real data, (autoregressive) WSM estimates results consistent with MLE; while existing (autoregressive) SM estimator fails in both scenarios.

2 Preliminaries

Now we provide knowledge on Poisson and Hawkes processes and (autoregressive) score matching.

2.1 Poisson Process and Hawkes Process

The Poisson process [9] is a stochastic point process that models the occurrence of events over a time window [0 , T ] . A trajectory from Poisson process can be represented as an ordered sequence T = ( t 1 , . . . , t N T ) where N t = max { n : t n ≤ t, t ∈ [0 , T ] } is the corresponding counting process and thus N T is the random number of events in [0 , T ] . The inhomogeneous Poisson process has a timevarying intensity λ ( t ) representing the instantaneous rate of event occurrence at t . Mathematically, the intensity function is defined as λ ( t ) = lim δ t → 0 E [ N t + δ t -N t ] /δ t . The probability density function of Poisson process is:

The Hawkes process [3] is a self-excitation point process where the occurrence of an event increases the likelihood of more events in the future. A trajectory from Hawkes process is similarly represented as T = ( t 1 , . . . , t N T ) on [0 , T ] . The conditional intensity function of Hawkes process, representing the instantaneous rate of event occurrence at t given the history up to but not including t , is:

where µ ( t ) is the baseline intensity, g ( · ) is the triggering kernel representing the self-excitation effect, the summation expresses the accumulative excitation from all past events, F t -is the historical information up to but not including t . λ ∗ ( t ) means the intensity is dependent on the history.

Poisson process assumes the independence of event occurrences, while Hawkes process extends it by introducing an autoregressive structure, making subsequent events dependent on prior events. Given

the history F t n = ( t 1 , . . . , t n ) , the conditional probability density of ( n +1) -th event at t > t n is:

Here, we introduce the definition of the univariate Hawkes process. However, multivariate Hawkes processes also exist. For ease of notation, we use the univariate case for illustration, but we also provide solutions for the multivariate case.

2.2 Score Matching

MLE is a classic estimator that minimizes Kullback-Leibler divergence between a model distribution and data distribution. However, a drawback is the intractable computation of the normalizing constant. Approximating it through numerical integration can be computationally demanding. In contrast, SM [5] offers an alternative by minimizing Fisher divergence between model and data distributions:

where p ( x ) represents the data distribution, p θ ( x ) is the parameterized model distribution, the gradient of the log-density is called the score, and ∥·∥ represents a suitable norm, such as the ℓ 2 norm. Minimizing the Fisher divergence above provides the parameter estimate. The advantage of SM lies in its ability to bypass the computation of the normalizing constant since the score no longer contains this constant: p θ ( x ) = 1 Z ( θ ) ˜ p θ ( x ) where Z ( θ ) = ∫ ˜ p θ ( x ) d x , ∇ x log p θ ( x ) = ∇ x log ˜ p θ ( x ) .

Under certain conditions, we can use integration by parts to replace the explicit SM objective, which involves an unknown distribution p ( x ) , with an equivalent implicit one,

2.3 Autoregressive Score Matching

An autoregressive model defines a probability density p ( x ) as a product of conditionals using the chain rule: p ( x ) = ∏ N n =1 p ( x n | x <n ) , where x n is the n -th entry and x <n denotes the entries with indices smaller than n . The original SM is not suitable for autoregressive models because the autoregressive structure introduces challenges in gradient computation in Equation (4). To address this issue, [13] proposed autoregressive score matching (ASM). Unlike SM, which minimizes the Fisher divergence between the joint distributions of the model p θ ( x ) and the data p ( x ) , ASM minimizes the Fisher divergence between the conditionals of the model p θ ( x n | x <n ) and the data p ( x n | x <n ) :

Similarly, the above explicit ASM objective involves an unknown distribution p ( x n | x <n ) . Under specific regularity conditions, we can apply integration by parts to derive an implicit ASM objective:

3 Score Matching for Poisson Process

Weanalyze the application of SM for Poisson process and its failure in achieving consistent estimation. Subsequently, we propose a provably consistent WSM estimator.

3.1 Failure of Score Matching for Poisson Process

Consider a Poisson process T = ( t 1 , . . . , t N T ) on [0 , T ] . Let p ( T ) represent the data distribution, which is uniquely associated with an intensity function λ ( t ) . Let p θ ( T ) represent the parameterized

model distribution, which is uniquely associated with a parameterized intensity function λ θ ( t ) . In the following, we denote the score as ψ ( t n ) = ∂ ∂t n log p ( T ) .

Previous works [18; 24] have both attempted to apply SM to the Poisson process:

In order for SM to be practical, [18; 24] assumed that specific regularity conditions are satisfied. Therefore, they employed an implicit SM objective similar to Equation (4):

In practical applications, we have found that the above estimator works only for specific Poisson processes and fails for more general Poisson processes. The reason for its failure lies in the fact that, for more general Poisson processes, the specific regularity conditions cannot be satisfied.

Such conditions require the probability density function of the random variable is zero when it approaches infinity in any of its dimensions. However, for point processes, such requirement is not satisfied, because the random variable in point process T = ( t 1 , . . . , t N T ) is not of fixed dimension and takes values in a subset of R N T + . Therefore, for general Poisson processes, we cannot derive the implicit SM in Equation (7) based on the explicit SM in Equation (6).

Proposition 3.1. Assume that all functions and expectations in L SM ( θ ) and J SM ( θ ) are well defined, we have,

Therefore, L SM ( θ ) is equivalent to J SM ( θ ) if and only if the sum of the last two terms is a constant not containing θ .

For specific Poisson processes, the sum of the last two terms can be zero. However, for more general cases, this sum contains θ . This implies that J SM fails for general Poisson processes.

3.2 Weighted Score Matching

To address the situation where SM fails, inspired by [6; 22], we introduce the WSM for Poisson process. The core idea of WSM is to eliminate the two intractable terms by adding a weight function that takes zero at the boundary of the integration region. The weight function is designed to be a vector-valued function h : R N T + → R N T + with the n -th element denoted as h n ( T ) . Here, we present the conditions that a valid weight function should satisfy:

One can verify that such weight functions are easy to find for most p ( T ) and ψ θ ( t n ) . With a valid weight function h , the explicit WSM objective can be defined as:

The introduction of the weight function allows control over the values of the integrand at the boundaries of the integration domain, thereby eliminating the last two terms in Equation (8).

Theorem 3.2. Assume the true intensity is in the family of the model intensity, denoted as λ ( t ) = λ θ ∗ ( t ) , where θ ∗ ∈ Θ . We further assume that ∂ log λ θ 1 ( t ) ∂t = ∂ log λ θ 2 ( t ) ∂t a.s. gives θ 1 = θ 2 . Then the unique minimizer of L WSM ( θ ) is θ ∗ .

The explicit WSM objective is not practical as it depends on the unknown data distribution p ( T ) , so we further derive the implicit WSM objective which is tractable.

Theorem 3.3. Assume that all functions and expectations in L WSM ( θ ) and J WSM ( θ ) are well defined, Equation (9) is satisfied, we have,

For general Poisson processes, Equation (11) is always valid with a suitable weight function. Thus, we do not need to worry about the issues of failure that may arise when using Equation (7).

4 Autoregressive Score Matching for Hawkes Processes

Similarly, we analyze the usage of ASM for Hawkes processes and its failure in achieving consistent estimation. Subsequently, we propose a provably consistent autoregressive WSM (AWSM) estimator.

4.1 Failure of Autoregressive Score Matching for Hawkes Process

The original SM, even when adjusted by a weight function, is not suitable for point processes with autoregressive structures, such as Hawkes process. Because in such cases, directly calculating the score still includes the intensity integral, which is precisely what the use of SM aims to avoid. Therefore, an ASM method is proposed for parameter estimation for Hawkes process in [10].

Consider a Hawkes process T = ( t 1 , . . . , t N T ) on [0 , T ] with the underlying conditional probability density of t n denoted as p ( t n |F t n -1 ) . The parameterized conditional probability density model of t n is p θ ( t n |F t n -1 ) . We denote the conditional score as ψ ( t n |F t n -1 ) = ∂ ∂t n log p ( t n |F t n -1 ) = ∂ ∂t n log λ ( t n |F t n -1 ) -λ ( t n |F t n -1 ) , n = 1 , . . . N T . An explicit ASM objective is defined as:

Similarly, to make ASM practical, [10] assumed that specific regularity conditions are satisfied. Therefore, an implicit ASM is proposed accordingly:

However, the same issue as in the Poisson process arises here. The regularity conditions required to eliminate the unknown data distribution do not hold. Therefore, we cannot derive the implicit ASM in Equation (13) based on the explicit ASM in Equation (12).

Proposition 4.1. Assume that all functions and expectations in L ASM ( θ ) and J ASM ( θ ) are well defined, we have,

Therefore, L ASM ( θ ) is equivalent to J ASM ( θ ) if and only if the sum of last two terms is a constant not containing θ .

Generally speaking, for most Hawkes processes, the sum of the last two terms in Equation (14) still contains θ , even for a common Hawkes process with an exponential decay triggering kernel. We illustrate this example in Section 6.2. This implies that J ASM fails for general Hawkes processes.

4.2 Autoregressive Weighted Score Matching

Similarly, to address the situation where ASM fails, we introduce the AWSM for Hawkes process. We present the conditions that a valid weight function h should satisfy :

With a valid weight function h , the explicit AWSM objective can be defined as:

Theorem 4.2. Assume the true conditional density is in the family of the model conditional density, denoted as p ( t n |F t n -1 ) = p θ ∗ ( t n |F t n -1 ) , where θ ∗ ∈ Θ . We further assume that p θ 1 ( t n |F t n -1 ) = p θ 2 ( t n |F t n -1 ) a.e. gives θ 1 = θ 2 . Then the unique minimizer of L AWSM ( θ ) is θ ∗ .

The explicit AWSM objective is not practical as it depends on the unknown data distribution p ( t n |F t n -1 ) , so we further derive the implicit AWSM objective which is tractable.

Theorem 4.3. Assume that all functions and expectations in L AWSM ( θ ) and J AWSM ( θ ) are well defined, Equation (15) are satisfied, we have,

For general Hawkes processes, Equation (17) is always valid with a suitable weight function. Thus, we do not need to worry about the issues of failure that may arise when using Equation (13).

Multivariate Hawkes Processes For the multivariate case, events are { ( t 1 , k 1 ) , . . . , ( t N T , k N T ) } with k n ∈ 1 , . . . , K denoting the event type of the n -th event. The history up to the ( n -1) -th event is denoted by F t n -1 . We need to consider both the distributions of event times and event types. For the temporal distribution, we use the AWSM objective with the temporal score ψ ( t n |F t n -1 ) = ∂ ∂t n log p ( t n |F t n -1 ) as before. For the type distribution, since we do not need to compute the intensity integral, we directly use the cross-entropy objective:

where λ = ∑ K k =1 λ k . The final loss is J ( θ ) = J AWSM ( θ ) + α J CE ( θ ) ; α is a balancing coefficient.

5 Theoretical Analysis

In this section, we analyze the statistical properties of AWSM estimator of univariate Hawkes process. Similar conclusions also hold for the WSM estimator of Poisson process, as discussed in Appendix C.5. We consider M i.i.d. sequences { t ( m ) 1 , . . . , t ( m ) N m } M m =1 from p ( T ) of a Hawkes process. We assume the true density is in the family of the model density, denoted as p ( T ) = p θ ∗ ( T ) , where θ ∗ ∈ Θ ⊂ R r . The estimate ˆ θ is obtained by ˆ θ = arg min θ ∈ Θ ˆ J AWSM ( θ ) where ˆ J AWSM represents the empirical loss. Below we omit the subscript AWSM as it does not cause any ambiguity.

5.1 Asymptotic Property

We first establish the consistency of ˆ θ for a Hawkes process.

Theorem 5.1. Under mild regularity Assumptions C.1 to C.3, we have ˆ θ p - → θ ∗ as M →∞ .

5.2 Non-asymptotic Error Bound

Then, we establish a non-asymptotic error bound for ˆ θ . We define

Assumption 5.2. Assume there exists α > 1 such that,

holds for any small δ . Here, C h is a positive constant that depends on the weight function h such that C a h = aC h for any positive constant a . ∥ · ∥ is the euclidean norm.

Assumption 5.3. For ∀ n ∈ N + , there exists ˙ A n ( T ) , ˙ B n ( T ) such that,

Theorem 5.4. Given that ˆ θ converges to θ ∗ in probability, combined with Assumptions 5.2 and 5.3, for δ < CK α √ r 2 α -1 Γ( h ,A,B ) C h , we have

where Γ( h , A, B ) = √ E p ( T ) { ∑ N T n =1 [ ( ˙ A n ( T ) h n ( T )) + ( ˙ B n ( T ) ∂h n ( T ) ∂t n ) ] } 2 , C is a universal constant, K α = 2 2 α 2 α -1 -1 , and r is the number of dimensions of θ .

5.3 Discussion on Optimal Weight Function

In Sections 3 and 4, we only provide the conditions that the weight function needs to satisfy. In fact, there are many weight functions that satisfy these conditions. The optimal weight function should minimize the error bound in Equation (19), which is equivalent to minimizing the coefficient Γ( h ,A,B ) C h . The numerator cannot be analytically computed as it involves an unknown distribution p ( T ) , but we can maximize the denominator C h in a predefined function family.

Theorem 5.5. Define h 0 to be a weight function with its n -th element defined as the distance between t n and the boundary of its support [ t n -1 , T ] :

where H is a family of functions that is rigorously defined in Equation (27) .

Combined with Assumption 5.2, it can be observed that h 0 maximizes C h in H . Though it does not necessarily optimize Γ( h ,A,B ) C h , it is an adequate choice without using any information on p ( T ) . We also discuss it heuristically in Appendix C.4. It is worth noting that h 0 n is not continuously differentiable; however, it is weakly differentiable. Its weak derivative is continuous, allowing both integration by parts and statistical theory to hold. In subsequent experiments, we consistently employ this optimal weight function when T is available or can be approximated for the dataset.

6 Experiments

In this section, we validate our proposed (A)WSM on parametric or deep point process models. For parametric models, we focus on verifying whether (A)WSM can accurately recover the groundtruth parameters. For deep point process models, we confirm that our new training method is also applicable to deep neural network models. 2

We have,

6.1 Baselines and Metrics

We consider three baseline parameter estimators: (1) MLE (2) implicit (A)SM [18; 24; 10] (3) Denoising Score Matching ( DSM ) [10]. We briefly introduce DSM in deep point process models.

For deep Hawkes process training, DSM is employed as follows. For observed timestamps t ( m ) n in m -th sequence, we sample L noise samples ˜ t ( m ) n,l = t ( m ) n + ϵ ( m ) n,l , l = 1 , . . . , L, where Var ( ε ( m ) n,L ) = σ 2 and get the DSM objective:

where J CE ( θ ) is the cross-entropy loss defined in Equation (18).

To compare the performance of different methods, for parametric models on synthetic data, we use the mean absolute error ( MAE , | ˆ θ -θ | ) between the ground-truth parameters and the estimates as a metric since the ground-truth parameters are known. For deep point process models, we use the test log-likelihood ( TLL ) and the event type prediction accuracy ( ACC ) on the test data as metrics.

6.2 Parametric Models

Datasets We validate the effectiveness of (A)WSM using three sets of synthetic data. (1) Poisson Process : This dataset is simulated from an inhomogeneous Poisson process with an intensity function λ ( t ) = exp( θ sin( t )) with T = 2 , θ = 2 . (2) Exponential Hawkes Processes : This dataset is simulated from 2 -variate Hawkes processes with exponential decay triggering kernels g ij ( τ ) = α ij exp( -5 τ ) , τ > 0 with T = 10 , µ 1 = µ 2 = 1 , α 11 = 1 . 6 , α 12 = 0 . 2 , α 21 = α 22 = 1 . (3) Gaussian Hawkes Processes : This dataset is simulated from 2 -variate Hawkes processes with Gaussian decay triggering kernels g ij ( τ ) = α ij √ 2 πσ exp( -τ 2 2 σ 2 ) , τ > 0 with T = 10 , µ 1 = µ 2 = 1 , α 11 = 1 . 6 , α 12 = 0 . 2 , α 21 = α 22 = 1 , σ = 1 .

Training Protocol We assume that we know the ground-truth model but do not know its parameters. Therefore, we use the ground-truth model as the training model. The purpose is to verify whether the estimator can recover the ground-truth parameters. For each dataset, we collect a total of 1000 sequences. We run 500 iterations of gradient descent using Adam [8] as the optimizer for all scenarios. For MLE, the intensity integral is computed through numerical integration, with the number of integration nodes set to 100 to achieve a considerable level of accuracy. We change the random seed 3 times to compute the mean and standard deviation of MAE.

Results In Table 1, we report the MAE of parameter estimates for three models trained by MLE, (A)SM, and (A)WSM on the synthetic dataset. We can see that both MLE and (A)WSM achieve small MAE on three types of data. However, the MAE of (A)SM is large. As we have theoretically demonstrated earlier, this is because MLE and (A)WSM estimators are consistent. In contrast, (A)SM, due to the absence of the required regularity conditions in the three cases, has an incomplete objective and cannot accurately estimate parameters. In Figure 1, we showcase the learned intensity functions. Both MLE and (A)WSM successfully captured the ground truth, while (A)SM fails.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Time

0

5

10

15

20

25

30

35

Intensity

Truth

WSM

SM

MLE

Event

(a) Poisson

0

10

20

30

40

Time

1

0

1

2

3

4

5

6

Intensity

Truth

AWSM

ASM

MLE

Event

(b) Exp-Hawkes Dim-1

0

10

20

30

40

Time

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Intensity

Truth

AWSM

ASM

MLE

Event

(c) Gau-Hawkes Dim-1

20

40

60

80

100

120

Number of Monte Carlo Samples

14.58

14.56

14.54

14.52

14.50

14.48

14.46

Average Log Likelihood

runtime: 79.4sruntime: 31.0s

runtime: 28.3s

runtime: 16.0s

MLE-ExpHawkes

Score-ExpHawkes

(d) TLL and Runtime

6.3 Deep Point Processes Models

Datasets We consider four real datasets. (1) Half-Sin Hawkes Process : This is a synthetic 2-variate Hawkes process with trigerring kernel g ij = α ij sin ( τ ) , τ ∈ (0 , π ) , K = 2 . (2) StackOverflow [7]: This dataset has two years of user awards on StackOverflow. Each user received a sequence of badges and there are K = 22 kinds of badges. (3) Retweet [25]: This dataset includes sequences indicating how each novel tweets are forwarded by other users. Retweeter categories serve as event types K = 3 . (4) Taobao [21]: This dataset comprises user activities on Taobao (in total K = 17 event types). For each dataset, we follow the default training/dev/testing split in the repository.

Training Protocol In recent years, many deep point process models have been proposed. Here, we focus on two of the most popular attention-based Hawkes process models: SAHP [23] and THP [27]. We deploy AWSM and ASM on THP and SAHP. For each dataset, we train 3 seeds with the same epochs and report the mean and standard deviation of the best TLL and ACC. When using MLE, we adopt numerical integration to calculate the intensity integral. To ensure model accuracy, the number of integration nodes is set to be large enough as we sample 10 nodes between every two adjacent events. When using DSM, we tune the variance of noise for better results. When using AWSM, since for real datasets, the true observation endpoint T is unknown. We choose the maximum event time of each batch as the observation endpoint for weight function h 0 . This may lead to unsatisfying results since real datasets may not be sampled during a unified time window. We provide a remedy for this as discussed in Appendix D.1. Details of training and testing hyperparameters are provided in Appendix D.2.

Results In Table 2, we report the performance of SAHP and THP trained using three different methods, namely MLE, AWSM, and DSM, on four datasets. It is evident from the results that models trained with MLE and AWSM exhibit very similar performance in terms of both TLL and ACC on the test data. This indicates consistency between MLE and AWSM, as they yield comparable model parameters. For DSM, it is significantly inferior to the performance of MLE and AWSM. This may result from the fact that the DSM objective is a biased estimation of the original SM objective and fails to produce consistent estimation when σ > 0 as discussed in [20]. For ASM, it completely fails in the scenarios mentioned above. It is unable to estimate the correct parameters, and its results are not reported. Generally, for complex point process models such as deep Hawkes processes, the necessary regularity conditions are not satisfied, meaning that ASM's objective is incomplete.

6.4 Advantage of (A)WSM over MLE

The key advantage of (A)WSM over MLE is its avoidance of computing intensity integrals, which can be computationally intensive for complex point process models and impact MLE accuracy. We evaluate the test log-likelihood of MLE and AWSM on the Exp-Hawkes dataset as the number of integration nodes varies. As shown in Figure 1d, with a limited number of nodes, MLE is faster but exhibits substantial estimation errors. Increasing the number of nodes reduces the error but significantly increases computation time. In this scenario, AWSM is much faster than MLE with the same accuracy, thus offering better computational efficiency.

MAE

DATASET

SAHP (TLL

↑

)

↑

)

MLE

AWSM

DSM

MLE

AWSM

DSM

HALF-SIN

1

.

542

±

0

.

038

1

.

703

±

0

.

014

0

.

804

±

0

.

353

1

.

161

±

0

.

031

1

.

271

±

0

.

036

-

0

.

385

±

0

.

033

STACKOVERFLOW

-

2

.

428

±

0

.

14

-

2

.

541

±

0

.

461

-

2

.

629

±

0

.

068

-

2

.

368

±

0

.

003

-

2

.

508

±

0

.

007

-

2

.

782

±

0

.

034

TABAO

-

1

.

050

±

0

.

100

-

1

.

373

±

0

.

091

-

1

.

911

±

0

.

049

-

1

.

052

±

0

.

012

-

0

.

948

±

0

.

004

-

1

.

791

±

0

.

040

RETWEETS

0

.

454

±

0

.

009

0

.

411

±

0

.

077

0

.

110

±

0

.

186

0

.

421

±

0

.

012

0

.

419

±

0

.

009

-

0

.

183

±

0

.

197

DATASET

SAHP (ACC

↑

)

THP (ACC

↑

)

MLE

AWSM

DSM

MLE

AWSM

DSM

HALF-SIN

0

.

502

±

0

.

001

0

.

505

±

0

.

001

0

.

501

±

0

.

001

0

.

508

±

0

.

016

0

.

523

±

0

.

010

0

.

503

±

0

.

001

STACKOVERFLOW

0

.

461

±

0

.

001

0

.

462

±

0

.

01

0

.

421

±

0

.

042

0

.

461

±

0

.

001

0

.

462

±

0

.

001

0

.

445

±

0

.

016

TABAO

0

.

572

±

0

.

022

0

.

455

±

0

.

011

0

.

421

±

0

.

017

0

.

594

±

0

.

001

0

.

592

±

0

.

002

0

.

435

±

0

.

010

RETWEETS

0

.

454

±

0

.

009

0

.

411

±

0

.

077

0

.

590

±

0

.

009

0

.

594

±

0

.

001

0

.

592

±

0

.

002

0

.

556

±

0

.

011

200

400

600

800

1000

Sample Size

0.1

0.2

0.3

Optimal

Natural

Squareroot

(a) On parameter

α

11

200

400

600

800

1000

Sample Size

0.1

0.2

0.3

MAE

Optimal

Natural

Squareroot

(b) On parameter

α

21

200

400

600

800

1000

Sample Size

0.05

0.10

0.15

MAE

Optimal

Natural

Squareroot

(c) On parameter

µ

1

6.5 Comparison Between Weights

Though we provide theoretical insight into the choice of an optimal weight function for AWSM, its validity still needs to be testified by experiments. Here, we compare the near-optimal weight h 0 with natural weight h 1 and squareroot weight h 2 satisfying Equation (15),

All three weight functions can be applied in AWSM to recover ground-truth parameters, however with different convergence rates. We carry out experiments on synthetic data for exponential-decay model with the same setting as Section 6.2 in our paper. We measure their MAE for different sample sizes in Figure 2 and find that h 0 does achieve the best results among the three weight functions.

7 Limitations

The current limitation of the methodology is that some real data are collected from multiple time intervals [0 , T 1 ] , . . . , [0 , T L ] or collated in a fixed time interval [0 , T ] with unknown T . However, for a score matching to be valid, the required weight function must involve knowledge of T . Currently, our remedy including approximate T or performing data truncation as discussed in Appendix D.1.

8 Conclusions

In conclusion, the SM estimator for point processes can overcome the challenges associated with intensity integrals in MLE. While existing works have proposed SM estimators for point processes, our investigation reveals that they prove effective only for specific problems and fall short in more general cases. To address this issue, our work introduces a novel approach: the (A)WSM estimator for point processes, offering both theoretical soundness and empirical success.

Acknowledgments and Disclosure of Funding

This work was supported by NSFC Project (No. 62106121), the MOE Project of Key Research Institute of Humanities and Social Sciences (22JJD110001), the fundamental research funds for the central universities, and the research funds of Renmin University of China (24XNKJ13).

References

Emmanuel Bacry and Jean-François Muzy. Hawkes model for price and trades high-frequency dynamics. Quantitative Finance , 14(7):1147-1166, 2014.

Feng Chen and Peter Hall. Inference for a nonstationary self-exciting point process with an application in ultra-high frequency financial data modeling. Journal of Applied Probability , 50(4):1006-1024, 2013.

Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika , 58(1):83-90, 1971.

Alan G Hawkes. Hawkes processes and their applications to finance: a review. Quantitative Finance , 18(2):193-198, 2018.

Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res. , 6:695-709, 2005.

Aapo Hyvärinen. Some extensions of score matching. Computational statistics & data analysis , 51(5):2499-2512, 2007.

Leskovec Jure. Snap datasets: Stanford large network dataset collection. Retrieved December 2021 from http://snap. stanford. edu/data , 2014.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

John Frank Charles Kingman. Poisson processes , volume 3. Clarendon Press, 1992.

Zichong Li, Yanbo Xu, Simiao Zuo, Haoming Jiang, Chao Zhang, Tuo Zhao, and Hongyuan Zha. Smurf-thp: Score matching-based uncertainty quantification for transformer hawkes process. 2023.

Scott Warren Linderman. Bayesian Methods for Discovering Structure in Neural Spike Trains . PhD thesis, Harvard University, 2016.

Song Liu, Takafumi Kanamori, and Daniel J Williams. Estimating density models with truncation boundaries using score matching. The Journal of Machine Learning Research , 23(1):84488485, 2022.

Chenlin Meng, Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Autoregressive score matching. Advances in Neural Information Processing Systems , 33:6673-6683, 2020.

Paul Milgrom and Ilya Segal. Envelope theorems for arbitrary choice sets. Econometrica , 70(2):583-601, 2002.

George Mohler. Modeling and estimation of multi-source clustering in crime and security data. The Annals of Applied Statistics , pages 1525-1539, 2013.

Yosihiko Ogata. Space-time point-process models for earthquake occurrences. Annals of the Institute of Statistical Mathematics , 50(2):379-402, 1998.

Yosihiko Ogata. Seismicity analysis through point-process modeling: A review. In Seismicity patterns, their statistical significance and physical meaning , pages 471-507. Springer, 1999.

Maneesh Sahani, Gergo Bohner, and Arne Meyer. Score-matching estimators for continuoustime point-process regression models. In Francesco A. N. Palmieri, Aurelio Uncini, Kostas I. Diamantaras, and Jan Larsen, editors, 26th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2016, Vietri sul Mare, Salerno, Italy, September 13-16, 2016 , pages 1-5. IEEE, 2016.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Siqiao Xue, Xiaoming Shi, James Zhang, and Hongyuan Mei. Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. Advances in Neural Information Processing Systems , 35:34641-34650, 2022.

Shiqing Yu, Mathias Drton, and Ali Shojaie. Generalized score matching for non-negative data. The Journal of Machine Learning Research , 20(1):2779-2848, 2019.

Qiang Zhang, Aldo Lipani, Ömer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 11183-11193. PMLR, 2020.

Yixuan Zhang, Quyu Kong, and Feng Zhou. Integration-free training for spatio-temporal multimodal covariate deep kernel point processes. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Qingyuan Zhao, Murat A Erdogdu, Hera Y He, Anand Rajaraman, and Jure Leskovec. Seismic: A self-exciting point process model for predicting tweet popularity. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1513-1522, 2015.

Feng Zhou, Quyu Kong, Zhijie Deng, Jichao Kan, Yixuan Zhang, Cheng Feng, and Jun Zhu. Efficient inference for dynamic flexible interactions of neural populations. Journal of Machine Learning Research , 23(211):1-49, 2022.

Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 11692-11702. PMLR, 2020.

A Proof of Results in Section 3

A.1 Proof of Proposition 3.1

Proof. First consider the cross term in L SM ( θ ) and expand the expectation,

The third equation uses an integral-by-part trick. All integrations above are taken within the area of { 0 ≤ t 1 ≤ . . . ≤ t N ≤ T } or { 0 ≤ t 1 ≤ . . . ≤ t n -1 ≤ t n +1 ≤ . . . ≤ t N ≤ T } when t n has been integrated. For the first term in the right side of the last equation, notice that,

Therefore, we can see that, for n ∈ { 2 , . . . , n } ,

Using the above equation, we manage to cancel out most of the terms being summed in the right side of the thrid equation in Equation (20) and only leave the first and last term, which completes the proof.

A.2 Proof of Theorem 3.2

Proof. First, since L WSM ( θ ) ≥ 0 and L WSM ( θ ∗ ) = 0 , we see θ ∗ is a minimizer. If there exists another minimizer θ 1 , then we have

By the definition of h ( T ) , for any N , since h ( T ) > 0 a.s. elementwisely on { 0 ≤ t 1 ≤ . . . ≤ t N ≤ T } . So L WSM ( θ ) = 0 implies ∂ log λ θ ∗ ( t ) ∂t = ∂ log λ θ 1 ( t ) ∂t a.e. on [0 , T ] . By assumption, this implies θ 1 = θ ∗ , which completes the proof.

A.3 Proof of Theorem 3.3

Proof. The proof is basically the same as the proof of Proposition 3.1. We first expand the expectation and consider the cross-term in L WSM ( θ ) ,

We denote t N +1 = T and t 0 = 0 here. Using the first two equations in Equation (9), we have:

Therefore, the first intractable summation term in Equation (22) will disappear, and the second term equals -E p ( T ) [ ∑ N T n =1 ∂ ∂t n ψ θ ( t n ) h n ( T ) + ψ θ ( t n ) ∂ ∂t n h n ( T ) ] . The existence of such an expectation is due to the last two equations in Equation (9). Therefore, we complete the proof.

We can see from the proof that, in Equation (9), the first two equations ensure that the integration by parts trick does not produce an intractable term, and the last two equations are simply regularity conditions that ensure all terms are well-defined.

B Proof of Results in Section 4

Lemma B.1. Let f ( t n , F t n -1 ) be a function of t n , F t n -1 , where n ∈ { 1 , . . . , N T } . Then we have

where p ( t 1 , . . . , t n ) is the density of observing these timestamps, the integration is taken over { 0 ≤ t 1 ≤ . . . ≤ t n ≤ T } .

Proof. We first expand the expectation and obtain,

At this point, we need to first integrate out t n +1 , . . . , t N , which uses

Plug this in and we obtain,

The thrid equation adopts the exchange of summation. The feasibility is ensured by the assumption that the expectation in the left side of the equation exists and Fubini's theorem. The fourth equation use the fact that ∑ ∞ N = n p ( N T = N |F t n ) = 1 .

B.1 Proof of Proposition 4.1

Proof. We use Lemma B.1 to the cross term of L ASM ( θ ) and obtain,

For the fifth equation, we simply rearrange the terms. We recall that the notation p ( T :0 ) equals one. For the last equation, we use Lemma B.1 again. This will be sufficient to complete the proof.

B.2 Proof of Theorem 4.2

Proof. First, since L AWSM ( θ ) ≥ 0 and L AWSM ( θ ∗ ) = 0 , we see θ ∗ is a minimizer. If there exists another minimizer θ 1 , then we have

By the definition of h n ( T ) , we have h n ( T ) > 0 , a.s. on 0 ≤ t 1 ≤ . . . , ≤ t N ≤ T for any n ≤ N and N ∈ N + . This implies, for any n ≤ N and N ∈ N + , ψ θ ∗ ( t n |F t n -1 ) = ψ θ 1 ( t n |F t n -1 ) , a.e. on { 0 ≤ t n -1 ≤ t n ≤ T } . Therefore we have

And we conclude that C = 0 since ∫ T t n -1 p θ ( t n |F t n -1 ) dt n = 1 . Then by assumption, we have p θ ∗ ( t n |F t n -1 ) = p θ 1 ( t n |F t n -1 ) a.e. ⇒ θ 1 = θ ∗ .

B.3 Proof of Theorem 4.3

The proof is basically the same as the proof of Proposition 4.1. We first use the Lemma B.1 to the cross term of L AWSM ( θ ) ,

Between the second and the third line above, we omit the steps used in the derivation of Proposition 4.1 to make it concise. For the term in the third line above, it will be eliminated using Equation (15). For the term in the fourth line above, using Lemma B.1, we have:

The existence of the expectation is ensured by the last two terms in Equation (15).

C Proof of Results in Section 5

We present all the regularity condition needed for establishing the consistency of our estimator:

Assumption C.1. The parameter space Θ is a compact set in R d and contains an open set which contains θ ∗ .

Assumption C.2. Both the µ θ ( t ) and g θ ( t ) are twice continuously differentiable w.r.t. t for all θ ∈ Θ and those derivatives are continuous w.r.t. θ .

We remind readers that µ θ ( t ) and g θ ( t ) are the mean intensity function and the triggering kernel for a Hawkes process, first defined in Equation (2).

Assumption C.3. If both µ θ 1 ( t ) = µ θ 2 ( t ) a.s. and g θ 1 ( t ) = g θ 2 ( t ) a.s. on [0 , T ] , then θ 1 = θ 2 .

C.1 Proof of Theorem 5.1

Proof. As shown in Theorem 5.7 in [19], if we can prove a uniform in probability convergence for ˆ J AWSM ( θ ) to J AWSM ( θ ) , then using the fact that θ ∗ is a unique minimizer of J WSM ( θ ) in a compact set in R d , the consistency is proved. Therefore, we only prove the uniform in probability convergence.

For any θ ∈ Θ , for each sampled sequence ( t ( m ) 1 , . . . , t ( m ) N m ) , we perceive the following value as a random variable,

One can verify that it is a measurable map from the sample space to the real line, therefore indeed a random variable. And its expectation is J AWSM ( θ ) , which is finite by assumption. Since different sequences are i.i.d. samples with finite expectation, using the weak law of large numbers, we have:

Now we prove that this convergence is uniform in Θ . Similar to [2], we first prove that

First, by Assumption C.2, we know λ θ ( t n |F t n -1 and ∂ ∂t n λ θ ( t n |F t n -1 are continuous w.r.t θ . Therefore ψ θ ( t n |F t n -1 ) = ∂ ∂t n log λ θ ( t n |F n -1 ) -λ θ ( t n |F t n -1 ) and ∂ ∂t n ψ θ ( t n |F t n -1 ) = ∂ 2 ∂t 2 n log λ θ ( t n |F t n -1 ) -∂ ∂t n λ θ ( t n |F t n -1 ) are both continuous w.r.t. θ , therefore the J AWSM ( θ ) is a continuous function of θ . Since Θ is a compact set in R d , we know J AWSM ( θ ) is uniform continuous.

Therefore, we can bound |J AWSM ( θ 1 ) -J AWSM ( θ 2 ) | using ∥ θ 1 -θ 2 ∥ . Using the result in Equation (23), we know that for any ε > 0 , we can find a uniform δ so that |J AWSM ( θ 1 ) -J AWSM ( θ 2 ) | < 1 6 ε, ∀|| θ 1 -θ 2 || < δ . So that,

Now we follow exactly the same steps as [2] for the uniform in probability convergence. Since Equation (24) hold, for such a δ in that equation, since Θ is a compact set in R d , there exists a finite number of open balls with radius δ whose union covers Θ . Let ϑ 1 , . . . , ϑ i , . . . , ϑ L denote the centers of these balls. We denote ϑ i ( θ ) the center of a ball which contains θ . Since we have

The third term on the right equals 0 because of its definition and the uniform continuous of J AWSM ( θ ) . The first term converges to 0 , M →∞ by Equation (24). For the second term, we write

Finally we obtain sup θ ∈ Θ | ˆ J AWSM ( θ ) -J AWSM ( θ ) | p - → 0 , which completes the proof.

C.2 Proof of Theorem 5.4

First we restate the technical assumptions needed for the proof.

Assumption 5.2. Assume there exists α > 1 such that,

holds for any small δ . Here, C h is a positive constant that depends on the weight function h such that C a h = aC h for any positive constant a .

For this assumption, we assume that the optimal parameter θ ∗ is well-separated from other neighbouring parameters in terms of the population objective values. This is a standard assumption for Theorem 5.52 in [19]. This assumption will satisfy if ∇ 2 θ J h ( θ ) is positive definite at θ ∗ .

Assumption 5.3. For ∀ n ∈ N + , there exists ˙ A n ( T ) , ˙ B n ( T ) such that,

For this assumption, we assume the objective function has certain level of continuity. This assumption will realize if sup θ ∈ Θ ||∇ θ A n ( T , θ ) || exisits for any n and the same condition also applies to B n ( T , θ ) .

We also define,

we see E [ J h ( T ; θ )] = J h ( θ ) .

Now we begin the proof of Theorem 5.4.

Proof. First define

Next we evaluate the bracketing number of such function class,

a

Denote N [] ( ε, F , L 2 ( P )) as the bracketing number of F with the radius ε under the norm of L 2 ( P ) . P is the underlying probability measure induced by T . Use Example 19.6 in [19] we have,

Since therefore ˙ J h ( T ) δ is the envelope of F δ , we denote it as F δ . Therefore we obtain,

After obtaining this quantity, the next step is to upper bound the empirical process defined as,

Using Corollary 19.35 in [19], since we have || ˙ J h || L 2 ( P ) = Γ( h , A, B ) < ∞ , then,

where C and C ′ are universal constant and J [] ( · , · , · ) is the entropy integral.

Finally, given that ˆ θ converges to θ ∗ in probability, combined with Assumption 5.2 and Equation (26), using Theorem 5.52 in [19] we have, for δ < CK α √ r 2 α -1 Γ( h ,A,B ) C h , we have,

where C is a universal constant and K α = 2 2 α 2 α -1 -1 .

C.3 Proof of Theorem 5.5

First we give a rigorous definition of H ,

In other words, H contains functions whose component function h n is 1 -Lipschitz continuous w.r.t. its last dimension t n . It's easy to verfity h 0 ∈ H .

Proof. First we prove that,

Notice that,

Notice that, in H , for any z = t -n -1 or z = T ,

Therefore,

and equation 28 is proved.

Finally, we have,

The second equality is due to Equation (28) and L h ( θ ∗ ) = 0 . The inequality is due to max-min inequality.

C.4 Continued Discussion on Optimal Weight Function

Given h 0 maximizes C h , it is still unclear whether it is still preferable considering Γ( h , A, B ) . This is indeed a tough question that we do not yet have a satisfying answer. For specific parametric models, ˙ A n ( T ) , ˙ B n ( T ) can be computed analytically (see line 478-480) and then Γ( h , A, B ) can be computed via Monte Carlo. Then we can study how sensitive Γ( h , A, B ) is to h . For general models, especially when ψ θ is a deep neural network like THP or SAHP, Γ( h , A, B ) is intractable to compute.

However, heuristically speaking, our choice of near-optimal weight function h 0 should be a good choice even concerning Γ( h , A, B ) . To make Γ( h , A, B ) small, a natural idea is to make | h n ( T ) | and | ∂ ∂t n h n ( T ) | small. The weight function we chose and its derivative have relatively positive low powers with respect to t n , therefore making | h 0 n ( T ) | and | ∂ ∂t n h 0 n ( T ) | small. For weight functions h 1 n ( T ) = ( T -t n )( t n -t n -1 ) , the power w.r.t. t n is two. And for h 2 n ( T ) = √ ( T -t n )( t n -t n -1 ) , its derivative is ∂ ∂t n h 2 n ( T ) = T -t n -( t n -t n -1 ) √ ( T -t n )( t n -t n -1 ) , the numerator is usually a bounded quantity and the denominator may be close to zero, making its derivative large. In conclusion, h 0 is a better choice compared with h 1 or h 2 concerning Γ( h , A, B ) .

C.5 Discussion on Poisson Process

For Poisson process, results in Section 5 also holds, including the consistency and the convergence rate. For the choice of weight function, a reasonable choice is still the distance function presented below.

Consider a Poisson process, since the support of T is V = {T ∈ N T , 0 ≤ t 1 ≤ . . . ≤ t N T ≤ T } . Define A ∈ R ( N T +1) × N T to be a coefficient matrix used below as,

Denote a n ∈ R N t as the n -th row vector of A . Let b n = 0 , n ∈ [ N T ] , b N T +1 = -T . Then V is a convex Polytope which can be rewritten as,

Therefore, the distance between T and ∂V is ,

Let h 0 n ( T ) = dist ( T , ∂V ) , h 0 ( T ) = ( h 0 n ( T ) , h 0 n ( T ) , . . . , h 0 n ( T )) T . Again, h 0 ( T ) is only weak derivative. The envelope theorem in [14] yields,

wehre n ∗ is the is the minimizer of the last minimization in Equation (29).

Similar arguments as in Section 5.3 also applies to Poisson process, with the distance weight function defined above. Such a weight function also maximizes the denominator for the convergence rate of the Poisson process. However, in the experiment, we adtops another weight function for easier implementation defined as h 1 ( T ) with its n -th component h 1 n ( T ) = ( T -t n )( t n -t n -1 ) .

D Additional Experimental Details

In this section, we present some experimental details. First we discuss our modification to the original AWSMfor when T for a dataset is not accessible. Then we provide addtional estimation results for the parametric model and hyperparameters for results in Table 2.

D.1 Weight for Equal Length Sequences

In this paper, we consider the setting of the temporal point process in a fixed time interval [0 , T ] consistently. Technically speaking, this poses extra difficulty since the dimension(number of events) would also be random. Therefore, the realized point processes should have a random length in this setting, which is the case for our dataset. However, in real data collection, different sequences may be sampled from different lengths of time interval lengths, and our proposed weight may fail under such a case, resulting in inconsistent estimation. In such a case, we propose to truncate the sequences to be of same length, and introduce the weight function for fixed-length temporal point processes.

We focus on AWSM for Hawkes process. A fixed N -length Hawkes process could be considered as a N dimensional random variable with density function as,

Since the number of observations is fixed, we could simply treat it as an N dimensional random variable and perform autoregressive weighted score matching. The valid weight function in this case should satisfy h n ( T ) | t n = t n -1 = h n ( T ) | t n = t n +1 = 0 . Similar nonasymptotic bounds could be derived here and a near optimal weight function would be the Euclidean distance between t n and the boundary of t n 's support, which would be { t n -1 , t n +1 } . Therefore, the near-optimal weight function would be h 0 n ( T ) = min { t n -t n -1 , t n +1 -t n } .

During experiments, when T is unknown and cannot be approximated well, we perform a data truncation for each batch. We naively drop all timestamps beyond the length of the shortest sequence to make length consistent within a batch. Then we perform an AWSM for the fixed-length Hawkes process. We specify the results with this modification when we show our hyperparameters.

D.2 Addtional Results and Hyperparameters

For parametric models, Table 3 provide the estimation results for some parameters and Figure 3 shows the learned intensity functions on the 2-nd dimension for the 2-variate Hawkes processes.

For deep point process experiments, we run 4 datasets with 3 methods deployed on 2 models. We show the hyperparameters of those experiments in Table 4. Epochs column represents the number

0

10

20

30

40

Time

1

0

1

2

3

4

5

Intensity

Truth

AWSM

ASM

MLE

Event

(a) Exp-Hawkes Dim-2

0

10

20

30

40

Time

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Intensity

Truth

AWSM

ASM

MLE

Event

of epochs for the experiments of a model (THP or SAHP) on a dataset. We train same epochs for three training methods (MLE, AWSM or DSM) and validate every 10 epochs to report the best result. When training MLE, the hyperparameter is the number of integral nodes, which is always 10 for all experiments. When training AWSM, we have two hyperparameters, the value of balancing coefficient for CE loss, shown in column α AWSM and whether data truncation is performed as shown in column Trunc . T represents performing data truncation and F represents no data truncation. For DSM, the hyperparameters are balancing coefficient denoted as α DSM and variance of noise denoted as σ DSM.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to abstract and introduction of this paper, where we list main contributions verified in following theory and experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our proposed method in the Limitations section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide comprehensive proof in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all the information needed to reproduce the main experimental results. Please refer to the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data is attached in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The brief instruction of train/test, hyperparameters, and the related training details are provided in the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experimental results reported in the tables includes the mean and standard deviation for multiple runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We list the necessary information about computer resource in the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have received NeurIPS Code of Ethics and conform with it in every respect. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper about assets, such as dataset, baseline and so on.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please refer to supplementation material, where we provide the details of code. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEDIT: I have changed my score from a \"weak accept\" to an \"accept.\"\n\nThe paper studies the potential use of score matching (SM) to do inference\nwith the model of interest is a Poisson point process or Hawkes\nprocesses. Because using SM in practice requires modifying the objective\nwith integration by parts, the paper shows for many common Poisson and\nHawkes processes a core assumption to make this modification possbile does\nnot hold. In order to remedy the situation, the paper proposes a random weighted\nversion of SM called WSM (along with an analog for Autoregressive Score\nMatching (ASM) called WASM), which reestablishes the validity of integration\nby parts for the aforementioned processes. The paper argues theoretically\nthat their new estimators are consistent, and it provides empirical\nanalyses to demonstrate that the new estimator outperforms the non-weighted\nvariant and matches the performance of alternatives. It also makes an\nargument to select a weighting function based on some set of assumptions.\n\nSTRENGTHS:\nThe paper points out a fundamental issue with trying to use SM for general\nPoisson point processes and Hawkes processes, and in addition, it recommends\na weighting scheme that patches up this problem based on some reasonable\nassumptions. This is a valuable observation, and the proposed remedy looks\nreasonable. There is ample empirical work to suggest that their tweak on SM\nand ASM does agree with the maximum likelihood estimator (MLE). The theory\nprovided in the paper also argues that their weighted estimator is\nconsistent if the model is well-specified.\n\nOriginality:\n\n[+]\n\n+ The authors have applied a weighting scheme to SM and ASM that permits them\nto use integration by parts for the standard objective and prove the\nconsistency of their new estimator under suitable regularity conditions.\n\nQuality:\n\n[+]\n\n+ The empirical work is thorough and does a good job checking the accuracy\n  of their new estimator.\n\nClarity:\n\n[+]\n\n+ The presentation of the theory and empirical work is very neat and\n  polished.\n+ Assumptions are clearly laid out to ensure consistency of the new estimator.\n\nSignificance:\n\n[+]\n\n+ The new estimator provides the ability to do consistent semiparametric\n  estimation for Poisson point and Hawkes processes with SM (instead of\n  MLE), which provides a new tool for very large dimensional problems.\n\nWEAKNESSES:\nThere are a couple critiques of the paper:\n\n- While the idea of using a weighting function to enable integration by\n  parts is a good one, it has been seen in other fields before\n  (e.g. weighting Stein kernels in compact settings), and hence is not\n  completely novel.\n- The argument in Section 5 outlining a weighting function is a great\n  addition to the paper, but given that it is only optimizing $C_h$ and not\n  $\\Gamma(h, A, B)$, it is hard to know how close it is to the optimum.\n\nOriginality:\n\n[-]\n\n- The only novel insight of the paper is that the trick of integration by parts,\n  so often using in score matching, is not possible for general point\n  processes, but it can be recovered by weighting the Fisher divergence a bit\n  differently. It is not clear if this idea alone is sufficient for publication.\n\nQuality:\n\n[-]\n\n- The argument to choose their weighting function $h$ is definitely better\n  than simply picking one. That being said, it still not clear how close to\n  optimal their choice of $h$ is given they are only optimizing one term in\n  the objective ($C_h$). The paper would be a bit stronger if they could\n  include a different (natural) weighting function to show their new choice\n  is an improvement.\n\nClarity:\n\n[-]\n\n- Equation 1: Can we at least have a sentence explaining the paper will be\n  slightly abusing notation by referring to $p(T)$ as the conditional\n  distribution of $p$ given $N_t = |T|$?\n\nQUESTIONS:\n[Q1] Given the choice of $h$ optimizes $C_h$ but ignores $\\Gamma(h, A, B)$,\nhow sensitive is $\\Gamma$ to the choice of $h$? This could be demonstrated\nempirically or theoretically, but it would be good to have some analysis of\nthis to argue why this weighting function is decent.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the use of score matching for estimation in point process models, and proves the incompleteness in the original score matching estimators due to the bounded support. Weighted score matching is use to address the issue and theoretical results are establish ed for the consistency and optimality of the proposed estimator. Experiments on different data demonstrate the effectiveness of the proposed estimator which yields results comparable to maximum likelihood estimation.\n\nSTRENGTHS:\n1. This paper theoretically shows the limitation in the use of original score matching estimators for point processes, pointing out their incompleteness and provide a solution.\n2. Theoretical analysis is given to support the claim and the convergence of the proposed method.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. As acknowledged in the limitation section in the paper, existing approaches that adopt denoising score matching on the point process is not considered. Even with theoretical gaurantee, experimentally, such baselines would be necessary to include in order to fully demonstrate the superiority of the method.\n2. For Hawkes processes, the problem cause by the bounded support can be solved by applying log-normalization to transform the bounded temporal domain into an unbounded one [1]. One could achieve this by removing the right bound and transform the $(0,+\\infty)$ to $(-\\infty,+\\infty)$. Although the real data is observed within finite window, the effect of removing the right bound should be negligible as it only affect the last event.\n3. Including more metrics on event prediction, such as MSE of event time, would be helpful, as the practical use of point process model is for prediction.\n\n[1] Lin, H., Wu, L., Zhao, G., Liu, P., & Li, S. (2022). Exploring generative neural temporal point process. Transactions on Machine Learning Research.\n\nQUESTIONS:\n* Did the author observe any instability during the score matching training?\n\nLIMITATIONS:\nIncluded in weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of utilizing score-matching approaches for point processes. The main motivations for the paper are the Poisson and Hawkes processes. Both of these model the repeated occurrences of events over a finite time interval $[0, T]$. The Poisson process models the occurrence of an event within a fixed infinitesimal interval with a fixed homogenous rate $\\lambda$ while a Hawkes process is a self-excitation process where past events increase the likelihood of future ones. Score-matching is a natural approach in settings where the computation of the normalizing constant of a parameterized model are challenging to compute. However, to ensure traceability, one utilizes an implicit approach where the score matching objective is approximated by a tractable alternative which may be easily computed through the derivatives of the approximating model. \n\nThe main contribution of this work is the identification of simple scenarios, even for the simple setting of the generalized Poisson process, where the score function optimization function of prior approaches fails as the regularity condition underlying them does not apply. The paper derives an expression showcasing the bias (Proposition 3.1) and is related to the evaluation of the estimated score function (for fixed number of occurrences) at the end points of the interval of interest ($t = 0$ and $t = T$). The paper then proposes an alternative where a weight function $h$ is used to augment the score function loss which ensures that these biasing terms evaluate to $0$. Furthermore, these functions may be chosen independently of the parameters of the underlying model. In their experiments, it is shown that the use of such functions still satisfies desirable properties such as the optima corresponding to the true values of the parameters while also allowing for a tractable score function formulation which avoids the bias incurred by prior approaches. This is a natural approach which suggests an avenue towards improving the performance of such models.\n\nOverall, the problem considered in the paper and the solution they identify are interesting and likely relevant to the NeurIPS community. However, I do have concerns regarding the writing of the paper. For example, there are several assumptions on the weight function in Equation 9 but it is not clear which properties are used in the proof of Proposition 3.3. For instance, Line 400 features an explicit expression for $h_1$ and $h_N$ while the expression in Equation 9 features none of these. It is my understanding that the assumptions in Equation 9 ensure that the biasing terms equate to $0$ for these elements. Furthermore, I believe that under the assumptions in Equation 9, both terms appearing in the display between Lines 399 and 400 are $0$. Please clarify whether this is in fact the case. The same issues persist for the proofs of the results in Section 4. I would like clarification from the authors on these points before updating my review.\n\n************************************************************************************************************************************************\n\nI thank the authors for their response and have amended my review accordingly.\n\nSTRENGTHS:\nSee main review\n\nWEAKNESSES:\nSee main review\n\nQUESTIONS:\nSee main review\n\nLIMITATIONS:\nSee main review", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}