EarthquakeNPP: A Benchmark for Earthquake Forecasting with Neural Point Processes
Authors: Samuel Stockman
Daniel John Lawson
Maximilian J. Werner
Venue: Accepted by TMLR ()

Abstract: For decades, classical point process models, such as the epidemic-type aftershock sequence (ETAS) model, have been widely used for forecasting the event times and locations of earthquakes. Recent advances have led to Neural Point Processes (NPPs), which promise greater flexibility and improvements over such classical models. However, the currently-used benchmark for NPPs does not represent an up-to-date challenge in the seismological community, since it contains data leakage and omits the largest earthquake sequence from the region. Additionally, initial earthquake forecasting benchmarks fail to compare NPPs with state-of-the-art forecasting models commonly used in seismology. To address these gaps, we introduce EarthquakeNPP: a benchmarking platform that curates and standardizes existing public resources: globally available earthquake catalogs, the ETAS model, and evaluation protocols from the seismology community. The datasets cover a range of small to large target regions within California, dating from 1971 to 2021, and include different methodologies for dataset generation. Benchmarking experiments, using both log-likelihood and generative evaluation metrics widely recognised in seismology, show that none of the five NPPs tested outperform ETAS. These findings suggest that current NPP implementations are not yet suitable for practical earthquake forecasting. Nonetheless, EarthquakeNPP provides a platform to foster future collaboration between the seismology and machine learning communities.

Published in Transactions on Machine Learning Research (03/2026)
EarthquakeNPP: A Benchmark for Earthquake Forecasting with Neural Point Processes
Samuel Stockman
kam.stockmannBbritto..cco
School of Earth Sciences University of Bristol, UK
Daniel Lawson
www.laousnn@bristool.co
School of Mathematics University of Bristol, UK
Maximilian Werner School of Earth Sciences University of Bristol, UK
mad..uerner&@bristtooocc
Reviewed on OpenReview: https: // openreview. net/ net/forum? forum? id= dIcNAA622Z
Abstract
For decades, classical point process models, such as the epidemic-type aftershock sequence (ETAS) model, have been widely used for forecasting the event times and locations of earthquakes. Recent advances have led to Neural Point Processes (NPPs), which promise greater flexibility and improvements over such classical models. However, the currently- used benchmark for NPPs does not represent an up-to-date challenge in the seismological community, since it contains data leakage and omits the largest earthquake sequence from the region. Additionally, initial carthquake forecasting benchmarks fail to compare NPPs with state-of-the-art forecasting models commonly used in seismology. To address these gaps, we introduce EarthquakeNPP: benchmarking platform that curates and standardizes existing public resources: globally available earthquake catalogs, the ETAS model, and evaluation protocols from the seismology community. The datasets cover range of small to large target. regions within California, dating from 1971 2021, and include different methodologies for dataset generation. Benchmarking experiments, using both log- likelihood and generative evaluation metrics widely recognised in scismology, show that none of the five NPPs tested outperform ETAS. These findings suggest that current NPP implementations are not yet suitable for practical earthquake forecasting. Nonetheless, EarthquakeNPP provides platform to foster future collaboration between the seismology and machine learning communities.
1 Introduction
Operational earthquake forecasting by global governmental organisations such the US Geological Survey (USGS) necessitates the development of models which can forccast the times and locations of damaging earthquakes. While model development is ongoing in the seismology community, recent progress has relied upon refinement of a apatio-temporal point process model known as the Epidemic-Type Aftershock Sequence (ETAS) model (Ogatal 1988 1998). This continued reliance on low-dimensional parametric framework
stands in contrast the substantial growth in available earthquake data Takanami al., 2003 Shelly
2017
Ross et al. 2019 White et al. 2019 Mousavi al. 2020
Tan al. 2021 Mousavi & Beroza, 2023

---

Published in Transactions on Machine Learning Research (03/2026)
In contrast, the machine learning community has offered promising advancements over classical point process models like ETAS with Neural Point Process (NPP) models, showcasing greater flexibility (Du et al. 2016:
Omi et al.) 2019a; Shhchur et al. 2019 Jia & Benson, 2019 Chen et al.) 2021 Zhou et al. 2022 Zhou &
Yu 2024). While some initial benchmarking of these models has boen conducted on an earthquake dataset
in Japan, these experiments lack relevance for stakeholders in the seismology community. The benchmark
omits the largest earthquake sequence from the region, introduces data leakage with non-sequential train-test splits, and does not compare against state-oftthe-ar models like ETAS.
Here, we introduce EarthquakeNPP:: curated collection of datasets designed for benchmarking NPP models in earthquake forecasting, accompanied by statc-of-the-art benchmark model. These datasets are derived from publicly available raw data, which we process and configure within our platform to facilitate meaningful forecasting experiments relevant to stakeholders in the seismology community. Covering various regions of California, these datasets represent typical forecasting zones and encompass data commonly utilized by forecast issuers. Morcover, employing modern techniques, some datasets include smaller magnitude earthquakes, exploring the potential of numerous small events to enhance forecasting performance through flexible NPPs. To unify efforts, we present operational-level implementation of the ETAS model alongside the datascts, serving as the benchmark for NPPs.
Although initial benchmarking shows that none of the five tested NPP implementations outperform ETAS, NaPthquakeNPP designed to support ongoing model development and evaluation. In addition to the standard log-likeling ligelikel metric common in the NPP literature, the platform incorporates the generative eval- uation procedures used in seismology for more rigorous benchmarking. This ensures that future NPPs (and other models such time series approaches [Wang et al. 2017) and Bayesian point processes (Serafini et al., 2023 ) can have direct relevance to seismological stakeholders. All datasets, experiments, and documentation are available at ottppp/s//ggghoo..o/////////////
ggtbbb.ttuu.... com wot.///s1889//arrtaaaaaacccaeeee
1.1
Related Work
1.1.1 Benchmarking by the NPP Community
Chen et al. (2021) introduced an carthquake dataset for benchmarking the Neural Spatio-temporal Point Process (NSTPP) model using a global dataset from the U.S. Geological Survey, focusing on Japan from 1990 to 2020. They considered carthquakes with magnitudes above 2.5, splitting the data into month-long segments with a 7-day offset. They exclude earthquakes from November 2010 to December 2011, deeming these sequences 'too long" and 'outliers'. However, this period includes the 2011 Tohoku earthquake [Mori et al. 2011), the largest earthquake recorded in Japan and the fourth largest in the world, at magnitude 9.0. This exclusion renders the benchmarking experiment irrelevant for seismologists, as is preciscly these large earthquakes and their aftershocks that are crucial to forecast due to their damaging impact.
The dataset is partitioned into training, testing, and validation segments. Rather than following a chrono- logical split that would reflect operational forccasting, the segments are assigned in alternating pattern. This design introduces data leakage it misrepresents a realistic forecasting setup and artificially inflates performance measures due to the nature of earthquake triggering (Freed 2005). Specifically, because the model is evaluated on windows that immediately precede its training windows, it can exploit backward-in- time cansal dependencies. Section B.2 quantifies the resulting performance inflation, expressed in terms of information gain.
Although earthquakes with magnitudes above 2.5 are considered by Chen al. (202)), following a change in USGS policy global data collection, from 2009 onwards, only events above magnitude 4.0 are recorded in the dataset. For carthquake forecasting in Japan, scismologists use datasets from Japanese data centers since they are more comprehensive and complete than global datasets. Section describes the biases incurred from such data missingness.
Chen et al. 2021) benchmark their model against another spatio-temporal model, Neural Jump SDEs (Jia & Benson 2019) and a temporal-only Hawkes process, even though spatio-temporal Hawkes process would provide more rigorous benchmark. Subsequent papers adopting this benchmark (Zhou et al. 2022
Yuan
2

---

Published in Transactions on Machine Learning Research (03/2026)
et al 2023
Zhou & Yu 2024) similarly lack comparisons to spatio-temporal Hawkes process, benchmarking
instead against temporal-only or spatial-only baselines or other spatio-temporal NPPs.
1.1.2
Benchmarking by the Seismology Community.
Model comparison has been crucial in the development of carthquake forecasting modcls since their inception
Kagan & Knopoll 1987 Ogala 1988) The Collaboratory for the Sludy of Earthquake Predictability
CSEP) (Michael & Werner 2018 Schorlemmer et al.) 2018 [aavaan et al] 2022 Iturrieta et al. 2024) org/ aims to unify the framework for earthquake model testing and evaluation, hosting retrospective and fully prospective forecasting experiments globally. CSEP benchmarks short-term models using performance metries that require forecasts to be generated by simulating many repeal sequences over a specified time horizon (typically one day). These simulated forecasts are compared by discretizing time and space intervals, with test statistics calculated for event counts, magnitudes, locations, and times. The simulation- based approach allows the inclusion of generative models that do not output explicit earthquake probabilities (i.e., likelihood), and enables evaluation of the full distribution of entire sampled sequences.
lhttps://csepteee
Two exisling works benchmark NPPs for earthquake forecasting within the seismology community. The first by Dascher- Cousineau et al. (2023) extends a temporal-only NPP from [Shchur et al. (2019) to include carthquake magnitudes. The second by Stockmaa et al. (2023) extends another temporal-only model by Omi al. (2019a) to target larger magnitude events. Both models are benchmarked against. a temporal ETAS model, showing moderate improvements over the baseline. Extending these models to include spatial data is necessary for further testing and potential operational use in the seismological community.
2 Background
2.1 Spatio-Tempore Point Processes
A spatio-temporal point process is a continuous- time stochastic process that models the random number of events N(S x (f), which occur in a space-time interval S x (tarts), S CRR (ta, tol C RR. This process is typically defined by non-negative conditional intensity function
[[N At) B(x, Ax)
X(t,x(H())
lim ÁI,Ax
(1)
At B(x, Ax)
where H. [14, x:) < 1) denotes the history of events preceding time I and [B(x, Ax)] is the Lebesgue measure of the ball Bxx, Ax) with radius Ax. Given we observe a history of events up t,, the probability density function (pdf) of observing an event at time and location x is given by
plt, x HH,) = x(H,x - exp (-f. M(s, z HH (ddds
(2)
Most models specify the conditional intensity function, though some (c.g. Shehur et al. 2019 Chen et al. 2021 Yuan et al. 2023) directly model this pdf. Model parameters are typically estimated by maximizing the log- ligliioll of observed events within a training time interval [To, Til and spatial region S,
logp(H(r)= = Σ log ((t.)H))) - [na 2.5.
2( dzdds + log fxxl(ti, H4.),
(3)
Temporal ligeliiel
Spatial log- likelhkodoo
where the decomposition of the spatio-temppral conditional intensity function, X(t,,x,HHa (x,H), A(t,/Hl,)) f (x,lt,, H,,,, allows the log-likelihood to be written as contributions from the temporal and spatial compo- nents. In practice, this exact function is often not maximized directly during training: for models specified through the conditional intensity function, an analytical solution to the integral term is generally not possible and is approximated numerically.
For model evaluation and comparison, the likelhooddo of observing events in the test set can be used as a performance metric. This is consistent with wealth of literature in the seismology community (see Zechar
3

---

Published in Transactions on Machine Learning Research (03/2026)
et al. 2010 and references therein) as well as the wider general point process literature (Daley & Vere-Jones, 2004), which now includes neural point processes (Shchur et al. 2021). The metric evaluates models that output probability distributions over their predictions and consequently penalises models that are overcon- fident. Although evaluating on events in the test set, the test log- lioeliio,,ood, log ((t,, x:), it, E [T2, Ta), Hr,), may still contain dependence upon events prior lo the lest window [72, Ta], lypically contained in the his- tory Hr, of the intensity function. Comparing the difference mcan log-likelihond per event provides the information goin from one model to another (Daley & Vere-Jones 2004p.
Point processes are the dominant modeling approach in the seismology community, used extensively in both real-time operational carthquake forecasting (Mizrahi et al. 2024ap and established benchmarking experi- ments (CSEP) (Troni et al.) 2018 Rhoades et al. 2018). The point process representation of earthquake data aligns naturally with their occurrence as discrete events in time (Kagan 1994). Furthermore, this modeling approach favored over discretized forecasting models (e.g., time series) because it eliminates the need for optimizing binning strategies and allows for immediate updates, rather than waiting until the end of time bin a delay that could miss critical, potentially darmaging events.
2.2 ETAS
The Epidemic Type Aftershock Sequence (ETAS) model (Ogata 1998 is spatii-temppraal Hawkes process
Hawkes 1971 Siviero et al. 2024]; |; Bernabeu & Mateu 20255 which models how carthquakes cluster in time
and space. It has been adopted for operational earthquake forecasting by government agencies in California
Miilner et al. 2020), New-Zealand @Christophersen et al.) 2017). Italy (Spassiani et al. 2023), Japan (Omi
et al. 2019b) and Switzerland (Mizrahi et al. 2024b), and performs consistently well in CSEP's 's retrospective
and fully prospoctive forecasting experiments (c.g. Wocssner et al. 2011 Rhoades ct al. 2018 Taroni ct al.
2018
Callania et al., 2018 Mancini et al.. 2019 2020 2022). The general formulation of the model
X(1, x X(1, = +
Σ <0
g(l - 4, ||x x, x113. m)),
where μ is constant background rale of events, :)) is a non-negative excitation kernel which describes how past events contribute to the likelihood of future events and m, are the associated magnitudes of each event. The equivalent formulation as a Hawkes branching process accompanics a causal branching structure B. This concept broadly aligns with the understanding of the physics of earthquake triggering and interaction, e.g. via dynamic wave triggering (Brodsky & van der Elst 2014) and static stress triggering Goomberg 2018 Mancini et al. 2020).
Although ETAS can be fit by maximizing the log-likelihood function directly, parameter estimation is typ- ically performed by simultaneously estimating the branching structure B. Veen & Schoenberg (2008) de- veloped an Expectation Maximisation (EM) procedure, which maximises the marginal likelihood over the unobserved branching structure, log p(Hr, BB (BB())) through the iteration
(k+1) = arg max EB (B-p(11Hr,..l 000) (log p (Hr, B[0)]
(5)
This avoids the need to numerically approximate the integral term in the likelihood, provides more stability during estimation, and simultaneously distinguishes background events from triggered events.
The formulation of the ETAS model we present in the Earthquake benchmark is implemented in the etas python package by Mizrahi al. (2022). It defines the triggering kernel as
e e-//T k (m))
Mc)
g m)
(6)
(t + c) (r2 + d (0(m-MM) l+p'
where 72 is the squared distance between events and k, a, c, w, T, 7, are the learnable parameters along
with the constant background rate μ. This triggering kernel is derived from statistical distributions found
through decades of obscrvational studies (Utsu & Scki 1955 Utsu 1970 Utsu et al.) 1995) and several of
the learnable parameters have been linked to physical properties of the carthquake rupture process (Utsu
et al.
1995 Ide 2013)
4

---

Published in Transactions on Machine Learning Research (03/2026)
Despite its widespread use, it is commonly accepted that ETAS is a misspecified model of scismicity. By construction, ETAS describes only self-exciting triggering behaviour and therefore cannot capture inhibitory effects or stress relaxation processes, such as those represented by stress-release models (Zheng & Vere-
Jones 1991 Xiaogu & Vero-Jones) 1994] Bebbington & Harte 2003), or by models based on elastostatic
streess Transler and Coulomb rale-and-slate friction (Dieterich 1994). In addition, foreshock activity has been observed to deviate from ETAS assumptions, both spatially and temporally (McGuire et al.) 2005; Brodsky 2011 Lippiello et al. 2012; Ogata & Katsura, 2014]. Finally, to simplify inference, ETAS typically assumes isotropic spatial triggering kernels, despite observational evidence for anisotropic and fault-aligned aftershock distributions (Page & van der Elst. 2022). Together, these limitations motivate the exploration of more flexible modelling frameworks capable of capturing richer spatio-temporal structure in earthquake
sequences.
2.3 Neural Point Processes
Neural point processes (NPPs) have emerged in recent years within the machine learning literature as flexible alternatives to classical parametric point process models. Their central motivation is to replace restrictive, hand-crafted parametric forms with neural network based components that can learn complex, non non-linear dependencics directly from data. This makes them particularly appcaling for carthquake forccasting overcome the known limitations of the ETAS model discussed in Section 2.2
Early developments focused on temporal point processes (Shchur et al. 2021 Du et al. (2016) introduced the use of recurrent neural networks (RNNs) to encode the event history into a fixed- dicendiimnsional latent state, replacing explicil summatton over past events with a learned representation of temporal dependence. Subsequent work explored alternative sequence encoders, including LSTMs (Mei & Fisner 2017) and Trans- formers (Zuo et al.) 2020), alongside a variety of decoding strategies for modelling the conditional intensity or inter-event time distribution (Du et al. 2016 Omi al.) 2019a] Shhcur et al. (201)). In most cases, model parameters are learned by maximising the log-likelihood of observed event. sequences, although alternative
training objectives have also been proposed (Xiao et al. 2017 Li et al. 2018).
These temporal formulations were later extended to spatio-temporal settings Mukkherjee et al. 2025) by incorporating event locations into the history encoder and introducing flexible decoders for the conditional spatial distribution of future events. Existing spatio-temporal NPPs can be broadly grouped into three mod- elling classes. The first class exlends Hawkes-lype formulations by replacing parametric triggering kernels with neural network based influence functions, allowing non-stationary and history- excitation Zhou & Yu 2024). A second class models event dynamics in contin- uous time using neural ordinary dillerential equations, jointly evolving latent temporal states and spatial distributions (Jia & Benson 2019 Chen et al. 2020 A third, more recent class adopts fully generative
(Zhou et al. 2022 Dong et al., 2022:
approaches based on diffussin or score matching, learning the joint spatio-temporal distribution of events without explicitly parameterising an intensity function (Yuan et al., 2023 Li ct al. 2023; Lüdke et al., 2024). These approaches differ substantially in their computational cost, interprelability. and suilability for simulation and likelihood- baseed evaluation; see Mukherjee et al. (2025) for detailed discussion.
3 EarthquakeNPP Datasets
The FarthquakeNPP datasets (Figure 1 encompass earthquake records, including timestamps, geographical coordinates, and magnitudes, documented within California from 1971 to 2021. California, with its dense network and high seismic hazard, has been extensively studied, demonstrating the utility of forecasting algorithms (Gerstenberger et al./ [2004] Field 2007 Field et al. 2021). It encompasses the San Andreas fault plate boundary system (Zoback et al. 1987 and includes modern high-resolution catalogs with numerous small magnitude earthquakes, offering potential for new, more expressive models.
A central challenge when working with earthquake catalogs is data missingness, referred to in seismology as catalog incompleteness (Mignan & Wocssner (0122). Earthquakes are assumed to be only fully detocted above a time- and region-dependent completeness magnitude Mc, which reflects limitations of the seismic network and changes in detection capability over time. Ignoring this incompleteness can introduce substantial bias
5

---

Published in Transactions on Machine Learning Research (03/2026)
44'N
White Mc: 0.6 #:58, 636 2008-2021
40TN
ComCat
Mc: 2.5
SanJac
SaltonSea
#: #2 102 1971-2020
Mc: 1.0 #:21, 2008 2008-2018
Mc: 1.0 #:45 570
30
2008 2018
SCEDC
34"
Mc: 2.0 #:130,
1981 - 2620
32"N
125"WI
W2
115"W
110"W
105
Figure 1: Earthquakes contained in the observational datasets found in FarthquakeNPP. Colours indicate the respective datasets, including the target region, magnitude of completeness Me, number events and the time period that the dataset spans. In red is fault map from the GEM Global Active Faults Database (Styron & Pagani 2020
into both model fitting and evaluation (Sornette & Werner 2005). particularly for methods that rely smaller magnitude events.
All EarthquakeNPP datasets are constructed from publicly available raw catalogs provided by their respec- tive data centres. To enable a consistent and realistic retrospective forecasting experiment, the raw data is preprocessed by restricting each dataset to larget spatial region, truncating events above datasel-
specific magnitude threshold Meut Me (c.g.. Mignan al. 2011 Mignan & Woessner 2012) and removing duplicate locations.
Notebooks to access and preprocess the datasets, along with the associated benchmarking experiments, are publicly available at umentation. A more in-depth discussion of earthquake catalog generation, completeness, and preprocessing choices is provided in Appendix A Table ! summarises the key characteristics of each EarthquakeNPP dataset.
accompanied by detailed dataset doc-
4 Benchmarking Experiment
We use EarthquakeNPP to benchmark five spatio-temporal Neural Point Processes (NPPs) against the ETAS model described in Section 2.2 Each of these NPPs has prior positive claims in earthquake forecasting, yet lacks performance comparison with the ETAS model.
Neural Spatio-Temppral Point Process (NSTPP) (Chen al., 2021 A probability density function base) NPP that parametrizes the spatial pdf with continuous-ting normalizing flows (CNFs). We evaluate their Attentive CNF model due to its superior computational efficiency and overall performance compared to the Jump CNF model (Chen et al. 2021).
Deep Spatio- Spatio-Temporal Temporal Point Process (DeepSTPP) [hou et al., 2022 A conditional intensity function-based NPP that constructs non-parametric space-time intensity function driven by deep la- tent process. This model features closed-form intensity function, climinating the need for numerical approximations.

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Summary of LarthquakeNPP datasets, including: region, dataset development, magnitude threshold (M,), number of training (combined with validation) events, and number of testing events. The chronological partitioning of training, validation, and testing periods is also detailed. An auxiliary (burn-in) period begins from the Start date, followed by the respective starts of the training, validation, and testing periods. All dates are given as 00:00 UTC January 1st unless noted (* refers to 00:00 UTC on January 17 Finally, we give our purpose for including each dataset.
ComCat
SCEDC
White
QTM
Region
Whole of California
Southern California
San
Jacinto
Fault-
QTM_ [Yy_SanJac:
Zone
San Jacinto Zone,
Fault-
QTN SaltonSea: Salton Sea
Development
The U.S. Geological Survey (USGS) National Earth- quake Information Center (NEIC) monitors global earthquakes (Mw 4.5 or larger) and provides com- plete selamic monitoring of the United States for all sig- nificant earthquakes Mw 3.0 or felt). Its contribut- ing seismic networks produced the Advanced National Seismic System (ANSS) Comprehensive Catalog Earthquake
The Southern Califor- nia Seismic Network (SCSN) has developed and maintained the standard earthquake Southern (Hutton ctt al. 2010 since the Caltech Seismological Laboratory began rou- operations 1932. Siguificant improvements sinoe the 1970s and 1980s reduced the catalog Mw 3.25 to Mw 1.8.
White et al. (201 created catalog focusing on
Using data collected by the SCSN, Ross et al 2019) gener- catalog by reanalysing the same waveform data with matching procedure that looks correlations with previonsly detected events.
enhancrd
the San Jacinto fault region, using deuse seismic network Southern California. This denser network, combined with auto- mated phase picking (STA/LTA), ensures 99% detertion rate for earthquakes greater than Mw 0.6 in specific subregion White al 2019
cataling California
template
network
Eventa and Products.
completeness
Me
Mw 2.5
SCEDC_ 20: Mw 2.0, SCEDC_30:
Mw 0.6
Mw 1.0
Mw 3.0
# Train/Test 79,037 23,059
SCEDC_20:
38,556 26,914
[TV_SenJach
Events
128.265 / 14,351,
18. 66.6664 4,837,
SCEDC 26:
43,221 / 5,466,
STM_SanJac: 44,042 4,393
SCEDC_30:
12,426 2,065
Mart-Train- Val-Test-End
1771-1981-1998-200 2007-2000-
1881-1985-2005-2014 2020
2008-2009-201--2017 2021
1008-2009-2014- 2016-2018
Purpose
Example of data currently in use for operational forerasting (USGS ntilines ComCat in aftershook fore- they public).
Three magnitude thresholds (Mw 2.0,
To explore if newly detected low magni- tude earthquaes tain additional predic- tive information.
To explore if newly detected low mag- nitude earthquakes contain additional predictive informa- tion (with different detection methodol- ogy).
2.5, 3.0 for SCEDC 20. SCEDC_25, SCEDC_30)
explore effect truncation forecast- ing model performance.
Automatic Integration for Spatiotemporal Neural Point Processes (AutoSTPP) (Zhou & Yu 2024]: A conditional intensity function-based NPP that jointly models the 3D space-time integral of the intensity and its derivative (the intensity function) using dual network approach.
Spatio-temporal Diffusion Point Process (DSTPP) Yuan et al. 2023): : A generative NPP that does not have likelihood function. DSTPP employs diffusion models to capture complex spatio-temporal dynamics.

---

Published in Transactions on Machine Learning Research (03/2026)
ComCat
QTM Sal_Saatn
QTM_Sanjac
White
2.5
25
25-
2.5
20-
Temporal log logllkhood
1-2
1.-
1.0
1.0
23
-25-
-15
-15
-10
AWASSTP
DespSTPP
NSYPP
ER85
Poisson
Detgs799
asypp
ERS
Poiston
AdaliP
DespSTer
NDTPP
E755
Phisson
AddaSTPP
Diip3
ETAS
Poisson
SCEDC
SCEDC 25
SCEDC 30
22
2.0
70-
2.0
Toroporal log
0.5
-02
-05
14
16
AutosTPP
DrepssTP
NSTPP
BTAS
Phsson
AuteSTPP
Deeplrpp
NSTPP
ETAS
Pessln
NPTPP
Pricien
Figure 2: Test temporal log-likelihood scores for all the spatio-temporal point process models on each of the EarthquakeNPP datasets. SCEDC_20 SCEDC_ and SCEDC_ 30 correspond magnitude thresholds (Mw 2.0, 2.5, 3.0) of the SCEDC dataset. Error bars of the mean and standard deviation constructed for the NPPs using three repeat runs.
Score beiching--bsed Pseudolikelihood Estimation of Neural Marked Spatio-Tempora, Point Process (SMASH) (Li i et al. 2023): A generative NPP that also lacks a likelihood function. SMASH adopts normalization-fi free objective by estimating the pseudolikelihood of marked STPPs through score-matching.
Appendix provides details on the computational cost training and inference for all the models tested.
4.1 Likelihood Evaluation
Since generating repeated sequences over forecast horizons is computationally costly, the seismology commu- nity uses the mean log-liken lieeliill on held-out data for more streamlined metric during model development (Ogata 1988 Harte 2015). Other traditional next-event metrics like Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) are mislcading for earthquake forecasting (Hodson) 2022), earthquake occur- rence follows power law distributions (Kagan] 1994 Felzer & Brodsky 2006) that are heavy-tailed, making
the errors non-Gaussian and Laplacacn, contrary to the assumptions underlying RMSE and MAE (see
Section D_ For the three models with valid likelihood functions (NSTPP, DeepSTPP, and AutoSTPP), we present the mcan log-likelihood scores in Figures and 3 These scores are compared alongside the ETAS model (Section 2.2 and homogeneous Poisson process. The Poisson model is fit. to events in the auxiliary, training, and validation windows to provide baseline score for comparison.
Unlike the NPPs, which follow the standard machine lcarning procedure of training, validation, and testing, ETAS does not typically incorporate validation in its estimation procedure. Thus, it is fit using both the training and validation windows combined. For NPPs, the likelihood for training, validation, and testing depends on events occurring before the respective splits through memory their history. The exception is NSTPP, which lacks a direct dependency on prior events. Nevertheless, its likelihood is evaluated on the same data the other models.
8

---

Published in Transactions on Machine Learning Research (03/2026)
ComCat
QTM SaltonSea
QTM Sanjac
White
Spocial log bellhood
-30
12
-4
DespSToP
MSTAP
Poisso
Antorpp
DeepsTer
NSTP
ERA5
Poisson
Astoypp
Drigiip
Roisson
Pespi
NSpP
ETAS
Poiston
SCEDC 20
SCEDC 25
SCEDC 30
5potts log
Despl
RETP
Pessin
NETA
BTAS
Pesse
Amnd
DempSTPP
NSTPP
FTAS
Peicien
Figure 3: Test spatial ligelikel scores for all the spatio-temporal point process models on each of the EarthquakeNPP datasets. SCEDC_20 SCEDC_ and SCEDC 30 correspond magnitude thresholds (Mw 2.0, 2.5, 3.0) of the SCEDC dataset. Error bars of the mean and standard deviation constructed for the NPPs using three repeat runs.
To ensure that fitting ETAS on both the training and validation windows does not bias the comparison, we also tested an alternative configuration where ETAS was trained only on the training window. As shown in Appendix C ETAS performance remains effectively unchanged under this setup. The ETAS formulation (Equation also specifies how the magnitudes of prior earthquakes contribute the conditional intensity; this magnitude dependence is not implemented in any of the NPPs we benchmark, since it requires modelling choices beyond the scope of this work.
The ETAS model consistently achioves the highest temporal log-likelihodd, with NPPs performing compa- rably or, in some cases, marginally better, except at the higher magnitude thresholds of the SCEDC catalog, Among the NPPs, AutoSTPP and NSTPP perform well across several datasets, though their performance is more variable than that of DeepSTPP, which demonstrates consistent, comparable performance ETAS, Differences in Poisson performance across Figures 2 and 3 driven by variations in clustering strength, with weakly clustered catalogs appearing nearly Poisson-like and strongly clustered catalogs exhibiting larger departures.
Closer analysis of model performance over time (see Section reveals that relative performance to ETAS is poorest during large earthquake sequences. This likely due to ETAS leveraging the magnitude feature of the data, which enables it handle these sequences effectively. Conversely, model performance is strongest during "background" periods, when no large earthquakes occur. During these periods, ETAS models the background with constant rate, while the NPPs improve upon this by capturing the non-stationary nature of the background data. The improved relative temporal performance of all NPPs compared to ETAS, particularly when the magnitude threshold is lowered from 3.0 to 2.0 in the SCEDC dataset, indicates that low ude earthquakes provide valuable predictive information for NPPs.
ETAS consistently outperforms all NPPs in spatial ligeliccl Further analysis of model performance over space (see Section shows relative performance ETAS weakest in the most active and clustered areas (see Figures
12 and 13 likely due to the absence of magnitude feature in the NPPs. However,
9

---

Published in Transactions on Machine Learning Research (03/2026)
NPPs tend to perform more competitively in regions characterised by spatially complex diffuse scismicity, AutoSTPP achieves the highest spatial log-likelihod,, attributed to its ability to capture anisotropic Hawkes
kernels (see Figure 2 of Zhou & Yu (2024)), which are commonly observed in earthquake data (Page &
van der Elst 2022
4.2 CSEP Consistency Tests
EarthquakeNPP supports the earthquake forecast evaluation protocol developed by the Collaboratory for the Study of Earthquake Predictability (CSEP). In this procedure, a model generates 24-hour forecasts through 10,000 repeated simulations of earthquake sequences at the beginning of each day the testing period. This approach mirrors how earthquake forecasts are produced in operational settings (van der Elst al. 2022). Models are then evaluated by comparing the observed sequence with the distribution of forecasts generated by the simulations, Four test statistics assess the temporal, spatial, likelihood, and magnitude components of the forecasts. A test is considered failed if the observed statistic falls within pre-defined rejection region (Figure 14h. We apply this evaluation procedure to the two generative NPPs (DSTPP and SMASH) alongside ETAS (Table 2 and present a case study on the 2010 M7.2 El Mayor-Cucapah earthquake, using the forecasts from these models (Figure 4p. Appendix F provides an introduction to the CSEP consistency tests, with further details found at the simulated forecasts.
org/ and Appendix G provides further analysis
Table 2: CSEP consistency tests evaluate the calibration of daily forecasts from three models (ETAS, SMASH, DSTPP) on EarthquakeNPP datasets. A test performed the a 0.05 significance level on each day in the testing period. The pass rate indicates the proportion of testing days with non-rejected hypotheses. If the model is the data generator, quantile scores should be uniformly distributed. The KS- Statistic quantifies deviation from uniformity (see Appendix F). ETAS is the only model that forecasts carthquake magnitudes, so is the only model evaluated with the magnitude test.
Dataset
Model
Number Test
Spatial Teat
Pseudo Likelihood Teat
Magnitude Test
Pass Rate
KS-Stat.
Pass Rate
KS-Stat.
Pass Rate
KS-Stat.
Pass Rate
KS-Stat.
ETAS
95.8%
0.222
92.0%
0.029
97.6%
0.128
93.8%
0.113
ComCat
SMASH
86.2%
68.6%
0.217
87.6%
0.134
DSTPP
86.7%
0.116
88.6%
0.070
86.3%
0.138
ETAS
92.6%
0.347
88.3%
0.119
95.9%
0.233
90.0%
0.256
SCEDC
SMASH
69.6%
0.602
51.1%
0.471
68.0%
0.611
DSTPP
27.4%
0.840
0.935
0.8%
0.988
ETAS
95.4%
0.151
91.7%
0,095
96.6%
0.123
94.8%
0.076
Jac
SMASH
81.7%
0.290
55.6%
0.385
53.4%
0.584
DSTPP
85.7%
0.110
85.7%
0.240
85.3%
0.136
ETAS
93.6%
0.210
90.9%
0.206
96.4%
0.119
90.6%
0.136
QTM, QTM_Sal SonSon
SMASH
75.8%
0.186
53.6%
0.371
73.7%
0.451
DSTPP
87.7%
0.154
88.8%
0.136
85.6%
0.127
ETAS
88.3%
0.167
86.6%
0.225
90.8%
0.233
88.8%
0.052
Vhite
SMASH
69 69.3%
0.274
84.3%
0.150
67.7%
0.246
DSTPP
0.5%
0.987
30.9%
0.691
32.3%
0.892
ETAS consistently performs best across all datasets and tests. Tt achieves the highest pass rates and lowest KS statistics, indicating strong calibration and reliability. SMASH shows modcrate performance, often outperforming DSTPP but trailing ETAS. Its results vary more across datasets and tests, with occasional strengths (e.g. in White for spatial KS). DSTPP generally performs worse, with lower pass rates and higher KS statistics, especially for the SCEDC and White datascts. However, it achieves relatively good spatial calibration in some cases (e.g,, ComCat).
Further analysis of the simulated forecasts in Appendix G provides insight into the consistency test results. Temporally, all models struggle capture the highost-rate scismicity days, indicating substantial room for improvement in modelling the most hazardous periods. SMASH exhibits highly variable, spiky daily rate forecasts that result. in frequent over- and underprediction, while DSTPP produces much smoother
10

---

Published in Transactions on Machine Learning Research (03/2026)
36-94
35
36'N
-05
35/N
35"
53'1
34'N
3414
Mognb.00c
M4
2.3
N.85
22
22'N4
22%
S00e DIET M T
-35
1500
Magrituoo
1000
500
Forecast humber Distribttion Foewound Distrisution
01-1p-2020
05-Ap-2010
05-0g-2010
13
17-1o 2010
21-Aph-2030
Figure 4: Forecasts from ETAS, SMASH, and DSTPP during the 2010 M7.2 El Mayor-Cucapal earthquake contained in the ComCat dataset. Top: Spatial forecasts for the day following the mainshock. ETAS ac- curately captures the primary aftershock zone along the Laguna Salada fault system. SMASH produces smoother forecasts with broader spatial spread, while DSTPP concentrates its probability mass north of the mainshock epicenter. Bottom: Cumulative carthquake counts over time, with magnitudes shown as scaled orange circles. Forecast number distributions from each model are plotted with 95% confidence intervals. All models initially underestimate aftershock activity. ETAS and SMASH begin to recover after the first week, whereas DSTPP continues to systematically underpredict event counts throughout the sequence.
forecasts that systematically underestimate seismicity across both background and active periods. Spatially, ETAS assigns concentrated rates along known fault structures through its explicit clustering mechanism. In contrast, SMASH generates diffuse spatial forecasts with weak contrast between active and inactive regions, whereas DSTPP more accurately follows fault-aligned structure but often assigns uniformly low spatial rates, particularly the SCEDC and White datasets.
We were unable apply the CSEP evaluation procedure for NSTPP, AutoSTPP and DeepSTPP, since the models not explicitly formulated to be generative and therefore suffer from slow sampling (see details
11

---

Published in Transactions on Machine Learning Research (03/2026)
in Appendix D This limitation significantly hinders their ability to be applied to real-time operational earthquake lorecasting.
5 Discussion and Conclusion
We introduce EarthquakeNPP, a benchmarking platform designed to evaluate Neural Point Process (NPP) models against the state-of-the-art ETAS model for carthquake forecasting. The platform hosts datascts from diverse regions of California, both standard forecasLing zones and datasets that incorporate modern detection techniques. We establish two evaluation frameworks tailored to seismology: standard log-likelihood metrics and the generative consistency tests developed by the Collaboratory for the Study of Earthquake Predictability (CSEP), ensuring that successful models can be directly relevant to operational forecasting.
In benchmarking five neural point process (NPP) models against ETAS, we found that none outperformed the baseline, indicating that current NPP architectures are not yet suitable for operational earthquake forecasting. While several NPPs achieve competitive performance during low-activity background periods, they consistently struggle during highly active phases following large earthquakes. Our results highlight several concrete architectural and methodological gaps, which we summarise below as actionable directions for future NPP development.
Action 1: Encode explicit magnitude dependence. ETAS explicitly encodes magnitude dependence, whereby larger earthquakes exponentially increase both the rate and spatial extent of subsequent seismic- ity. None of the benchmarked NPPs incorporate such explicit magnitude scaling, which limits their ability to capture the dominant influence of large events. Future NPP architectures could address this by in- troducing magnitudo-aware design choices, such as hicrarchical encodings that distinguish small and large events, magnil weighted attention mechanisms, or parameterisations aligned with the logarithmic fre- queney- magnitude scaling observed in seismicity (Richter 1935). These approaches would allow NPPs to retain flexibility while incorporating structure that has proven critical for ETAS performance.
Design scalable long-term memory mechanisms. All evaluated NPPs truncate the con- ditioning history due to the computational cost of sequence encoders, with models such as DeepSTPP and AutoSTPP conditioning on as few as 20 past events. In contrast, ETAS integrates the full event history, al- lowing long-past earthquakes, including large or spatially distant events, to inffuence future rates. Designing NPPs with scalable long-term memory is therefore critical avenue for improvement. Promising directions include sparse or dilated attention mochanisms (Child et al., 2019: Hassani & Shi 2022) to reduce quadratic complexity, hierarchical coarse-to-fine representations of earthquake histories (Yagg el al. 2016), and ex- plicit memory compression modules (Kim et al.] 2023) that preserve the influence of distant but significant events. Advances in long-context modelling within the NLP literature suggest that such mechanisms are technically feasible (Liu al. 2025) and may translate naturally to earthquake triggering dynamics.
Action 2:
Align generative training with operational evaluation. Third, our results reveal a mis- match between how generative NPPs are trained and how they are evaluated. Models such as SMASH and DSTPP are trained to predict sample the next event, whereas CSEP consistency tests require simulating complete event sequences over fixed forecasting windows. This discrepancy helps explain why some gener- ative models show reasonable short-term accuracy (Yuan et al. 2023 Li et al. 2023) but perform poorly in our multi-event simulation tests. Future generative NPPs may therefore benefit from training objectives that explicitly target long-horizon trajectory behaviour, for example by optimising multi-event. simulation losses (c.g. Lüdke et al., 2024) or designing statistics (c.g. equation 13 aligned with the CSEP evaluations used in this study.
Action 3:
Action 4: Incorporate empirically supported scaling laws. Finally, results suggest that the com- plete removal of physically motivated structure may be counterproductive. While NPPs aim to move beyond parametric models, ETAS kernels encode power-law scaling relationships that are strongly supported by em- pirical seismology (Kagan 1994 Felzer & Brodsky 2006). Hybrid architectures that combine neural density ostimation with ETAAS-inppred power-law kernels or magnitude-dependen triggering functions may offer a productive middle ground, retaining empirical laws while allowing greater flexibility than purely parametric formulations. For example, replacing Gaussian spatial kernels in existing NPPs (e.g in DeepSTPP) with
12

---

Published in Transactions on Machine Learning Research (03/2026)
learned power-law forms could improve their ability to represent aftershock clustering without sacrificing expressiveness.
provides platform for future NPP developments to be benchmarked against these initial results. The platform is under ongoing devel- opment and in the future will see the direct comparison of emerging and other existing models developed within the seismology community, as well as an expansion of datasets included other seismically active global regions. Successful NPP models these datasets, for both log-likelihood and CSEP metrics, will be directly impactful to stakeholders in seismology, ultimately enabling their integration into operational earthquake forecasting by government agencios.
EarthquakeNPP, available at https://github. www.com///ao//e /s Beer11589//atthuuaaeece
Acknowledgments
This project is funded by Compass Centre for Doctoral Training in Computational Statistics and Data Science (EPSRC Grant Ref EP PSP//S5359//// Compass is funded by United Kingdom Research and Innovation (UKRI) through the Enginecring and Physical Sciences Research Council (EPSRC), https:// //ww:: ukri org/councils/epsrc This project also has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant 821115, Real-time earthquake rlsk reduction for a reSilient Europe (RISE), http: http://www.. lasseeeu.ory and by United States Geological Survey (USGS) EHP grants G24AP00059 and G25AP00379.
References
Duncan Carr Agnew. Equalized plot scales for exploring seismicity data. Seismological Research Letters, 86 (5):1412 1423, 2015.
Rex Allen. Automatic phase pickers Their present use and future prospects. Bulletin of the Seismological Society of America, S2(6 S242, 1982.
Mark Bebbington and David Harte. The linked stress release model for spatio-temporal scismicity: formu- lations, procedures and applications. Geophysical Journal International, 154(3):925-99) 946, 2003.
Alba Bernabeu and Jorge Mateu. Spatio-temporal hawkes point processes: statistical inference and simula- tion strategies. arXiv preprint arXio:2511.14500 2025.
Emily E Brodsky. The spatial density of foreshocks. Geophysical Research Letters, 38(10). 2011.
Family Brodsky and Nicholas van der Flst. The uses of dynamic earthquake triggering. Annual Review of Earth and Planetary Sciences, 42:317 339, 2014.
Camilla Cattania, Maximilian J Werner, Warner Marzocchi, Sebastian Hainzl, David Rhoades, Matthew Gerstenberger, Maria Liukis, William Savran, Annemarie Christophersen, Agnès Helmstetter, et al. The forecasting skill of physics-based seismicity models during the 2010-2012 canterbury, new zealand, earth- quake sequence. Seismological Research Letters, 89(4):1235 1250, 2018.
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representationss 2021. URL httpp:://pperrrrreee.neeeefoeeee
XQQA6-So14
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spalio-temporal point processes. arXiv preprint arXi::2011.04588 2020.
Rewon Child, Scott Gray, Alec Radford, and Nya Sutskever. Generating long sequences with sparse trans- formers. arXiv preprint arXix: 10509, 2019.
A Christophersen, DA Rhoades, MC Gerstenberger, Bannister, J Bocker, SH Potter, and S McBride. Progress and challenges operational earthquake forecasting in new zealand. In New Zealand society for earthquake engineering annual technical conference, 2017.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Daryl J Daley and David Vere-Jones. Scoring probability forecasts for point processes: The entropy score and information gain. Journal of Applied Probability, 41 (1 (A):297-3322 2004.
Kelian Dascher-Cousineau, Oleksandr Shehur, Emily E. Brodsky, and Stephan Günnemann. Using deep learning for flexible and scalable earthquake forecasting. Geophysical Research Letters, 50(17): c2023GL103909, 2023. doi: https://doi.ors https///ddboorgg0.102////// 10.1009//0222L11030000
James Dieterich. A constitutive law for rate of carthquake production and its application to carthquake clustering, Journal of Geophysical Research: Solid Earth, 99(B2):2001-2) 16001-268,, 1994.
Zheng Dong, Xiuyuan Cheng, and Yao Xie. Spatio-temporal point processes with deep non-stationary kernels. arXiv preprint arXiv:2211.111999 2022.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigues and Le Song, Recur- rent marked temporal point processes: Embedding event history to vector. In Proceedliggs of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1555-1564, 2016.
Karen R Felzer and Emily Brodsky. Decay of aftershock density with distance indicates triggering by dynamic stress. Nature, (41(7))))7735-733 2006.
Edward Н Field. Overview of the working group for the development of regional earthquake likelihood models (relm). Seismological Research Letlers, 78(1):7 16, 2007.
Edward H. Field, Kevin R Milner, Morgan T Page, William H Savran, and Nicholas van der Elst. Improve- ments to the third uniform california earthquake rupture forecast etas model (ucerf3-etas). The Seismic Record, 122)1117- 2021.
Andrew M Freed. Earthquake triggering by static, dynamic, and postscismic stress transfer. Anna. Rev. Earth Planet. Sci., 33:335 367, 2005.
Matt Gerstenberger, Stefan Wiemer, and Lucile M Jones. Real-time forecasts of lomorrow's earthquakes in Califormia: A new mapping tool. US Geological Survey, 2004.
Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1):125-15,, 2014.
Joan Gomberg. Unsettled carthquake nucleation. Nature Geoscience, 11(7):463-4644 2018.
Beno Gutenberg and Charles Francis Richter. Magnitude and energy of carthquakes. Science, 83(2147): 183 185, 1936.
Sebastian Hainzl. Apparent triggering function of aftershocks resulting from rate-dependent incompleteness of earthquake catalogs. Journal of Geophysical Research: Solid Earth, 121(9):6494) 16499-6509, 2016a.
Sebastian Hainzl. Rate-dependent incompleteness of earthquake catalogs. Seismological Research Letters, 87(2A):337-344 2016b.
Scbastian Hainzl. Etas-approach accounting for short-term incompleteness of carthquake catalogs. Bulletin of the Seismological Sociely of America, 112(1):494 507, 2022.
Sebastian Hainzl, A Christophersen, and B Enescu. Impact of earthquake rupture extensions on parameter estimations of point-process models. Bulletin of the Seismological Society of America, 98(4):2066 200-622277,7, 2008.
Thomas C Hanks and Hiroo Kanamori. A moment magnitude scale. Journal of Geophysical Research: Solid Earth, 23(B5))2348-232) 2350, 1979.
DS Harte. Log- likellhod of carthquake models: evaluation of models and forecasts. Geophysical Journal International, 201 (2):7111 1-723, 2015.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Ali Hassani and Humphrey Shi. ar 2rXii:2209.1500 2022.
Dilated neighborhood attention transformer.
arXiv preprint
Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58((): 83 83-90, 1971.
Agnes Helmstetter, Yan Y Kagan, and David Jackson. Comparison of short-term and time-independent earthquake forecast models for southern california. Bulletin of the Seismological Sociely of Americn, 96 (1):90-100) 106, 2006.
Marcus Herrmann and Warner Marzocchi. Inconsistencies and lurking pitfalls in the magnituude-frqquency distribution of high-resolution earthquake catalogs. Seismological Research Letters, 92(2A)):009992) 2021.
Timothy O Hodson. Root mean square error (rmse) mean absolule error (mae): When to use them not. Geoscientific Model Development Discussions, 2022:1-10, 2022.
Kate Hutton, Jochen Woessner, and Egill Hauksson. Earthquake monitoring in southern california for seventy-seven years (1932-2008), Bulletin of the Seismological Society of America, 100(2):423-44) 2010.
Satoshi Ide. The proportionality between relative plate velocity and seismicity in subduction zones. Nature Geoscience, 6(9):780-77) 784, 2013.
Pablo Iturrieta, José A Bayona, Maximilian J Werner, Danijel Schorlemmer, Matteo Taroni, Giuseppe Falcone, Fabrice Cotton, Asim M Khawaja, William H Savran, and Warner Marzocchi. Exaluation of decade-long prospective earthquake forecasting experiment in italy. Seismological Research Lellers, 2024.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
Yan Y Kagan. Likelihood analysis earthquake catalognes. Geophysical journal international, 106(1):
135 148, 1991.
Yan Kagan. Observational evidence for earthquakes nonlinear dynamic process. Physica D: Nonlinear Phenomena, 77(1-3):160- 192, 1994.
Yan Y Kagan and L. Knopoff. Statistical short-term earthquake prediction. Science, 236(4808):1563 1567, 1987.
Jang-Hye Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. arXiv preprint arXiv:2312.08/ 034114 2023.
Sacha Lapins, Berhe Goitom, J-Michael Kendall, Maximilian J Werner, Katharine V Cashman, and James OS Hammond, A little data goes a long way: Automating seismic phase arrival picking at nabro volcano with transfer learning. Journal of Geophysical Research: Solid Earth, 12011)1100,,, 2021.
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song. Learning temporal point processes via reinforcement learning. Advances in neural information processing systems, 31, 2018.
Zichong Li, Qunzhi Xu, Zhenghao Xu, Yajun Mei, Tuo Zhao, and Hongyuan Zha. Score matching- based pseu- dolikelihood estimation of neural marked apato-tempoot point process with uncertainty quantification. arXiv preprint orXio:2810.163100 2023.
Eugenio Lippiello, Warner Marzocchi, I. De Arcangelis, and C Godano. Spatial organization of foreshocks as a tool to forecast large earthquakes. Scéentific reports, 2(1):1 6, 2012.
Jiaheng Liu, Dawci Zhu, Zhiqi Bai, Yancheng Hc, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, al. A comprehensive survey on long context language modeling. arXiv preprint Xix:22500. 17407, 2025.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Anthony Lomax, Jean Virieux, Philippe Volant, and Catherine Berge-Thierry. Probabilistic carthquake location in 3d and layered models: Introduction of a metropolis-gibls method and comparison with linear locations. Advances in seismic event location, pp. 101 134, 2000.
David Lüdke, Enric Rabasseda Raventós, Marcel Kollovieh, and Stephan Günnemann. Unlocking point processes through point set diffusion. arXiv preprint 11.:22400.22400 2024.
S Mancini, M Segou, MJ Werner, and C Cattania. Improving physics-bass amiisrrossner22011nssuuummm title=What is an instrumental seismicity catalog, Community Online Resource for Statistical Scismic- ity Analysis, doi: 10.5078/ 10.5078/ccrssa-38778 cor99a-38784330,, author-Woessner, and Hardebeck, JL and Haukkson, E, year=2010 d aftershock forecasts during the 2016-2017 central italy earthquake cascade. Journal of Geo- physical Research: Solid Earth, 22(88)8626-8644 2019.
Simone Mancini, Margarita Segou, Maximilian Jonas Werner, and Tom Parsons. of elastic coulomb rate-and-state aftershock forecasts during the 2019 ridgecrest, california, earthquake sequence. Bulletin of the Seismological Sociely of America, 110(4):1736-11 2020.
The predictive skills
Simone Mancini, Margarita Segou, Maximillan J Werner, Tom Parsons, Gregory Beroza, and Lauro Chiar- aluce. On the use of high-resolution and deep-learning scismic catalogs for short-term carthquake fore- casts: Potential benefits and current limitations. Journal of Geophysical Research: Solid Earth, 127(;1): e2022JB02500,, 2022.
Jeffrey McGuire, Margaret S Boettcher, and Thomas H Jordan. Foreshock sequences and short-term earthquake predictability on east pacific rise transform faults. Nature, 434(7032):457 461, 2005.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017.
Andrew Michael and Maximilian J Werner. Preface to the focus section on the collaboratory for the study of carthquake prodictability (csop): New results and future directions. Seismological Research Letters, 89 (4):1226 1228, 2018.
A Mignan, MJ Werner, S Wiemer, C-C Chen, and Y-M Wu. Bayesian estimation of the spatially varying completeness magnitude of earthquake catalogs. Bulletin of the Seismological Society of America, 101(3): 1371-1385, 2011.
Arnaud Mignan and Jochen Woessner. Theme in-dddesttanding seismicity catalogs and their problems. Communily online resource for slatistical seismicily analysis, 2012.
Kevin R Milner, Edward H Field, William H Savran, Morgan T Page, and Thomas H Jordan. Operational earthquake forecasting during the 2019 ridgecrest, california, earthquake sequence with the ucerf3-etas model. Seismological Research Letters, 91 (3)):115771558 2020.
Leila Mizrahi, Shyam Nandan, and Stefan Wiemer. Embracing data incompleteness for better earthquake forecasting. Journal of Geophysical Research: Solid Earth, 20221..323211,, 2021.
Leila Mizrahi, Nicolas Schmid, and Marta Han. Imizrahi/ctas, 2022. URL
zenodo. 6583992
Leila Mizrahi, Irina Dallo, Nicholas J. van der Elst, Annemarie Christophersen, Ilaria Spassiani, Maxi- milian J. Werner, Pablo Iturrieta, José A. Bayona, lunio lervolino, Max Schneider, Morgan T. Page, Jiancang Zhuang, Marcus Herrmann, Andrew J. Michael, Giuseppe Falcone, Warner Marzocchi, David Rhoades, Malt Gerstenberger, Laura Gulia, Danijel Schorlemmer, Julia Becker, Marta Han, Lorena Kuratle, Michèle Marti, and Stefan Wiemer. Developing, testing, and communiating earthquake fore- casts: Current practices and future directions. Reviews of Geophysics, 6223332302,, 2024a. doi: https:/ //tp://///....r//00///////////// /2033R000022.. URL https://agupubs..nnl www.:ppwww..s...n...e...a..aa.aa www.eeelirrrrr.ellrelleeeee.eeee wwwey. com/doi/abs/ 10. 1029/2023R00008223 22023RG000223 2023RG000823..
16

---

Published in Transactions on Machine Learning Research (03/2026)
Leila Mizrahi, Shyam Nandan, Banu Mena Cabrera, and Stefan Wiemer. suiETAS: Developing and Testing ETASS-aaeed Earthquake Forecasting Models for Switzerland. Bulletin of the Seismologionl Sociely of America, 05 2024b. doi: 10.1785/0120240000.
Nobuhito Mori, Tomoyuki Takahashi, Tomohiro Yasuda, and Hideaki Yanagisawa. Survey of 2011 tohoku earthquake tsunami inundation and run-up. Geophysical reseorch letters, 38(7), 2011.
S Mostafa Mousavi and Gregory C Beroza. Machine learning earthquake scismology. Annual Review of Earth and Planetary Sciences, 51:105 129, 2023.
S Mostafa Mousavi, William Ellsworth, Weiqiang Zhu, Lindsay Y Chuang, and Gregory C Beroza. Earth- quake transformer an attentive deep-learning model for simultaneous earthquake detection and phase picking. Nature communications, 11(1):3952, 2020.
Sumantrak Mukherjee, Mouad Elhamdi, George Mohler, David A Selby, Yao Xie, Sebastian Vollmer, and Gerrit Grossmann. Neural spatiotemporal point processes: Trends and challenges. arXiv preprint arXiv:2502.0934/ 2025.
Yosihiko Ogata, Statistical models for earthquake occurrences and residual analysis for point processes. 1988.
Journal of the Americon Statistical association,
Yosihiko Ogata. Space-time point-process models for earthquake occurrences. Annals of the Institute of Statistical Mathematics, 50(2):379-402, 1998.
Yosihiko Ogata and Koichi Katsura. Comparing foreshock characteristics and foreshock forecasting in ob- served and simulated earthquake catalogs. Journal of Geophysical Research: Solid Earth, 119(11):8457- 8477, 2014.
Takahiro Omi, Yosihiko Ogata, Yoshito Hirata, and Kazuyuki Aihara. Estimating the etas model from an early aftershock sequence. Geophysical Research Letters, 41(3):850-83 857, 2014.
Takahiro Omi, Kazuyuki Aihara, et al. Fully neural network based model for general temporal point pro- cossos. Advances in neural information processing systems, 32, 2019a.
Takahiro Omi, Yosihiko Ogata, Katsuhiko Shiomi, Bogdan Enescu, Kaoru Sawazaki, and Kazuyuki Aihara. Implementation a real-time system for automatic aftershock forecasting in japan. Seismological Research Letters, 90(1):242 250, 2019b.
Morgan T. Page and Nicholas J. van der Elst. Aftershocks preferentially occur in previously active areas. The Seismic Record, 2(2): 106, 04 2022. ISSN 2694-4006. doi: 10.1785/0320222005 URL https: //doi org/10. 10. 1785 0320020002200
Morgan T Page, Nicholas van der Flst, Jeanne Hardebeck, Karen Felzer, and Andrew J Michael. Three ingredients for improved global aftershock forecasts: Tectonic region, time-dependent catalog incomplete- ness, and intersequence variability. Bulletin of the Seismological Society of America, 106(5):2290 2301, 2016.
David A Rhoades, Annemarie Christophersen, Matthew C Gerstenberger, Maria Liukis, Fabio Silva, Warner Marzocchi, Maximilian J Werner, and Thomas H Jordan. Highlights from the first ten years of the new zcaland carthquake forccast testing center. Seismological Research Letters, 89(4):1229-1231 2018
Charles F Richter. An instrumental earthquake magnitude scale. Bulletin of the seismological society of America, 2511):133, 1935.
Zachary E Ross, Daniel Trugman, Egill Hauksson, and Peter M Shearer. Searching for hidden earthquakes in southern california. Science, 361(6444)7767-771, 2019.
William Savran, Maximilian J Werner, Warner Marzocchi, David Rhoades, David D Jackson, Kevin Milner, Edward Field, and Andrew Michael. Pseudoprospectivee evaluation of ucerf3-ctas forecasts during the 2019 ridgecrest sequence. Bulletin of the Seismological Sociely of America, 110(4):1799-1880 2020.
17

---

Published in Transactions on Machine Learning Research (03/2026)
William H Savran, José A Bayona, Pablo Iturricta, Khawaja M Asim, Han Bao, Kirsty Bayliss, Marcus Herrmann, Danijel Schorlemmer, Philip J Maechling, and Maximilian J Werner. pycsep: a python toolkit for earthquake forecast developers. Seismological Society of America, 93(5):2858 2870, 2022.
Danijel Schorlemmer and Jochen Woessner. Probability of detecting earthquake. Bulletin of the Seismo- logical Society of America, 98(5):2103 2117, 2008.
Danijel Schorlemmer, Maximilian J Werner, Warner Marzocchi, Thomas H Jordan, Yosihiko Ogata, David D Jackson, Sum Mak, David A Rhoades, Matthew Gerstenberger, Naoshi Hirata, et al. The collaboratory for the study of earthquake predictability: Achievements and priorities. Seismological Research Letters, 89(4):1305-1313, 2018.
Stefanie Seif, Arnaud Mignan, Jeremy Douglas Zechar, Maximilian Jonas Werner, and Stefan Wiemer. Estimating etas: The effects of truncation, missing data, and model assumptions. Journal of Geophysical Research: Solid Earth, 122(1):449 469, 2017.
Francosco Scrafini, Finn Lindgren, and Mark Naylor. Approximation of bayesian hawkes process with inlabru. Eneironmellicss 34(5), 2023.
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. arXiv preprinl Xiv: XXxx:1000 1909. 12127, 2019.
Oleksandr Shehur, Ali Caner Türkmen, Tim Januschowski, and Stephan Günnemann. Neural temporal point processes: A review. In Zhi-Hua Zhou (ed.), Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI 2021, IJCAI International Joint Conference on Artificial Intelligence, pp. 4585 4593. International Joint Conferences on Artificial Intelligence, 2021. Publisher Copyright: 2021 International Joint Conferences on Artificial Intelligence. All rights reserved.; 30th International Joint Conference Artificial Intelligence, IJCAT 2021 Conference date: 19-08-2021 Through 27-08-2021.
Peter M Shearer. Introduction to seismology. Cambridge university press, 2019.
David R Shelly. A 15 year catalog of more than million low-frequency earthquakes: Tracking tremor and slip along the deep san andreas fault. Journal of Geophysical Research: Solid Earth, (23 2017.
Emilia Siviero, Guillaume Staerman, Stephan Clémengon, and Thomas Moreau. Flexible parametric inference for space-time hawkes processes. arXiv preprint arXiv:2406. 06849, 2024.
Didier Sornette and Maximillian J Werner. Apparent clustering and apparent background earthquakes biased by undetected seismicity. Journal of Geophysical Research: Solid Earth, 110(B9), 2005.
Ilaria Spassiani, Giuseppe Falcone, Maura Murru, and Warner Marzocchi. Operational earthquake forecast-
ing in italy: validation after 10 yr of operativity. Geophysical Journal International, 234(3)):501-2518 2023.
Seth Stein and Michael Wysession. An introduction to seismology, earthquakes, and earth structure. John Wiley Sons, 2009.
S. Stockman, D. J. Lawson, and M. J. Werner. SB-ETAS: using simulation based inference for scalable, likelihood-free inference for the ETAS model of earthquake occurrences. Statistics and Computing, 34 (174), 2024. doi: 10. 10.1007/s11222-024 020-222-22----8--- 10486-6. URL
Samuel Stockman, Daniel J Lawson, and Maximilian J Werner. Forccasting the 2016-2017 central apennines earthquake sequence with a neural point process. Earth's 's Future, 11)) 2023.
Richard Styron and Marco Pagani. The gem global active faults database. Earthquake Spectra, 36(1 (up_]) 160-180, 2020.
Tetsuo Takanami, Genshiro Kitagawa, and Kazushige Obara. Hi-net: High sensitivity scismograph network, japan. Methods and applications of signal processing in seismic network operations, pp. 79-87 88, 2003.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Yen Joe Tan, Felix Waldhauser, William L Ellsworth, Miao Zhang, Weiqiang Zhu, Maddalena Michele, Lauro Chiaraluce, Gregory C Beroza, and Margarita Segou. Machine-learning-basee based high-resolution earthquake catalog reveals how complex fault structures were activated during the 2016 2017 central italy sequence. The Seismic Record, (11):11-19, 2021.
Matteo Taroni, Warner Marzocchi, Danijel Schorlemmer, Maximilian Jonas Werner, Stefan Wiemer, Jeremy Douglas Zechar, Lukas Heiniger, and Fabian Euchner. Prospective csep evaluation of 1-day, 3-month, and 5-yr earthquake forecasts for italy. Seismological Research Letters, (9(4):1251-126) 2018.
Clifford H Thurber. Nonlinear earthquake location: theory and examples. Bulletin of the Seismological
Society of America, 75(3):779-788 790, 1985.
Tokuji Utsu. Aftershocks and earthquake statistics (1): Some parameters which characterize an aftershock sequence and their interrelations. Journal of the Faculty of Science, Hokkaido University. Series 7, Geo- physics, 3(3):129 195, 1970.
Tokuji Utsu and Akira Seki. A relation between the area of after-shock region and the energy of main-shock. Journal of the Seismological Society of Japan, 7:233-240, 1955. URL
org/CorpusID:133444
Tokuji Utsu, Yosihiko Ogata, Ritsuko S, and Matsu'ura The centenary of the omori formula for a decay law of aftershock activity. Journal of Phyyscs of the Earth, 43(1):1-33 -3,, 1995. doi:
Nicholas J van der Elst, Jeanne L Hardebeck, Andrew J Michael, Sara K McBride, and Elizabeth Vanacore. Prospective and retrospective evaluation of the geological survey public aftershock forecast for the 2019-2021 southwest puerto rico earthquake and aftershocks. Seismological Society of America, 93(2A): 620 640, 2022.
Alejandro Veen and Frederic P Schoenberg. Estimation of space time branching process models in seismology using an em-type algorithm. Journal of the American Statistical Association, 103(482):614-624 2008.
Qianlong Wang, Yifan Guo, Lixing Yu, and Pan Li. Earthquake prediction based spatio-temporal data mining: an lstm network approach. IEEE Transactions on Emerging Topics in Computing, 811):148-158 2017.
Maximilian J Werner, Agnès Helmsteuter, David D Jackson, and Yan Y Kagan. High-resolution long-term and short-term earthquake forecasts for california. Bulletin of the Seismological Society of America, 101 16444, 2011.
Malcolm CA White, Yehuda Ben-Zion, and Frank L Vernon. A detailed earthquake catalog for the san jacinto fault-zone region in southern california. Journal of Geophysioal Research: Solid Earth, 124(7): 6908-6930, 2019.
Stefan Wiemer and Max Wyss. Minimum magnitude of completeness in carthquake catalogs: Examples from alaska, the western united states, and japan. Bulletin of the Seismological Society of America, 90(4): 559-869, 2000.
J Woessner, JL Hardebeck, and E Haukkson. What is an instrumental seismicity catalog, community online resource for statistical seismicity analysis, doi: 10.5078/ 10.5078/corssa-3878..88 corssaa8884488 2010.
Woessner, Sebastian Hainzl, W Marzocchi, MJ Werner, AM Lombardi, Catalli, B Enescu, M Cocco, MC Gerstenberger, and S Wiemer. A retrospective comparative forecast test on the 1992 landers sequence. Journal of Geophysical Research: Solid Earth, 116(B5), 2011.
Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasserstein learning of deep generative point process models. Advances in neural information processing systems, 30, 2017.
Zheng Xiaogu and David Vere-Jones. Further applications of the stochastic stress release model to historical earthquake data. Tectonophysics,
1994.
19

---

Published in Transactions on Machine Learning Research (03/2026)
BELCE
magnitae
Vine
Wove
50033
ISTA/COC
TI0--1-0 T6LEN-2
2.75
042
TI07 TO
3154200
1922
Source
104404450
Station
115,4000
192-91-12
0.00
0300000
-17407900
190--9-11 215308
5845
A0100000
-19477900
1222-01-44000 00:11-510000:133300
5804
1900-11-
Figure Generating an earthquake catalog involves several key steps: seismic phase picking, magnitude estimation, and the association and location of seismic sources. This process transforms raw waveform data recorded at scismic stations to locations, times, and magnitudes of carthquakes.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for compulational linguislies: human language Lechnologies, pp. 1480-1485 2016.
Yuan Yuan, Jinglao Ding, Chenyang Shao, Depeng Jin, and Yong Li. Spatio-temppral diffusion point processes. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3173 3184,, 2023.
Douglas Zechar, Matthew C Gerstenberger, and David A Rhoades. Likelibood-based tests for evaluating space rate maguilude earthquake forecasts. Bulletin of the Seismological Sociely of America, 100(3): 1184-1195, 2010.
Xiao-Gu Zheng and David Vere-Jones. Application of stress release models to historical earthquakes from north china. Pure and Applied Geophysics, 135(4):559 576, 1991.
Zihao Zhou and Rose Yu. Automatic integration for spatiotemporal neural point processes. Advances in Newral Information Processing Systems, 36, 2024.
Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference, pp. 777-789, PMLR, 2022.
Weiqiang Zhu and Gregory C Beroza. Phasenet: a deeg-neural-networkkkee seismic arrival-time picking method. Geophysical Journal International, 216(1):261 273, 2019.
Mark D Zoback, Mary Lou Zoback, Van S Mount, John Suppe, Jerry P Eaton, John Healy, David Oppenheimer, Paul Reasenberg, Lucile Jones, C Barry Raleigh, al. New evidence the state of stress of the san andreas fault system. Science, 238(4830::1105-1111 1987.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. International conference on machine learning, pp. 11692 11702. PMLR, 2020.
A Earthquake Catalog Data
A.1 Earthquake Catalog Generation
Data missingness, referred to in seismology as catalog (in)completeness, is the primary challenge faced with earthquake catalogs. It is an important and unavoidable feature, and is a result of how earthquakes are detected and characterisod. Below, we briefly overview the process of generating an carthquake catalog illustrate the data quality issues. In the subsequent section, we review catalog incompleteness and its potential impact the performance and evaluation of forecasting models.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Seismometers and Scismic Networks. A scismometer is instrument that detects and records the vibrations caused by seismic waves (Stein & Wysession 2009 Shearer 2019p. Tt consists of a sensor to detect ground motion and a recording system to log chree-dimensional ground motion over time, typically vertical and horizontal velocitics. Scismic networks, comprising multiple scismometers, monitor scismic activity at regional, national or global scales (see, e.g., (Woessner et al.; 2010) and relerences therein). High-density networks with modern, sensitive equipment provide more detailed and accurate data, enhancing the ability to detect and analyse smaller and more distant earthquakes.
From Waveforms to Phase Picking. The process of converting raw continuous seismic waveforms into useful carthquake data begins with phase picking, which identifies the arrival times of the primary (P) and secondary (S) waves of an earthquake. Historically, this was done manually, but now automated algorithms, such as the STA/LI /LTA algorithm, detect wave arrivals by analyzing signal amplitude changes [Allen 1982) Recent algorithms, such as machine learning classifiers (e.g. Zhu & Beroza 2019 Lapins et al.) 2021] and template-matching (c.g. Ross et al. 2019), can process much higher volumes of data efficiently and are often able to detect. events of much smaller magniludes.
Earthquake Association and Location After phase picking, the next step is to associate phases from different seismometers with the same earthquake. Simple algorithms require at least four phase arrivals to be detected on different. stations within a short time interval to declare an event. Once phases are associated, location estimation determines the earthquake's hypocenter and origin time by minimizing travel- time residuals using linearized or global inversion algorithms (Thurber 1985 Lomax et al., (2000). Given the potential for misidentified or mis-associated phase arrivals due to low signal-to-noise of small events or the near-simultancous occurrence during very active aftershock sequences, an automated system typically first picks arrival times and determines a preliminary location, which is subsequently reviewed by seismologist (e.g. Woessner et al. 2010 and references therein). Locations are typically reported as the geographical coordinates and depths where earthquakes first nucleated (hypocenters), although some catalogs report the centroid location, a contral mcasure of the extended carthquake rupture.
Earthquake Magnitude Calculation The magnitude of an earthquake quantifies the energy released at the source and was originally defined in the seminal paper by Richter (1935). The original definition, now referred to the local magnitude (ML), is calculated from the logarithm of the amplitude of waves recorded by seismometers. This scale, however, 'saturates" at higher magnit udes, meaning it underestinates magnitudes for various reasons. This led to introduction of the moment magnitude scale (Mw) (Hanks & Kanamori 1979), which computes the magnitude based on the estimated seismic moment Mo, which can be related to the physical rupture process via
Mo = rigidity rupture area x slip,
where rigidity is mechanical properly of the rock along the fault, rupture area is the area of the fault that slipped, and slip is the distance the fault moved. Mw is determined seismologically via a spectral fitting process to the carthquake waveforms. In practice, it can be challenging to use a single magnitude scale for a broad range of maguitudes, therefore a range of scales may be present within a single catalog, and approxinate magnitude conversion equations may be used to homogenize the scales (e.g. Herrmann & Marzocchi 2021 and references therein).
A.2 Earthquake Catalog Completeness
All of the EarthquakeNPP datascts are made publicly available by their respective data, centers in raw format. However, constructing suitable retrospective forecasting experiment from this raw data requires appropriate pre-processing. This typically involves truncating the dataset above magnitude threshold Mcut and within target spatial region to address incomplete data, known as catalog completeness Ma (c.g..
Mignan et al. 2011 Mignan & Woessner 2012
There are several reasons why an earthquake may not be detected by a seismic network. Small events may be indistinguishable from noise at a single station, or insufficiently corroborated across multiple stations. Another significant cause of missing events occurs during the aftershock sequence of large earthquakes, when the seismicty rate is high (Kagan & Knopoff 1987
Hainz] 2022). Human or algorithmic detection abilities
21

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Summary additional datasets, including: magnitude threshold (Me), number of training events, and number of testing events. The chronological partitioning of training, validation, and testing periods is also detailed. An auxiliary (burn-in) period begins from the 'Start' date, followed by the respective starts of the training, validation, and testing periods. All dates are given 00:00 UTC on January 1st, unloss January 17th).
noted (* refers to 00:00 UTC
Catalog
Mc
Mlart-Train-Val-Tests
Train Events
Test. Events
ETAS
2.5
1971-1188-1998--207222000
117,550
43,327
ETAS_ incompleeee
2.5
1977-1981-198--007222000
115,115
42,932
Deprecate
2.5
1990-1992-2007-200 1777-2011-2000 1-2020
22,213
15,368
are hampered when numerous events occur in quick succession, e.g, when phase arrivals of different events overlap different stations or the amplitudes of small events are swamped by those of large events. Since catalog incompleteness increases for lower magnitude events, lypically the lask is to find the value M, above which there is approximately 100% detection probability. Choosing truncation threshold Meut that is too high removes usable data. Where NPPs have demonstrated an ability to perform well with incomplete data (Stockman et al. 2023), typically threshold below the completeness biases classical models such as ETAS (Seif et al. 2017 Seismologists often investigate the biases of different magnitude thresholds by performing repeat forecasting experiments for different thresholds (e.g. Mancini et al. 2022 Stockman et al. 2023 which we also facilitate in our datascts.
Typically Me is determined by comparing the raw earthquake catalog to the Gutenberg-Riee law (Guten- berg & Richter 1936], which states that the distribution of earthquake magnitudes follows an exponential probability density function
faR(m) = -e 18mmmMm. m 2 M
(8)
where 3 is a rate parameter related to the b-value by blg 10. Histogram-basod approaches, such as the simple Maximum Curvature method (Wiemer & Wyss 2000) well many others (e.g. Herrmann) & Marzocchi, 2021, and references therein), identify the magnitude at which the observed catalog deviates from this law, indicating incompleteness (See Figure PD).
In practice, catalog completeness varies in both time and space Mc(l, x) (e.g. Schorlemmer & Woessner
2008). During aftershock sequences, Mc(t) can be very high (e.g., Agnew 2015
Hainzl 2016b) (See Figure
6)) Thresholding at the maximum value might remove too much data.
Instead, modelers either omit
particularly incomplete periods during training and testing (Kagan 1991
Hainzl al., 2008). model the
incompleteness itself (Helmstetter et al. 2006 Werner et. al. 2011 Omi et al. 2014: Hainzl
201 6a Mizrahi
et al. 2021 Hainzl 2022), or accept known biases from disregarding this issue (Sornette & Werner 2005
Spatially, catalogs are less complete farther from the scismic network (Mignan et al. 201 so the spatial
region can be constrained to remove outer, more incomplete areas (See Figure 6z).
B Additional Datasets
Beyond the official EarthquakeNPP datasets, we include 3 further datasets that eit provide additional scientific insight or continuity from previous benchmarking works.
B.1 Synthetic ETAS Catalogs.
We simulate a synthetic catalog using the ETAS model with parameters estimated from ComCat, at M, 2.5, within the same California region. A second catalog emulates the time-var varying data-missingness present in observational catalogs by removing events using the time-dependent formula from Page et al. (0066).
Mc(M. M/2 0.25 log)o(!);
22

---

Published in Transactions on Machine Learning Research (03/2026)
Mj(t)
Magnitude of Completeness (Mc)
a)
0B:00
09:00
10:00
11:00
12:00
13:00
14:00
Time
34,50
Catalog GR to Suocatalog
Target region
Mt, inside the
34.25
Magnitude of Completeness (Ma)
Mc catalog
2.0
Frequency (log scale)
10a
Events the Raw Catalog
34,00
Events the Target Region
33.75
Latitude
1.5
10a
10a
33.00
0.5
10a
32.75
32.50
117.5
-117.0
-116.5
-116.0
-115.5
b)
c)
ML
Longitude
Figure a) the June 10, 2016 Mw5.2 Borrego Springs earthquake and aftershocks, which occurred on the San Jacinto fault zone and is recorded in the WHITE catalog. An estimate of the magnitude of completeness Mc(t) over time using the Maximum Curvature method reveals more incompleteness immediately following the large earthquake. b) magnitude-frequene histograms reveal that truncating the raw WHITE catalog to inside the target region decreases M.. Each histogram is fit to the Witenbberg-iccheee (GR) law and an estimate of Mc for each catalog occurs whore the histogram deviates from the (GR) linc. c) An estimate of Me for gridded regions of the San Jacinto fault zone, using the raw WHITE catalog.
where M the mainshock magnitude. Events below this threshold are removed using mainshocks of Mw 5.2 and above. The inclusion these datasets allows us to test whether NPPs are inhibited by data missingness the same extent that ETAS is.
B.2 Deprecated Catalog of Japan.
To provide continuity from the previous benchmarking for NPPs on earthquakes, we also provide results the Japanese dataset from Chen et al. (2021), however with chronological train-test split and with- out removing any supposed outlier events. To reflect our recommendation not to use this dataset in any future benchmarking following the dataset completeness issues mentioned above, we name this dataset Japan_ Dearanteeeeceeed..
We can use this corrected dataset to quantify the inflation of performance caused by the non-chronological sranningvalidation-tooee splitting in the Chen et al. (2021) dataset. Table presents the information gain (difference in total log-likel,hood, section 2.1) relative to Poisson process for the three NPP models
23

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4:
NPP performance comparison on the Original Chen et al. 2021 dataset versus the
Japan_Deee Deprecated dataset. Values are reported in terms of information gain from son process.
homogencous Pois-
Model
Original
Chen et al. (2021 dataset Deprecaade
NSTPP
11.26
1.99
DeepSTPP
11.77
2.95
AutoSTPP
12.16
2.76
across the two datasets. The dramatic drop in information gain highlights how the original data split and omission of the 2011 Tohoku earthquake inflates model performance.
B.3 Likelihood Evaluation
Figures and S report the temporal and spatial log-likelihodd scores of all the benchmarked models on additional datasets. On synthetic data generated by the ETAS model the performance of NPPs mirrors the results on the obsorvational data (Figures 2 and 3). The performance of NPPs more comparable to ETAS in terms of temporal log-likelihood however they cannot capture the distribution of earthquake locations. Change in temporal performance of models between the ETAS and ETAS TTTS_incomplets datasets reveal each model's robustness to the missing data typically present in earthquake catalogs (See section A.2 Auto- STPP and ETAS reduce in performance upon the removal earthquakes during aftershock sequences, whereas DeepSTPP and NSTPP maintain the same performance indicating a robustness to the data missingness.
On the Jaapa_Deereeaaeeeeeeeeaeeeeeeeea temporal prediction performs comparably to NSTPP and is even marginally outperformed by DeepSTPP. This performance can be attributed to the data completeness issues of the Japan Deprecated dataset (sce
Deprecated dataset, whilst ETAS remains the best performing model for spatial prediction, for
section 1.1 where the test period is missing all carthquakes bellow magnitude 4.0.
ETAS
ETAS. incomplete
Japan Deprecated
Temporal log Bhafrood
1.0
0.9
0.5
DespsTop
ETAO
HSTW
naissen
Autesypp
DeepSTPP
ETAS
HSTP
Peisen
AutosTPP
DeeesTR Pp
ETAS
HSTPP
Persum
Figure 7: Test temporal log-likelihood scores for all the nantio-temporal point process models on each of the additional datasets. Error bars of the mean and standard deviation are constructed for the NPPs using three repeat runs.
C Effect of Training Window on ETAS Performance
To verify that fitting ETAS on both the training and validation windows does not artificially improve its performance relative to the NPPs, retrained ETAS using only the training window and re-evaluated its
24

---

Published in Transactions on Machine Learning Research (03/2026)
FTAS
ETAS Incomplete
japan
-10
Spatial log-Resthod
13
-30
-24
AubSTeP
Deepstop
NSTPP
Paism
Autostre
Desppi
ETAS
NSTR
Priston
Daaps
ETAS
NSTP
Possun
Figure 8: Test spatial log- log-likcl scores for all the spatio-temporal point process models each of the additional datasets. Error bars of the mean and standard deviation are constructed for the NPPs using three repeat runs.
test lig-iikelshoods. As shown in Figures & 10 the resulting log- likelihood scores are effectively unchanged across all FarthquakeNPP datasets.
2.0
Nabutte
8079
Hime
25
APP
Figure Test temporal log-likelihood scores for all the spatio-temppral point process models on each of the EarthquakeNPP datasets. Error bars of the mean and standard deviation are constructed for the NPPs using three repeat runs. ETAS (orange) is trained on both training and validation windows, whereas ETAS* (light blue) is trained only using the training window.
D Computational Efficiency
D.1
Training
Table
reports the training times for each model across all datasets. We ran all the NPP models using a HPC node with Nvidia Ampere GPU with 4x Nvidia A100 40GB SXM EXM-Ami "Ampere" GPUs and AMD EPYC 7543P 32-Core Processor "Milan" CPU using torch==1.12.0 and cuda==11.3.
ETAS training scales O(n²) with the total number of events, since for every event contribution the intensity function is computed from summation all previous events. This scaling, coupled with the
25

---

Published in Transactions on Machine Learning Research (03/2026)
ComCat
QTM SaliorSes
QTM Sardet
White
Sp00 hoy
Now
Partoon
200
SCEOC 20
SCHDC 25
SCEDC 30
Scutta Ao
400
Figure 10: Test spatial log- likelieiihood scores for all the spatio-temporal point process models on each of the EarthquakeNPP datasets. Error bars the mean and standard deviation are constructed for the NPPs using three repeat runs. ETAS (orange) is trained on both training and validation windows, whereas ETAS* (light blue) is trained only using the training window.
Table Training times for each model across all datasets, including the number training events. Times are formatted as HH-MM:SS, with days included for durations exceeding 24 hours. SMASH times are estimated 1.5x AutoSTPP, and DSTPP times are extrapolated assuming linear scaling from Salton Sea.
Dataset
# Training Events
ETAS
Deep-STPP
AutoSTPP
NSTPP
SMASH
DSTPP
Poisson
79,037
08:59-04
00:15:35
01:34:09
days, 05:10:17
2:21:13
20:05:57
second
QTN_SaltonSea
14,012
07:28:28
00:26:16
01:15:31
days, 00:26:15
2:38:21
11:12:00
<1 second
QTN_SanJac
18,084
00:32:40
00:09:31
00:37:03
day, 22:06:33
0:53:34
4:44:46
<1 sccond
SC5DC_20
days, 02:20:40
day, 8:37:05
30RDC_26 26
43,221
03:09-14
00:09-34
00:56:05
2 days, 16:33:55
1:24:07
10:59-28
<1 second
SCIDC_30
12,428
00:12:25
00:02:44
00:16:01
day. 16:39:04
0:24:01
3:10:26
second
White
38,556
03:55:10
00:08:21
01:10:51
days, 01:03:57
1:16:17
9-18-17
second
Degrectdd
22,213
06:09.08
00:13:45
01:02:07
2 days, 05:32:03
1:33:06
5:39:32
<I second
00:15:24
01:10:22
days, 03:09:17
1:45:33
day, 1:27:44
second
ETAS, Incorplate
116,115
00:35:14
00:15:29
01 01:09:43
3 days, 11:39-51
1:44:34
1 day, 2:28:42
<1 sccond
lack of parallelization in the current implementation, results in long training times for larger datasets. Poorer scaling will likely hinder ETAS if dataset sizes continue to grow in the future SSockman et al., 2024).
Encouragingly, both DeepSTPP and AutoSTPP are significantly faster to train due to GPU acceleration and their use of sliding window of the most recent k 20 events. While exact complexity analyses are not provided in Zhou et al. (2022) or Zhou & Yu (2024), we can infer that DeepSTPP likely scales as O(kn) since it benefits from closed-form expression for the likelihood. AutoSTPP, though requiring automatic
26

---

Published in Transactions on Machine Learning Research (03/2026)
integration to compute the likelihood, still scales with (kn) because the additional integration cost does not affect the overall scaling.
NSTPP, on the other hand, incurs significant training costs, rendering it impractical for real-time forecasting. Unlike the sliding window mechanism used in DeepSTPP and AutoSTPP, NSTPP partitions the event sequence into fixed time intervals, leading sequences that are much longer than the k: 20 events used by the other models (as shown in Figure 11 of Chen et al. (2021)). ). Furthermore, solving an ODE for each event time adds significant computational burden, even with the use of their faster attentive CNF architecture.
Whilst SMASH and DSTPP are built on the same backbone architecture, SMASH is much quicker to train than DSTPP, even faster than ETAS. This efficiency arises from its use of a single-step, normalization-free score-matching objective, which avoids the costly demoising and sampling loops required in diffusion-based training. SMASH directly learns the gradient of the log-density via pseudolikelihood estimation, enabling efficient GPU parallelization and bypassing the need for repeated evaluations over diffusson steps. In contrast, DSTPP simulates a sequential generative process over hundreds of intermediale steps per sample, significantly increasing computation and memory costs.
D.2 Inference
Whilst log-likelihood computation for ETAS, Poisson, DeepSTPP, AutoSTPP, and NSTPP is fast (< 30 sec- onds per dataset), real-time earthquake forecasting and CSEP evaluation require simulating many repeated event sequences (at least 10,000) over a fixed forccasting horizon. While ETAS training scales as O(n)) with the number of training events, its simulation is considerably more efficient, scaling approximately as O(n log n) due to its equivalent formulation as a Hawkes branching process (see Section 2.2 which enables immigration birth sampling.
Fast simulation is not currently feasible for several NPP architectures. NSTPP is not Hawkes- and requires solving a neural ordinary differential equation to generate each new event, rendering simulation prohibitively slow even for small datasets. DeepSTPP and AutoSTPP, while inspired by Hawkes processes, employ non-stationary neural triggering kernels that depend on the full event history. As a result, standard immigration birth sampling cannot be applied, since triggering relationships change after each event and new events cannot be generated independently or in parallel. Simulation via thinning also problematic: AutoSTPP does not enforce monotonic or decaying kernels, meaning the conditional intensity X X*(1, (1,x) cannot be safely upper-bounded, while DeepSTPP can in principle be simulated via thinning but is extremely slow in practice. For these reasons, DeepSTPP, AutoSTPP, and NSTPP are excluded from CSEP generative evaluation. For the models where large-scale simulation is tractable (ETAS, SMASH, and DSTPP), we report inference and simulation limes in Table D Based on simulation times, DSTPP is fastest, followed by ETAS, with SMASH consistently the slowest across datasets, reflecting the fact that DSTPP samples events via a fixed-length diffusion process with closcd-form updates, whercas SMASH relies on itcrative Langevin dynamics requiring many gradient evaluations per event.
Table 6: Simulation time for a batch of 100 repeated simulations across datasets. Reported times correspond to individual forecast days and are summarised minimun, median, mean, and maximum over the testing window, reflecting varialion in daily event counts rather than runtime stochasticity. Times are formalled as HH.MM:SS.
Dotesct
No. Days
Daily Counts
ETAS
SMASH
DSTPP
(min / med/ meon / mmx)
(mii med / meon max]
(eniis/ / medl / mean max)
[min med
mean maxe)
0/ /// 4.89 /
00:01:33/
01:09/25
00002:00
00004:00 (8:26500
0363120
Saltonbes
781
//2 5.03 2222
00:00.10 00:01.20
00:01:80
00:26-38
00:02:15 / 00:08.17 00:04:65/
02:0057
00:00/00 / 00/01.05 / 00:01:14 00:06:12
Sanlac
751
0. //b. 6.02 261
0000034 / 00:01:38/ / 00:01-41
00:22:11
00:00:25/ / 00:04:00 / 00x06:55
01:40:15
00:00112 / 0000L:15 / 80501:14
80000
2191
0./ 5.00 2134
00:00:23 06:01:26/
00:01:40
08:07.19
00:02:40 / 00:08-41
00.04:54
14:28211
00:00.07
0D00L110 / 00/0L:14/
02/58605
19/ 18.48
00:01:15
00:02:30
00:08:43
08:39:26
00:0:25
00113:66
27

---

Published in Transactions on Machine Learning Research (03/2026)
E Analysis of Likelihood Scores
E.1 Temporal Information Gain
To better interpret the temporal likelihood results presented in Figure 2 we analyse how model performance evolves over time in cach dataset in Figure 11 presenting the cumulative information gain (log-likclihood difference) of each NPP model over ETAS.
In Figure Ila on the ComCat dataset, the largest decreases in relative performance occur during the two largest sequences in the testing window, namely the 2010 El Mayor Cucapah M7.2 and the 2019 Ridgecrost M7.1 earthquakes. ETAS performs strongly in these periods, likely because it incorporates magnitude scaling which enables it to model large aftershock cascades effectively. In contrast, the 2014 South Napa M6.0 and 2014 Offshore Eureka M6.8 do not produce such a marked drop in relative performance. This suggests that capturing the largest aftershock sequences is a key limitation of current NPP models, while during background periods NPPs perform relatively better due to their ability to capture non-stationarity, a property directly not modelled by ETAS. Despite the overall worse performance of NSTPP, the relative decrease during large events not as sharp as the other models, suggesting superior performance.
A similar pattern appears in Figure T1b for the SCEDC dataset. All models show a sharp reduction in performance during the 2019 Ridgecrest M7.1 sequence. During the quicter period leading up to Ridge- crest, DeepSTPP performs comparably to ETAS. This again suggests that the lack of explicit magnitude conditioning limits NPP performance during large sequences and highlights NPP models ability to capture background non-staionarities.
Figures 11c 11d and [eessow results for the smaller regional datasels, SanJac, SaltonSea and White. These smaller magnitude, more regionally concentrated datasets, don't display the same constrasting 'background' "mainshock" behaviour present in ComCat and SCEDC. Although the overall performance of the NPP models is below that of ETAS, improvements occur during the 2017 Brawley swarm in the SaltonSea dataset, whereby DeepSTPP achieves higher temporal likelihood than ETAS. This behaviour aligns with the known difficulty ETAS has in modelling swarm-like sequences that are not initiated by a large mainshock.
E.2 Spatial Information Gain
To better interpret the spatial likelihood results presented in Figure 3 we analyse how model performance evolves over space in each dataset in Figures 12 13
visualising geographically where NPPs outperform or
fall behind ETAS.
Figures 12 and 13 show the spatial distribution of log-likelihood information gain across all datasets. Across both figures, consistent pallern emerges. ETAS performs best in regions dominated by large, magnitude driven mainshock aftershock sequences, while NPP performance degrades the immediate vicinity of these events. In contrast, NPPs tend to perform more competitively in regions characterised by spatially complex diffuse scismicity.
In the larger regional catalogs shown in Figure 12 this distinction is most apparent in the ComCat dataset. Near the Ridgecrest and El Mayor Cucapah sequences, NPPs exhibit reduced information gain relative to ETAS, consistent with the absence of explicit magnitude driven triggering. However, in the complex tectonic setting of the Mendocino Triple Junction, NPPs achieve comparatively strong spatial performance, with frequent positive information gain relative to ETAS. This region characterisod by interacting fault systems and persistent background activily rather than a single dominant mainshock, suggesting that NPPs are better suited to modelling such non-stationary and spatially heterogeneous seismic regimes.
Figure 13 presents results for the smaller regional datascts San Jacinto, Salton Sea, and White. These regions are dominated by lower magnitude seismicity and swarm like behaviour, and here NPPs again perform more competitively relative to ETAS. AutoSTPP assigns probability more smoothly along fault structures similarly to ETAS, resulting in information gain values concentrated near zero, while DecpSTPP and NSTPP show more heterogeneous behaviour. NSTPP produces more extreme and spikier information gain values that are spatially scattered, indicating less precise spatial localisation despite occasional high
28

---

Published in Transactions on Machine Learning Research (03/2026)
0.00
0.0
-0.1
Curulaative IG
Cumulative
-0.05
-0.2
-0.10-
DeepSTPP
DeepSTPP
AutoSTPP
-0.5
AutoSTPP
-0.6
Magnitude
7.5
Magnitude
7.5
5.0
2,5
2.5
01-jen-2009
o1 01-jan-2010
01-jan-2012
o1-jen-2014
01-jen-2016
D1-jen-2018
01-jan-2020
01-jen-2014
01-jan-2015
01-jan-2016
01-jan-2017
01-jan-2018
01-jan-2019
D1-jen-2020
Time
Time
(a) ComCat
(b) SCEDC
0.0
Cumulattive iG
Cumulative IG
-0.02
-0.04
-0.3
DeepSTPP
-0.4
DeepSTPP
-0.0
AutoSTPP
AutoSTPP
NSTPP
-0.5
Magnitude
5,0
Magnitude
5.0
2.5
2.5
01-jen-2016
2016
01-1ol-2016
01-0ct-0116
2017
0117
01-jd-20017
01-0kt-2017
01-jan-2018
01-jan-2016
0111p-2016
01-ju--2016
01-0ct-2016
01-jan-2017
o1 2017
01-juk-2017
01-0st-2017
01 01-jan-2018
Time
Time
SanJac
(d)
SaltonSea
0.00
Cumulative
DeepSTPP
AutoSTPP
-0.25
NSTPP
Magnitude
01-jam-2017
01-jul-2017
01-jam-2018
01Juu-2018
01-jen-2019
01Ju0-2019
21-jan-2020
21-0al-2220
01-jam-2021
Time
(e) White
Figure 11: Cumulative information gain (IG) plots for the temporal performance of all the NPP models with respect to ETAS on a) ComCat,b) SCEDC, e) QTM San Jac, d) QTM Salton Sea, White.
29

---

Published in Transactions on Machine Learning Research (03/2026)
likelihood assignments. Instability during training led to the drastic underperformance of NSTPP the White dataset and consequent low likelihood scores distributed across the entire region.
Overall, the spatial results mirror the temporal analysis. Current NPP architectures struggle most in main- shock dominated regimes but show clear promise in modelling spatially complex background scismicity and swarm driven activity, motivating futurr work on incorporating large magnitude triggering while preserving this flexibility.
DeopSTRP
4299
479
Soeial in Cai
Spekiistreeee Out
Scate inberatte Care
259
25%
245
22%
20OMT
120 1el5
127.0
HOW
100M
1175PM 1177 162'
1212
NOW HOO JEW 1055 UEW 112
(a) ComCat
DeepSTPP
30%
177
219
Social chendtmnann
(b) SCEDC
Figure 12: Spatial information gain per event, for NPP models relative to ETAS on (a) ComCat and (b) SCEDC. Scatter points correspond to the geographical location of the forecasted event, coloured by the value of the information gain over the ETAS model (green positive, red negative). For each model an inset plot displays the distribution of the spatial information gains for all events in the testing period.
30

---

Published in Transactions on Machine Learning Research (03/2026)
DeepSTRp
AutosTIR
NSTPP
660
Spale
26
PWIEER WISESPWEEE WWW.IGGWUSR
(a) SanJac
DeepSTPP
AutoSTiP
NSTPP
2224
105
DaY
30.7M
33.0M
195W 15575 DBE D5O 133W 13515
13F 115 DE 1ISAWW
DL 315
315P 11N 155M BIA JISS 3IE
(b) SaltonSea
DeepSTPP
AutoSTPP
NSTPP
3477
Spaaa vnrratin SE
DERATWIIN ROMM LSA 25A40 134A9H SELAN 20AUTM 1TR
137.0W STAIN SIEAS STAUPO SYY 316 135.57M
(c) White
Figure 13: Spatial information gain per event, for NPP models relative to ETAS on (a) SanJac, (b) SaltonSea, and (c) White. Scatter points correspond to the geographical location of the forecasted event, coloured by the value of the information gain over the ETAS model (green positive, red negativo). For cach model an inset plot displays the distribution of the spatial information gains for all events in the testing period.
F
CSEP Consistency Tests
F.1 Number (Temporal) Test
The number test evaluates the temporal component of the forecast by checking the consistency of the forecasted number of events, N with those observed in the forecast horizon, Nots- Upper and lower quantiles
31

---

Published in Transactions on Machine Learning Research (03/2026)
are estimated using the empirical cumulative distribution from the repeat simulations, FN.
P(N 2 Nobs) FN 1)
(10)
62 P(V I Nobss) FN
(11)
F.2 Pseudo-Likelihood Test
The pseuuo-likelihodd test evaluates the compatibility of forecast with an observed catalog using an ap- proximation to the space-time point process likelihood.
The test statistic is based on the pscudo-log-likclhoodd:
Nobe Σ
Lobs
log log (o:()) N,
(12)
where Å (A:)) is the approximate rate density in the spatial cell of the ith event, and N is the expected number of events.
Each forecast sinulation j provides lest statistic
N₂ f. log (k(g)) N, Σ
(13)
which is used to build the empirical cumulative distribution FL- The quantile score then computed
7L = IPLL; S Loos) = Fi(Co))).
(14)
F.3
Spatial Test
To evaluate the spatial component the forecast, a test statistic aggregates the forecasted rates of earth- quakes over a regular grid,
N
S=
Σ log A(ki)
N-1,
(15)
where Å(k:) is the approximate rate in the cell & where the ith event is located. Upper and lower quantiles are estimated by comparing the observed statistic
Nota
Sobs
Σ
log ((ks))
Nov-
(16)
with the empirical cumulative distribution of S using the repeat simulations, Fs
% = P < Sota) = Fs(Sobe).
(17)
The grid is constructed from {0.1%, 0.050,05000 0.00%,0.011} squares for ComCat, SCEDC and (QTM_Salton_ee) JTM__aaJac, White] respectively.
F.4
Magnitude Test
To evaluate the earthquake magnitude component of the forecast, test statistic compares the histogram of forecast's magnitudes A (m), against the mean histogram over all forecasts A (m))
Σ b
log Ā (m)) (k) + 1] log [ (m) (K) + 11)
(18)
32

---

Published in Transactions on Machine Learning Research (03/2026)
Test
Spatial
Magnitude Test
S0
300
SCBRC X
1799
1,00
u- 7.40
PV
536
of 0.15
POX at 0.05
MM (x)
By HMX 0.100.15
300
000
255
7.5
36.6
125
254
253
-11 -3
(a) Number test
(b) Spatial test
(c) Magnitude test
Figure 14: CSEP consistency tests on the ETAS model for the first day (01/01/2014) the testing period in the SCEDC catalog. A total of 10,000 simulations are generated to compute empirical distributions of the test statistics for each of the three consistency tests: (a) Number Lest, (b) Spatial test, and (c) Magnitude test. The test fails if the observed statistic falls within the rejection region (red), defined by the 0.05 and 0.95 quantiles of the distribution.
where ((m)(( (k) and A have the same total counts as the obscrved catalog. Upper and lower quantiles are estimated by comparing the observed statistic
(k) are the counts in the th bin of the forecast and mean histograms, normalised to
Dobs Σ (log A (m)) (A) + 1 -log A (md (R:) + 11)²
(19)
with the empirical distribution of D using the repeat simulations, FD
7m = PPD < Dans) FD((D
(20)
Histogram bins of size 6m = 0.1 are used across all datasets.
Although SMASH is, in principle, capable of modelling earthquake magnitudes, we restrict it to rate fore- casting in this benchmark, as exlending it to fine-grained magnitude prediction led to a deterioration in spatio-temporal performance metrics during training.
F.5 Evaluating Multiple Forecasting Periods
Savran et al. (2020) describe how to assess a model's performance across the multiple days in the testing period (Figure 15p. By construction, quantile scores over multiple periods should be uniformly distributed if the model the data generator (Gneiting & Katzfuss 2014). Therefore comparing quantile scores against standard uniform quantiles (y x), highlights discrepancies between the observed data and the forecast. Additional statements can be made about over-prediction
mnder-predictin of cach test statistic (quantile curves above/bellow y=x respectively). The Kolnogorov-Smirnoo (KS) statistic then quantifies the degree of difference to the uniform distribution for each of the tests.
33

---

Published in Transactions on Machine Learning Research (03/2026)
10a
(a)
10a
102
1.0
(6)
0.4
200
400
600
Frequency
[c)
2008
2010
2012
2014
2016
2018
2020
Figure 15: Daily mumber forecasts from SMASH the ComCat dataset. (a) Forecasted daily distributions for the number of earthquakes, with green lines indicating days where the observed count falls within the 95% forecast interval, and red lines where the forecast fails. Observed values are marked with dot sizes proportional to the number of earthquakes. (b) Quantile scores from the number test for each day, with red markers indicating failed forecasts. Marker size reflects the number of earthquakes observed on that day. (c) Temporal evolution of observed carthquakes during the testing period, with event magnitudes represented by marker size. (d) Histogram of quantile scores from the number test. Under ideal calibration, scores should follow uniform distribution. Red bars indicate failed forecasts, and the Saaeooooro---mmiiinnn (KS) statistic quantifies deviation from uniformity.
34

---

Published in Transactions on Machine Learning Research (03/2026)
G
Analysis of CSEP Tests
G.1
Temporal
To further interpret the CSEP consistency test results reported in Table Figures 16 18 show the daily event count forecasts produced by ETAS, SMASH, and DSTPP across all Earthquake datasets. These plots reveal systematic differences in how the models capture both background seismicity and sudden increases in carthquake rate.
Across all datasets, ETAS provides the most consistent. forecasts of daily event counts. It captures low- activity days well and responds more effectively than the Aaseb models to increases in seismicity rate, although it still underpredicts the largest rate excursions associated with major carthquake sequences. This behaviour is particularly evident in the ComCat and SCEDC datasets during the 2010 El Mayor-Cucappe and 2019 Ridgecrest sequences (Figures 16a and 16b and explains ETAS's consistently high consistency test pass rates.
SMASH exhibits highly variable daily rate estimates across all regions. While this variability occasionally allows it to match periods of elevated scismicity, such during El Cucapah in ComCat, it more often leads to pronounced over and under prediction. This spiky behaviour results in a substantial number of failed consistency lests.
DSTPP produces much smoother daily rate forecasts than SMASH, but this comes at the cost of systematic underprediction. Across all datasets, DSTPP underestimates both background seismicity and elevated rate periods, with the effect becoming especially severe in SCEDC and White (Figures 16b and 18). This persistent bias explains its low consistency test pass rates.
Performance differences across datasets largely reflect the dominant seismic regime. In smaller regions such as San Jacinto and Salton Sea (Figures 17a and 17b/. all models perform more competitively due to the absence of large mainshock-driven rate increases. However, even in these settings, SMASH remains overly variable and DSTPP continues to underestimate daily rates.
G.2 Spatial
19 and 20 show aggregated spatial forecasts over the entire testing period for ETAS, SMASH, and DSTPP across all FarthquakeNPP datasets. These plots summarise how each model distributes seismicity rate spatially when forecasts are integrated over time. While this provides a useful overview of long-term spatial structure, it does not distinguish whether high rates are forecast before events occur.
Figures
Across all datasets, ETAS produces the most spatially concentrated forecasts, with high rates aligned along known fault structures and low rates assigned to scismically inactive regions. This apparent spatial precision arises largely from ETAS's explicit modelling of clustering, whereby elevated rates are assigned in the vicinily of earthquakes that have already occurred. This behaviour is particularly evident in the ComCat and SCEDC datasots (Figures 19a and 19b where ETAS reproduces the largo-scale fault network and major clusters associated with past seismicity.
In contrast, SMASH consistently produces spatially diffuse forecasts. While sometimes captures regions of elevated activity, its rate is spread broadly across each domain, leading weaker spatial contrast between active and inactive regions. This effect is most pronounced in ComCat, where SMASH concentrates strongly around the southern end of the domain near the 2010 El Mayor-Cucapal sequence, while remaining diffuse elsewhere. Similar behaviour is observed in SCEDC, where SMASH is again dominated by the El Mayor region and assigns comparatively high rates across large areas of the domain.
DSTPP generally produces smoother spatial forocasts than ETAS but with greater structure than SMASH. In ComCat, San Jacinto, and Salton Sea (Figures [9a 20a and 20b DSTPP follows the main fault-aligned clustering while allenuating sharp spatial contrasts. However, in SCEDC and White, DSTPP substantially underestimates the overall seismicity rate, resulting in uniformly low spatial forecasts and visible boundary effects, particularly near the edges of the domains (Figures 19b and 20c
35

---

Published in Transactions on Machine Learning Research (03/2026)
M2.5 Events
10.-
SMASH
Pass % B6
MM2.5
DS:PP
Pass % m B5.7
10r
Date
ComCat
=222 Events
10a
ETAS
Pass % 92.6
10f
No. M2.0 Events
10a
SMARHH
Pass
100
M2.00 Events
10P
DSTRP
10a
2014
2015
2016
2017
2010 2019
2020
Date
(b) SCEDC
Figure 16: Daily number forocasts from ETAS, SMASH, and DSTPP over the full testing period for (a) ComCat and (b) SCEDC. Vertical lines show the forecasted daily distributions of earthquake counts. Green lines indicate days where the observed count falls within the 95% forecast interval, while red lines indicate failures. Observed daily counts are shown as dots, with marker size proportional to the number of carthquakes and colour indicating pass (green) or fail (red).
36

---

Published in Transactions on Machine Learning Research (03/2026)
ML. 1.0
SMASH
Pass 5 = 61.7
107
M..
101
DSTTP
Pass % U5.7
100
101
10r
Date
(a) SanJac
ETAS
Pass % 93.6
10f
No. M1.0 Events
SMAASH
Pass
10"
DSSRP
2016-01
2016-04
2016-07
2016-10
2017-01
2017-04 2017-07
2017-10
2018-01
Date
(b) SaltonSea
Figure 17: Daily number forocasts from ETAS, SMASH, and DSTPP over the full testing period for (a) SanJac and (b) SaltonSea. Vertical lines show the forecasted daily distributions of earthquake counts. Green lines indicate days where the observed count falls within the 95% forecast interval, while red lines indicate failures. Observed daily counts are shown dots, with marker size proportional to the number of carthquakes and colour indicating pass (green) or fail (red).
37

---

Published in Transactions on Machine Learning Research (03/2026)
ETAS
Pass % 38.3
MO.
SMASII
09.2
10/-
No. MO.6
107
No. Events
DSTPP
100
101
2017-01
2717
2018 01
2018 07
2019 01
2019-07 2020-01
2020
2023-03
Date
Figure 18: Daily number forecasts from ETAS, SMASH, and DSTPP over the full testing period for White. Vertical lines show the forecasted daily distributions of earthquake counts. Green lines indicate days where the observed count falls within the 95% forecast interval, while red lines indicate failures. Observed daily counts are shown as dots, with marker size proportional to the number earthquakes and colour indicating pass (green) or fail (red).
38

---

Published in Transactions on Machine Learning Research (03/2026)
ETAS
SMASH
DSTPP
42'N
-0.
0 worr
-1.
35'N
Hagrstudes
Magnitudas
Magniaate
-25
34
25
34'M
2.5
2.5
10010
43
-30
N2'
126 124" 322'W 120'W 116/V 130 134"H
126"W 324" 122'W 120'W 138" 136 110"W
120/W 124
1227W 120"W
118PW 116 114'
(a) ComCat
SMASH
DSTPP
37M
37TN
37TN
00
30'N
36'N
36TN
-1.0
N
35
34N
Hagnitudes
34"
Magnitudes
Magnibudos
2.8
-5
N
M3T
23
32'N
1"1.5WW 120" 118.'W 117" 1155W 134"
1215"W 120"W 119.5W 117" 115.5W 114
121.5WW 120' 118:'W 117"W 115.W 114"
(b) SCEDC
Figure 19: Spatial forecasts from ETAS, SMASH and DSTPP, aggregated across the entire testing period of (a) ComCat and (b) SCEDC. For each day in the testing period, every model simulates 10,000 repeated earthquake catalogs within the boundary region. All simulated catalogs aggregated across the entire testing period and the expected carthquake rates are plotted within each grid cell. Observed carthquakes are overlaid with marker size proportional lo magnitude.
39

---

Published in Transactions on Machine Learning Research (03/2026)
ETAS
SMASH
DSTPP
00
33.9
-0.5
pu 0.003 W02S
33.75%
33.75%
33.6%
13.6
33.674
-1
-20
080
33.2%
33.3
33.3%M
Hagnihdex
1.0
1.0
10010
33.15%
2.4
31-15'9
2.4
33.1574
2.4
-30
3.5
3.8
116.5'W
136.65'W
116.35W
116.0'W
116355W
116.65'W
136,35'W
136.05WW
136.50
136.65TW
116.35W
116.059
-35
(a) SanJac
ETAS
SMASH
DSTPP
33.3%
33.3'M
3..5
0.0
33.15%
33.1
23-15'N
-0.5
1.0
35%
257
357
$2.85%
12 $2.25
-2.0
32.2%
Nagnh.des
37.7%
Magnitudes
2,3%
Magnitues
23
211
32,587W
3.2
32.55'N
11.55N
-3.5
115.9W115.75 W11.1......11. 4S W
115.9 www.wa5.... 75 W115 W115 3"15
(b) SaltonSea
ETAS
SMASH
DSTPP
33.97N
13.0%
11.97M
-0.5
03 0.003 vears
33.75'N
23.751%
35.75/N
-1.0
-2.0
33.45'N
Magribudes
33.451
KeqnELddes
33.45'N
Magsibides
D.
0.6
0.6
2.1
2.1
33.15'N
15
11711W
116.'W
116.'W
116.2W
1171.
116.'W
1155.W
116.2'W
11711W
11621W
116.5WW
116.2WW
(c)
White
Figure 20: Spatial forecasts from ETAS, SMASH and DSTPP, aggregated across the entire testing period of (a) SanJac, (b) SaltonSea, and (c) White. For each day in the testing period, every model simulates 10,000 repeated earthquake catalogs within the boundary region. All simulated catalogs are aggregated across the entire testing period and the expected earthquake rates are plotted within each grid cell. Observed earthquakes are overlaid with marker size proportional magnitude.
40

---

Published in Transactions on Machine Learning Research (03/2026)
H
Further Dataset Figures
H.1 ComCat
Buneka Mww 7:3 North Coast
Landers My 7.3 Inland Empire
Hector Mine 7.2. Fastern
Baja Californ Mw 7.2 Boja Cal formia
Aldgecrest
Auxillory Wainingg Velidelion
Testing
b00Uy
Hoqnitde
20000
Start
Train
Vall Start
Test
End
Figure 21: Times and magnitudes events in ComCat dataset (with key events labeled). The size of the points are plotted on a log scale corresponding Mw. Auxiliary, training, validation and testing periods are indicated by colour and further cumulative count of events is indicated in red.
44%
Auxiliary
Training
Validation
Testing
36TN
32'N
127.5'M
125'W
122.5W
100W
117.5'W
115'W
112.5PW
Figure 22: Locations of events in the ComCat dataset, laboled by their partition into auxiliary, training, validation and lesting periods.
41

---

Published in Transactions on Machine Learning Research (03/2026)
H.2 SCEDC
Elmore Ranch Mw 6.5 Imperial Valley
Landers May 7.3 Inland Empire
Hector Mine fow Eastern
Uaja Califomia 7.2
Ridoecrest 7.1
Eastern
Taining
Validaaiin Testing
90000
Mapritode
O.mulatwe
60000
40000
20000
1980
1905
1950
1995
2000
Start
Irain Start
Val Start
lest Start
End
Figure 23: Times and magnitudes of events in the SCEDC dataset (with key events labeled). The size of the points plotted log scale corresponding to Mw. Auxiliary, training, validation and testing periods are indicated by colour and further cunulative count of events is indicated in red.
Auxiliary
Training
Validation
37
Testing
36
341N
33*N
32'N
31'N
121.5"W
120"W
118.5'W
117"W
115.5"W
114"W
112.5'W
Figure Locations of events the SCEDC dataset, labeled by their partition into auxiliary, training, validation and testing periods.
42

---

Published in Transactions on Machine Learning Research (03/2026)
H.3 White
Borrego Springs
Anza Borrege Mw 4.7
Bornego Springs 5:2
Triining Validaaioo Westing
Mocntude
30000
2010
2014
2010
Figure 25: Times and magnitudes of events in the White dataset (with key events labeled). The size of the points plotted on log scale corresponding to Mw. Auxiliary, training, validation and testing periods are indicated by colour and further cunulative count of events is indicated in red.
Auxiliary
Training
34.25
Validation
Testing
34NN
33.75'N
33.5'N
33.25'N
39"N
32.75
W 117"W 116.75 ww1.62555.666. 116"W 115.75"W
Figure Locations of events the White dataset, labeled by their partition into auxiliary, training, validation and testing periods.
43

---

Published in Transactions on Machine Learning Research (03/2026)
H.4 _an_aa
SanJac
Auxilarr Training Validation Testing
15000
Camulstive Count
Megnitute
3-
10000
2008
2009
2030
2011
2012
2013
2014
20055
2016
2017
2018
Time
Train
Val Start
Test Start
End
Figure 27: Times and magnitudes of events in the _TM_JanJac dataset. The size of the points are plotted on log scale corresponding to Mw. Auxiliary, training, validation and testing periods are indicated by colour and a further cumulative count of events is indicated in red.
H.5 QTM_SaltonSea
SaltonSea
Audiaary Training Validation Testing
30000
Oum dtive Court
Megnitutc
20000
10000
2008
Time
Start
Train Start
Val Start
Test Start
End
Figure 28: Times and magnitudes events in the STM_SaaonSSea dataset. The size of the points are plotted on a log scale corresponding Mw. Auxiliary, training, validation and testing periods are indicated colour and a further cumulative count of events indicated in red.
44

---

Published in Transactions on Machine Learning Research (03/2026)
Auxiliary
Training
34
Validation
Testing
San Jacinto Fault Zone
33.75'N
Salton Sea
33.25
32,751N
32.5%
Figure 29: Locations of events in the @TM_SanJac and QTM STM_SaltonSea datasets, labeled by their partition into auxiliary, training, validation and testing periods.
Error Distributions & Next-event metrics
Narmal JIN5O
35
Rornal R. (MCD
Semma (MRDE)
Lepace
34-
63-
22
Denoty
02-
42
2.1
3.0
2.5
Errona
Erars
Figure 30: The distribution of errors (Yobe pree for the Normal(O, 1), Exponential(1), and Pareto(2) distributions. Maximum likelihood cstimation is used to fit Normal and Laplace distributions to cach error histogram. Normal errors (Normal x Normal) are best approximated by the Root Mean Square Error (RMSE), while Laplacian errors (Exponential x Exponential) are best approximated by the Mean Absolute Error (MAE). However, neither RMSE nor MAE effectively capture the errors for the heavy-tailed Parcto distribution.
45


Reviews:
[{"review_id": "2lFR6HSAK5", "reviewer": "Reviewer_RSG5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces EarthquakeNPP, a benchmark to evaluation neural point processes on seismic data.\nThis benchmark includes multiple datasets on California earthquakes publicly available, as well as domain specific baselines (ETAS) and evaluation protocols tailored to the seismic domain.\nThe main evaluation is based on the classical log-likelihood on a separate test set, as well as seismic specific evaluation CSEP, which use 10000 simulations and compute various statistical tests.\nThe key conclusion is that none of the existing neural point process models outperform the ETAS model on this benchmark, demonstrating the need for further research in this area, and offering a reliable way to assess the progress of future models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- **S1:** The exposition of the paper is clear and easy to follow.\n- **S2:** The purpose of the benchmark is well motivated, and I think this is a step in the right direction to show that usual benchmarks can be misleading for real world applications. The performances of classical tools like ETAS need to be included in benchmarks to give the full picture on the performance of the model.\n- **S3:** The evaluation protocol, based on two metrics is well justified. The addition of CSEP is relevant for the seismic domain.\n\n---\n\n- **W1:** One weakness of the benchmark is the lack of common evaluation for all methods. In particular, likelihood is only reported for some methods, and CSEP only for others. This makes to completely grasp the strength and weaknesses of the method (See **C1** and **C2**).\n- **W2:**  Moreover, the evaluation proposed in the code is not centralized, making it very error prone (**C3**). The code for the evaluated methods is also not present in the supplementary materials. Overall, I feel more care should be spend on designing the benchmark to make it easy to use and extend for future methods, with common objects to load data and common evaluation protocol.", "audience_yn": "Yes", "audience_explanation": "It is somewhat relevant to TMLR audience that is developing spatio temporal PP models but may be missing some more complete evaluation, and some code to easily extend this benchmark.", "requested_changes": "- **C1:** While I understand that simulation might be slow with DeepSTPP, AutoSTPP, and NSTPP, I think it should be run at least on the smallest dataset, to show which order of magnitude this is. As these models relies on Hawkes model, it seems it is possible to run immigration birth algorithm for simulation, which should be faster than thinning. But even if it is not possible, evaluating on a small dataset would be useful to have a more complete comparison of the methods. For the SMASH method, the method seems able to predict magnitude but has been used with quantized magnitude in the original paper (small/medium/large). Wouldn't it be possible to have predict magnitude directly with the method? For the NLL metric, two methods are excluded. However, looking at the papers introducing these methods, DSTPP uses a variational lower-bound to estimate the NLL. This could also provide some insights on the performance of the DSTPP and SMASH models.\nOverall, I think having at least one common metric (possibly on a subset of the datasets for the slowest methods, or using approximations such as variational bounds) would be useful to make this benchmark more complete.\n- **C2:** I appreciate the fact that the authors report training time for the models. Adding runtime evaluation for inference would also be a useful addition to the benchmark, as in real world applications, inference time is often a bottleneck.\n- **C3:** The code provided does not include the methods other than ETAS. From the `README.md`, it seems that each method uses it own script to run. This should be improved as I think the benchmark would benefit from a more modular codebase, with common data loaders and evaluation protocols for all methods. With the current design, it is very error prone to add a new method, as a complete script needs to be written. Having a more modular approach would make it easier to extend the benchmark in the future, and avoid errors in the evaluation. I would recommend to have a common evaluation script that would take as input the predictions of the models, and compute all the metrics (NLL and CSEP). Note that some python packages such as [`benchopt`](https://benchopt.github.io/) provide dedicated tools to make it easier to build reusable benchmarks.", "broader_impact_concerns": "N/A", "additional_comments": "- Abstract: `seismology.To` -> `seismology. To`\n- *\"relied upon refinement of [...] ETAS model, despite significant growth in available data\"*:: It is unclear why the growth in available data would imply that ETAS should be outperformed by other models. This could be clarified, saying that this is a simple model, which may be limited in its expressiveness compared to neural networks.\n- NPP references: Neural Hawkes Process [A] would be a good reference to include in the intro.\n- Recent papers proposed parametric models for seismic data that could be mentioned in the related work, e.g., [B], [C].\n- Figure 2 and 3: It would be helpful to reorder the methods, to have the three NPP models grouped together, and ETAS/Poisson baselines together.\n\n**References:**\n\n[A] Mei, H., and Eisner, J. [\"The neural hawkes process: A neurally self-modulating multivariate point process.\"](https://arxiv.org/abs/1612.09328) NeurIPS (2017).  \n[B] Bernabeu A, Mateu J. [\"Spatio-temporal Hawkes point processes: statistical inference and simulation strategies\"](https://arxiv.org/pdf/2511.14509). arXiv (2025).  \n[C] Siviero, E., Staerman, G., Clémençon, S. & Moreau, T. [\"Numerically Efficient Parametric Inference for Learning Space-Time Hawkes Processes\"](https://openreview.net/pdf?id=VqfvOXyTPj). DSAA (2025).", "extra_scores": {}, "extra_text": {}}, {"review_id": "MzS5bq7sAu", "reviewer": "Reviewer_m3e9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThe authors provide EarthquakeNPP, a collection of public datasets together with a benchmark for evaluating neural point process (NPP) models on the task of earthquake forecasting.\n\nThey evaluate 5 different spatio-temporal NPPs, two of which are generative models, and compare their performance against ETAS (a model commonly used within the seismology community).\n\nEarthquakeNPP consists of 4 different datasets, documenting earthquakes within California from 1971 to 2021.\n\nThey find that none of the 5 evaluated NPP models outperforms the ETAS baseline, indicating that _``current NPP architectures are not yet suitable for operational use''_. They also briefly discuss a few directions for how NPPs potentially could be improved based on their findings. \n\n\n\n\n***\nStrengths:\n- The paper is quite well written and easy to follow overall.\n- The studied problem is quite interesting, and I definitely think that EarthquakeNPP could be a useful resource for some parts of the seismology and machine learning communities.\n\n\n\n\n\n***\nWeaknesses:\n- The technical contribution is limited. This is not necessarily an issue for TMLR, but still something that makes the paper less interesting than it could be. The paper would definitely be stronger if the authors also evaluated at least some type of potential model modification/improvement based on their findings and the discussion in Section 5.\n- The model evaluation/comparison is not overly detailed, and the analysis could definitely be more in-depth. For example, the 5 NPP models are not compared in terms of computational cost, and the trade-off between performance and computational cost for the past event history could be analyzed.\n\n\n\n\n\n\n***\nQuestions/suggestions:\n- Section 3 is very short, you are not close to the 12-page limit, and the appendix contains quite a lot more information about the datasets etc. Would perhaps make sense to add a bit more details to the main paper?\n- What is meant by SCEDC_20/25/30 in Figure 2 & 3? This is not explained (at least not in the main paper)?\n- Could you perhaps comment on the performance relative to the Poisson baseline in Figure 2 & 3? For example, the Poisson performance is poor on SCEDC_30 but basically the same on White in Figure 2, does this make sense?\n- In Figure 3, the performance of NSTPP on White seems to stand out. Does it make sense that this model performs poorly on this particular dataset? If so, why is its performance not poor on White also in Figure 2?\n- You write that DeepSTPP and AutoSTPP condition on just the 20 past events. How is the performance affected if this conditioning window is decreased even further (e.g., to 10 or 5 past events)? And what happens if you increase it to 40? Or, is this not possible at all due to computational cost reasons?\n- Section 5 is quite interesting, with the potential directions for how to improve NPPs. Could make sense to emphasize this more, e.g. by creating a list of \"Main actionable takeaways\" or similar?\n\n\n\n\n\n\n***\nMinor things:\n- Abstract, \"seismology.To address\": typo.\n- Abstract: \"future collaboration between the seismology and machine learning\" --> \"future collaboration between the seismology and machine learning communities\"?\n- Section 1: \"doesn’t compare against state-of-the-art models like ETAS\" --> \"does not compare against state-of-the-art models like ETAS\"?\n- 1.1.1: 'these sequences \"too long\" and \"outliers.\" However, this' --> 'these sequences \"too long\" and \"outliers\". However, this'.\n- 1.1.2: \"models that don’t output\" --> \"models that do not output\"?\n- Table 1: \"casts they release to the public.)\" --> \"casts they release to the public).\".", "claims_and_evidence_yn": "No", "claims_explanation": "The current model evaluation/comparison is not quite detailed enough, the analysis is a bit too shallow. See \"Weaknesses\" and \"Questions/suggestions\" above.", "audience_yn": "Yes", "audience_explanation": "The studied problem is quite interesting, and I think EarthquakeNPP could be a useful resource.", "requested_changes": "This is a quite well-written paper overall that I think could be relevant for the TMLR audience. I definitely think that EarthquakeNPP could be a useful resource.\n\nHowever, I think the current version requires some clarifications and tweaks, see \"Weaknesses\" and \"Questions/suggestions\" above.", "broader_impact_concerns": "No concerns.", "additional_comments": "No additional comments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mx4566bipb", "reviewer": "Reviewer_YgWk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a large earthquake dataset designed for evaluating Neural Point Processes (NPPs). In contrast to previous datasets used in this area, the proposed datasets cover target regions ranging from small to large scales. The paper also compares recent NPPs with the classical ETAS model, demonstrating that ETAS performs comparably to, or even outperforms, the NPPs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Although the discussion in this paper has room for improvement, I think its claims are supported by experimental evidence.", "audience_yn": "Yes", "audience_explanation": "1.\tProviding a comprehensive and large-scale dataset for evaluating Neural Point Processes is an important and timely contribution.\n2.\tThe surprising result that the classical ETAS model performs comparably to—or even better than—modern Neural Point Processes provides new insights and may guide future research directions in this field.", "requested_changes": "Major Points;\n\n1.\tThe review of Neural Point Processes is severely lacking. The paper does not clearly explain the motivation behind introducing Neural Point Processes, making it difficult to fully appreciate the significance of the experimental results. I strongly encourage the authors to include a clear explanation of the motivation and development of Neural Point Processes, and to discuss why the previous datasets were insufficient to demonstrate this motivation. \n\n2. The paper should elaborate on the insights and promising research directions inspired by the experimental findings. For instance, the authors say  in Section 5 that \n\n“Designing NPPs with scalable, long-term memory mechanisms is therefore a critical avenue for improvement.” \n\nHowever, it remains unclear how such mechanisms can be designed, or whether it is even feasible. Since this is one of the main implications of the paper, the authors are encouraged to elaborate on these points in more depth, with a clearer explanation of Neural Point Processes.\n\n\nMinor Points:\n\n3.. Abstract: “seismology.To address” → should be “seismology. To address”\n\n4. Section 2.2: “(ta,tb) ∈R” → should be “(ta,tb)⊂R” ?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)