EarthquakeNPP: Benchmark Datasets for Earthquake Forecasting with Neural Point Processes

Authors: Samuel Stockman, Daniel John Lawson, Maximilian J. Werner
Primary area: datasets and benchmarks
Keywords: Point Processes, Earthquake Forecasting, Benchmarking, Datasets

Abstract:
Classical point process models, such as the epidemic-type aftershock sequence (ETAS) model, have been widely used for forecasting the event times and locations of earthquakes for decades. Recent advances have led to Neural Point Processes (NPPs), which promise greater flexibility and improvements over classical models. However, the currently-used benchmark dataset for NPPs does not represent an up-to-date challenge in the seismological community since it lacks a key earthquake sequence from the region and improperly splits training and testing data. Furthermore, initial earthquake forecast benchmarking lacks a comparison to state-of-the-art earthquake forecasting models typically used by the seismological community. To address these gaps, we introduce EarthquakeNPP: a collection of benchmark datasets to facilitate testing of NPPs on earthquake data, accompanied by a credible implementation of the ETAS model. The datasets cover a range of small to large target regions within California, dating from 1971 to 2021, and include different methodologies for dataset generation. In a benchmarking experiment, we compare three spatio-temporal NPPs against ETAS and find that none outperform ETAS in either spatial or temporal log-likelihood. These results indicate that current NPP implementations are not yet suitable for practical earthquake forecasting. EarthquakeNPP also provides generative evaluation metrics, enabling broader model classes to be benchmarked and facilitating the future collaboration between the seismology and machine learning communities.

Decision: reject
Reviewer scores: 6, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730190194358

thread_type:
official_review

<thread_object>
creator: reviewer_nwwy

<content>
summary:
This paper constructs a benchmark dataset for earthquake forecasting using machine learning. It constructs datasets of various periods, regions, and sizes and provides an evaluation method for them. It presents the results of predicting the dataset using the epidemic-type aftershock sequence (ETAS), a classical prediction method, and three Neural Point Processes (NPPs) models.

strengths:
- Provides data over a more extended period of time, a wider area, and a wider variety of magnitude than existing benchmark datasets.
- The process of constructing the dataset was specifically explained. (Data period, scale, characteristics of each institution's data and etc.)
- The differences between the existing deep learning data split setting and the existing method were well explained.
- Various seismic domain-related experiments were conducted, such as number test, spatial test, and magnitude test.

weaknesses:
- An analysis should be added on which period, time, and location the NPPs models predicted accurately/inaccurately (Qualitative results).

- It would be better to provide specific experimental results or references rather than “Since the NPPs lack any direct dependence on magnitudes” as the reason why NPPs perform worse than ETAS.

- If a new methodology appropriate for the task is proposed through the above analysis, it would be a better study.

- It would be good to organize the advantages more useful in the machine learning community than the existing benchmark datasets and present them in a table.

- Adding real-world earthquake forecasting and response scenarios using NPPs will improve applicability.

questions:
- Can you explain the specific benefits of using classic ETAS compared to NPPs or other machine learning methods? For example, how do they differ in terms of computational load, efficiency, and cost? It would also be helpful to include the motivation for why solving this task with machine learning methods could be beneficial.
- Is there a technical reason for applying only NPPs among the various machine learning methodologies? It would be helpful if you could justify the choice of NPPs by providing a direct technical comparison with other machine learning methods. 
- Have there been any previous studies, or do you have any experimental results, that applied other machine learning methods (e.g., spatio-temporal methods) besides NPPs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615208794

thread_type:
official_review

<thread_object>
creator: reviewer_7xrx

<content>
summary:
The paper introduces EarthquakeNPP, the benchmark datasets designed to evaluate neural point processes (NPPs) for earthquake forecasting. It compiles earthquake data from California spanning 1971 to 2021 for model assessment. The study highlights that current NPP methods are not yet practical for real-world applications, as none outperform the Epidemic Type Aftershock Sequence (ETAS) model in terms of spatial or temporal log-likelihood performance.

strengths:
1. The work highlights the gap between the seismology and machine learning communities in earthquake prediction, providing potential directions for future development of neural point processes (NPPs) to improve earthquake forecasting.
2. It gathers earthquake data from California from 1971 to 2021, creating a valuable resource for research in both seismology and machine learning.
3. The benchmark datasets and experiments are publicly available, facilitating further research in the field.

weaknesses:
1. The scope of the work is limited. While the paper focuses on NPPs, various deep learning methods [1, 2] can also be applied to earthquake prediction. Including these methods would enhance the comprehensiveness of the evaluation.

2. The paper employs log-likelihood as the evaluation metric. Including metrics such as MAE and accuracy would offer a more comprehensive assessment from multiple perspectives.

3. The description of the benchmark dataset's statistics and characteristics is insufficient. A more detailed analysis, including aspects such as spatial nodes, the evolution of seismic activity, and the pattern similarity between training and testing earthquakes for each dataset, would strengthen the paper.

4. The analysis of model performance is limited. For example, why does Deep-STPP generally perform best in terms of temporal log-likelihood, while AutoSTPP excels in spatial log-likelihood?

5. Terms such as SCEDC_20, SCEDC_25, and SCEDC_30 are unclear and need clarification.

6. The paper's structure is somewhat inconsistent, with descriptions of additional datasets provided without corresponding results in the main text.

[1] Wang, Qianlong, et al. "Earthquake prediction based on spatio-temporal data mining: an LSTM network approach." IEEE Transactions on Emerging Topics in Computing 8.1 (2017): 148-158.
[2] Vardaan, K., et al. "Earthquake trend prediction using long short-term memory RNN." International Journal of Electrical and Computer Engineering 9.2 (2019): 1304-1312.

questions:
1. Why does the paper focus only on neural point processes and exclude other model types? 
2. Could you justify the use of log-likelihood as the primary metric while excluding other metrics such as MAE and R-score? 
3. Could you provide a more detailed description of the benchmark dataset's statistics and characteristics? 
4. Could you clarify the meaning of dataset labels such as SCEDC_20, SCEDC_25, and SCEDC_30? Do these refer to SCEDC data with magnitude thresholds of 2.0, 2.5, and 3.0, respectively?
4. Is there experimental evidence indicating that the degraded performance of NPPs compared to ETAS is due to their lack of direct dependence on magnitudes, rather than other factors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730633870497

thread_type:
official_review

<thread_object>
creator: reviewer_pnto

<content>
summary:
This work introduces EarthquakeNPP, a benchmark dataset for earthquake forecasting using Neural Point Processes (NPPs), aiming to enhance existing models. It covers seismic records from 1971 to 2021 in various California regions, including timestamps, coordinates, and magnitudes. Comparing NPPs with the ETAS model, it finds that current NPP implementations do not outperform ETAS in spatial or temporal log-likelihood, indicating that NPPs are not yet ready for practical earthquake forecasting. However, EarthquakeNPP serves as a platform for collaboration between the seismology and machine learning communities to improve predictability.

strengths:
(1) The dataset spans multiple regions in California over an extended period and includes earthquakes of various magnitudes, including smaller ones. This enhances data diversity and increases the dataset's granularity.

(2) It provides a direct comparison with the popular ETAS model, promoting collaboration between the seismology and machine learning communities.

(3) By utilizing modern seismic detection technologies, the dataset offers improved data quality.

weaknesses:
(1) The application of NPPs in practical earthquake forecasting is limited. For example, the high computational cost of generating repeated sequences may hinder real-world applications. I noticed that you used four A100 GPUs for training. Have you conducted any computational complexity analysis, such as comparing the training times between the baseline and NPPs? If you have other ways to present the computational cost, please include them as well.

(2) This work lacks sufficient detail on the reproducibility of experimental results, which may affect other researchers' ability to replicate the study. For instance, issues with the division of training and testing data in the existing NPPs benchmark dataset may prevent an accurate reproduction of real operational environments. Additionally, the interpretation of experimental results relies heavily on statistical data, lacking an in-depth analysis of the underlying physical mechanisms.

questions:
Refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680190562

thread_type:
official_review

<thread_object>
creator: reviewer_qezt

<content>
summary:
The authors have presented EarthquakeNPP, a group of datasets designed to evaluate/benchmark NPP models for earthquake forecasting. The authors evaluate some state-of-the-art NPP-based models against ETAS and have identified that these models demonstrate comparable temporal performance with the ETAS  model. Based on their analysis, the authors note that NPP-based models do not have a direct dependence on magnitude, in contrast to ETAS.

strengths:
**Originality**: While all the datasets are already available, the authors have created a pipeline and baseline and state-of-the-art methods and evaluation metrics to formalize the process of evaluating earthquake forecasting methods. Standardizing the evaluation method by comparing against the ETAS method that is used in the real world is a good idea.

**Significance**: The dataset collection created by the authors can serve as a benchmark for future research in NPP-based earthquake forecasting models. The authors have highlighted the limitations of existing benchmark datasets and why they need to be improved. 

**Experiments**: The authors have also provided initial assessments of state-of-the-art NPP-based models and highlighted the limitations of existing methods.

weaknesses:
Clarity: 
1. Section 1.1 The three categories defined in related work are hard to follow. A table highlighting the differences/limitations, or better defining the categories would help. For instance, "benchmarking efforts by NPP/ML community" , "benchmarking efforts by seismology community" etc. The existing benchmark dataset can then go into the first category. 
2. Section 5: Since each of the three tests is aimed at evaluating the temporal, spatial, and magnitude components of the forecast, renaming section 5.1 to "temporal test" is suggested.
3. Section 3 is better summarized as a table and includes additional information such as the number of training, validation, and test data. (This is currently included in the anon. github README). Additionally, the strengths/weaknesses/challenges of each dataset can be elucidated in this format, allowing the user to understand how each dataset is different from the other.
4. From the title, it seems as if the dataset is purely for benchmarking NPP-based methods, which might narrow the scope of the dataset. What needs to be done to generalize it for other methods capable of generating log-likelihoods such as bayesian inferencing-based methods?

questions:
1. Is it possible that since ETAS was trained using both training + validation data while the NPP models were trained on the training data, it is performing better? What is the impact of training data size on the evaluated models? Please conduct experiments such as evaluating ETAS using only training data.
2. Including a comparison of model complexity of the NPP-based models vs ETAS models in the evaluation metrics would be of great importance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732289726195

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated manuscript has been uploaded.

comment:
We sincerely thank all reviewers for their thoughtful feedback and valuable suggestions. We have now uploaded an updated version of the manuscript in which we have carefully considered each comment and made significant updates to the paper as a result. We have made both minor changes in response to the comments, as well as the following major changes:

1. Section 3 which introduces our datasets has been instead summarised as a table. (Reviewer: Qezt)
2. An expanded discussion of results has been included at the end of Section 4. (Reviewer: PNTo, 7xrX)
3. A "Computational Efficiency" section has been included as appendix Section C. (Reviewer: Qezt, PNTo, nWWY)
4. A justification on our focus on point processes has been included at the end of Section 2.1. (Reviewer: Qezt, 7xrX, nWWY)
5. The abstract and the end of introduction (Section 1) now clarify that our CSEP metrics broaden the scope of models beyond point processes. (Reviewer: Qezt, 7xrX, nWWY)
6. We have copied dataset information from the anon. github repository to the text. This includes dataset statistics included in the table in Section 3 and a new "Further Dataset Figures" appendix Section E. (Reviewer: Qezt, PNTo, 7xrX)
7. A new appendix, Section F, titled "2019 M7.1 Ridgecrest Earthquake Case Study," presents a real-time forecasting example, illustrating how to apply CSEP metrics for more qualitative analysis. (Reviewers: 7xrX, nWWY)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a new benchmark for earthquake forecasting by collecting datasets of various seismic records in different California regions across decades. The paper then specifically evaluates NPP models and show that these models show good performance compared to a current SOTA statistical model called ETAS, but fall short demonstrating the value of the benchmark in practical utility of NPP models.

Strengths: comprehensive collection of datasets and evaluation pipeline/workflow/metrics that can serve as a useful benchmark for seismology ML researchers

Weaknesses: The main weakness raised was that the paper was either narrow in its scope to focus only on NPP models without enough experimental evidence to discard other models or that the narrow focus on NPP was also limited in deeper analyses of these models. 

Points of improvement suggested:
* Broadened focus:  It would be beneficial to include other types of DL models as possible benchmarks (time series models and general sequence models). If these models are subpar by design, then the benchmark created by the authors would be an ideal vehicle to demonstrate this.
* Alternatively, the focus could be narrowed to NPPs but the analysis expanded on their performance. Suggested ideas were:

    - Analysis on which period, time, and location the NPPs models predicted accurately/inaccurately
    - Analysis connecting underlying physical mechanisms to the performance and in general, an analysis of the performance difference 
      between NPP models and how they connect to the underlying physics
    - Real-world earthquake forecasting and response scenarios using NPPs (while this is partially in the revised manuscript with the case 
      study, it does not evaluate NPPs)