OpenForecast: An Open-Domain and Large-Scale Dataset for LLM-Based Event Forecasting
Authors: Zhen Wang, Zhou Xi, Yating Yang, Bo Ma, Azmat Anwar, Lei Wang, Rui Dong


Reviewer paper summaries: The paper utilizes an open-source Large Language Model (LLM) to automatically construct a large-scale event prediction dataset in both English and Chinese. This dataset is extensive, avoiding data leakage for the LLM, and includes a set of complex prediction tasks. The paper conducts experimental analysis on existing open-source large models and proposes a new evaluation metric.
Summary of strengths: This paper
1. Constructed a large-scale event prediction dataset and proposed a corresponding set of tasks, and conducted experimental analysis on existing open-source large models.
2. Proposed a new evaluation metric, and the metric has been proven effective through experimental analysis.
3. Analyzed the reasons for the weak predictive performance of large models and the failure of fine-tuning large models, providing some help for subsequent improvements in the event prediction performance of Large Language Models (LLMs).
Summary of weaknesses: 1. The authors claim that one of the contributions of OpenForecast is the use of data from 2023 and 2024 as the test set, but this data was collected from public data and APIs, so it can only ensure that there might be no "Knowledge leakage" at the current point (when the data was captured). It cannot guarantee that subsequent LLMs will not use this data, and it is possible that some large models have already used this data for training or fine-tuning. Therefore, the contribution value of the dataset in this aspect is not obvious.

2. The authors claim that one of the contributions of OpenForecast is the proposal of three open-domain and three close-domain tasks, which were defined in section 2.1, but they did not list what tasks were already in the existing forecasting datasets. Are all six tasks proposed by the authors?

3. Although the paper has evaluated the quality of the dataset, it only assessed "event timeline on the completeness and the temporal correctness across 40 complex events" and provided that "approximately 77.5% of events simultaneously meet both criteria." But for such a large-scale event prediction dataset, is it sufficient to only assess the quality of 40 complex events? Are there other angles that affect the quality of the dataset? In addition, the paper did not provide specific methods for event selection, evaluation process, evaluation criteria, and the number of evaluators, etc. In summary, how to ensure the credibility of the dataset assessment results and the quality of the dataset?

4. In the experimental part, the LLM for Chinese was not evaluated in the "open-domain" and lacked necessary comparison with the results of the English LLM. The reason the authors claimed is "Considering the instability of automatic evaluation." If "automatic evaluation" is indeed very unstable, then the phenomena and conclusions observed in this paper are questionable.

5. In the dataset constructed, the Chinese time coverage is from 2022 to 2024, which is much less than the English from 1995 to 2024, and the two time spans are not balanced.

6. The paper did not use closed-source large models such as GPT for experiments and necessary comparisons.
Comments, suggestions and typos: 1. The dataset constructed in this paper includes both "open-domain" and "close-domain," and it is necessary to provide the necessary definitions and explanations for both.

2. There is an error in the caption of Table 2; the CE in the third line should be changed to AE.

3. In the table of experimental results, the best results are bolded to facilitate reading.

4. Typographical errors: 
   - 81 "been" should be "be";
   - 298 "show" should be "shows";
   - 315 "spaning" should be "spanning";
   - 553 "an" should be "a".
Overall assessment scores: 2.5