Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The Autocast dataset was introduced, a dataset for measuring the ability of neural networks to forecast future world events. Autocast poses a novel challenge for large language models and improved performance could bring large practical benefits. Additionally, Autocast questions were written by experienced forecasters and are always unambiguous given the full question description, in contrast to related work.
strengths: Well structured and mostly well described, easy to read, the difference from ForecastQA is significant, so the contribution is considerable.
weaknesses: It is not clear how to use "source_links" field and how CC news corpus is used.

Some article URLs from the dataset are unavailable: some redirect to another page, some require subscription.

correctness: Overall dataset structure is fine, but it is not clear for me how to retrieve articles from provided links correcty.

Evaluation methods and experiment design looks correct.
clarity: The paper is well written and easy to read, no objections here.
relation to prior work: It is described clearly how AutoCast dataset differs from ForecastQA dataset.
additional feedback: I would add a more detailed example from the dataset, i.e. how the crowd decision distribution changed over time, which news caused that and why.

Also the instruction for "source_links" field processing would be great.
review_title: Brief review of "Forecasting Future World Events With Neural Networks" paper.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces Autocast dataset that contains thousands of  **high-quality** forecasting questions and an accompanying  **temporal** news corpus. The paper is interesting and well-written.

strengths: - The paper is interesting and well-written.
- The dataset has **high-quality** forecasting questions and it has an associated **temporal** news corpus. The problem is somehow well-defined.
- It design some simple baselines and the results seem making-sense.
weaknesses: - I am not sure if all these questions are **predictable** given the *temporal* news corpus (especially top ten news based on a ranker). Could  the top-ranked news  provide enough inductive bias to predict the questions？This might not make sense from agnosticism point of view.
- FiD Temporal does not (significantly) outperform FiD Static. One might wonder what does this really mean for  incorporating the auxiliary crowd predictions.
- In line 235, should Table 3 be Table **2**?
correctness: it is constructed in a sound way. But the soundness should be further discussed, for exmaple, should all these questions be predictable from these temporal news?
clarity: yes
relation to prior work: seems good
additional feedback: see weakness above
review_title: Interesting paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose a new Autocast dataset for future events forecasting under the QA format. The raw data comes from public forecasting tournaments, which ensures the quality of the event forecasting questions. The dataset also includes news corpora corresponding to each question that can be leveraged by forecasting models. A benchmark including human performance is also introduced.
strengths: This paper introduces a new dataset based on public forecasting tournaments. It expands the existing future events forecasting QA research. It will bring new research material to the related communities.

weaknesses: 1. About human performance comparison. Based on my understanding, the human aggregated performance is provided by domain experts or enthusiasts (crawled from forecasting tournaments?). Different categories probably have different experts with lots of domain knowledge. However, for a deep learning model, it is very difficult to gain knowledge of all domains. So, the large gap between models and human performance reported in the table is no surprise. From my point of view, a more reasonable human performance collection at the current stage is collecting the performance from multiple crowdsourcing workers and each worker should answer questions from multiple categories (at least one question from one category) using news content before the closing date as input information. I agree that this extra human performance collection would take much extra effort. But I think it would also be a more practical indicator for evaluating further Autocast models.
2. It seems that no information about the distribution of the question types (i.e., T/F, MCQ, and Numerical) in Autocast. It could further strengthen the selection of the combined Score metric. For example, if the majority of the dataset is MCQ as well as T/F questions, a model with a strong ability in these types but worse Numerical performance would also have a relatively high combined score. 
3. Why not report the error bars of the investigated models? The results would be more convincing if different random seeds are applied for running experiments multiple times.
4. For UnifiedQA. According to the paper, the reported result is the zero-shot performance (without any fine-tuning?). Is there a particular reason for not fine-tuning UnifiedQA on Autocast but T5 is fine-tuned? 
correctness: 1. About unresolved questions. Will these questions be included in the future Autocast versions if they are resolved?
2. About the claim in Line 137. It points out that the unresolved questions can still be used as trained data with crowd forecasts as labels. However, the experiments conducted in the paper didn't investigate this claim (Line 67 Supp, unresolved questions are not used in baselines). Maybe considering a comparison (T5 with using unresolved questions in fine-tuning VS. without unresolved questions) to further justify this.
clarity: Overall, the paper is well written. 
Please consider reorganising the paper to make the corresponding tables/figures closer to the paragraphs that discuss the tables/figures.
relation to prior work: 1. Missing prior related work in the discussion:
[1] A Dataset for Answering Time-Sensitive Questions (neurips21, dataset track)
[2] Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs (ACL)
Especially, I think TimeQA (reference [1]) is a relevant paper that should be considered.

2. Since this work is very close to the previous ForecastQA, I suggest including a statistical comparison (e.g., number of data instances, date ranges, etc.) of Autocast and ForecastQA.
additional feedback: 1. Please add the venue information for ForecastQA reference in the bib (Line 373). It is accepted here: https://aclanthology.org/2021.acl-long.357/
2. Since the proposed dataset is based on public forecasting tournaments, it would be better to introduce the licensing information of the tournaments (if available).
3. Line 147 states that if a model is pre-trained on more recent data, it will not simulate forecasting faithfully. Any possible measures to avoid this? Otherwise, I think this would affect the future usage of the proposed dataset. For example, future researchers are highly likely to design DL models based on pre-trained language models for Autocast and they probably won't pay much attention to the data used for pre-training (e.g., using off-the-shelf models). It would lead to an unfair benchmark comparison.
review_title: Official Review of Dataset Track Paper441 by Reviewer yQuF

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: In this paper, the authors describe their efforts to (1) curate a dataset of forecasting questions (taken from forecasting tournaments) and answers (drawn from a news corpus) and (2) provide an early evaluation of popular modeling for at answering these forecasting questions. Great care was taken so that there is no time-based data leakage (i.e., machines can simulate learning facts about the world in the order they occurred).

The framing of the problem is very convincing, i.e. the paper makes a strong argument that this dataset will be important for a variety of applications and fields. The Introduction does a great job motivating the work and summarizing the key contributions. Related Work is focused on situating related technical work on forecasting, IR, QA.

As noted below, the clarity of the work is also great, which enhances the overall contribution.
strengths: - Important, well documented contribution of a novel dataset with very high potential for real world impact.

- Clear and concise paper that explains why this dataset is useful and provides details about how it was collected, and how similar datasets might be collected in the future.

- The paper is convincing that this is a pretty important domain for research, i.e. it is fairly important to have discussions in the community about benchmarks in the forecasting space.

- Early stage evaluations are also provided which may save researchers working with this dataset substantial time.
weaknesses: Minor: the main paper is sometimes a bit vague in discussing the specific scraping and manual verification process. Specifically, any additional details that can be provded about how the human curation of data was performed will probably be helpful for this kind of work, given that this (1) a dataset contribution and (2) this is the kind of task where people with different backgrounds may make different curation decisions. As I understand, full details on the questions side of the data about how the *questions* were curated may not be possible, as the platforms hiring moderators to curate questions are private platforms.

That said, the general topic of scraping and verification is briefly discussed several times in the supplement and code is available for replication. The info provided in Section 3 is very likely enough for independent replication.

Additionally, were there major differences in the curation aspects of the main dataset vs. the calibration dataset?

It may be worth noting that the test set only lasts for one year (in slight contrast to the example given of using the collapse of the Soviet Union as a motivating example), just to avoid the small chance that readers may think that examples like Soviet Union example appear prominently in this test set.

Finally, it may help to discuss whether a more complete “backtesting” approach from finance is feasible here. Does the LM training cost make this prohibitively expensive? Should researchers in this space be aiming to work towards comprehensive backtests in the long run?

All these concerns are very minor. Hopefully these are helpful in the strengthening the paper, but I’d argue for accepting the paper even with very minor revisions.
correctness: The described train/test split and baseline evaluation seem reasonable. The modeling choices also seem well justified. Evaluation metrics also make sense. A good amount of details about training are provided, (e.g. training epochs, etc.), which I assumed was mainly for reproducibility as no specific results seem to be dependent on training choices here.

The actual data scraping process also seems correct. As noted above, more details about manual checking process would be helpful (random sample? how many observations were manually inspected checked? etc.)
clarity: Excellent. All parts of the main paper and supplement are very clear.
relation to prior work: Good coverage of recent work in information retrieval, which I think gives reader necessary context to understand how the dataset contribution adds to existing work. Overall, as I understand, this dataset is very novel.
additional feedback: Thanks to the authors for this contribution!
review_title: Strong dataset contribution for work on forecasting models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a dataset of questions and answers for predicting world events. The dataset includes the questions, the start and close dates, the answers (if the questions have been resolved), the time-series of crowd forecasts by human experts, and an accompanying news corpus sorted by date collected from CommonCrawl. The authors state that they intend to update the dataset regularly with new questions and resolutions.

This dataset has diverse question types including multiple choice questions (MCQ), True/False questions (T/F), and numerical questions. The dataset covers diverse topics such as politics, economics, society, science, and a category for everything else called *other*, and altogether they have a total of 44 subcategories. The questions have different time horizons; some questions are resolved while some of them are still unresolved. The dataset is collected from public human forecasting tournaments and amounts to 6,706 questions in total. The authors balance the True/False questions by adding negated questions using GPT-3 and then manually checking for correctness. Moreover, they include an auxiliary dataset of numerical questions that includes about 30,000 questions. This auxiliary dataset provides the groundwork for measuring calibration.

Train/test split is done based on the resolution date of questions so that questions in the test set all resolve after a specific date, and the training set includes the questions that resolve before that specific date.
The authors prevent the models from overfitting/memorization by using only those news articles that were available before the resolution date of a forecast.

The authors train two sets of models. The first set is models without retrieval including UnifiedQA-v2 and T5, and the second set is retrieval-based models including *FiD static* and *FiD temporal* that use news articles. FiD static uses the top 10 retrieved articles, while the FiD Temporal uses the top 1 article and in addition leverages the intermediate crowd predictions (before the question is resolved) as auxiliary supervision. Their results show that their temporal method neither harms nor helps.
They find that retrieval-based models that take advantage of relevant news articles outperform the models without retrieval and become even more effective as the number of parameters increases. However, all these language models still perform significantly worse than the aggregate human forecast (which is a time series of forecasts from start to resolution date).

strengths: 1.  The problem that the authors address is of great importance. Forecasting future world events can be helpful in a variety of domains.
2.  The authors show that the state-of-the-art language models are significantly worse than the human baseline, and this dataset can open up a great opportunity for researchers to improve the language models for forecasting.
3.  The paper is very well written and details are explained.
4.  This dataset is more accurately collected and is more realistic compared to prior works.
5.  The authors take the necessary precautions to avoid leakage from the training set to the test set.

weaknesses: Most of the weaknesses are minor.

1.  In the "Metrics" section, it is worth mentioning that for T/F and MCQ metrics the higher is better, while for the numerical metric the lower is better to avoid confusion for readers. Also, in the combined score metric which is (T/F + MCQ - Numerical)/2, why is the denominator two instead of three?
2.  Are collected news from CommonCrawl only used for training or do you use them for the test as well? Also, are they only used for retrieval-based models? Please clarify these and state them in the paper.
3.  Would you please clarify why you use the top 1 article in the FiD Temporal and top 10 articles in the FiD Static? That does not seem to be a fair comparison.
4.  On lines 209-211 you say that you limit the article length to 512 tokens? Please specify whether you consider "tokens" to be words or characters.
5.  In table 2, column Numerical, why 19.5 is bold given that it is not the max?
6.  X axis in figure 1 does not have a label. Although it can be inferred by the reader, it would make it more clear to indicate that the X axis is the date. Also, is the prediction percentage for model (blue line) an average of multiple models? Since the graph is the performance for a single binary question, I assume it is an average of multiple models, but it needs to be clarified what models.
7. Additional graphs similar to figure 4 illustrating the FiD static model and the two models without retrieval would be helpful for understanding the differences among all these models.

correctness: The dataset is constructed in a correct method. The evaluations are thorough and valid. The only concern is mentioned in item three of the "Weaknesses" section.
clarity: The paper is very easy to read and keeps the reader engaged. The methodology is clearly explained for most parts. Refer to "Weaknesses" section for more details.
relation to prior work: The authors position their work very well with respect to different works related to the topic of forecasting including calibration, information retrieval, and forecasting itself.
additional feedback: 1.  Are there any open-ended questions in the dataset such as "who is more likely to win the next Nobel prize?" where the answer is not numerical and is not also limited to multiple choices to choose from? If yes, please mention it in the paper. If not, why did you choose not to have them, and do you have any plans to add them?
2.  On line 138, you mention that your dataset also includes metadata. Would you please state whether you use these metadata for training or not?
3.  Do you have any insights or hypothesis as to why the performance decreases for numerical questions when you increase the number of parameters of both FiD models?
4.  I understand your argument regarding leakage from training to test if we have access to information/news after the fixed date in the training set. However, I am confused by the following statement on lines 40-42. "When it comes to learning, humans cannot be trained on past data (e.g. How likely was the Soviet Union’s collapse from the viewpoint of 1980?) because they know the outcomes – but past data can be used for ML models." Humans study history and can be trained (in a supervised manner) using past data. Also, data by definition is something that has already happened (in the past), and the term "past data" is confusing to me. With that said, I guess maybe you meant that humans cannot be **evaluated** on predicting events that have already happened, which I agree with. Please correct me if I am wrong.
review_title: An interesting and challenging dataset for forecasting

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The submission provides clear motivated and well written contribution to establishing datasets that test the ability of learning algorithms to build models forecasting future world events of importance from available text news corpora. The authors gather the dataset (Autocast) comprising 6707 questions by making use of three public forecasting tournaments (Metaculus, Good Judgment Open and CSET Foretell) that feature human forecast experts crowd. Questions contain true/false, multiple choice and numerical ground truth answers. Text news corpora is curated from Common Crawl and is organized by date, allowing to mimic the conditions under which humans make past forecasts, avoiding leakage from the future information.

Authors further present baseline approaches for learning forecast models from the presented dataset. They compare approaches without retrieval (based on zero-shot UnifiedQA-v2 and T5 fine-tuning) and with retrieval (based on BM25 for retrieval and on Fusion-in-Decoder, FiD, for encoding retrieved news articles). The retrieval-based method is either considering final output and top-10 ranked retrieved articles (FiD static) or takes into account human crowd predictions evolving in time between start and closing of a question (FiD temporal, using GPT-2 autoregressive model for intermediate forecast outputs).


Comparing baseline methods to human crowd performance, authors observe advantage for retrieval-based methods (FiD static, FiD temporal) against non-retrieval ones (that perform close to random guess level on the task). The observed gap to the human performance is substantial, which make authors call for further development of forecast learning algorithms. The authors do not observe substantial difference between FiD static and FiD temporal retrieval-based methods, with the conclusion that further efforts have to be undertaken to make better use of auxilliary losses for the forecasting task. 

Further, the authors  provide an additional dataset to calibrate output of the forecasting models on numerical answer, where they source the questions from NLP datasets covering diverse topics and with answers varying across orders of magnitude. They observe that with increasing model size (using DeBERTa-v3 model backbone), calibration improves along with other performance metrics.
strengths: The presented work deals with important and challenging question of constructing datasets that can be used to test the ability of learning algorithms to build models that forecast future world events from large corpora of text data. Authors take care that training procedure and train/test split respect the notion of forecasting problem and cleanly separate past and future data. Provided baselines check dataset sanity and show clear gap to human crowd performance, providing motivation to use the dataset for further development and improvement in this important direction.
weaknesses: One concern in my opinion is a small scale of the questions dataset. Few thousands of those may by far not be conclusive enough to evaluate performance of state-of-the art algorithms at scale. 

Further limitation (minor, as it can be readily lifted) is in my eyes decision to use only news text corpora for training. Arguably, other available sources of information (eg stock exchange behavior, weather, text corpora like Wikipedia dated before the relevant events to forecast) may be helpful to enhance learning.
correctness: The claims put forward by authors have sound back up in the conducted experiments.
clarity: The paper provides good motivation for the subject of building datasets for benchmarking algorithms that learn future world events forecasting models and is well and clearly written.
relation to prior work: Prior work is properly addressed and discussed thoroughly.
additional feedback: - Would it be possible to extend questions dataset in semi-automated manner by taking some of the already available questions and rephrasing those in different time context (for instance, Will X become a president in Y year can be re-used for multiple time points and multiple person candidates; same procedure applied to other topics may help to expand the dataset profoundly)
review_title: A dataset composed of human forecasting tournaments questions and a curated news text corpus (date sorted, to be released soon), as benchmark for learning future world events forecasting models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduced a judgmental forecasting question dataset Autocast (accessible) along with a news corpus (not released yet) with date information for the purpose of facilitating the research of using language models to forecast future events. The authors also provided several benchmarks constructed using various popular language models and time-series models as well as human crowd predictions. The benchmarks performance also revealed that retrieving relevant articles could help improve the performance of machine forecasts, while there still existed a large improvement space for machine forecasts when compared to human crowd forecasts. Although there are several existing mature forecasting datasets, Autocast along with the news corpus with date information is still a progress as they provide an environment for language models to simulate the real forecast environment (if the news corpus is released and can be verified). 

strengths: ### Significance of the contribution

The Autocast dataset along with the news corpus have the following features, which, as a combination, is unique. 
1. Forecasting questions are professional, i.e., coming from three well-known public forecast tournaments (GJP, Metaculus, and CSET Foretell)
2. Forecasting questions and news articles have date information so that they allow ML models to simulate the real forecasting environment and avoid information leakage. 

### Relevance to the broader research community
I believe forecasting future events is an important topic that have many real-world applications. The provided dataset will help the general forecasting community as well as the machine forecasting community. 

### Accessibility and accountability
Autocast is easily accessible on Github, with most fields clearly defined. 


weaknesses: ### Significance of the contribution

It seems that the dataset and the news corpus are all tailed from existing datasets and projects. However, the effort should still be meritted. 

### Accessibility and accountability
* The accompanying news corpus dataset is not released according to the github page.  

* Resolved date is missing in the dataset, which I believe is very important as forecasts only make sense before resolution and the resolved date may be earlier than the given close date. In other word, the information leakage problem is not entirely solved if the resolved date is not provided. 

* The human crowd forecasts is an important feature as well as a good target to learn in this dataset. However, the authors did not clarify how the human crowd forecasts are generated or obtained. 

* It would be better if the authors introduced the distribution of the time scope of the forecasting questions, e.g., how many of them are short-term questions, e.g., taking weeks to resolve, and long-term questions, e.g., taking one or two years to resolve. 
correctness: The forecasting question dataset Autocast is constructed in a sound way. The forecasting questions are from three serious forecast tournaments and most of the important information, including the open date, close date and resolution status. The questions are well-defined and covered a wide range of important topics and geographic areas. The authors also added negated questions to balance the ground truth labels. A small question is that why the authors added these negated questions instead of replacing the original questions, as adding negated questions may change the distribution of the original questions. 

I believe that the benchmark evaluation is well-conducted. The authors used percent accuracy as the metric, which is intuitive and straightforward. However, for evaluating forecasting for T/F for multiple-choice questions, the proper scoring rules, especially the Brier score (MSE) are the most widely used metric, which also evaluate the calibration of forecasts. I was wondering why the authors did not try this metric. 
clarity: The paper is well-writen. 
relation to prior work: The related work is properly discussed. 
additional feedback: Seen questions left in the above aspects.
review_title: A nice forecasting question dataset accompanied by a news article corpus (not released yet) with date information and fair benchmarks

Meta-review: metareview: The paper presents a novel dataset on the task of judgmental forecasting. Autocast was collected by scraping questions from public forecasting tournaments, accompanied by a news corpora corresponding to each question. The reviewers agree that this is a high quality and novel dataset with diverse questions and tasks. It has a potential for high impact given the benefit of the datasets. The paper is clearly written, accompanied by a detailed data sheet.
After consulting the ethics chairs, I would like to recommend conditional acceptance of this paper, pending minor revision.

The authors clarify that they received permission from the source websites when possible, directly addressing the concerns of Ethics Reviewer nK2v and Reviewer qpTi.  This additional context needs to be mentioned explicitly in the paper and supplementary materials. 
Authors should state the following in an appendix or footnote to the main text:
Full permission for Metaculus CEO, Gaia Dempsey
CSET-Foretell website does not have a Terms of Use but the data is only being used for academic, non-commercial purpose, which is consistent with CSET’s purpose as a policy research organization and fair use.
The Terms of Use for the Common Crawl website are being used. However, to further compliance, the link to the cc_news dataset is removed from our GitHub page. The dataset is fully reproducible by including the script to download and filter cc_news on GitHub.
 
The terms of use of the involved platforms addresses the concern for the presence of potentially offensive content. 


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept