AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval
Authors: Qi Yan
Raihan Seraj
Jiawei He
Lili Meng
Tristan Sylvain
Venue: ICLR 2024 poster (2024)
Keywords: world event prediction
large language models
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a zero-shot ranking-based context retrieval technique to enhance performance on the AutoCast world event prediction dataset.

Abstract: Machine-based prediction of real-world events is garnering attention due to its potential for informed decision-making. Whereas traditional forecasting predominantly hinges on structured data like time-series, recent breakthroughs in language models enable predictions using unstructured text. In particular, (Zou et al., 2022) unveils AutoCast, a new benchmark that employs news articles for answering forecasting queries. Nevertheless, existing methods still trail behind human performance. The cornerstone of accurate forecasting, we argue, lies in identifying a concise, yet rich subset of news snippets from a vast corpus. With this motivation, we introduce AutoCast++, a zero-shot ranking-based context retrieval system, tailored to sift through expansive news document collections for event forecasting. Our approach first re-ranks articles based on zero-shot question-passage relevance, honing in on semantically pertinent news. Following this, the chosen articles are subjected to zero-shot summarization to attain succinct context. Leveraging a pre-trained language model, we conduct both the relevance evaluation and article summarization without needing domain-specific training. Notably, recent articles can sometimes be at odds with preceding ones due to new facts or unanticipated incidents, leading to fluctuating temporal dynamics. To tackle this, our re-ranking mechanism gives preference to more recent articles, and we further regularize the multi-passage representation learning to align with human forecaster responses made on different dates. Empirical results underscore marked improvements across multiple metrics, improving the performance for multiple-choice questions (MCQ) by 48% and true/false (TF) questions by up to 8%. Code is available at 
https://github.com/BorealisAI/Autocast-plus-plus.

Published as a conference paper at ICLR 2024
AUTOCAST++: ENHANCING WORLD EVENT PREDIC- TION WITH ZERO-SHOT RANKING-BASED CONTEXT
RETRIEVAL
Qi Yan Raihan Seraj" Jiawei He Lili Meng Tristan Sylvain²
University of British Columbia qi yandece. ubc. ca 1il lilimengl 1000mmmmmm....omommmmmmm.mmmm beihannseeaae jiawei he, tristan. www.ssttsssvaannnnaooo.o syLrain)eborreaiisa..coo
Borealis Al $Independent Researcher
ABSTRACT
Machine-based prediction of real-world events is gamering attention due to its potential for informed decision-makinng Whereas traditional forecasting predom- inantly hinges structured data like time-series, recent breakthroughs in lan- guage models enable predictions using unstructured text. In particular, (Zou et al., 2022 unveils AutoCast, new benchmark that employs news articles for answer- ing forecasting queries. Nevertheless, existing methods still trail behind human performance. The cornerstone of accurate forecasting, we argue, lies identify- ing a concise, yet rich subset of news snippets from vast corpus. With this moti- vation, we introduce AutoCast++, a zero-shot ranking-based context retrieval sys- tem, tailored to sift through expansive news document collections for event fore- casting. Our approach first re-ranks articles based on zero-shot question-passage relevance, honing in on semantically pertinent news. Following this, the chosen articles are subjected to zero-shot summarization to attain succinct context. Lever- aging pre-trained language model, conduct both the relevance evaluation and article summarization without needing domain -omaii--peccffc training. Notably, recent articles sometimes be odds with preceding ones due new facts unan- ticipated incidents, leading to fluctuating temporal dynamics. To tackle this, our re-ranking mechanism gives preference more recent articles, and we further regularize the multi-passage representation learning to align with human fore- caster responses made on different dates. Empirical results underscore marked improvements across multiple metrics, improving the performance for multiple- choice questions (MCQ) by 48% and true/false (TF) questions by up to 8%. Code is available at http:/ggww...oooooobo/lls///////
INTRODUCTION
In the realm of machine learning research, event forecasting has emerged as an area of both the- oretical intrigue and practical importance. The intersection of these two dimensions has placed event forecasting at the forefront of artificial intelligence inquiries, holding immense promise for real-world applications.
The initial focus of machine learning research in forecasting was predominantly on the prediction of time-series data (Shabani et al., 2023). a relatively straightforward task when compared to the complexity of real-world events. However, as the demand for more accurate forecasts in diverse domains has grown, the need to integrate data from beyond the structured time-series modality has become apparent. One such critical modality is the continuous stream of news articles, often presented in lengthy textual formats. In the pursuit of predicting future events, the analysis and interpretation of news articles have become central to the endeavor.
Recent advancements in this field, exemplified by initiatives like the Autocast dataset, have demon- strated the potential of utilizing news articles to provide probabilistic estimates of real-world events. Nevertheless, it is evident that the field of event forecasting through machine learning is still in its
Work performed while interning at Borealis AT.

---

Published as a conference paper at ICLR 2024
early stages. Despite promising results, these methods have yet reach the level of proficiency exhibited by human forecasters. A considerable gap exists between the theoretical potential and the practical feasibility of machine dearning-based event forecasting.
In this research paper, we posit that progress in event forecasting methods relies on addressing three fundamental questions concerning the analysis of news articles. These questions form the core of our investigation:
How can the selection of the most relevant news articles for event forecasting be improved?
How can the processing of selected news articles be optimized to extract information effec- tively?
How can the learning dynamics be refined enable efficient model training and leverage the enriched information from relevant news articles?
This paper introduces a series of novel components designed provide answers these critical questions, ultimately enhancing the entire event forecasting process. Our contributions encompass the following key aspects:
We present a task-aligned retrieval module that aligns the information retrieval process with the central task of answering critical questions, thereby increasing the relevance of selected news articles.
TasneAl Retrieval Module:
We propose an improved neural article reader, en- hanced through unsupervised distillation techniques, including summarization. This aug- mentation enables the model to delve deeper into news articles, efficiently extracting perti- nent insights.
Enhanced Neural Article Reader:
Human-Aligned Loss Function: We introduce novel human-aligned loss function that considers human preferences and judgments, bridging the gap between machine learning models and human intuition.
Through thorough testing and comprehensive evaluation, our study demonstrates that our proposed components play vital role, supported by ablation studies, and also contribute substantial performance improvements. Our findings indicate that the proposed technique significantly en- hances baseline models across various metrics, resulting in a noteworthy 48% boost in accuracy for multiple-choice questions (MCQ), an appreciable 8% improvement for true/false (TF) questions, and substantial 19% enhancement for numerical predictions. These results emphasize the progress we have made in advancing event forecasting through machine learning. Our research represents significant step toward realizing the full potential of machine learning in event forecasting, aligning thcoretical exploration with practical applications in the domain of Al-driven predictions.
2 RELATED WORK
Event Forecasting. Using large language models (LLMs) to make predictions is an increasingly active area of research (Lopez-Lira Tang 2023). One of the pioneering datasets event forecast- ing is ForecastQA (Jin et al. 2021). However, highlighted by (Zou et al. 2022). this dataset has shortcomings, such as the presence of non sensical questions or potential data leakage in retrospec- tive human forecasting labels. To address this, (Zou et al. (022) introduced a large-scale, balanced dataset tailored for event forecasting, which serves as the primary evaluation setting in our study.
Information Retrieval. Information retrieval (Guo et al.
2016 Mitra & Craswell 2017 Zhao
et al. 2022) aims to extract pertinent information in response to specific queries from a vast textual repository. Earlier methods such as BM-25r (Robertson et al. 1995) relied on bag-of-wone ofoddds re- trieval function. More recently, there has been a growing interest in leveraging information retrieval to enhance the question-answeriigg capabilities of L.LMs (Lewis et al. 2020 Shuster et al. 2021) In parallel vein, L.MS can also be directly employed for the purpose of information retrieval (Zhu ct al. 2023).
LLMs for Text Summarization. Recent studies (Liu al. 2023 Zhang al. 2023b) have high- lighted the capabilities of large language models (LLMs) in text summarization. Notably, the GPT series (Brwnn al. 2020
OpenAl 2023) and the T5 model (Raffel al. 2020) have emerged as
2

---

Published as a conference paper at ICLR 2024
leaders in this domain, showcasing the advancements and versatility of LLMs in condensing textual content. In addition to these generalist architectures, some approaches such as (Liu et al. 2022 Liu
& Liu 202T are dedicated to summarization.
Open-domain Question Answering. Open domain question answering (OpenQA) focuses on generating factual answers from expansive document collections without explicit evidence point- ers (Zhang et al., (023a). Unlike many QA tasks, OpenQA relies solely on the posed question, devoid of any associated documents that might house the answer (Sachan et al.] (02T). Typical OpenQA methodologies incorporate both retriever and reader mechanisms. The retriever's role is to
pinpoint a concise set of documents to assist the reader (Chen et al.) 2017). and subsequently, the
reader is calibrated to produce succinct answer to the query (Izacard & Grave 202T The Un-
supervised Passage Re-ranker (UPR) stands as an exemplar in this field, emphasizing unsupervised passage re-ranking for open-domain retrieval (Sachan al. 2022).
Rebriever Pipeline
Naws Corpus
Question: q. News: (s) Please rate the relevance between the news articlo and the quastion.
Top-k naws articles
Re-rankad articles
f Question
Crude Retriever
Pre trained Language
Relavance Re-rank
Recency Re-rank
Text Summarization
Fly
News: (n) Please write concise summary for this news article.
Retriever model with zero-shot re-ranking and summarization.
Model Pipeline
Question
News
encoder
feature dy
concat
decoder
text answer
prediction loss
Question
News
encoder
feature d;
Question
News its
encoder
feature 4;
Question
News
encoder
feature 4yg
casual attention
temporal conficence
alignment loss
(b) Reader model based on Fusion-in -ssionni--Decoder Decoder (FiD) Izacard & Grave 2021
Figure I: Illustration the Autocast++ components. Top: For each question, the retriever employs articles from
zero-shot relevance re-ranking and recency re-ranking to pinpoint relevant large corpus, subsequently using unsupervised text summarization to establish concise context. Bottom: Our FiD-based reader utilizes the generative decoder for predicting event outcomes. We also introduce an auxiliary alignment loss to synchronize with the responses of human forecasters.
METHOD
Event forecasting through news articles demands three specific capabilities: accurately pinpointing relevant articles, adept processing of these articles, and ensuring the learning dynamics optimally harness the acquired information. To enhance these capacities, we present zero-shot ranking- based retriever-reader model that emphasizes effective context retrieval and alignment with human forecasters. A comprehensive depiction of our architecture is illustrated Figure [ Each of the following subsections provides key improvement that we introduce to address limitations of past approaches in the three areas listed above.
3.1 PRELIMINARY
Our model is grounded in the retriever-readee reader framework. For given query q from the question set Q, the retriever module strives select subset of pertinent articles, N₂, from the news passages database represented by 'D = [M: Nai: n. I The articles retrieved in relation to question q are represented as Ng (n1, n2, ....... ng). where Ng E D is the retrieved subset of news
We represent the canonical order of news articles in the dataset D using the notation {is, f2. ..M). This aims differentiate from the article index in the retrieval set Ng-
3

---

Published as a conference paper at ICLR 2024
articles. Each query q includes the question text, potential answer choices, start and end dates of the question, and the type of question. Each news article n4 within the retrieval set Ng comprises published date, title, text, and a relevance score relative to the query q. This paired data (q.NVg) is then input into the reader network to forecast an event outcome o. The event outcome o is defined as discrete variable corresponding to one of the allowable answers. For example, in the case of True/False questions, the outcome is represented as o E {True, False). We discretize responses for continuous-valued numerical questions make this condition hold, which will be further detailed later on. Our reader employs an encoder-decoder transformer, designed to produce answers for the forecasting question using the generative model PNo: θ), where é means the reader parameters. The reader objective is to maximize the likelihood of generating the actual outcome Og: arg maxe p(o = Og N. q Q) Specifically, our reader model adopts the Fusion-in- Decoder (FiD) (Izacard & Grave 202T) on top T5 (Raffel et al., 2020).
Significantly, the Autocast dataset provides intermediate forecasting results from human experts for every query. These forecasts are documented at various timestamps spanning the start and end of the question period. The notation ph (olg, t) represents the probabilistic human prediction made at the timestamp for the specific question q. Specifically, (o agalq, t) denotes the accuracy of the human forecasters. We employ such human feedback to steer both our retriever and reader models towards effective forecasting, which will be detailed in the following sections.
3.2 TASK-ALIGNED RETRIEVAL MODULE
Callegrry
Natano according National Hurricane Center? 2019 07 2019- 2019-11-30)
Recency score A
0.3
hall hmmm mpman mmmm
BM25 Retiever
Rubriewal
Re-raticing
0.2-
"Nbovernoma hurkkane (2019 09)
N1: 2019 Atantic Humiaane Season Fast Facts. (201 9-1199)
projected
N2: NOAA propcill forwoist saia more stamm this hur nuriiane... (2015-08-06)
N2 Humiaane season 2019 Will there be any more humiaane this year? (2019 11-25)
6:1
NS NDAA incoesses prediction for hamriaine actidly 2010, "ateeeroomal aciiviy now expectad (2019-05-08)
NO: A neere Attantic humcane sesson is just about over (2019-11-29)
0.0
0.2
0.6 Nornalized time
0.4
0.8
1.0
Figure 2: Left: retrieval results of BM25 and our re-ranking retriever. Given a query seeking in- formation about a hurricane until its expiration date, the BM25 identifies articles based on lexical similarity. However, these articles, closely aligned with the query start date, lack depth for retro- spective answer. In contrast, our re-ranking retriever focuses on more recent, relevant passages to effectively address the query. Right: Visualization of news recency score st (t). As the query expiry date nears, human forecaster accuracy typically increases more rapidly. This score statistical measure capturing general patterns across the whole dataset.
We posit the existence of a preliminary retriever (specifically BM25 as adopted in previous research), which consumes the question q and the document set 'D to approximate the K most relevant articles Ng. While many forecasting studies (Zou et al. [022 Jin et al. 2021) depend on BM25, it often falls short in differentiating between articles that directly address the question and those that merely focus on a related theme. As a result, we aim to improve the retrieval module.
Zero-shot News Relevance Assessment. From the top- K news articles Ny identified by the initial retriever, our goal to derive robust quantitative measures on the relevance between questions and news to enhance article retrieval. We subsequently re-rank these articles based on the relevance score 8, (n,, q). Vn, E Ng.. obtained in a zero-shot manner. This means we forgo the need for task-specific training data, leveraging a pre-trained LLM for ease of implementation.
Estimating the inherent value of sy (n,, q) is admittedly intricate (Sachan et al. 2021 2022). given the absence of definitive ground- truth. To bring clarity to relevance-based re- re-ranking, we design
Adhering to the conventions estahlished in the Autocast dataset as detailed in Zou al. 2022 we treat the aggregated human crowd judgment a prohabilistic measure.
4

---

Published as a conference paper at ICLR 2024
binning-base -bssed relevance score estimation for (n, q). Instead of estimating the continuous- score directly, we first assess a discrete relevance label g that corresponds to up to bins, evenly distributed in the range of 0 to 1. The LLM assessment output g represents the relevance, which is quantified on a scale from to G:
Eígl G G-1 g q: V)
s, s, 9)
(I)
Here, v denotes the parameters of the pretrained LLM, and g (0. 1} represents the discrete relevance metric. We utilize the LLM in a zero-shot manner, refraining from fine-tuning on the task- task-specific dataset. To the question and article tokens, we append straightforward language prompt such as: "Please rate the relevance between the news article and the question on a scale of 0 to 4, with being the most relevant." ", where set G = 5. We run LLM inference multiple times 1/ 9: and evaluate 8, (n,, q)
(denoted by to estimate E gl
Accounting for News Recency based on Human Feedback. The timeliness of context passages pivotal determining their usefulness. For instance, when addressing forecasting queries about natural disasters, news reports closer to the question ending date often hold more value compared to etrly-stage reports. The ever-changing dynamics of the real world profoundly impact the favored responses to forecasting queries. Thus, solely considering the content-based news-question rele- vance 5, (m,, q) is inadequate for effective retrieval. To address this, we leverage human-feedback statistics gauge temporal truthfulness and prioritize more recent news.
We arrange the articles in Ng chronologically as ni,, Th...'.. nxx where the publication date for news n2, is t4 and t I ...... tk. with tx being the most recent date. We postulate that the recency score 8t (7i7k= 9) for news dated i correlates with the temporal performance enhancement observed in human forecasters:
se (T q) (o = ogr q, (x) - Ph Ph ogl tx-1)
tK 1K-1
Human forecasters provide responses at time & using information available up to that point, encom- passing the articles nrsk. We assess (y(T q) by examining the variation in human forecaster accuracy, averaged over the time gaps between two successive articles. To derive temporal dynamics that are agnostic to specific query-news pairings, we calculate the expectation across the empirical question distribution q E Q and its top-K news articles N₂ distribution:
st) & EggglE,,es eN,, (nr, 4)
To cope with queries spanning multiple years, we normalized time t relative to the question start and expiry dates instead of absolute time. Putting things together, we combine the scores defined in Eq. T1 and Eq. (3) to obtain the retrieval score:
s nn; 9) = 8y snin ①
St Sem
contentt relevance
recency
where f, denotes the normalized time of news n.. After computing the retrieval score s(n,, q) for each g E Q and n, E Ng. we reorder the articles in NG and select the top-N out of the K passages. Fig. 2 shows the effectiveness of our retriever against the vanilla BM25 and the recency score st))).
3.3 NEURAL READER WITH UNSUPERVISED DISTILLATION
Following the retrieval, our reader processes the question q along with its associated articles Ng to formulate answers to the forecasting inquiry. To refine context clarity, we advocate for an unsuper- vised distillation on the retrieved news articles, achieved through abstractive summarization. These distilled articles are then fed to the FiD reader for acquiring textual representations and for learning to generate forecasting-relan related answers.
Zero-shot News Summarization. News articles often encompass lengthy segments, including interviews and analyses, which might not directly provide factual insights. Extracting significant information from these potentially relevant sections poses a challenge. As solution, rather than processing the raw articles Ng in the reader network, we turn to an abstractive summarization tool to produce succinct and pertinent summaries. Our method leverages a pre-trained LLM, prompting with "Please write a concise summary for this news article." " to carry out text summarization without the necessity of supervised signals.
5

---

Published as a conference paper at ICLR 2024
Fusion-in-Decoder Reader Architecture. In the reader model, combine the tokens of question q with the tokens from a single news article sourced from its top-N retrieval results, Ng. Each question-news pair undergoes the prepending process independently, denoted by æ; prepend(q. n.). We consider temporal data, including the start and end dates of queries and the publication dates of news in the prepending step. Please the appendix for details.
Subsequently, the T3 encoder fe processes the concatenated tokens {2; |2; resulting in a textual representation for the sequence:
Vi E [N], 2; = fa,;; ,); X = concat(21, 22, ,N
(5)
Thereafter, the T5 decoder fa derives the answer for the forecasting question, employing both cross- attention and causal self-attention mechanisms. These mechanisms account for the tokens in X. and the tokens generated previously, respectively. The underlying principle of concatenating repre- sentations from diverse documents is to provide the decoder with holistic view of the information. The answer generation process is modeled by the autoregressive decoder p(olq.-Nq: 9).
3.4 TRAINING OBJECTIVES WITH HUMAN ALIGNMENT LOSS
Unlike prior research on generative question answering, our focus lies on real-world event prediction tasks which necessitate concise and definitive predictions rather than elaborate textual responses. Notably, our model is designed to predict numerical values, an area where traditional language models often falter (Lightman et al. 2023). Moreover, our model acknowledges the temporal dy- namics intrinsic to real-world events, as evidenced by the sentiment shifts observed in news articles over time. To accommodate these unique challenges, we introduce multiple modifications to the conventional autoregressive generative decoder loss, aiming to enhance performance.
Binning Numerical Questions. The generative decoder p(o) g. ; +) inherently struggles with direct numerical value prediction. The earlier baseline (Zou al., 2022) attempted to address this by integrating linear layer based on the attention layer features for regression. We propose an alternative approach that retains the objective the textual token space and avoids introducing such regression layers. Let Onum E R be the continuous-valued nuemrical outcome, which is categorized Ozam
into R groups: o'umm + (omns oztn 1/1l E (1, R). During training, we use the discrete o'um to serve as a proxy training target. During inference, to revert to the numerical value space, we employ (o oftit
the median value of each category: Onum +- O'um
In the Autocast dataset, with values given as o num = 1 and o mii = 0, this discretization process is substantially simplified.
2K
Human Annotation Alignment Loss. The Autocast dataset offers intermediate probabilistic sponses from human forecasters, denoted by (olq. (), gathered over various dates. Utilizing these labels, we harmonize the T5 encoder text representations {z:): with the beliefs held by human forecasters. Consider the merged question-news token sequences {x,} to be arranged chronolog- ically, following tz. S tæ.._ .... We employ a self-attention mechanism integrated with causal mask, founded upon the text features {zi} This layer aims to infer the contextual confidence up time instant t: p(u (g: +). Here, d, E [0, 1] symbolizes the confidence, while denotes the self-attention layer parameters. Our objective is to synchronize the inferred confidence with the ac- curacy of the human forecaster, as represented by ph (Ogz q, !), thereby regularizing the learning of the text representation. Consequently, the overall training goal is to minimize the following loss:
.V
L = logplolq.Ngi -
Σ
DEL (px (p.llgg (aall.t)ll t)ppp))l 1)),
(6)
N
decoder loss
alignment loss
where À is weighting coefficient. We use cross-entropy loss for both terms in implementation.
4 EXPERIMENTS
4.1 DATASET AND METRICS
We assess model on the Autocast dataset (Zou et al. 2022). benchmark comprising diverse set annotated questions, including True/False (TF), Multipl-choice Questions (MCQ), and nu- merical predictions. These span various domains, such economics, politics, and technology. We

---

Published as a conference paper at ICLR 2024
Table 1: The number of questions in Autocast dataset, grouped by question type.
Question Type
Train
Test
Total
T/F
3187
775
3962
MCQ
753
176
929
Numerical
471
341
812
incorporate news articles from the Common Crawl corpu corpusspaanning spanning 2016 to 2022 for retrieval pur- pose. For performance evaluation, we use accuracy metries for T/F and MCQ questions and absolute error for numerical prediction questions. The dataset is partitioned with cut-off in mid-2021 I and questions the test set span from mid-2021 to mid-2022.
4.2 EXPERIMENTAL SETUP
Implementation Details. For relevance evaluation and text summarization, we utilize the pre- trained GPT-3 model (Brown et al. 2020). We initially retrieve K = 50 news articles using BM25 10 unless otherwise specified. The reweight- ing coefficient À in Eq. (6) is fixed at 0.1. For most of our experiments, especially hyper-paramen and architecture optimization, we employ LoRA (Hu al. 2022), fine-tuning over the pre-trained T5 backbone. More results can be found in the appendix.
and proceed with our re-ranking process to select
Baselines. Following the Autocast (Zou et al. 2022) benchmark, we incorporate these baselines: 1) UnifiedQA-v2 (Khashabi et al., 2022). 2) T5 (Raffel et al.] 2020), 3) FiD Static (Izacard & Grave 2021). and 4) FiD Temporal built upon GPT-2 (Radford et al. 2019). UnifiedQA-v2 and T5 are no-retrieval models, meaning they do not extract information from news articles. Specifi- cally, UnifiedQA-v2 uses zero-shot prompting for answers, while T5 undergoes fine-tuning on the training data. In contrast, the FiD Static (T5-based) retrieves the top 10 news articles throughout the period, whereas the FiD Temporal (GPT-2 with T5 encoder) sources the top article daily. Although newer LLMs might demonstrate enhanced capabilities and potentially outperform in forecasting question answering, we avoid using or comparing them to avoid potential contamination from their more recent training data. It is important to highlight that model pre-trained on data post-mid-22 mid-2021 (training/test cut-off time) will not genuinely replicate the real-world forecasting condition.
4.3
MAIN RESULTS
Table 2 showcases the primary results of our approach alongside the baselines highlighted in (Zou al. 2022). It's pivotal to note that our reader's architectural framework aligns with FiD Static, with both leveraging pre-trained T5 weights. Notably, our proposed model eclipses the FiD Static baseline in multiple-choice questions (MCQ) by a margin 48% (43.8 versus 29.6), true/false (TF) questions by 8% (66.7 versus 62.0) and numeircal predicion by 19% (19.8 vs 24.5) within the 0.2B model size category. On the other hand, numerical predictions remain a challenging task. For the 0.2B and 0.8B model sizes, our models achieve the best results in the class, leading by margin of approximately 1%.
Furthermore, more compact models, such as the 0.2B and 0.8B variants, already surpass larger baselines like the FiD Static with 2.8B parameters. This suggests that model size may not be the primary determinant for forecasting question answering. In similar vein, we observe that the performance boost from utilizing more extensive models in our scenario isn' It as remarkable. For instance, our 2.8B parameter model marginally outperforms the 0.2B model for T/F and MCQ, yet delivers identical outcomes for numerical predictions. While the 4.3B parameter FiD Temporal baseline achieves the best performance numerical prediction, is significantly larger and more computationally intensive to train. This is because it processes news articles for each day within the active period of a question, which averages 130 days in the Autocast dataset. Consequently, its retriever deals with considerably larger volume of news articles. However, FiD Temporal does not perform satisfactorily on T/F and MCQ questions in contrast to FiD Static or our method. This
CCommon Crawl- Open Repository of Web Crawl Data, httpt:tt///////. httpas://commocraan/ .org/

---

Published as a conference paper at ICLR 2024
Table 2: Performance of our approach using three different model sizes. We take the baseline results from (Zou et al. 2022). Across various model sizes, our method demonstrates a substantial performance boost, notably for smaller models. It significantly surpasses the retrieval baselines of FiD Static and FiD Temporal, underscoring the efficacy of our proposed retriever-reader model.
Model
Full Context Length
Model Size
T/F1
MCQ
Num.
Small-size Models
UnifiedQA (Khashabi al. 2022
n/a n/a 10
0.2B
45.4
23.5
34.5
Raffel ci al.
2020
0.2B
61.3
24.0
20.5
FiD Statee Zou et al.
2022
0.2B
62.0
29.6
24.5
FiD Temporal (ou
2022
query-specific 10
0.6B
62.0
33.5
23.9
Autocast+ (ours)
0.2B
66.7
43.8
19.8
Middle-size Models
UnifieQQA Khashabi et al. T5 RRffel al.
2022
n/a n/a
0.8B
48.2
23.5
34.5
0.8B
60.0
29.1
21.7
FiD Statie Zou ct al. 2022
10
0.8B
64.1
32.4
21.8
FiD Temporal Zou et al. 2022
query-specific 10
1.5B
63.8
32.4
21.0
Autocast++ (ours)
0.8B
67.3
44.0
19.9
Large-size Models
UnifiedQA KKashhaii et al. 2022
n/a
2.8B
54.9
25.1
34.5
T5 Raffel et al. 2020
n/a
2.8B
60.0
26.8
21.9
FiD Statie Zou al. 2022
10
2.8B
65.4
35.8
19.9
FiD Temporal Zou et al. 2022
query-specific 10
4.3B
62.9
36.9
19.5
Autocast++ (ours)
2.8B
67.9
44.1
19.8
highlights the critical importance of effectively extracting relevant information from news articles, rather than indiscriminately processing all available data.
4.4 ABLATION STUDIES
To substantiate the effectiveness of our introduced components, we perform ablation studies partic- ularly in comparison to the FiD Static model. In the context of retrieval, the FiD Static integrates the BM25 retriever enhanced with cross-encoder reranking. The architecture both the FiD Static and model's reader networks align until the self-attention module tailored for the human alignment loss and the training loss for binning numerical question. By implementing modifications progres- sively, can evolve from the FiD Static performance benefits of each component. In Table 3 we detail the ablation experiments utilizing the 0.2B model size, with the baseline model being the vanilla FiD Static. To ease exposition, name FiD Static with numerical question binning FiD Static". We employ distinct markers to clearly differentiate between various ablations: ! represents experiments LLM-enhanced components, ② denotes experiments on numerical question binning, and 3 indicates experiments on alignment loss.
design, facilitating
evaluation of the incremental
Table 3: Effects of different retriever-reade components on performance.
Model
Retriever Componeni
Reader Component
Metrics
Zero-shot Relevance Re-rank
Recency Re-rank
Zero-shot Sum.
Binning Num. Questions
Alignment Loss
T/F
MCQ*
Num]
FiD Static
K
62.0
29.6
24.5
FID Static rel. re-rank & sum.
X
65.5
42.1
21.3
Statie alignment
62.5
31.2
23.5
2 F&D Statie*
X
x
/
62.1
29.6
23.2
① FID Static" rel. re-raok
/
64.7
39.8
21.2
! FID Static" full retriver
65.8
42.4
20.2
② A w/o binning
X
66.7
43.6
21.0
3 A w/o alignment
/
66.7
43.5
19.9
+ fall model
66.7
43.8
19.8
8

---

Published as a conference paper at ICLR 2024
① Effects of LLM Enhancement. The zero-shot relevance re-ranking and text summarization techniques, facilitated by the pre-trained LLM, stand out as the most effective among our proposed methods. Implementing these techniques in FiD Static FiD Static" baselines significantly im- proves performance across various question types, with a marked enhancement in MCQ accuracy, elevating their performance to nearly match that of the full Autocast++ model.
② Effects of Binnning Numerical Question. We propose binning numerical questions to transform the regression task in continuous space into a classification problem. This approach allows to apply the same cross-entropy loss used for T/F and MCQ. Empirically, we observe that this method generally enhances the performance of numerical questions (e.g., in FiD Static or the full +utocast++ model) without adversely affecting other question types. Although its impact on overall metrics is less significant, it complements the L.M enhancement components well.
3 Effects of Alignment Loss. The alignment loss leverages human crowd forecasting results to regulate the text encoder representations, thereby simulating the progression of information and knowledge over time. This alignment loss is particularly beneficial for the basic FiD Static base- line, which lacks any LLM-enhanced components. However, in the full Autocast++ model, where relevance re-ranking and text summarization are employed, the alignment loss appears to have diminished role in enhancing performance.
4.5 QUALITATIVE EXAMPLES
Question: Between 5 July 2019 and 30 November 2019, how many Category or above turricanes will occur in the Atlantic Ocean acconding to the National Answer:
Humiicane Center? (from 2019 05 2019:11 30y Choices A:O. I, C. 2, D: Our retriever:
NI: 2019 Arlantic Hurzicane Senson Hast Focts. (2019-11 1229)- Text cauned
N2: Hurricane season 2019: Will there be more burricness this year? (2019 11 (5) Text omitted
N3; fieree Atlantic haricane
just about (2019-11-22) Text. hurricanes, three reaching major hurneane status (winds of
hurricnee
which ended recearly, marked sigaificant and
mphi. Additionall,, there were 12 tropical or subtrogical storms and two trpical depressions. Noubly, this sezson included two Category 5 hurricanes: Derian and Larenzo, This marks the fourth cossecutive shove- noniaal Atlantic hurricate sensos, patteon last observed from 1995 to 2001. In the Gulf of Moxico, five tropical cyclones lieeda. and Nestor
historic destruction vanoos reguons. It pooduced
fomsed, tying record 2003 and 1957. Three to favorable conditions hurricanes during season. These factors included multi-docade cycle of high hunizane activity since 1995, stronger African monsoca, warmer Atlantic waters, and weak wind shear, which is crosswinds that can hinder burricune formation. Hurricane Dorian, one the most significant storas of the season. brought devasiation to the Bahamas with near record strength. reaching winds of 185 mph. It continued to impact the cast with
landfall the United
Several
contributed
and Nova terais of wind speed. Another Category harricane, Loresao, caased less destruction than Deciat quickly voered tubbaat and bcought strong winds to Irelund, resulting Booding and power outages. The 2019 hurncane season officially runs from June to Novamber 30, though stanis can occar outside these dates, As the senoon has come to in and, the Atlonric Ocean is currestly fiee of tropical storm activity. and the National Atlantic, Galf of Mexico, or Carilbean Sea the remsiiing days of
Hurriane Derin''s streegtt
historic hunicanes
strongest huricane record the Atlantic basin However, it
a French
Hunicane Center does anicipate any
Figure 3: An example of our proposed re-ranking retrieval with text summarization.
In Fig. 3 we present an expanded view of qualitative results corresponding to the retriever compar- ison depicted Fig. Specifically, we focus on the news context after summarization for the most informative news article. which we identify as N3 (news with third highest relevance score). With the key information highlighted in orange, the model can readily make an informed inference about the answer, which is C (2 Category 4 or above hurricanes during this period). The integration of pretrained LLM into the retrieval and summarization processes significantly improves the extraction of pertinent textual information while filtering out less relevant content. This enhancement proves particularly beneficial, as demonstrated in our ablation experiment labeled as (.
5 CONCLUSION
In this research paper, we introduced three key components that significantly enhance the event fore- casting process: 1) task-aligned retrieval module, 2) enhanced neural article reader, and 3) human- aligned loss function. Our work has significantly boosted accuracy in various question types, such as a 48% improvement in multiple-choice questions, 8% enhancement in true/false questions, and a substantial 19% increase in numerical predictions. These results highlight our progress in advancing event forecasting through machine learning, bridging the gap between theory and practi- cal Al-driven applications. In conclusion, our research represents a significant step forward in the field of event forecasting through machine learning. As the field continues to evolve, we anticipate further breakthroughs and innovations that will shape the future of event forecasting and its practical applications.

---

Published as a conference paper at ICLR 2024
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International Conference Information and Knowledge Management, CIKM 16, pp. 55-55-6666 64, New York, NY, USA, 2016. Association for Computing Machinery.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguistics.
Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. ForecastQA: question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4636-4650, Online, August 2021. Association for Computational Linguistics.
Daniel Khashabi, Yeganch Kordi, and Hannaneh Hajishirzi. Unifiedga-v2: Stronger generalization via broader cross-format training. arXiv preprint 1.XXxx2222.113322 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenttdd generation for knowledge-intensive nip tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 2020.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2005/ 2023.
Yixin Liu and Pengfei Liu. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Anmual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 1065-1072, Online, August 2021. Association for Computational Linguisties.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. BRIO: Bringing order abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 2890-2903, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 4140-4170 Toronto, Canada, July 2023. Association for Computational I. Linguistics.
10

---

Published as a conference paper at ICLR 2024
Alejandro Lopez-Lira and Yuehua Tang. Can chatgpt forecast stock price movements? return pre- dictability and large language models. arXiv preprint arXiv:2304.07619, 2023.
Bhaskar Mitra and Nick Craswell. Neural models for information retrieval. arXiv preprint 1iXiv:1705.010% 2017.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(1), ) jan 2020. ISSN 1532-4435.
Stephen Robertson, Walker, Jones, M. Hancock- Beaulieu, and M. Gatford. Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3), pp. 109-126. Gaithersburg, MD: NIST, January 1995.
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. In Pro- ceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp- 3781-3797, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Devendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer, and Dani Yogatama. End- to-end training of multi-document reader and retriever for open-domain question answering. In Advances in Neural Information Processing Systems, 2021.
Mohammad Amin Shabani, Amir H. Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In The Eleventh International Con- net forum?
ference on Learning Representations, 2023. URL
id=sCrallct
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. In Findings of the Association for Computational Lingnistics: EMNLP 2021, pp. 3784-3803, Punta Cana, Dominican Republic, November 2021.
Association for Computational Linguisties.
Qin Zhang, Shangsi Chen, Dongkuan Xu, Qingqing Cao, Xiaojun Chen, Trevor Cohn, and Meng Fang. A survey for efficient open domain question answering. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14447- 14465, Toronto, Canada, July 2023a. Association for Computational Linguistics.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848 2023b.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey, arXiv preprint urXiv:2211.14876 2022.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, 2023.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. In siiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
11

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
Question Appending. In FiD, we concatenate the question and its top-N retrieved articles Ng encode the input whole:
q SEP date(() [SEP] choices(q).
n title (n:) [SE]] date( [SEP] ni,
x; [CLS] q' [SEP] n/ [SEP],
where [CLS] is the beginning the sequence, and [SEP] is the separator token.
To provide comprehensive description of both the query and the context, the question is aug- mented with its starting news article is prefixed with its title and date. Notably, the notation nu; is repupposed to denote the summarized article.
and ending dates, along with its allowable choices. Also, the beginning of
B ADDITIONAL EXPERIMENTAL RESULTS
B. REPRODUCING RESULTS LORA
In this subsection, demonstrate that LoRA (Hu et al. 2022) enables us to reproduce the out- comes presented in the original paper (Zou et al., 2022) while maintaining performance at virtu- ally indistinguishable level. We use LoRA to explore different model design choices and carry out hyperparameter search to save time. We carry out fine-tuning without LoRA for final experiments.
Table 4: Effects of LoRA on FiD Static model performance.
Model
Model Size
T/F1
MCQ1
Num
FiD Static
Zou ct al.
2022
0.2B
62.0
29.6
24.5
FiD Static
Zou et al.
2022
0,8B
64.1
32.4
21.8
FiD Static
Zou et al.
2022
2.8B
65.4
35.8
19.9
FiD Static (LoRA)
0.2B
62.1
29.3
22.2
FiD Static (LoRA)
0.8B
64.5
32.9
20.4
B.2 IMPACT OF REMOVING NOISY DATA
The significance the quality of news articles appended to each question is paramount facilitating accurate predictions. In our empirical investigation, find that filtering mechanism, which elim- inates less-relevant news articles based on retrieval scores, can result in performance enhancements, even though it may entail reduction in the overall number of training instances. The score-based thresholding is based on the overall retrieval score defined in Eq. (4), which means we employ both the relevance re-ranking and recency re-ranking. Nevertheless, is crucial to acknowledge that the inclusion of noisy or irrelevant news articles may have counterproductive effect, potentially lead- ing confusion rather than bolstering the model's capacity. This underscores the imperative nature of meticulous pre-processing procedures and the necessity of guiding the model to focus its attention on informative news sources. We perform these comparison experiments based on the vanilla FiD Static (Zou et al. 2022) configurations without changing the retriever or reader module.
Table 5: Effects of removing noisy data on model performance.
Model
Score Threshold
Model Size
T/F1
MCQ1
Num
FiD Static
Zou et al. 2022)
n/a
0.2B
62.0
29.6
24.5
FiD Static (re-run)
0.0
0.2B
62.1
29.3
22.2
FiD Static (re-run)
0.5
0.2B
64.8
43.0
20.3
12

---

Published as a conference paper at ICLR 2024
B.3 SPLITTING NUMERICAL AND NON-NUMERICAL QUESTIONS
Numerical questions entail outputting continuous values, which stands in contrast to the discrete choice tasks of T/F and MCQ. Through empirical analysis, observed that jointly training these three tasks can potentially impede performance in one area or another. However, when we segregate the numerical questions within the training set, we observe a noteworthy enhancement on T/F and MCQ. We conduct the ablation experiments with the relevance re-ranked and summarized news, and keep the same retriever and reader as the vanilla FiD Static (Zou al. 2022). Please note that we do not conduct numerical question binning in these experiments.
Table 6: Effects of splitting numerical questions model performance.
Model
Conlig
Model Size
T/Ft
MCQ1
Num.
FiD Static Zou et al. 2022
0.2B
62.0
29.6
24.5
FiD Static (re-run)
rel. re-rank sum. news
0.2B
65.5
42.1
FiD Static (re-run)
rel. re-rank sum. news num.
0.2B
65.9
43.1
FiD Static (re-run)
rel. re-rank sum. news num. only
0.2B
n/a
n/a
In comparison to the results Table we can find that using binning for numerical questions can consistently lead to better performance metrics.
B.4 IMPACT OF CONTEXT SIZE
The context size denotes the quantity of news articles we input into our model to facilitate future event forecasting. In an ideal scenario, greater number of articles can offer more information, potentially enhancing performance. However, this choice also entails the trade-off of increased computational resources and nosier information. To determine the optimal context size, we perform ablation studies with the relevance re-ranked and summarized news, and the outcomes in Table 7 We carry out the ablation experiments using the same retriever and reader network as the vanilla FiD Static (Zou et al. 2022). Indeed, larger context windows do not necessarily enhance performance as they may include less pertinent news articles, potentially leading to confusion the model's responses.
presented
Table 7: Effects of news article context size model performance.
Model
Context
Model Size
T/Ft
MCQ1
Num
Small-size Models
FiD Static (Zou al. 2022
raw news
0.22
62.0
29.6
24.5
FiD Static (re-run)
rel. re-rank + sum. news, ctx 10
0.2B
65.5
42.1
21.3
FiD Static (re-run)
rel. re-rank + sum. news, ctx @ 30 rel. re-rank + sum. news, ctx @ 100
0.2B
66.0
42.8
20.7
FiD Static (re-run)
0.2B
65.3
40.2
21.3
Middle-size Models
FiD Statie (Zou al. 2022
raw
0.88
64.1
32.4
21.8
FiD Statie (re-run)
rel. re-rank + sum. news, ctx 10 rel. re-rank + sum. news, ctx 30
0.8B
65.8
41.0
19.8
FiD Static (re-rum)
0.8B
66.4
43.4
19.9
FiD Static (re-run)
rel. re-rank + sum. news, ctx 100
0.8B
64.2
40.1
23.3
13

---

Published as a conference paper at ICLR 2024
C ADDITIONAL QUALITATIVE RESULTS
We utilize the pre-trained GPT-3 model (Brown et al., 2020) (gpt-3. 1.0t-3.5-tue 5-turbo API) for the rele- vance assessment and text summarization. We show some qualitative results below.
C.I QUESTION-NEWS RELEVANCE ASSESSMENT
Intuitively, news articles that are more semantically relevant contribute more to the prediction of future event outcomes. This motivates us select the most relevant news articles.
Question: Will Apple release new iMac based on Apple silicon in 2021? Choices: Yes or No. Start time: 2021-02-19 19. Expiry time: 2021-05-01.
News: Title: A new iMac with Apple silicon may be coming even sooner than expected. Date: 2021-03-25. Article: Credit: IDG We all know an Apple silicon-based iMac is on the way sometime this year, but might be coming very very soon. After retiring the iMac Pro and discontinuing some higher-end storage configurations of the iMac 4K, the latest macOS Big Sur 11.3 beta contains some clues that new models might be landing very soon. As reported by 9to5Mac, the fifth Big Sur 11.3 beta that landed this week contains references to iMac21,1 and iMac21,2 code identifiers that haven't 't been seen before. 9to5Mac says they match the Apple silicon codenames previously reported by Bloomberg, though there's 's nothing out of the ordinary about the numbering. Last year's iMac refreshes were labelled iMac20, iMac20,2. But it still points necessarily mean new iMacs are around the corner, it certainly lends some more fuel to that particular fire. A new iMac form factor has been rumoured for more than year, bringing Pro XDR Display-inspired redesign that shrinks the bezels and eliminates the chin, and it stands to reason that the next Apple allicon-based Mac will be the iMac. New MacBooks are seemingly slated for the second half of 2021 and the Mac Pro isn't 't rumoured to arrive until 2022. It's been nearly 10 years since the iMac received its thin-edge design and another eight before it had its last major redesign. The current iMac form factor is basically variation of the iMac G5 that replaced the iconic iMac G4 in 2004.
possibly imminent release. While the existence of the identifiers doesn't 't
Relevance Assessment: [4, 4. 4, 4, 4] (We prompt the pre-trained LLM multiple times to pro- vide a relevance score in the range of 0 to 4, with larger numbers indicating higher relevance.)
Figure 4: Example 1 of question- relevance assessment.
Question: Will Apple release new iMac based on Apple silicon in 2021? Choices: Yes or No. Start time: 2021-02-19. Expiry time: 2021-05-01.
News: Tite: The next Apple event is actually happening on April 20 for real. Date: 2021-04- 14. Article: So there an event after all. Cre.it: Apple. After rumours, missed predictions, shaved eyebrows, and surprising Siri leak, we know officially know when Apple's spring event will be held: April 20 at 10 am PT. What will reveal is another question. Apple call- ing the event Spring Loaded and the tagline is accompanied by an Apple logo in the shape of spring. The colours match those in the original Apple rainbow, but it's hard to decipher much of anything else. There's 's also an AR component that features the coloured lines swirling around the room and converging to form the Apple logo in the invitation. As they have done with other recent events, theres AR experience https:/tt.c/xffY/2600I Iv on your iPhone and tap the event image. (Sleeping kitty optional.) Jason Cross @ (@JssnCrrsssO0) April 13, 2021 If we're specu- lating, it could point to the release of a new Apple Pencil alongside a revamped iPad Pro or possibly new Apple silicon iMac in colourful case options. We're also expecting Apple Tags, an updated Apple TV, and AirPods, but speculation has suggested that some of the products may be held until the fall. The event will once again be virtual and streamed Apple's 's web- site and YouTube channel.
the
#SpringLoaded Apple Event. Visit
nic.twitter..cmm/9f//kwwww
Relevance Assessment: [3, 2. 3. 3. 1] (We prompt the pre-trained LLM multiple times to pro- vide relevance score in the range of 0 to 4, with larger numbers indicating higher relevance.)
Figure 5: Example 2 of question- relevance assessment.
14

---

Published as a conference paper at ICLR 2024
C.2 TEXT SUMMARIZATION
Below, we showcase additional qualitative results of the pretrained LLM-enhanced text summariza- tion. Typically, the summarization process truncates URL links and less vital details like the report's location. The resulting summarized text is generally logically coherent, significantly more concise, and preserves most of the key information from the original text.
Original Text: Title: Beyond Outperforms US in SMB Processing Growth & Retention. Ar- ticle: PRINCETON, N.J. Sept. 16, 2020 /PRNewswire/ - Beyond outperformed the SMB market in both year year growth in processing volume and in merchant retention, when compared to a recent report published by top electronic payments analytics and consulting firm The Strawhecker Group (TSG). Beyond increased Net Revenue by 23 percent during Q2, ex- ceeding the aggregate U.S. small and medium business (SMB) market's performance, which had a net revenue reduction of 19 percent during the same time period. Analysis on the U.S. SMB market was conducted using TSG's proprietary Acquiring Industry Metries (AIM) plat- form, which tracks more than 45 percent of all U.S. processing merchants and more than one trillion dollars in annualized transactions. "I am so proud of all the hard work our team put in to make this happen," " said Beyond founder and CEO Robert O. Carr. "We make promises to both clients and sales team that we will not arbitrarily raise prices (for clients) or change our com- pensation structure (for sales). The fact that we could get through the worst of the COVID-19 crisis without breaking these promises, is a testament the fact that people (whether sales- people or customers) still appreciate being treated fairly and with respect." "We have built the most effective team in the business," said Chief Sales Officer Brent Rose. "We give our sales and service teams the tools to succeed and we provide them with the best support team in the industry." Information about sales careers Beyond can be found here. About Beyond Be- yond is financial technology and transactions company offering a suite of business tools and services including payment processing, employee management, business capital, and point-of- sale. About The Strawhecker Group The Strawhecker Group (TSG) is fast-growing analytics and consulting firm. The company serves the entire payments ecosystem, from fintech startups to Fortune 500 companies. The firm provides its clients with advisory services, research and analytics to help them plan and execute their strategic initiatives. Based Omaha, a recognized payments industry hub, TSG is established leader in this high-growth, ever-evolving space. (email protected] Sources: https:/thestrawgropp /
After Summarization: Beyond, a financial technology and transactions company, has sur- passed the small and medium business (SMB) market in both year-over-year growth in process- ing volume and merchant retention, according Beyond achieved a 23% increase in Net Revenue during Q2, outperforming the overall U.S. SMB market, which experienced net revenue reduction of 19% in the same period. The anal- ysis was conducted using TSG's proprietary Acquiring Industry Metrics (AIM) platform, which tracks a significant portion of U.S. processing merchants and transactions. Beyond's s success is attributed to its commitment to not raising prices for clients or altering its compensation structure for sales amid the COVID-19 crisis. The company's effective team and support frastructure have also contributed to its accomplishments. Beyond offers suite of business tools and services, including payment processing, employce management, business capital, and point-of-sale solutions. The Strawhecker Group (TSG) is growing analytics and consulting firm that serves the entire payments ecosystem, providing advisory services and research to assist clients in their strategic initiatives.
report by The Strawhecker Group (TSG).
Figure 6: Example 1 of text summarization.
15

---

Published as a conference paper at ICLR 2024
Original Text: Title: Charles Barkley Said The Bucks Will Win The 2019 NBA Champi- onship. Article: Twitter The Milwaukee Bucks put together an impressive Game 4 victory in Boston and, since a disappointing start to the Eastern Conference Semifinals, Mike Buden- holzer's 's team has quieted doubters with improved play. In the immediate aftermath of 12- point win to take 3-1 series lead, Milwaukee found itself with (very) famous advocate, with TNT's Charles Barkley loudly proclaiming that the Bucks "are going to win the world cham- pionship." "The @ Bucks are going to win the world championship." Sir Charles has spoken. w.c.twitter.co/RShXNNNVI
- NBA on TNT (@NBAonTNT) May 7, 2019 Of course, it has be noted that Barkley likes to make bold stances and, as you can hear in the clip above, he predicted that the Portland Trail Blazers would represent the Western Conference Finals, even with the Golden State Warriors still alive and kicking. Still, this was definitive from Barkley and, in some ways, it isn't I a crazy thought. After all, the Bucks were pretty easily the best team in the NBA during the regular season, racking up 60 wins with a league-leading net rating and dominant units on both ends of the floor. In addition, Giannis Antetokounmpo has played at an MVP level from wire to wire and isn't wild to say that he would be able to go toe-to-toe with any player in the league today. The Bucks entered Monday evening the team with the second-bes, betting odds to win the title and, frankly, that seems right. Barkley isn't 't one to shy away from blazing opinion, though, and he laid this one out with perfect clarity. After Summarization: Charles Barkley, prominent NBA analyst, confidently declared on Twitter that the Milwaukee Bucks would win the 2019 NBA Championship. He made this sertion following the Bucks' strong performance in Game 4 against the Boston Celtics, where they took 3-1 series lead. Barkley's statement notable given his tendency to make bold predictions, as he had previously predicted the Portland Trail Blazers to reach the NBA Finals despite the presence of the Golden State Warriors. While his proclamation about the Bucks winning it all might seem audacious, it's not entirely unreasonable. The Bucks had been the dominant team during the regular season with the best record, impressive net rating, and strong performances both offense and defense. Led by Giannis Antetokounmpo, who was playing an MVP level, the Bucks were considered among the top contenders for the championship. reflected in their favorable betting odds. Despite Barkley's 's penchant for bold opinions. he confidently expressed this prediction based on the Bucks' impressive season and strong roster.
the NBA
Figure 7: Example 2 of text summarization.
16

---

Published as a conference paper at ICLR 2024
Original Text: Tite: Paytm employee tested positive with coronavirus after returning from Italy, Article: A Paytm employee in Haryana's Gurgaon (now officially known Gurugram) has been tested positive for the novel coronavirus, the company said in statement on Wednes- day. The employee had recently returned from a vacation Italy. Italy has been emerging as one of the worst-hit countries from coronavirus. The company has advised its employees to work from home for the next couple of days while the Gurugram unit gets sanitised, according to the statement. It also said that as a precautionary measure, they have suggested their team members get health tests done immediately. Meanwhile, Paytm employee who has been tested positive is receiving appropriate treatment, the company as quoted by news agency ANI. Paytm: One of our colleagues based out of Gurgaon office who recently returned from Italy has sadly been tested positive for Coronavirus. He is receiving appropriate treatment. As precautionary measure, we have suggested his team members to get health tests done immedi- ately.
ANI @ ANI) March 4, 2020 article continues after ad -article resumes Following today and day before's developments, where Noida man was tested positive for coronavirus, Paytm, one of India's leading digital payment banks has decided to shut its offices in Gurugram and Noida for at least two days. Today it was reported that the Noida man's 's (who was recently tested positive after returned from Italy) six relatives in Agra have been tested positive for COVID-19 19, He had travelled to Agra to meet them. Morever, after an Italian man and his wife tested positive in Jaipur, today 14 Italian members from the tourist group and their Indian driver also tested positive, making total 17 infections in the group. Yes- terday, Shriram Millennium school in Sector 135 Noida decided to shut down the premises for two days and also postpone its annual examinations amid concerns that their student's 's parent, had tested positive with COVID-19 and has been isolated on Monday. This followed reports that the individual who contracted coronavirus in Italy had attended birthday party Febru- ary 28 (Friday) with his child studying in the Noida school. This party was also attended by other students of the school and their family members. Prior this, techie from Hyderabad was suspected to have contracted the virus. With these, the total count of known coronavirus infected cases in India rose to 29, including sixteen Italians. Coronavirus has infected more than 89,000 people globally and caused over 3,000 deaths. The World Health Organization has named the illness COVID-19, referring to its origin late last year and the coronavirus that causes it.
After Summarization: An employee of Paytm in Gurgaon, India, tested positive for coron- avirus after returning from Italy, a country severely affected by the virus. Paytm advised its employees to work from home temporarily while the office sanitized. The company also recommended health tests for its team members precaution. Paytm's offices in Gurugram and Noida were closed for two days due to positive case in Noida and related developments. A man in Noida who had returned from Italy tested positive, along with six of his relatives in Agra. Additionally, a group of Italian tourists and their Indian driver tested positive in Jaipur. The virus spread to a Noida school where a parent had attended party with other students and family members. The total number of confirmed coronavirus cases in India reached 29, including 16 Italians. Globally, COVID-19 has infected over 89,000 people and caused 3,000 deaths. The World Health Organization named the illness COVID-19.
Figure 8: Example 3 of text summarization.
17


Reviews:
[{"review_id": "xm3mXRDzCn", "reviewer": "Reviewer_KQUb", "summary": "The authors present AutoCast++, a system for world event prediction relying on three components: a task-aligned retrieval module; a news summarisation module (text summarisation on retrieved news); a fusion-in-decoder model that is aligned to perform the event predictions.\nThey evaluate the system on the AutoCast dataset by grouping the tasks in numerical, multiple choice and true/false; considering as baselines a collection of methodologies suggested by the benchmark.\nThe results show that the proposed system is able to outperform the considered baselines considering different model sizes.", "questions": "Which GPT3 version was considered in the work?\nWhat is the impact of binning numerical questions? Is the binning applicable also to the baselines? If yes how would results change?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed system shows remarkable performance presenting a limited impact from the model size.\nThe only tasks where it does not excel are the numerical ones, but it's anyway a close call with a baseline that is almost two times larger.", "weaknesses": "While the exclusion of baselines relying on new LLMs including data post mid 2021 is understandable, the ablation studies seem to suggest that relying on LLM for retrieval reranking and summarisation play a huge role in the performance of the system.\nWhat would be convincing is to build/revamp the baselines considered using the GPT3 pre-trained version that the authors leverage in their experiments.\nThis would surely make the submission much stronger and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h15e7sIRvW", "reviewer": "Reviewer_mh89", "summary": "This paper introduces AutoCast++, an event prediction system designed to address forecasting questions by analyzing news documents. AutoCast++ comprises three key components: the Task-Aligned Retrieval Module, which re-ranks news documents based on relevance (using GPT-3 in a zero-shot manner) and recency; the Enhanced Neural Article Reader, which summarizes pertinent news content; and the Human-Aligned Loss Function, aligning system confidence with human forecaster accuracy. These components collectively led to improvements in addressing various forecasting question types in the AutoCast dataset, notably achieving a 48% enhancement in handling multiple-choice questions.", "questions": "Please see the weakness section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Ablation study is done to evaluate the effectiveness of different components of the system.\n\n- Their proposed system achieved 48% improvement on multiple choice forecasting questions, which is significant.\n\n- Even the smaller version of their proposed system (with 0.2 and 0.8 billion parameters) outperforms larger baselines (with 2.8 billion parameters).", "weaknesses": "- Limited reproducibility: The source code associated with the research paper is unavailable, so it is not possible to reproduce the results.\n\n- In ablation study, no specific experiment has been conducted to demonstrate the isolated effectiveness of the Alignment Loss component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcKP5GhjJo", "reviewer": "Reviewer_vd8q", "summary": "This paper proposes a zero-shot ranking-based retriever-reader model for event forecasting. The model can generate the effective answers by the proposed task-aligned retrieval module and enhanced neural article reader. Experimental results on a public dataset verify the effectiveness of the proposed method.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.The logic of the paper is sound.  \n2.The description of the methodology is relatively clear.", "weaknesses": "1.The writing of the paper needs to be further improved and all the symbols need to be interpreted.  \n2.The work focuses on a single textual modality, so why mention multiple data sources, multiple modalities in the introduction section?  \n3.N_q is a subset of D. Why are the elements in N_q n and not d?  \n4.The dataset used is too small and results on more as well as larger datasets are needed to be validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)