SlowGuess/NeurIPS_2023_OCR huggingface dataset:
REALTIME QA: What's the Answer Right Now?
Jungo Kasai $^{*,*}$  Keisuke Sakaguchi $^{*,*}$  Yoichi Takahashi $^{*,*}$  Ronan Le Bras $^{*,*}$
Akari Asai\* Xinyan Velocity Yu\* Dragomir Radev
Noah A. Smith $\diamondsuit$  Yejin Choi $\diamondsuit$  Kentaro Inui $\triangle \spadesuit$
Toyota Technological Institute at Chicago Tohoku University RIKEN Allen Institute for AI
$\spadesuit$  University of Washington  $\spadesuit$  University of Southern California  $\spadesuit$  Yale University  $\spadesuit$  MBZUAI
REALTIME QA realtimeqa.nlp@gmail.com @realtimeqa
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond. $^2$
How many home runs has Shohei Ohtani hit so far this season? A user of a question answering (QA) system might ask such time-sensitive questions and seek out answers in real time. Widely-used evaluation benchmarks of QA systems, however, implicitly assume that answers are static regardless of the time of inquiry. Several recent works (Jia et al., 2018; Chen et al., 2021; Zhang and Choi, 2021; Liška et al., 2022) challenged this assumption and proposed QA datasets that specify the temporal context (e.g., who was the President of the U.S. in 1940?). We extend these recent efforts on time-sensitive QA to fulfill real-time, more instantaneous information needs from users: we establish
Q: How many home runs has Shohei Ohtani hit? A: 24
Figure 1: REALTIME QA establishes a framework to benchmark question answering at the present time: answers (e.g., the number of Shohei Ohtani's home runs) change in real time. Source: https://thecomeback.com/mlb/ shohei-ohtani-home-runs-tommy-john.html.
Q: How many home runs has Shohei Ohtani hit? A: 25
* Work was done during JK's internship at AI2.
$^{2}$ https://realtimeqa.github.io/.
a dynamic benchmark based on newly-published news articles—REALTIME QA—and provide a regularly-updated (weekly in the current version) evaluation platform for the research community.
We develop an annotation framework ( $\S 2$ ) and a benchmarking timeline for real-time QA system submissions. Every week, REALTIME QA retrieves news articles and human-written, multiple-choice questions from news websites (CNN, THE WEEK, and USA Today), covering a wide range of topics, including politics, business, sports, and entertainment. We upload these data, as well as our baseline results, to our website, and any model submission can be evaluated until the next set of questions is posted. This dynamic scheme contrasts with the well-established QA annotations (Chen et al., 2017; Chen and Yih, 2020) that are performed only once with information available at the time. Such annotations are effective for factoid (Berant et al., 2013; Hermann et al., 2015; Rajpurkar et al., 2016; Joshi et al., 2017) or commonsense questions (Zellers et al., 2018, 2019; Talmor et al., 2019; Sakaguchi et al., 2020), but not the real-time information needs that are our target.
We present two classes of real-time baseline systems that are built on strong, recent models (GPT-3: Brown et al., 2020; T5: Raffel et al., 2020; BART: Lewis et al., 2020a): open-book and closed-book QA models. We present a prompting method to use GPT-3 for open-domain QA. The former class uses an external knowledge source, such as Wikipedia (Min et al., 2019; Guu et al., 2020; Lewis et al., 2020b; Izacard and Grave, 2021) or news articles. The latter class of
Figure 2: REALTIME QA data statistics as of June 2, 2023. We started our real-time baselines on June 17, 2022 (§2.4). We also provide past 2,886 QA pairs that can be used by model developers (e.g., finetuning).
closed-book models directly outputs an answer to each question. By design, these closed-book baselines have no access to information more recent than the time of pretraining or finetuning, thereby helping us understand the degree to which real-time information is truly necessary. Notably, a small number of questions in REALTIME QA ( $\sim 12\%$ ) do not strictly require recent information; for example, Shohei Ohtani hits a home run today, leading one to ask where he was born. This is consistent with information-seeking, naturally-occurring scenarios that we target in this work, as seen in Clark et al. (2020). Most users of a QA system do not exclusively ask time-sensitive questions, even though these questions may be stimulated by current events; QA systems should aim to address these questions as well.
Figure 3: REALTIME QA annotation framework and submission workflow. At 3 am GMT on every Saturday, we extract questions from news websites and post them on our website. We immediately run API search for these questions (Google custom search) and share the results as a document pool. The use of this document pool is optional (indicated by a dashed line); participants are allowed to retrieve evidence documents by themselves. All evaluations are done on our website, and the submission window closes when the next set of questions is announced.
We evaluate six baselines both in multiple-choice and generation settings in real time and report the results over the period of June 17 through June 2, 2023. These evaluation data resulted in a total of 1,470 QA pairs (Fig. 2). Further, we provide 2,886 QA pairs that are collected in the same way but preceded our real-time evaluations. These can be used in later work for model development (e.g., finetuning). Our results show that an open-book GPT-3 model augmented with up-to-date text retrieval substantially outperforms closed-book baselines, as well as open-book models with retrieval from a past Wikipedia dump (Lewis et al., 2020b). This result illustrates that large language models
Figure 4: Examples of weekly quizzes from CNN and THE WEEK that are extracted during annotations of REALTIME QA. They span diverse genres, including politics, business, and entertainment.
can adjust their knowledge, based on the retrieved passages (§3). Nonetheless, we find that they still struggle, especially when the multiple choices include uncertainty (e.g., “none of the above”). Most of the errors originate from retrieval, rather than reading comprehension. The REALTIME QA benchmark, therefore, highlights the importance of fast, up-to-date text retrieval (Seo et al., 2019) to better serve instantaneous information needs. We share all data and code to reproduce our baselines so that follow-up work can build upon our first attempts to tackle this task.
REALTIME QA can also serve as an important step toward much-needed, broader, real-time applications of NLP. For example, a QA system with timely updates can improve emergency management of natural disasters (Imran et al., 2013, 2015, 2016; Nguyen et al., 2016) or pandemics (e.g., COVID-19; Wang et al., 2020; Lee et al., 2020; Møller et al., 2020; Alzubi et al., 2021). With the advent of online news, prior work developed automated systems that regularly retrieve and summarize news articles from the Internet (Allan et al., 2001; Radev et al., 2001; McKeown et al., 2002, 2003; Evans et al., 2004). Models developed for the REALTIME QA task can be further enhanced with such retrieval/summarization systems. We hope that our REALTIME QA interface and baseline models will serve as a useful platform for research and real-world applications.
2 REALTIME QA Framework
Our current version announces questions every week, based on news articles published within the past week. Here we establish the workflow ( $\S 2.1$ ) and the framework for annotations ( $\S 2.2$ ) and evaluations ( $\S 2.3$ ). We then discuss our built-in baselines ( $\S 2.4$ ) that are continually evaluated every week. Our user interface and more detailed statistics (e.g., genres and answer types) are available in Appendices B and C.
2.1 Workflow
Fig. 3 depicts the REALTIME QA workflow for each week. We announce  $\sim 30$  multiple-choice questions at 3 am GMT every Saturday. We internally run API search (Google custom search, GCS) for these questions and share a set of documents (mostly news articles) with the URLs that are available at that time. Participants run their model on these questions, optionally using the documents from our API search as a knowledge source (indicated as dashed lines in Fig. 3). While we provide our document set to lower barriers to submission, participants are also allowed to create and use knowledge sources by themselves (e.g., custom retrieval models or other external APIs such as Twitter API). System submissions are shared on our website with their performance and submission time. The submission window closes when the new set of questions is announced the next week.
Note that fair, retroactive comparisons of systems are also possible, as long as they use data available when the submission window was still open. For instance, participants might be interested in evaluating their model against a past submission on the Week N questions. In this case, they can do so by ensuring that their system only relies on data up to Week N and simulating how their system would have performed at that time. Our platform still focuses on real-time evaluations and encourages every participant to submit real-time results to better reflect real-world applications.
2.2 Annotation
Question Extraction The authors of this paper perform weekly annotations in a human-in-the-loop way. We first find web pages for "weekly quizzes" from three news websites: CNN (US-based), USA Today, and The WEEK (UK-based).<sup>3</sup> Shown in Fig. 4 are examples that span politics and business genres. We then execute our extraction script to collect multiple-choice questions. Each of these three websites posts  $\sim 10$  questions per week, resulting in  $\sim 120$  questions in total every month. Weekly quizzes are also available from the New York Times and ABC Australia, but they are not included in the current version, due to issues with automatic extraction or a paid subscription system.
API Search Using each of these questions as a retrieval query, we run Google custom search<sup>4</sup> to collect the top-10 documents from the web. The retrieval target is all articles from CNN, USA Today, and THE WEEK. We then parse every document using the newspaper3k package<sup>5</sup> and store the text as well as metadata, such as the publication date and author name. In some rare cases, articles from the search get taken down, in which case we disregard them. This indeed illustrates a unique challenge of real-time applications with constantly-changing, dynamic information.
2.3 Evaluation
Multiple Choice Since REALTIME QA is a multiple-choice question dataset, we can simply measure performance by accuracy. We also explored a NOTA (none of the above) setting: one of the original choices is randomly replaced with "none of the above," thereby helping prevent models from exploiting heuristics (Rajpurkar et al., 2018). As expected, the NOTA setting resulted in performance degradation across the board ( $\S 3$ ). NOTA choices can be found in other multiple-choice QA or reading comprehension datasets (Richardson et al., 2013; Lai et al., 2017).
Generation We also experiment with a generation setting where no choices are given, to better reflect real-world applications. Under this setting, we evaluate performance with exact matching and token-based F1 scores, following the standard practice in question answering (Rajpurkar et al., 2016).
Human Performance We randomly sampled 10 weeks from June 17, 2022 through January 13, 2023 (300 questions in total), and the authors of this paper answered multiple-choice questions using Google search. This resulted in the accuracy of  $96.7\%$ . Most questions in REALTIME QA are straightforward (e.g., single-hop questions) and a human with Internet access can easily answer them. For the sustainability of the dynamic benchmark, we do not provide an estimate of human performance on a regular basis.
2.4 Real-time Baselines
REALTIME QA executes six baselines in real time that are based on strong pretrained models: four open-book and two closed-book models. These six models are evaluated and made publicly available when weekly questions are announced. Any submission to REALTIME QA is compared against them. Participants can also build their model upon our baselines. See Appendix §A for more detail.
2.4.1 Open-book QA Models
Open-book QA models follow a two-step pipeline: document retrieval that finds evidence documents from an external knowledge source (e.g., Wikipedia) and answer prediction (or reading comprehension) that outputs an answer conditioned on the question and evidence documents. For either step, we experiment with two variants, resulting in a total of four configurations. Open-book systems have the advantage of being capable of updating the external knowledge source at test time (Lewis et al., 2020b). This property is particularly crucial for questions in REALTIME QA that inquire about information at the present time.
$^{3}$ Fair use is allowed under Section 107 of the Copyright Act in the U.S.: https://www.copyright.gov/title17/92chap1.html#107.
<sup>4</sup>https://programmablesearchengine.google.com/.
<sup>5</sup>https://github.com/codelucas/newspaper.
<sup>6</sup>In fact, USA Today has a record of human top scorers every week, and they all get perfect scores. E.g., https://www.usatoday.com/storytelling/quiz/news-quiz/2022-07-01/.
Document Retrieval For the retrieval step, we experiment with two configurations: top-5 Wikipedia documents from dense passage retrieval (DPR; Karpukhin et al., 2020) and top-5 news articles from GCS ( $\S 2.2$ ). In DPR, English Wikipedia articles from the December 2018 dump are segmented into 100-word documents (Wang et al., 2019). DPR encodes the question and every document into 768-dimensional vectors; it then computes the inner product to obtain a matching score and selects documents with top-5 matching scores. We use the BERT-based model (Devlin et al., 2019), finetuned on the Natural Questions dataset (Kwiatkowski et al., 2019) from the Hugging Face Transformers library (Wolf et al., 2020). GCS uses an external API, and we found that it sometimes returned fewer than five documents ( $\sim 10\%$  of the time); in this case, we add top documents from DPR to create a top-5 document set.
Answer Prediction We explore two methods for answer prediction, conditioned on the question and the corresponding retrieved text: retrieval-augmented generation (RAG; Lewis et al., 2020b) and a prompting method with GPT-3 (text-davinci-002; Brown et al., 2020). In the multiple-choice setting, we compute the log probability of every choice and normalize it by the generation sequence length. We then select the choice with the best score. For the generation setting, we simply perform text decoding.
For the RAG baseline, we use the BART-based (Lewis et al., 2020a) RAG-sequence model, again finetuned on Natural Questions from the Transformers library. This model predicts the answer sequence  $\mathbf{y}$  autoregressively from left to right while marginalizing over the set of top-5 retrieved documents  $(\mathcal{Z})$ :  $P(\mathbf{y}) = \sum_{z \in \mathcal{Z}} P(z) \prod_{t=1}^{|\mathbf{y}|} P(y_t | z, \mathbf{y}_{\leq t})$ . Here  $P(z)$  is given by the matching score from the retrieval step. In the equation, the conditioned-upon question is suppressed for brevity.
We propose a straightforward GPT-3 prompting method with temporal contexts (Fig. 5). We prepond to every question the title and the first two paragraphs of the top-5 articles from the document retrieval step. The publication date is inserted, using the metadata of each retrieved article (e.g., "Article on November 2, 2021" in Fig. 5). For Wikipedia passages retrieved by DPR, we prepend "December 31, 2018," based on the Wikipedia dump date (Karpukhin et al., 2020). Our ablation studies on date insertion will show that the open-book GPT-3 system benefits from specifying the dates of the question and the retrieved articles to some extent (\$3.2).
2.4.2 Closed-book QA Models
Closed-book QA models directly answer questions without access to external knowledge. They have proven competitive with open-book models on some QA datasets (Roberts et al., 2020; Guu et al., 2020). Since these models are trained/finetuned on the data available at that time, they cannot address questions about new events or updated information. Nonetheless, some of the real-time information needs do not necessarily require up-to-date information. Indeed, REALTIME QA contains a small portion of such questions ( $\sim 10\%$ ). For instance, Microsoft retired its Internet Expl questions are triggered by a new event but recently. Most users of a QA system do systems should aim to address these ques degree to which up-to-date information is n the following two strong methods for close
Figure 5: Example prompt for answer generation with the open-book GPT-3 baseline. For the closed-book GPT-3 baseline, the top-5 articles are not given. We perform ablation studies on the date information (§3.2).
${}^{7}$  Unlike DPR, GCS does not provide matching scores. We treat top-5 documents with equal probabilities.
See Lazaridou et al. (2022) for other prompt templates.
${}^{9}$  This substantially reduces the inference cost. They contain most of the key information in each article.
Finetuning Method We use the T5 model (T5-11B; Raffel et al., 2020) finetuned on the Natural Questions data, again from the Transformers library. Following the open-book baseline, we select the choice with the maximum average log score in the multiple-choice setting.
Prompting Method Similar to the open-book baselines (§2.4.1), we apply a prompting method to GPT-3 (text-davinci-002). We use the same prompt as Fig. 5, except that no articles are inserted before the question. Again following the open-book baselines, we select the choice with the maximum average log score in the multiple-choice setting.
We started our real-time experiments on June 17 2022, spanning a year as of June 2 2023 (1470 questions in total). We will continue our weekly annotations, but here we report our experimental and analysis results so far and give guidance to future participants.
Table 1: Results from the past year (from June 17, 2022 through June 2, 2023). GCS: Google custom search; DPR: dense passage retrieval (Karpukhin et al., 2020); RAG: retrieval-augmented generation (Lewis et al., 2020b).
Real-time Baselines	Multi-choice	Generation
Retrieve	Predict	Orig.	NOTA	EM	F1
Open	DPR	RAG	27.4	24.8	2.4	4.1
DPR	GPT-3	43.9	35.8	13.3	19.7
GCS	RAG	46.9	37.9	17.5	22.1
GCS	GPT-3	66.5	58.4	34.6	45.3
Closed —	T5	39.1	35.3	9.7	14.7
—	GPT-3	44.9	34.1	15.3	22.3
Table 2: Ablation studies on date insertion in the prompt for the open-book (Google custom search; GCS) and close-book GPT-3 baselines. All results are averaged over the first six weeks: June 17 through July 22, 2022.
Date Insert	Multi-choice	Generation
Articles	Qs	Orig.	NOTA	EM	F1
✓	✓	69.3	59.8	28.7	39.4
✓	✗	66.5	62.6	24.7	36.3
✗	✓	67.0	57.5	28.1	38.2
✗	✗	65.9	61.5	28.7	38.3
Closed—	✓	39.7	31.3	7.3	15.2
—	✗	45.8	38.5	9.0	15.9
Seen in Table 1 are the results from the past year. In all three settings (original/NOTA multiple choice and generation), GPT-3 with Google custom search (GCS) retrieval achieves the best performance. In particular, GPT-3 with GCS substantially outperforms both closed-book GPT-3 and GPT-3 with DPR (from a December 2018 Wikipedia dump): e.g., 34.6 vs. 15.3/13.3 in generation exact matching. This suggests that GPT-3 is able to answer questions based on the given prompt, rather than relying on past information from pretraining. Nevertheless, we still see a large performance drop of all baselines from the original multiple-choice setting to NOTA ("none of the above"): e.g., 58.4 vs. 66.5 for GPT-3 with GCS retrieval. Future work can further improve GPT-3's ability of reading comprehension, especially regarding answer uncertainty.
3.2 Analysis and Ablations
Date Insertion for Prompting Our prompt for the GPT-3 baselines presupends date information both to the articles and question (Fig. 5). Table 2 shows results from ablation studies on date insertion for the open-book (GPT-3 with Google custom search) and closed-book GPT-3 models. Temporal specification almost always helps the open-book GPT-3 model. Interestingly, it hurts the performance of the closed-book model, perhaps because the specified date is generally unseen during pretraining and the prompt becomes "out-of-domain."
Error Breakdown We conducted a manual error analysis of the results so far. In particular, we categorized answers from the best generation model (open-book GPT-3 with GCS) into three categories: correct, retrieval error, and reading comprehension error. For the questions from the first six weeks, the breakdown was the following: correct  $(52\%)$ , retrieval error  $(34\%)$ , and reading comprehension error  $(13\%)$ . This suggests that the key to instantaneous applications of question answering is accurate, up-to-date information retrieval.
Performance vs. Submission Time Fig. 6 plots the performance of the open-book GPT-3 baseline with Google custom search (GCS) over varying submission (i.e., GCS retrieval) time. All results are averaged over the questions between June 17 and July 22, 2022. We see a consistent pattern: the performance remains high (or improves) up to around 24 hours after the announcement but substantially degrades later. While the performance can improve when GCS starts to retrieve more recent articles, it eventually suffers from temporal gaps. Our website provides the submission time of every system as well as its performance.
Examples Table 3 shows some examples that compare the closed-book and open-book GPT-3 models. The first three examples illustrate that GPT-3 can correctly update its answer based on the retrieved documents across diverse genres: natural disasters, the COVID-19 pandemic, and entertainment. The last three cases, on the other hand, demonstrate a critical limitation of current large language models in temporal understanding: the retrieved documents do not suffice to answer the questions due to a temporal gap, and GPT-3 still generates an outdated answer. Ideally, GPT-3 should inform the user or even the retrieval module that it does not have enough evidence to answer the question. This way, the retrieval module can expand its search, or the user can consult other resources.
Figure 6: Performance vs. submission time (hours after the announcement of questions, 3 am GMT on Saturday) over the three evaluation settings (A: original multiple choice; B: none of the above; C: generation). All results are from open-book GPT-3 with Google custom search (GCS) and averaged over the questions from June 17, 2022 through July 22, 2022.  $\Delta t = 0$  for all of our six real-time baselines by default.
Note that it is possible to limit the retrieval target