RealTime QA: What's the Answer Right Now?
Authors: Jungo Kasai
Keisuke Sakaguchi
yoichi takahashi
Ronan Le Bras
Akari Asai
Xinyan Velocity Yu
Dragomir Radev
Noah A. Smith
Yejin Choi
Kentaro Inui
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: question answering
time sensitivity
information retrieval
large language models
TL;DR: We introduce RealTime QA, a dynamic, time-sensitive question answering benchmark that evaluates systems on a regular basis.

Abstract: We introduce RealTime QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). RealTime QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that RealTime QA will spur progress in instantaneous applications of question answering and beyond.

REALTIME QA: What's the Answer Right Now?
Jungo Kasai
Keisuke Sakaguchi +0- *0-
Yoichi Takahashi
Ronan Le Bras"
Akari Asai
Xinyan Velocity Yu
Dragomir Radev
Noah A. Smith 0o
Yejin Choi
Kentaro Inui 4
Toyota Technological Institute Chicago
Tohoku University
RIKEN
Allen Institute for Al
University of Washington
University of Southern California
Yale University
ABBZAAI
REALTIME @A
realtimeqa.. htwpiiimmaaa...maaaaaa... wwl.nnlpppmm..mmmmmmmmmmmmmmmm.m
@realtimeqa
Abstract
We introduce REALTIME QA, dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications question answering and beyond
Introduction
How many home runs has Shohei Ohtani hit?
How many home has Shohei Ohtani hit so far this season? user of a question answering (QA) system might ask such time-sensitive questions and seek out answers in real time. Widely- eval- uation benchmarks of QA systems, however, im- plicitly assume that answers are static regardless of the time of inquiry. Several recent works (Jia et al. 2018 Chen et al. 2021 Zhang and Choi 2021
A: 24
Q: How many home runs has Shohei Ohtani hit?
25
Liška a (002) challenged this assump- tion and proposed QA datasets that specify the tem- poral context (e.g., who was the President of the U.S. in 1940?). We extend these recent efforts on time-sensitive QA to fulfill real-time, more instan- taneous information needs from users: we establish
Figure 1: REALTIME QA establishes
framework to benchmark question answering at the present time: answers (e.g., the number of Shohei Ohtani's home runs) change in real time. Source: https::////// ttttps:///hecomebacc. com/mb//
home runs www.u-utmny--aaatee tommy-john
Work done during JK's internship AT2. https://realttimqaa. wwwpeeeeeegggtttbugaeeeege github.
html
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

a dynamic benchmark based on newly-published news articles-REALTTI QA-and provide regularly-updated (weekly in the current version) evaluation platform for the research community.
We develop an annotation framework ($2) and a benchmarking timeline for real-time QA system submissions. Every week, REALTIME QA retrieves news articles and human-written, multiple-choice questions from news websites (CNN, THE WEEK, and USA Today), covering a wide range of topics, including politics, business, sports, and entertainment. We upload these data, as well as our baseline results, to our website, and any model submission can be evaluated until the next set of questions posted. This dynamic scheme contrasts with the well-established QA annotations (Chen et al. 2017;CChen [hen and Yih, 2020) that are performed only once with information available at the time. Such annotations are effective for factoid (Berant et al.] [013]] Hermann et al. 2015
Rajpurkar et al., 2016 Doshi et al. (017) or commonsense questions (Zellers et al. [2018] 2019
Talmor et al. Sakaguchi et al. 2020 but not the real-time information needs that are our target.
We present two classes of real-time baseline systems that are built on strong, recent models (GPT-3: Brown et al. 2020; T5: Raffel et al., 2020:
1/3/2020
6/17/2022
6/2/2023 (NeurIPS Submission)
2886 QA Pairs
1470 QA Pairs
120 Questions/month
BART: Lewis et al. 2020ap: open-
Real-time Baselines
book and closed-book QA models. We present prompting method to use GPT-3 for open-domain QA. The former class uses an external knowl- edge source, such as Wikipedia (Min)
Figure 2: REALTIME QA data statistics as of June 2, 2023. We started our real-time baselines on June 17, 2022 ($2.4). We also provide past 2,886 QA pairs that can be used by model developers (e.g., finetuning).
ctt al.
2019; Guu et al. 2020; Lewis
al. 2020b Izacard and Grave 2021)
or news articles. The latter class of
closed-book models directly outputs answer to each question. By design, these closed-book baselines have no access to information more recent than the time of pretraining or finetuning, thereby helping us understand the degree to which real-time information is truly necessary. Notably, small number of questions in REALTIME QA (~12%) do not strictly require recent information; for example, Shohei Ohtani hits a home run today, leading one to ask where he was born. This is consistent with informationnseekiing in Clark et al. (2020). Most users of QA system do not exclusively ask time-sensitive questions, even though these questions may be stimulated by current events; QA systems should aim to address these questions as well.
scenarios that we target in this work, as seen
Week
Week
Ansurtatin
Annotation
Expaci
API Scacch
Extrct
API Search
ReafTime QA
CMN
WEWEX
documeits
Submit
CWM
TiCMEEC
questions
documeits
Partiipasss
modelay
Figure 3: REALTIME QA annotation framework and submission workflow. At 3 am GMT on every Saturday, we extract questions from news websites and post them on our website. We immediately run API search for these questions (Google custom search) and share the results The use of this document pool is optional (indicated by a dashed line); participants are allowed to retrieve evidence documents by themselves. All evaluations are done on our website, and the submission window closes when the next set of questions is announced.
document pool.
We evaluate six baselines both in multiple-choice and generation settings in real time and report the results over the period of June 17 through June 2, 2023. These evaluation data resulted in a total of 1,470 QA pairs (Fig. (2). Further, we provide 2,886 QA pairs that are collected in the same way but preceded our real-time evaluations. These can be used in later work for model development (e.g., finetuning). Our results show that an open-book GPT-3 model augmented with up-to-date text retrieval substantially outperforms closed-bood baselines, as well as open-book models with retrieval from past Wikipedia dump (Lewis et al. 2020b). This result illustrates that large language models
2

---

Ci
Things newa GMit
April 1, 2022
THE WEEK
Quiz of The Week
May 18, 2022
China is betting iis biggest Covid 19 weve yet. Wrich lerge city there implomented masa corcnaviius tosting this wook?
hernng deployed by the contrrol Ultrainc?
Targ
Twitter agreed be Tesla Spacex Elon Musk
Temminatee
Hell Drivers
64.4 Bon
$440 mi lon
Vakin
$4.4 ballion
944 billoo
Figure 4: Examples of weekly quizzes from CNN and THE WEEK that extracted during annota- tions of REALTIME QA. They span diverse genres, including politics, business, and entertainment.
can adjust their knowledge, based on the retrieved passages 3 Nonetheless, we find that they
still struggle, especially when the multiple choices include uncertainty (e.g., "none of the above"). Most of the errors originate from retrieval, rather than reading comprehension. The REALTIME QA benchmark, therefore, highlights the importance of fast, up-to-date text retrieval (Seo et al. 2019) to better serve instantaneous information needs. We share all data and code to reproduce our baselines so that follow-up work can build upon our first attempts to tackle this task.
REALTIME QA can also serve as an important step toward much-needed. broader, real-time applica- tions of NLP. For example, a QA system with timely updates can improve emergency management of natural disasters (Imran et al., 2013 2015 2016 Nguyen et al. 2016) or pandemics (e.g.. COVID-19 Wang et al.] 2020; Lee et al. 2020; Möller et al. [2200] Alzubi et al. (021). With the advent of online news, prior work developed automated systems that regularly retrieve and summarize news articles from the Internet (Allan et al., 2001] Radev et al., 2001 McKeown et al. [0022] 2003 [vans (ct al., 2004). Models developed for the REALTIME QA task can be further enhanced with such retrieval/summarizatiin systems. We hope that our REALTIME QA interface and baseline models will serve as a useful platform for research and real-world applications.
2 REALTIME QA Framework
Our current version announces questions every week, based on news articles published within the past week. Here we establish the workflow ($2.1) and the framework for annotations ($2.2) and evaluations ($2.3). We then discuss our built-in baselines ($2.4) that are continually evaluated every week. Our user interface and more detailed statisties (e.g., genres and answer types) available in Appendices C
2.1 Workflow
Fig. depicts the REALTIME QA workflow for each week. We announce ~30 multiple-choice questions 3 am GMT every Saturday. We internally run API search (Google custom search, GCS) for these questions and share set of documents (mostly news articles) with the URLs that are available at that time. Participants run their model on these questions, optionally using the documents from our API search as a knowledge source (indicated as dashed lines in Fig. 3 While we provide our document set to lower barriers to submission, participants are also allowed create and use knowledge sources by themselves (c.g., custom retrieval models other external APIs such as Twitter API). System submissions are shared on our website with their performance and submission time. The submission window closes when the new set of questions is announced the next week.
Note that fair, retroactive comparisons of systems are also possible, as long as they use data available when the submission window was still open. For instance, participants might be interested in evaluating their model against past submission on the Week questions. In this case, they can do so by ensuring that their system only relies on data up Week N and simulating how their system would have performed at that time. Our platform still focuses on real-time evaluations and encourages every participant to submit real-time results to better reflect real-world applications.

---

2.2 Annotation
Question Extraction The authors of this paper perform weekly annotations in a human-in-the-loop way. We first find web pages for "weekly quizzes" from three news websites: CNN (US-based) USA Today, and The WEEK (UK-based) Shown in Fig. are examples that span politics and business genres, We then execute our extraction script to collect multiple-choice questions. Each of these three websites posts 10 questions per week, resulting in 120 questions in total every month. Weekly quizzes are also available from the New York Times and ABC Australia, but they are not included in the current version, due issues with automatic extraction paid subscription system.
API Search Using each of these questions retrieval query, we run Google custom search to collect the 1op-10 documents from the web. The retrieval target is all articles from CNN, USA Today, and THE WEEK. We then parse every document using the newspaper3k package and store the text as well as metadata, such as the publication date and author name. In some rare cases, articles from the search get taken down, in which case disregard them. This indeed illustrates a unique challenge of real-time applications with constantly-haanging dynamic information.
2.3 Evaluation
Multiple Choice Since REALTIME QA is multiple-choice question dataset, we can simply measure performance by accuracy. We also explored a NOTA (none of the above) setting: one of the original choices randomly replaced with "none of the above," thereby helping prevent models from exploiting heuristics (Rajpurkar et al. 2018). As expected, the NOTA setting resulted in performance degradation across the board (33) NOTA choices can found in other multiple-choice QA or reading comprehension datasets (Richardson et al. 2013 Lai et al. 2017).
Generation We also experiment with a generation setting where no choices given, to better reflect world applications. Under this setting, we evaluate performance with exact matching and token-based Fl scores, following the standard practice in question answering (Rajpurkar et al. 2016).
Human Performance We randomly sampled 10 weeks from June 17, 2022 through January 13, 2023 (300 questions in total), and the authors of this paper answered multiple-choice questions using Google search. This resulted in the accuracy of 96.7%. Most questions in REALTIME QA straightforward (e.g., single-hop questions) and a human with Internet access can easily answer them For the sustainabilityy of the dynamic benchmark, we do not provide an estimate of human performance on regular basis.
2.4 Real-time Baselines
REALTIME QA executes six baselines in real time that are based on strong pretrained models: four open-book and two closed-book models. These six models evaluated and made publicly available when weekly questions are announced. Any submission to REALTIME QA is compared against them. Participants can also build their model upon our baselines. See Appendix A for more detail.
2.4.1 Open-book QA Models
Open-book QA models follow two-step pipeline: document retrieval that finds evidence doc- uments from an external knowledge source (e.g.. Wikipedia) and answer prediction (or reading comprehension) that outputs an answer conditioned on the question and evidence documents. For either step, we experiment with two variants, resulting in a total of four configurations. Open-book systems have the advantage of being capable of updating the external knowledge source test time (Lewis et al. (2020b). This property is particularly crucial for questions in REALTIME QA that inquire about information the present time.
Fair use allowed under Scction 107 of the Copyright Act in the U.S.: https://ww.cooprrigttttt (tslee77/9cchap..htmlilt
http:://ppopr.mmaaaeeerrrrrreeee google com/
https://github.cr http://hhhhh..ooo///eee////////p
In fact, USA Today has a record of human top scorers every week. and they all get perfect scores. E.g..
https://www. uwwwuday.aaoouu.. http://wwwwwccoooo/ooooooo/oo///

---

Document Retrieval For the retrieval step, we experiment with two configurations: top-5 Wikipedia documents from dense passage retrieval (DPR: Karpukhin et al., 2020) and top-5 news articles from GCS ($2.)) In DPR, English Wikipedia articles from the December 2018 dump are segmented into 100-word documents (Wang et al., 2019). DPR encodes the question and every document into 768-dimensional vectors; it then computes the inner product to obtain a matching score and selects documents with top-5 matching scores. We use the BERT-based model (Devlin et al., 2019), finetuned on the Natural Questions dataset (Kwiatkowski et al.) (019) from the Hugging Face Transformers library (Woll et al. 2020). GCS uses an external API, and we found that it sometimes returned fewer than five documents (~ 10% of the time); in this case, add top documents from DPR to create top-5 document set.
Answer Prediction
We explore two methods for answer prediction, conditioned on the question and the corresponding retrieved text: retrieval-augmented generation (RAG: Lewis et al. 2020b) and a prompting method with GPT-3 (text-davinci-O0)) (Bownn et al. (020). In the multiple-choice setting, we compute the log probability of every choice and normalize by the generation sequence length. We then select the choice with the best score. For the generation setting, we simply perform text decoding.
For the RAG baseline, we use the BART-based (Lewis et al. 2020a) RAG-sequence model, again finetuned on Natural Questions from the Transformers library. This model predicts the answer sequence autoregressively from left to right while marginalizing over the set of top-5 retrieved
documents (Z): (y) Eeez P (z) [Mv P (we) Here P(z) is given by the matching score from the retrieval step In the equation, the conditioned-upon question is suppressed for brevity.
We propose straightforward GPTT- prompting method with temporal contexts (Fig. 5 We prepend to every question the title and the first two paragraphs of the top-5 articles from the document retrieval step The publication date is inserted, using the metadata of each retrieved article (e.g.. "Article on November 2, 2021" in Fig. 5). For Wikipedia passages retrieved by DPR, we prepend "December 31, 2018," based the Wikipedia dump date (Karpukhin et al., 2020 Our ablation studies on date insertion will show that the open-book GPT-3 system benefits from specifying the dates of the
R
9
question and the retrieved articles to some extent ($.22.
2.4.2 Closed-book QA Models
Article November 2021: CDC recommends Pfizer COVID-19 19 vaccine for kids 5-11, shots expected to roll out this week...
Top-1 Articlo
Closed-book QA models directly answer questions without access to external knowl- edge. They have proven competitive with open-book models on some QA datasets (Roberts et al., 2020; Guu et al. 2020). Since these models are trained/finetuned on the data available at that time, they can- not address questions about new events or updated information. Nonetheless, some of the real-time information needs do not necessarily require up-to-date information. Indeed, REALTIME QA contains a small portion of such questions (~ 10%). For in-
Article June 23, 2022: Covid-19 vaccinations begin for US children under S...
Top-2 Article
Question: Today is June 24, 2022, Covid-19 vaccinations the US began for which age group this week?
Question
Answer: Children under
Answer Prediction
Figure 5: Example prompt for answer generation with the open- spen-book GPT-3 baseline. For the dlosed-book GPT-3 baseline, the top-5 articles are not given. We perform ablation studies on the date information (33.2) stance, Microsoft retired its Internet Explorer browser this week. What year did it debut? Such questions are triggered by a new event but inquire about facts in the past that have not changed recently. Most users of QA system do not exclusively raise time-sensitive questions, and QA systems should aim to address these questions as well. Closed-bool baselines thus quantify the degree to which up-to-date information is necessary to answer questions in REALTIME QA. We use the following two strong methods for closed-booe -bok QA.
Unllie DPR, GCS does not provide matching scores. We treat top-5 documents with equal probabilities.
*See Lazaridou et al. (2022) for other prompt templates.
This substantially reduces the inference cost. They contain most of the key information in each article.
5

---

Finetuning Method We use the T5 model (T3-11B; Raffel et al. 2020) finetuned on the Natural Questions data, again from the Transformers library. Following the open-book baseline, we select the choice with the maximum average log score in the multiple-choice setting.
Prompting Method Similar to the open-book baselines 224.11 apply prompting method to GPT-3 (lext-davinci-002). We use the same prompt as Fig. except that no articles are inserted before the question. Again following the open-book baselines, select the choice with the maximum average log score in the multiple-choice setting.
3 Experiments and Analysis
We started our real-time experiments on June 17 2022, spanning year of June 2 2023 (1470 questions in total). We will continue our weekly annotations, but here we report our experimental and analysis results far and give guidance to future participants.
Table 1: Results from the past year (from June 17, 2022 through June 2, 2023). GCS:
Table 2:
Ablation studies date insertion in the prompt for the open-book (Google custom search; GCS) and close-book GPT-3 baselines.
Google custom search; DPR: dense passage trieval (Karpukhin et al.] 2020); RAG: retrieval-
All results are averaged over the first six weeks: June 17 through July 22, 2022.
augmented generation (Lewis al. 2020b).
Real-time Baselines
Multi-choice
Generation
Date Insert
Multi-choice
Generation
Retrieve Predict
Orig.
NOTA
EM
Fl
Articles Qs
Orig.
NOTA
EM
F1
DPR
RAG
27.4
24.8
2.4
4.1
/
69.3
59.8
28.7
39.4
DPR
GPT-3
43.9
35.8
13.3
19.7
/
x
66.5
62.6
24.7
36.3
Open
Open
GCS
RAG
46.9
37.9
17.5
22.1
/
67.0
57.5
28.1
38.2
GCS
GPT-3
66.5
58.4
34.6
45.3
x
65.9
61.5
28.7
38.3
39.1
35.3
9.7
V
39.7
31.3
7.3
15.2
Closed
Closed
GPT-3
44.9
34.1
15.3
22.3
x
45.8
38.5
9.0
15.9
3.1 Results
Seen T are the results from the past year. all three settings (original/NOTA multiple choice and generation), GPT-3 with Google custom search (GCS) retrieval achieves the best performance. In particular, GPT-3 with GCS substantially outperforms both closed-book GPT-3 and GPT-3 with DPR (from a December 2018 Wikipedia dump): e.g., 34.6 15.3/13.3 3.3 in generation exact matching. This suggests that GPT-3 able to answer questions based on the given prompt, rather than relying on past information from pretraining. Nevertheless, baselines from the original multiple-choice setting to NOTA ("none of the above"): e.g., 58.4 vs. 66.5 for GPT-3 with GCS retrieval. Future work further improve GPT-3's ability of reading comprehension, especially regarding answer uncertainty.
still
large performance drop of all
3.2 Analysis and Ablations
Date Insertion for Prompting Our prompt the GPT-3 baselines prepends date information both to the articles and question (Fig. 5). Table shows results from ablation studies on date insertion for the open-book (GPT-3 with Google custom search) and closed-book GPT-3 models. Temporal specification almost always helps the open-book GPT-3 model. Interestingly. it hurts the performance of the closed-book model, perhaps because the specified date is generally unseen during pretraining and the prompt becomes "out-of-domain."
Error Breakdown We conducted manual error analysis of the results so far. In particular, we categorized answers from the best generation model (open-book GPT-3 with GCS) into three categories: correct, retrieval error, and reading comprehension error. For the questions from first six weeks, the breakdown was the following: correct (52%), retrieval error (34%), and reading comprehension error (13%). This suggests that the key to instantaneous applications of question answering is accurate, up-to-date information retrieval.

---

Performance vs. Submission Time Fig. plpoos the performance of the open-book GPT-3 baseline with Google custom search (GCS) over varying submission (i.e.. GCS retrieval) time. All results are averaged over the questions between June 17 and July 22, 2022. We see a consistent pattern: the performance remains high (or improves) up to around 24 hours after the announcement but substantially degrades later. While the performance can improve when GCS starts to retrieve more recent articles, it eventually suffers from temporal gaps. Our website provides the submission time of every system well as its performance.
Examples Table shows some examples that compare the closed-book book and open-book GPT-3 models. The first three examples illustrate that GPT-3 can correctly update its answer based on the retrieved documents across diverse genres: natural disasters, the COVID-19 pandemic, and entertainment. The last three cases, on the other hand, demonstrate a critical limitation of current large language models in temporal understand- ing: the retrieved documents do not suffice to answer the questions due to a temporal gap, and GPT-3 still generates outdated answer. Ideally, GPT-3 should inform the user or even the retrieval module that it does not have enough evidence to answer the question. This way, the retrieval module user can consult other resources.
Original
NOTA
Generation
70
Ave
20
600
24
36
00
35
50
botrs
Figure 6: Performance vs. submission time (hours after the announcement of questions, 3 am GMT on Saturday) over the three evaluation settings (A: original multiple choice; B: none of the above; C: generation). All results are from open-book GPT-3 with Google custom search (GCS) and averaged over the questions from June 17, 2022 through July 22, 2022. = 0 for all of our six real-time baselines by default.
expand its search, or the
Note that it is possible to limit the retrieval target to recent articles T0
but there are potential failure modes. Firstly, some questions in REALTIME QA inquire about the past, and models can benefit from older articles when answering such questions. Further. the appropriate date range for retrieval varies from question to question in real-world applications; some questions inquire about this year, while others about this week. We thus do not implement such filtering for the current real-time baselines.
4 Related Work
REALTIME QA has time sensitivity, which several prior works addressed on various NLP tasks. Here we discuss its relation to long-standing summarization and text retrieval tasks, as well recent work on temporal misalignment between training and evaluation. We then discuss its connections to dynamic evaluations and open- open-domain QA.
CaiimarrizaaiooRetrrivee over Time Temporal (or timeline) summarization is task that retrieves documents from the web and provides their summary over time (Catizone et al. 2006 Aslam et al.
2013 2014, 2015 Martschat and Markert 2017 2018). Update summarization (Witee al. |007]] Dang and Owezarzak 2008) and new event detection/track (Allan et al.) 1998 Li et al.) 2005) are tasks that monitor and track newly-added information. Prior work created datasets and systems for these tasks (Tran et al. 2013 2015 Wang et al.] 2015 Chen et al. [2019] Gholipour Ghalandari and Ifrim 2020). Their evaluations are usually executed statically, with information available at the time of data collection.
In contrast, the TREC real- -ime summarization track evaluates systems real time during a 1-2 week evaluation period (Lin et al.) [016]] 20177) /Sequiera et al. 2018). Several other works and initiatives
focused particularly on financial news summarization (Filippova et al. 2009 Passali et al.) (0.T) or emergency management technology (Temnikova et al. 201 4 Ghosh et al. [017] McCreadie et al. 2019), including the COVID-19 19 pandemic (Buntain et al. 20200 Pasquali et al., 2021). This work regularly evaluates question answering systems over diverse topics, but we share the goal of dealing with novel and evolving information over time; retrieval or summarization methods from these tasks (e.g.. Yan et al. 2001aa b 2012 Shou et al. 2013) can be combined with models in REALTIME QA to serve various Lime-sensitive information needs from users. REALTIME QA can also be used to evaluate time- sensitive retrieval systems by the downstream QA performance.
10 Indeed, GCS has paid version with date range feature that filters retrieval results by date.
7

---

Table 3: Examples that compare closed-book and open-book GPT-3 answers with top-5 articles from Google custom search (GCS) retrieval. As the first three examples, GPT-3 can adjust its answer based newly-retrieved documents. When the retrieved documents outdated unrelated, however, GPT-3 ignores the temporal gap and yields an outdated answer.
Question
Retrieved Documents (Top-5)
June 14, 2022
Yellowstone National Park
Historic rainfall led flooding, mudslides and visitor evacuations at which national park?
flooding 'still raging" ...
June 15, 2022
June 13, 2022
Yellowstone still closed flooding
Date: June 17, 2022
Yellowstone National Park closes entrances, evacuates visitors amid 'unprecedented" rainfall...
recedes and thousands evacuate...
Answer: Yellowstone National Park
June 14, 2022
Closed GPT-3: Yosemite National Park
Home swept away as
Open GPT-3: Yellowstone National Park
June 15, 2022
Yellowstone National Park is hit
Dozens evacuated as unprecedented by major floods and mudslides... flooding forces Yellowstone National Park to close...
July 22, 2021
November 2021
Biden says kids under 12 could be eligible for COVID vaccines weeks...
Covid- 19 vaccinations in the US began for
CDC recommends Pfizer COVID- 19 vaccine for kids 5-11, shots expected to roll our this week...
which age group this week?
Date: June 24, 2022
November 10, 2021 COVID-19
Answer: Children under 5
Closed GPT-3: 18 and up
June 23, 2022
the again
in lowa...
Open GPT-3: Children under 5
Covid- 19 vaccinations begin for US children under 5...
November 1, 2021
Everything to know about COVID-19 vaccine and children...
June 12, 2022
Which wildly popular show recently green
Netflix green lights "Squid Game" season 2...
February 4, 2022
lit for season?
The Busch Light Clash goes green this weekend...
Date: June 17, 2022
June 17, 2022 5 things to know for June 17...
Answer: Squid Game
September 26, 2018
Closed GPT-3: The show "Game of Thrones"
June 4, 2019
Dip into 4 new mysteries for fall, including Kate Atkinson's spy novel "Transcription'..
was recently green lit for a new
'Looking Alaska' details revealed for Hulu limited serics...
Open GPT-3: Squid Game
January 10, 2022
The IRS announced it will what this week? Date: June 24, 2022
IRS 2022 tax season set to begin 2 weeks carly on Jan. 24...
February 11, 2022
Answer: Finish processing the backlog of 2021
March 12, 2021
Don't panic you got seary IRS notice...
tax returns
When will get my third stimulus check?...
Closed GPT-3: The IRS announced it will
January 2022
begin processing tax returns this week. Open GPT-3: The IR$ announeed will begin processing 2021 tax returns as soon as
March 22, 2021
IRS will begin processing 2021 tax returns soon Jan. 24
IRS says more stimulus checks the way...
Jan. 24
March 20, 2022
March 2, 2022
Half of US hotels could close amid coronavirus crisis... hotels around the country go bankrupt...
Gun manufacturers are not entirely excmpt from being sued... the now- now-bankrupt gun manufacturer... March 12, 2021
Which country is now "bankrupt," according statement this week from its administration?
September 26, 2013
Date: July 2022
Colo. farmers arrested...
Answer: Sri Lanka
Mitch McConnell seeks to end Democrat's ''ss 'crazy policy' of heefed-up unemployment benefits... states go bankrupt...
now-banke now-bankrupt Jensen Farms...
Closed GPT-3: Greeve
January 2022
Open GPT-3: Venezuela
Trump administration restrictions
asylum cruel... Immigration policy is morally bankrupt...
Which head state announced his resignation
August 11, 2021
this week?
NY Gov. Andrew Cuoma will
March 23, 2021
Date: July 2022
resign in weeks...
Oregon State University President
September 21, 2021 Maricopa County Supervisor Steve Chucri to resign...
King Alexander resigns...
Answer: UK Prime Minister Boris Johnson
August 10, 2021
Closed GPT-3: Japanese Prime Minister
NY Gov. Andrew Cuomo to resign amid scandal...
Shinzo announced resignation
January 25, 2016
this week.
Ball State president Ferguson resigns...
Open GPT-3: Andrew Cuomo

---

Temporal Misalignment and Degradation While not particularly motivated by instantaneous information needs like REALTIME QA. prior work also explored temporal aspects of variety of NLP tasks. A flurry of recent work analyzed performance degradation from temporal misalignment
between (preinraining and evaluation/deploymemn on many NL.P tasks (Lazaridou al. 2021 Röttger
and Pierrehumbert 2021 Luu et al., 2022 Onoe et al. 2022) and proposed mitigation methods (Huang and Paul| 2018 2019 Dhingra et al, 2022, Jang et al. 2022a bj Lee et al. 2022). An
open-book QA model conditions answer generation upon newly-retrieved documents (Lewis et al.
2020b), but the extent to which answer generation can be updated based on the retrieved documents
limited (Longpre et al., 2021b). Temporal degradation is, therefore, one of the challenges that models in REALTIME QA need to address.
Dynamic Benchmarks Unlike the majority of datasets in natural language processing, REALTIME QA evaluates systems dynamically and its evaluations change over time. Several other prior works update challenge test sets (Kiela et al. [222] 1, Potts et al., [021] Ma et al., (021), evaluation tasks (Thrush al. p.22), or metrics (Gehrmann al., [02] 2022 Mishra and Arunkumar 2021 Kasai
al. 2022). ) REALTIME QA hosts similar online platform and adopts dynamic scheme specifically
to pursue instantaneous applications.
Open-Domain QA Much prior work proposed datasets for open-domain QA for English and beyond
(Clark al., 2020 Asai et al. 2021 2022 Longpre al., 202 Zhang al. 2021). Several recent works challenged the conventional problem setups (Chen and Yin (0200) where correct answers can be found from a fixed, external knowledge source, such as Wikipedia. Similar to REALTIME QA, 7Zhan and Choi (2021); Liška et al. (2022) focused on temporal geographical contexts that can change the answer to the same question. Consistent with these prior efforts, ReALTIME QA aims toward broader applications of question answering beyond the conventional framework.
5 Conclusion and Future Work
We introduce REALTIME QA, dynamic, open-domain QA benchmark that asks questions the present time. Our platform announces questions every week and continually evaluates six real-time baselines. Our experiments from the first year suggest that accurate, up-to-date information retrieval is particularly important to serve speedy information needs. We hope that REALTIME QA encourages research efforts toward fast, accurate applications of natural language processing.
Limitations
This work aims to develop a QA benchmark for addressing instantaneous information needs, including emergency management. The current version of REALTIME QA has two major limitations due to annotation framework 22.2) 1) question/answer pairs are all written in English, and the covered topics tend be English-centric (US and UK); questions announced
weekly basis, rather than a truly instantaneous basis. Nevertheless, our benchmark departs from many static datasets from prior work and provides an important step towards the research goal. We hope to develop future versions of REALTIME QA that mitigate these limitations.
Acknowledgements
We thank Noriyuki Kojima, Alisa Liu, Ofir Press, Koji Shiono, Wenya Wang, the ARK group the UW, and the Mosaic team at the Allen Institute for Al for their helpful feedback on this work.
References
James Allan, Rahul Gupta, and Vikas Khandelwal. 2001. Temporal summaries of new topics In Proc. of SIGIR.
James Allan, Ron Papka, and Victor Lavrenko. 1998. On-line new event detection and tracking In Proc. of SIGIR.

---

Jafar Ahmad Abed Alzubi, Rachna Jain, Anubhav Singh, Pritee Parwekar, and Meenu Gupta. 2021. COBERT: COVID-19 question answering system using BERT Arabian Journal for Science and Engineering.
Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proc. of NAACL.
Akari Asai, Shayne Longpre, Jungo Kasai, Chia- Hsuan Lee, Rui Zhang, Junjie Hu, Ikuya Yamada, Jonathan H. Clark, and Eunsol Choi. 2022. MIA 2022 shared task: Evaluating cross-lingual open-retrieval question answering for 16 diverse languages In Proc. of MIA.
Javed A. Aslam, Fernando Diaz, Matthew Ekstrand-Abus Richard McCreadie, Virgil Pavlu, and Tetsuya Sakai. 2015. TREC 2015 temporal summarization track overview In Proc. of TREC.
Javed A. Aslam, Matthew Ekstrand-Abueg, Virgil Pavlu, Fernando Diaz, Richard McCreadie, and Tetsuya Sakai. 2014. TREC 2014 temporal summarization track overview In Pruc. of TREC.
Javed A. Aslam, Matthew Ekstrand-Abueg. Virgil Pavlu, Fernando Diaz, and Tetsuya Sakai. 2013. TREC 2013 temporal summarization In Proc. of TREC.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs In Proc. of EMNLP.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Proc. of NeurlPS.
Cody Buntain, Richard McCreadie, and Ian Soboroff. 2020. time of COVID-19 In Proc. of TREC.
Incident streams 2020: TRECIS in the
Roberta Catizone, Angelo Dalli, and Yorick Wilks. 2006. timelines from the web In Proc. of LREC.
Evaluating automatically generated
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions, In Proc. of ACL.
Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering In Proc. ACL: Tutorial Abstracts.
Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions In Proc. of NeurlPS Datasets and Benchmarks.
Xiuying Chen, Zhangming Chan, Shen Gao, Meng- Hsuan Yu, Dongyan Zhao, and Rui Yan. 2019. Learning towards abstractive timeline summarization) In Proc. of IJCAI.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages) TACL.
Hoa Trang Dang and Karolina Owezarzak. 2008. Overview of the TAC 2008 update summarization task] In Proc. of TAC.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding In Proc. of NAACL.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models temporal knowledge bases] TACL.
David Kirk Evans, Judith I.. Klavans, and Kathleen R. McKeown. 2004. Columbia Newsblaster; Multilingual news summarization the web In Proc. of NAACL: Demonstrations.
10

---

Katja Filippova, Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2009. Company oriented extractive summarization of financial news In Proc. of EACL.
Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Di- panjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur P. Parikh, Laura Perez-Beltrachin Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hen- drik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metries) In Proc. of GEM.
Sebastian Gehrmann, Abhik Bhattacharjee,, Abinaya Mahendiran, Alex Wang. Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoe, Juraj Juraska, Kaustubh D. Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini. Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Stajner, Sébastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. AMahidewuni, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, and Yufang Hou. 2022. GEMv2: Multilingual NLG benchmarking in
single line of code
Demian Gholipour Ghalandari and Georgiana Ifrim. 2020. Examining the state-of-the-ar in news timeline summarization In Proc. of ACL.
Saptarshi Ghosh, Kripabandhu Ghosh, Debasis Ganguly, Tanmoy Chakraborty, Gareth J.F. Jones, and Marie-Francine Moens. 2017. ECIR 2017 workshop on exploitation of social media for emergency
relief and preparedness (SMERP 2017) SIGIR Forum.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval-augmented language model pre-training In Proc. of ICMI..
REALM
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay. Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend In Proc. of NeurlPS.
Xiaolei Huang and Michael J. Paul. 2018 Examining temporality in document classification of ACL.
In Proc.
Xiaolei Huang and Michael Paul. 2019. Neural temporality adaptation for document classification Diachronic word embeddings and domain adaptation models] In Proc. of ACL.
Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015. Processing social media messages in mass emergency; A survey ACM Computing Surveys.
Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013. Practical extraction of disaster-relevan information from social media In Proc. of www.
Muhammad Imran, Prasenjit Mitra, and Carlos Castillo. 2016. Twitter as lifeline: Human-annotated Twitter corpora for NLP of crisis-related messages) In Proc. of LREC.
Gautier Izacard and Edouard Grave 2021. Leveraging passage retrieval with generative models for open domain question answering In Proc. of EACL.

---

Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022a. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Seo. 2022b. Towards continual knowledge learning of language models In Proc. of ICLR.
Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. TempQuestions: A benchmark for temporal question answering. In Companion of the www.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale
distantly supervised challenge dataset for reading comprehension In Proc. of ACL.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering In Proc. of EMNLP.
Jungo Kasai, Keisuke Sakaguchi. Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, and Noah A. Smith. 2022. BBdimenssiona leaderboards: Generate and evaluate language hand in hand In Proc. of NAACL.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger. Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP In Proc. of NAACI.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions; benchmark for question answering research TACL.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations In Proc. of EMNLP.
Angeliki Lazaridou. Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet augmented language models through few-shot prompting for open-domain question answering
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' A'ttume Tomás Kociský, Sebastian Ruder, Dani Yogatama, Kris Cao. Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural language models In Proc. of NeurlPS.
Jinhyuk Lee, Sean S. Yi, Minbyul Jeong, Mujeen Sung, WonJin Yoon, Yonghwa Choi, Miyoung Ko, and Jaewoo Kang. 2020. [Answering questions on COVID-19 in real-time In Proc. of the Ist Workshop NLP for COVID-19 (Pari 2) at EMNLP 2020.
Kyungjae Lee, Wookje Han, Seung-won Hwang, Hwaran Lee, Joonsuk Park, and Sang- Woo Lee. 2022. Plug-and-playy adaptation for continuously-uddatee QA In Findings of the ACL: ACL 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising
to--eeee-------eneeeeeee pre-training for natural language generation, translation, and comprehension, In Proc. of ACL.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. RRetrivval-augeee -aumeetel generation for knowledge-intensive NLP tasks. In Proc. of NeurIPS.
Zhiwei Li, Bin Wang, Mingjing Li, and Wei- Ying Ma. 2005. A probabilistic model for retrospective news event detection In Proc. of SIGIR.
Jimmy Lin, Salman Mohammed, Royal Sequiera, Luchen Tan, Nimesh Ghelani, Mustafa Abualsaud, Richard McCreadie, Dmitrijs Milajevs, and Ellen M. Voorhees. 2017. Overview of the TREC 2017 real-time summarization track In Proc. of TREC.
12

---

Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen M. Voorhees, and Fernando Diaz. 2016. Overview of the TREC 2016 time summarization track In Proc. of TREC.
Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan- McMahon Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A bench-
mark for adaptation to new knowledge over time in question answering models
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021a. MKQA: A linguistically diverse benchmark for multilingual open domain question answering TACL.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021b. Entity-based knowledge conflicts in question answering In Proc. of EMNLP.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022. Time waits for no one! analysis and challenges of temporal misalignment Proc. of NAACL.
Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts. Adina Williams, and Douwe Kiela. 2021. Dynaboard: An evaluation evaluation-as-a-sevvaa as-a-service platform for holistic next-generation benchmarking In Proc. of NeurIPS.
Sebastian Martschat and Katja Markert. 2017. Proc. of EACL.
Improving ROUGE for timeline summarization In
Sebastian Martschat and Katja Markert. 2018. A temporally sensitive submodularity framework for timeline summarization Proc. of CoNL.L.
Richard McCreadie, Cody Buntain, and Ian Soboroff. 2019. TREC incident streams: Finding actionable information on social media In Proc. of ISCRAM.
Kathleen McKeown, Regina Barzilay, John Chen, David Elson, David Evans, Judith Klavans, Ani Nenkova, Barry Schiffman, and Sergey Sigelman. 2003. CCoumbia's newsblaster: New features and future directions In Proc. of NAACL: Demonstrations.
Kathleen R. McKeown, Regina Barzilay, David Evans, Vasileios Hatzivassiloglou, Judith I.. Klavans, Ani Nenkova, Carl Sable, Barry Schiffman, and Sergey Sigelman. 2002. Tracking and summarizing news on a daily basis with Columbia's 's Newsblaster In Proc. of HLT.
Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Knowledge guided text retrieval and reading for open domain question answering
Swaroop Mishra and Anjana Arunkumar. 2021. How robust are model rankings : A leaderboard customization approach for equitable evaluation In Pruc. of AAAI.
Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020. COVID-QA: A question answering dataset for COVID-19. In Proc. of the 1st Workshop on NLP for COVID-19 at ACL 2020.
Dat Tien Nguyen. Kamla Al-Mannai, Shafiq R. Joty, Hassan Sajjad, Muhammad Imran, and Prasenjit Mitra. 2016. Rapid classification of crisis-related data on social networks using convolutional neural networks In Proc. of ICWSM.
Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What LMs know about unseen entities. In Findings of the ACL: NAACL 2022.
Arian Pasquali, Ricardo Campos, Alexandre Ribeiro, Brenda Santana, Alípio Jorge, and Adam Jatowt. 2021. TLS-CovidI9: A new annotated corpus for timeline summarization In Advances in Information Retrieval.
Tatiana Passali, Alexios Gidiotis, Efstathios Chatzikyriakidis, and Grigorios Tsoumakas. 2021. Towards human-centered summarization: A case study on financial news] In Proc. of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing.
Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2021. DynaSent: A dynamic benchmark for sentiment analysis In Proc. of ACL.
13

---

Dragomir R. Radev, Sasha Blair-Goldensse Zhu Zhang, and Revathi Sundara Raghavan. 2001. NewsinEssence: A system for domain- indeeeddennd real-time news clustering and multi-oocument summarization In Proc. of HLT.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified
text-to-text transformer JMLR.
Pranay Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable questions for SQuAD In Proc. of ACL.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of texi. In Proc. of EMNLP.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest:: A challenge
dataset for the open-domain machine comprehension of text In Proc. of EMNLP.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proc. of EMNLP.
Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media In Findings of the ACL: EMNLP 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd schema challenge at scale. In Proc. of AAAI.
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index In Proc. of ACL.
Royal Sequiera, Luchen Tan, and Jimmy Lin. 2018. Overview of the TREC 2018 real-time summa- rization track] In Proc. of TREC.
Lidan Shou, Zhenhua Wang, Ke Chen, and Gang Chen. 2013. Sumblr: Continuous summarization of evolving tweet streams) In Proc. of SIGIR.
Alon Talmor. Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA:: A question answering challenge targeting commonsense knowledge, In Proc. NAACL.
Irina Temnikova, Andrea Varga, and Dogan Biyikli. 2014. Building crisis management term resource for social media: The case of floods and protests In Proc. of LREC.
Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William
Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. 2022. Dynatask: framework for creating dynamic AT benchmark tasks In Proc. of ACL.: System Demonstrations.
Giang Tran, Mohammad Alrifai, and Eelco Herder. 2015. Timeline summarization from relevant headlines In Advances in Information Retrieval.
Giang Binh Tran, Mohammad Alrifai, and Dat Quoc Nguyen. 2013. Predicting relevant news events for timeline summaries) In Proc. of WWW Companion.
Lu Wang, Claire Cardie, and Galen Marchetti. 2015. Socially-informed timeline generation for complex events, In Proc. of NAACL.
Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19; The COVID-19 open research dataset In Proc. of the 1st Workshop on NLP for COVID-19 19 at ACL 2020.
14

---

Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering EMNLP.
Proc. of
René Witte, Ralf Krestel, and Sabine Bergler. 2007. Generating update summaries for DUC 2007 In Proc. of DUC.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. TTranformers: State-of-the-ar) natural language processing In Proc. of EMNLP: System Demonstrations.
Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a. Timeline generation through evolutionary trans-temporal summarization In Proc. of EMNLP.
Rui Yan, Xiaojun Wan, Mirella Lapata, Wayne Xin Zhao, Pu-Jen Cheng, and Xiaoming Li. 2012. Visualizing timelines: Evolutionary summarization via iterative reinforcement between text and image streams
In Proc. of CIKM.
Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 201 1b. Evolutionary timeline summarization: balanced optimization framework via iterative substitution In Proc. of SIGIR.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference In Proc. of EMNLP.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? In Proc. of ACL.
Michael J.Q. Zhang and Eunsol Choi. 2021. QA In Proc. of EMNLP.
SituatedQA: Incorporating extra- lstrustinguistic contexts into
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi- multi-lingual benchmark for dense retrieval In Proc. of MRL..
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments (e.g. for benchmarks)..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
15

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curatinn contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16

---

Appendices
A Baseline Configurations
We provide the configurations for our real-time baselines ($2.4) We use the open-source, Transform- ers library and ensure easy replication of our results. Table lists the configurations for dense passage retrieval (Karpukhin et al. 2020) and serteval-augmentne generation (Lewis et al. 2020b). We gener- ally follow the default settings from the Transformers library. Seen in Table 5 is the configuration for the closed-book baseline. We again generally follow the default setting.
Table 4: Configurations for dense passage retrieval (Karpukhin al. 2020 and retrieval-augmeented generation (Lewis et al. 2020b from the Transformers library (Wolf et al. 2020)
Option
Value
docs
max_combined____ length
300
retrieval _ectorreeerr__eeee _eetor_sizee sizee
768
retrieval_batee batch_sizz is encoder_decode prefix
8
True
None
bos token id
None
pad_token id
None
cos_token_
None
decoder_staie siaeeee_sttttttttteeeeeee tokettttoke_aat id
None
title_sep
'I
doe_sep
dataset
'wiki_dpr'
datasct_split
'train'
index
"compressed'
index path passages_paad
None
None
use_dummy_dattse
False
reduce_loss label_smoode label_smoothing do deduplication exclude bos score do marginalize output retrieved use cache forced cos token id
Falsc
0.0
True
False
False
False
True
None
B REALTIME QA Interface
Fig. shows our REALTIME QA interface. gets updated every week, and all six baselines are evaluated as soon as the questions are available. Submissions will be shown on the same page, together with their submission time.
C REALTIME QA Statistics
Table bableidprovide more detailed statistics of REALTIME QA from the first four weeks. We analyze the questions from the first four weeks along genres and answer types. We also found that ~10% of the questions were not strictly time-sensitive These questions include, for example, "Temperatures in Britain are set to soar this weekend, but what is the hottest UK temperature on record?" from June 17, 2022. We do not filter out these cases to simulate information-seeking naturally-occurric scenarios.
17

---

Table 5: Configuration for the closed-book T5 baseline (Raffel al. 2020) from the Transformer library.
Option
Value
name or pr_pathh
architectures d_f
("ISFaCConditionn@Ceeraiii
65536
d_vv
128
d model
1024
pocoder_start_token
O
dropout_rate
0.1
cos_token_id id
feed forward_ proj"
initializer factor
1.0
is becoder_decodde
True
layer_norm_epsilon
1c-06
model_type
num gam__dcoder_layen
24
num num_heads num layerayees
128
24
output_pan past pad_ token id
True
relative attention num buckets
32
tokenizer class
TSTokenizer
vocab_size
32128
REALTIME GA
Rocult::
2003 2002/16
Results January 13, 2023
Multiple Choice Track
2013/01/13
Model
Submission Time
Original
NOTA
2023/01/86
GPT-3 + Google Custom Search
2023-01-14 05:00:00
63.3
58.7
2022
2023
Papera
RAD + Google Custom Searth
2023 01-14 05:00:00
46.7
30.0
T5 (Closes-Botd
2023-01-18 05:00100
33.3
333
Generacion
Submlesion Time (GMT)
GPI-33 Google Custom Search
2023--11-16 035-32100
3124
Geogle Curdom
3073-01-18 COUICUOO
303
(Giosedi
2023-01-14 03:00:00
6P7-3 CboseciBeok)
2073-01-14 05:00:00
3.4
88
DFR
2023-01-14 03-00.00
34
RAG DPR
3023-01-14 05:00000
0.0
100
Figure 7: REALTIME QA interface. is updated every week, and all baselines are evaluated as soon the questions are available. Submissions will be shown on the same page, together with their submission time.
18

---

Table 6: Detailed statistics (%) of REALTIME QA. We analyze the questions from the first four weeks along genres and answer types. also found that -12% of the questions were not strictly time-sensitive. These questions include, for example, "Temperatures in Britain set to soar this weekend, but what the hottest UK temperature on record?" from June 17, 2022. We do not filter out these cases to simulate information-seeking naturally-occurring scenarios.
Genre
Politics
Business
Entertain
Science
Technology
Health
Disaster
36.9%
17.5%
17.5%
7.0%
7.0%
8.8%
5.2%
Answer Type
Person
Location
Time
Number
Organization
Event
Miscellancous
12.3%
19.2%
5.3%
22.8%
3.5%
8.8%
28.1%
19


Reviews:
[{"review_id": "YNFU7iQmxA", "reviewer": "Reviewer_SxTx", "summary": "", "questions": "", "limitations": "The details of choice extraction can be more clear in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The evalution of realtime question answering is important and this paper introduces a dynamic platform that announces questions and evaluates systems every week. Based on the collected questions, performance over varying submission time are also analysized.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jBHmAYXxNn", "reviewer": "Reviewer_nqj1", "summary": "", "questions": "", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This paper proposes initial efforts in resolving real-time QA with time-sensitive questions and provides a dynamic platform for future developments in this direction.\n\n- The authors provide a collection of time-sensitive QA pairs and the latest retrieved source documents from June 17, 2022, to June 2, 2023. Moreover, ~120 QA pairs/month on upcoming time. This can be useful in developing many real-time QA systems.\n\n- Here, a thorough evaluation of six baselines from open-book and close-book QA is conducted and results indicate that the GPT-3 + GCS (open-book system) has better real-time QA capabilities due to access to the most recent articles.\n\n- This work can pave the way for developing more real-world QA systems for many applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0U9z26FqWD", "reviewer": "Reviewer_4qFr", "summary": "", "questions": "", "limitations": "The biggest limitation of this paper is that it appears to lack novel discoveries for readers. Although real-time QA is a valuable topic, the difference between real-time QA and traditional document-grounded QA lies only in their data sources. As a result, the authors were unable to draw conclusions that differ significantly from previous works. In the final error analysis, the majority of errors originated from retrieval mistakes, which aligns with the conclusions of prior research.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "How QA systems utilize retrieval tools to answer real-time questions is an important research question and a highly valuable application scenario. REALTIME QA has developed an authentic evaluation system and conducted three years of experiments to assess the performance of current state-of-the-art models, which is a robust endeavor. The paper is well-written, presenting a clear overall structure, and all the figures and tables are highly intuitive.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9JjPIq9Ojv", "reviewer": "Reviewer_bnpu", "summary": "", "questions": "", "limitations": "The authors include discussion of limitations in the paper.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper introduces a practical proposal for evaluating QA systems on *unseen* information, given that new questions pertaining to current events are used. Given recent concerns over LLMs testing on the training data, this proposal is timely and important.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0rGgXqLr5k", "reviewer": "Reviewer_CU6H", "summary": "", "questions": "", "limitations": "Please see W2. I think this limitation is important to consider, but I think the dataset itself has enough merits in spite of this limitation.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "S1: Evaluating NLP systems against ever-changing information is a critical requirement for knowledge intensive tasks such as fact checking or question answering.\n\nS2: Updates are provided at a weekly frequency which balances comparison of long-term effects and short term rapidly changing information. \n\nS3: The study is well positioned with ongoing research about model performance degradation and studying models with dynamic benchmarking.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a platform for real-time question answering evaluation. Unlike previous approaches that treat question answering as a static task, this platform provides a dynamic dataset that updates weekly. This platform can facilitate the development of online QA systems that can handle time-sensitive questions. Although, there are a few concerns: 1) the challenges of model evaluation; 2) the absence of commercial system benchmarks such as Microsoft Bing and Google BARD. Overall, the paper makes a valuable contribution to the field of real-time question answering.
Decision: Accept (Poster)