SlowGuess/NeurIPS_2023_OCR huggingface dataset:
to recent articles, $^{10}$  but there are potential failure modes. Firstly, some questions in REALTIME QA inquire about the past, and models can benefit from older articles when answering such questions. Further, the appropriate date range for retrieval varies from question to question in real-world applications; some questions inquire about this year, while others about this week. We thus do not implement such filtering for the current real-time baselines.
REALTIME QA has time sensitivity, which several prior works addressed on various NLP tasks. Here we discuss its relation to long-standing summarization and text retrieval tasks, as well as recent work on temporal misalignment between training and evaluation. We then discuss its connections to dynamic evaluations and open-domain QA.
Summarization/Retrieval over Time Temporal (or timeline) summarization is a task that retrieves documents from the web and provides their summary over time (Catizone et al., 2006; Aslam et al., 2013, 2014, 2015; Martschat and Markert, 2017, 2018). Update summarization (Witte et al., 2007; Dang and Owczarzak, 2008) and new event detection/track (Allan et al., 1998; Li et al., 2005) are tasks that monitor and track newly-added information. Prior work created datasets and systems for these tasks (Tran et al., 2013, 2015; Wang et al., 2015; Chen et al., 2019; Gholipour Ghalandari and Ifrim, 2020). Their evaluations are usually executed statically, with information available at the time of data collection.
In contrast, the TREC real-time summarization track evaluates systems in real time during a 1-2 week evaluation period (Lin et al., 2016, 2017; Sequiera et al., 2018). Several other works and initiatives focused particularly on financial news summarization (Filippova et al., 2009; Passali et al., 2021) or emergency management technology (Temnikova et al., 2014; Ghosh et al., 2017; McCreadie et al., 2019), including the COVID-19 pandemic (Buntain et al., 2020; Pasquali et al., 2021). This work regularly evaluates question answering systems over diverse topics, but we share the goal of dealing with novel and evolving information over time; retrieval or summarization methods from these tasks (e.g., Yan et al., 2011a,b, 2012; Shou et al., 2013) can be combined with models in REALTIME QA to serve various time-sensitive information needs from users. REALTIME QA can also be used to evaluate time-sensitive retrieval systems by the downstream QA performance.
10Indeed, GCS has a paid version with a date range feature that filters retrieval results by date.
Temporal Misalignment and Degradation While not particularly motivated by instantaneous information needs like REALTIME QA, prior work also explored temporal aspects of a variety of NLP tasks. A flurry of recent work analyzed performance degradation from temporal misalignment between (pre)training and evaluation/deployment on many NLP tasks (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Luu et al., 2022; Onoe et al., 2022) and proposed mitigation methods (Huang and Paul, 2018, 2019; Dhingra et al., 2022; Jang et al., 2022a,b; Lee et al., 2022). An open-book QA model conditions answer generation upon newly-retrieved documents (Lewis et al., 2020b), but the extent to which answer generation can be updated based on the retrieved documents is limited (Longpre et al., 2021b). Temporal degradation is, therefore, one of the challenges that models in REALTIME QA need to address.
Dynamic Benchmarks Unlike the majority of datasets in natural language processing, REALTIME QA evaluates systems dynamically and its evaluations change over time. Several other prior works update challenge test sets (Kiela et al., 2021; Potts et al., 2021; Ma et al., 2021), evaluation tasks (Thrush et al., 2022), or metrics (Gehrmann et al., 2021, 2022; Mishra and Arunkumar, 2021; Kasai et al., 2022). REALTIME QA hosts a similar online platform and adopts a dynamic scheme specifically to pursue instantaneous applications.
Open-Domain QA Much prior work proposed datasets for open-domain QA for English and beyond (Clark et al., 2020; Asai et al., 2021, 2022; Longpre et al., 2021a; Zhang et al., 2021). Several recent works challenged the conventional problem setups (Chen and Yih, 2020) where correct answers can be found from a fixed, external knowledge source, such as Wikipedia. Similar to REALTIME QA, Zhang and Choi (2021); Liska et al. (2022) focused on temporal or geographical contexts that can change the answer to the same question. Consistent with these prior efforts, REALTIME QA aims toward broader applications of question answering beyond the conventional framework.
We introduce REALTIME QA, a dynamic, open-domain QA benchmark that asks questions at the present time. Our platform announces questions every week and continually evaluates six real-time baselines. Our experiments from the first year suggest that accurate, up-to-date information retrieval is particularly important to serve speedy information needs. We hope that REALTIME QA encourages research efforts toward fast, accurate applications of natural language processing.
This work aims to develop a QA benchmark for addressing instantaneous information needs, including emergency management. The current version of REALTIME QA has two major limitations due to our annotation framework (§2.2): 1) question/answer pairs are all written in English, and the covered topics tend to be English-centric (US and UK); 2) questions are announced on a weekly basis, rather than a truly instantaneous basis. Nevertheless, our benchmark departs from many static datasets from prior work and provides an important step towards the research goal. We hope to develop future versions of REALTIME QA that mitigate these limitations.
We thank Noriyuki Kojima, Alisa Liu, Ofir Press, Koji Shiono, Wenya Wang, the ARK group at the UW, and the Mosaic team at the Allen Institute for AI for their helpful feedback on this work.
James Allan, Rahul Gupta, and Vikas Khandelwal. 2001. Temporal summaries of new topics. In Proc. of SIGIR.
James Allan, Ron Papka, and Victor Lavrenko. 1998. On-line new event detection and tracking. In Proc. of SIGIR.
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In Proc. of EMNLP.
Réné Witte, Ralf Krestel, and Sabine Bergler. 2007. Generating update summaries for DUC 2007. In Proc. of DUC.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proc. of EMNLP: System Demonstrations.
Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a. Timeline generation through evolutionary trans-temporal summarization. In Proc. of EMNLP.
Rui Yan, Xiaojun Wan, Mirella Lapata, Wayne Xin Zhao, Pu-Jen Cheng, and Xiaoming Li. 2012. Visualizing timelines: Evolutionary summarization via iterative reinforcement between text and image streams. In Proc. of CIKM.
Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011b. Evolutionary timeline summarization: A balanced optimization framework via iterative substitution. In Proc. of SIGIR.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proc. of EMNLP.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proc. of ACL.
Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proc. of EMNLP.
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proc. of MRL.
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
A Baseline Configurations
We provide the configurations for our real-time baselines (§2.4). We use the open-source, Transformers library and ensure easy replication of our results. Table 4 lists the configurations for dense passage retrieval (Karpukhin et al., 2020) and retrieval-augmented generation (Lewis et al., 2020b). We generally follow the default settings from the Transformers library. Seen in Table 5 is the configuration for the closed-book T5 baseline. We again generally follow the default setting.
Table 4: Configurations for dense passage retrieval (Karpukhin et al., 2020) and retrieval-augmented generation (Lewis et al., 2020b) from the Transformers library (Wolf et al., 2020).
Option	Value
n_docs	5
max(combined_length	300
retrieval_vector_size	768
retrieval_batch_size	8
isEncoder Decoder	True
prefix	None
bos_token_id	None
pad_token_id	None
eos_token_id	None
decoder_start_token_id	None
title sep	'/'
doc sep	'/'
dataset	‘wiki_dpr’
dataset_split	‘train’
index_name	‘compressed’
index_path	None
passages_path	None
use_dummy_dataset	False
reduce_loss	False
label_smoothing	0.0
do_dedduplication	True
exclude_bos_score	False
do_marginalize	False
output_retrieved	False
use_cache	True
forced_eos_token_id	None
B REALTIME QA Interface
Fig. 7 shows our REALTIME QA interface. It gets updated every week, and all six baselines are evaluated as soon as the questions are available. Submissions will be shown on the same page, together with their submission time.
C REALTIME QA Statistics
Table 6 provide more detailed statistics of REALTIME QA from the first four weeks. We analyze the questions from the first four weeks along genres and answer types. We also found that  $\sim 10\%$  of the questions were not strictly time-sensitive. These questions include, for example, "Temperatures in Britain are set to soar this weekend, but what is the hottest UK temperature on record?" from June 17, 2022. We do not filter out these cases to simulate information-seeking, naturally-occurring scenarios.
Table 5: Configuration for the closed-book T5 baseline (Raffel et al., 2020) from the Transformer library.
Option	Value
_name_or_path	/home/patrick/t5/t5-11b-ssm-nq
architectures	[ "T5ForConditionalGeneration"]
d_f	65536
d_kv	128
d_model	1024
decoder_start_token_id	0
dropout_rate	0.1
eos_token_id	1
feed_forwardProj"	relu
initializer_factor	1.0
is EncoderDecoder	True
layer_norm_epsilon	1e-06
model_type	t5
num_decoder_layers	24
num_heads	128
num_layers	24
outputpast	True
pad_token_id	0
relative attentio_num_buckets	32
tokenizer_class	T5Tokenizer
vocab_size	32128
Figure 7: REALTIME QA interface. It is updated every week, and all six baselines are evaluated as soon as the questions are available. Submissions will be shown on the same page, together with their submission time.
Table 6: Detailed statistics (\%) of REALTIME QA. We analyze the questions from the first four weeks along genres and answer types. We also found that  $\sim 12\%$  of the questions were not strictly time-sensitive. These questions include, for example, "Temperatures in Britain are set to soar this weekend, but what is the hottest UK temperature on record?" from June 17, 2022. We do not filter out these cases to simulate information-seeking, naturally-occurring scenarios.
Genre
Politics	Business	Entertain	Science	Technology	Health	Disaster
36.9%	17.5%	17.5%	7.0%	7.0%	8.8%	5.2%
Answer Type
Person	Location	Time	Number	Organization	Event	Miscellaneous
12.3%	19.2%	5.3%	22.8%	3.5%	8.8%	28.1%