Title: RealTime QA: What's the Answer Right Now?
Benchmark: RealTime QA
Summary: This paper introduces REALTIME QA, a dynamic question answering platform that evaluates systems' ability to answer questions about the current world. New questions requiring up-to-date information are released weekly. The paper presents the platform and evaluates strong baselines built on large language models (like GPT-3 and T5) combined with information retrieval (web search, DPR). Results highlight the importance of timely retrieval but also show models may provide outdated answers when retrieval is insufficient.
Contribution: Key contributions include: (1) Proposing REALTIME QA, a novel dynamic benchmark for evaluating QA systems on their ability to use real-time information. (2) Establishing a regular (weekly) cycle for question release and evaluation. (3) Providing strong baseline results using LLMs augmented with different information retrieval techniques. (4) Analyzing the performance and failure modes of current systems on timely QA. (5) Making the platform and results publicly available.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Real-time question answering; Reasoning with up-to-date information; Temporal awareness in QA systems.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability of a QA system to correctly answer questions about novel events or rapidly changing information necessitates access to and processing of the most current information available, unlike systems relying solely on static knowledge snapshots.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: To overcome the limitations of static QA datasets and drive research towards systems capable of handling continuously evolving world knowledge and providing timely answers.
Task definition: Given a natural language question released at a specific time, whose answer depends on the current state of the world, provide the correct, up-to-date answer. This typically requires querying external, real-time information sources.
Task item definition: A question released weekly via the REALTIME QA platform, requiring a factual answer reflecting the world state at that time. The platform manages questions and evaluates submitted answers.
Task definition detail: Questions are manually generated by the benchmark organizers to specifically require timely information. They cover diverse topics and can be short-answer or yes/no. The benchmark is ongoing and dynamic.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: The authors manually create new questions each week designed to require knowledge of recent events or information that frequently changes.
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: The dynamic nature and focus on current events make it highly representative of real-time information needs.
Dataset size: Dynamic/Ongoing. The dataset grows each week. The paper reports on results gathered over a year.
Dataset metadata available: Yes
Dataset metadata detail: Question Release Timestamp, Question Type (Short-Answer/YesNo), Answer Type (Person, Org, Loc, Date, Num, Other), Required Timeliness category.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Response format detail: Answers are expected to be concise factual strings or "Yes" / "No".
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Exact Match (EM) and F1 score.
Metric aggregation: Simple Mean
Metric statistics: Exact Match (EM), F1 Score (%)
Metric subscores: Yes
Metric subscores detail: Performance analyzed by question type, answer type, required timeliness category, and baseline system configuration (retrieval method, base model)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The benchmark's dynamic, ongoing nature is its core validity claim for measuring real-time QA ability. Questions are manually created to ensure they test timely knowledge. Performance analysis based on timeliness requirements further supports its construct validity.
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating
Procedural notes: Benchmark operates on a weekly cycle. Baselines use GPT-3 (text-davinci-002), T5-11B, DPR, and Google Custom Search API. Details baseline configurations. Evaluation interface shown. 
Other notes: REALTIME QA's key innovation is its dynamic evaluation framework, moving beyond static datasets to continuously assess performance on questions requiring current knowledge. It highlights the challenges models face in staying up-to-date and avoiding reliance on potentially outdated parametric memory.