Title: I Could’ve Asked That: Reformulating Unanswerable Questions
Benchmark: CouldAsk
Summary: The paper introduces CouldAsk, a document‑grounded QA benchmark that first asks a model to detect when a user’s question is unanswerable from a given document and then reformulate that question so it becomes answerable while staying relevant to the user’s intent. COULDASK pools 6 sub‑datasets (3 existing Wikipedia‑based sets and 3 new GPT‑4‑generated–then–human‑verified sets from news, Reddit, and Yelp) and proposes reference‑free automatic metrics to score both the detection (F1) and reformulation (“success rate”) stages, revealing that today’s best LLMs still succeed less than 30 % of the time.
Contribution: - New task formulation: joint detection + reformulation of presupposition‑error questions.
- Broad, multi‑domain benchmark: Wiki (SQuADv2, QA2, BanditQA) plus BBC News, Reddit, Yelp.
- Reference‑free evaluation using an answerability classifier and entity‑overlap relevance, validated against human judgements (κ ≈ 0.94).
- Detailed error and span‑type analyses; public release of data, code, and answerability classifier.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Unanswerable‑question detection & reformulation
Phenomenon defined by authors: Yes
Phenomenon definition: The paper defines the phenomenon as the ability to detect when a user’s question is unanswerable based on a document and then reformulate it into a relevant and answerable question grounded in that same document. Specifically:
“Given a document and a user question, the system must determine if the question is unanswerable. Upon identifying the unanswerable question, it must reformulate the question such that the new question is answerable by the document while remaining relevant to the original question.” 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a document and a user question, decide if the question is unanswerable; if so, output a minimally edited, document‑answerable version that remains relevant to the user’s query.
Task item definition: A single item consists of a natural language question paired with a supporting document. The model must first determine whether the question is answerable based on the document and, if it is unanswerable, generate a minimally edited, document-answerable reformulation that remains relevant to the original query.
Task definition detail: Two subtasks; evaluation only proceeds to reformulation if the model flags the question as unanswerable.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Existing Wikipedia‑based datasets are adapted, while new BBC/Reddit/Yelp questions are generated by GPT‑4, filtered to confuse an automated checker, and then annotated by three MTurk workers (majority‑vote).
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 4,332
Dataset metadata available: Yes
Dataset metadata detail: domain label, answerable flag, entities list, document/question lengths.
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Simple mean is used for aggregation.
Metric subscores: Yes
Metric subscores detail: Per sub-dataset and domain
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors validate their automatic relevance metric by comparing it to human judgements on 200 question pairs, finding near‑perfect agreement (Fleiss κ = 0.94), and they report 95 % accuracy for their answerability classifier on a held‑out set, supporting construct validity of the “success rate” metric.
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding