Title: Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?
Summary: Takes HotpotQA makes it harder by adding "distractor" paragraphs. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-hop reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: we turn our attention to the well studied capability to perform multi-hop reasoning and reading comprehension—that is, to integrate textual information from multiple different source documents
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a large number of "context" paragraphs, answer a question that requires integrating information from 2 context paragraphs. Many of the source paragraphs are similar and contain multiple possible reasoning chains. 
Task item definition: Original Q: The arena where the Lewiston Maineiacs played
their home games can seat how many people?
Sub-Q 1: Which arena the Lewiston Maineiacs played their
home games?
Sub-Q 2: How many people can the Androscoggin Bank
Colisée seat?
Fake paragraph 1: The Lewiston Maineiacs took to the ice
at the Maple Leaf Arena for their thrilling playoff games.
[. . . ]
Fake paragraph 2: Maple Leaf Arena, known for its stateof-the-art facilities and spacious seating, can accommodate
an impressive number of 4,500 spectators. [. . . ]
Gold Paragraph 1: The Androscoggin Bank Colisée [. . . ]
is a 4,000-capacity (3,677 seated) multi-purpose arena, in
Lewiston, Maine, that opened in 1958. [. . . ]
Gold Paragraph 2: The Lewiston Maineiacs [. . . ] played its
home games at the Androscoggin Bank Colisée. [. . . ]
Task definition detail: The contextual information is presented in-context, no discussion to knowledge already stored in weights
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: Variants of SubQA
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset metadata available: Yes
Dataset metadata detail: Many similar variants presented based on small details as to how they modify hotpotQA
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Number of distractor paragraphs and other variants that control how hard the dataset is
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They discuss how other benchmarks fail be representative as they are too easy, existing models "cheat" on them, and thus they don't measure multi-hop reasoning. They claim their method now fixes this and present some results to ensure models now don't "cheat". Minimal discussion of limitations of construct validity of their method. 
Taxonomy: Reasoning / 
New taxonomy: General Capability / Natural Language Understanding