Title: Large Language Models Can Be Easily Distracted by Irrelevant Context
Benchmark: GSM‑IC 
Summary: The paper introduces GSM-IC, a variant of the GSM8K arithmetic reasoning dataset that includes irrelevant sentences to test large language models' distractibility. The authors evaluate several prompting strategies on LLMs, revealing significant performance drops when irrelevant information is present, and explore mitigation strategies including self-consistency decoding, exemplar design, and explicit instructions that partially restore performance.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: reasoning, distractibility (irrelevant context QA)
Phenomenon defined by authors: Yes
Phenomenon definition: Filtering out irrelevant information is essential for handling real-world tasks. Our evaluation indicates that despite the strong performance on challenging reasoning problems, state-of-the-art language models still have fundamental weaknesses in context understanding and identifying the relevant information from the input. Our findings suggest that in order to gain a more holistic understanding of the reasoning capability of language models, future work should also consider the model sensitivity to irrelevant context, in addition to solving more challenging problems. (pg 2.)
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a maths word problem containing one irrelevant sentence, models are expected to output the numeric answer.
Task item definition: Each item = (Problem and Answer). The problems are derived from GSM8K  and distractor sentences.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: First, derive from existing benchmark (GSM8K), 
Then, create distractors via templates.
Finally, manually verify grammaticality and answer invariance.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Because the distractor sentences are from controlled templates, the new benchmark may nt be realistic due to the superfluous information built from synthetic perturbations.
Dataset size: 4000 problems
Dataset size detail: 58052 problems total but 4000 instances randomly drawn from the full set as an evaluation subset for cheaper computational cost.
Dataset size notes: 100 problems for validation.
Dataset metadata available: Yes
Dataset metadata detail: Reasoning steps, distractor categories (topic/ overlap/ number range)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Response format detail: Answers are expected to be exact integers; authors mark incorrect if formatting differs.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Percentage, comparison
Metric subscores: Yes
Metric subscores detail: Accuracies by reasoning step or by distractor category
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Validty factors (topic, overlap, number rage) and the discussion for construct limitations show that the task isolates the distractability (irrelevant context), which authors address as the current limitation of the reasoning.
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning