Title: BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Benchmark: BRAINTEASER
Summary: The paper introduces BRAINTEASER, a multiple‑choice benchmark that probes large language models’ ability for lateral thinking, a creative, non‑linear reasoning that overrides default commonsense associations. It describes a three‑step construction pipeline (web crawling and filtering of puzzles, semi‑automatic distractor generation, and semantic + contextual reconstructions) that yields high‑quality items while controlling for memorization. 
Contribution: First publicly-‑available benchmark for lateral‑thinking evaluation,
Detailed error analysis regarding memorisation & commonsense traps.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: (Lateral/creativee) reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: The paper targets lateral reasoning, defined as a creative problem-solving approach that differs from vertical thinking (commonsense association and inference). It requires solving puzzles that cannot be resolved through straightforward commonsense associations alone, demanding non-linear reasoning patterns.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: - Although authors note lateral thinking comprises four skills (awareness, random stimulation, alternatives, alteration), they do not operationalize them separately; sub‑elements are not separately measured.
- Authors explicitly contrast lateral vs vertical thinking and position benchmark as complementary to commonsense QA suites.
Task definition: Multiple-choice QA where the model selects the correct explanation to a brain‑teaser puzzle among 4 options (one may be “None of the above”).
Task item definition: One puzzle = question stem + 4 answer choices
Task definition detail: Sentence/word puzzles -- each puzzle has 2 reconstruction variants (semantic/context) to resist memorization.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: 373 core puzzles crawled from riddles.com, rd.com, etc., 
distractors via COMET (commonsense pre-trained language models),
reconstruction prompts via GPT‑4
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The puzzles are artificial and deliberately engineered, so they are unlikely to appear in day‑to‑day user interactions. Rather, they approximate the kind of creative‑reasoning challenges that could arise across diverse downstream tasks. Thus ecological validity seems to be low, serving research evaluation rather than direct operational use.
Dataset size: 1,119 puzzles
Dataset size detail: 627 sentence, 492 word; originals + reconstructions
Dataset size notes: Core 373 originals expanded 3 times, with reconstructions.
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Group accuracy = 1 only if model answers all three variants correctly.
Metric aggregation: Simple Mean
Metric statistics: Simple proportion, human/model comparisons
Metric subscores: Yes
Metric subscores detail: Separate scores for Sentence/ Word and for Original/Semantic/Context splits.
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: - The human annotators rated 99% of the original QA pairs valid, and 97%-100% of the semantic/context reconstructions as consistent with the original QA pairs. 
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning