Title: When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models
Benchmark: FLUB
Summary: FaLlacy Understanding Benchmark (FLUB) contains multiple choice, classification, and explanation questions about "cunning texts." These are snippets from posts on a Chinese online forum, which human annotators filter and then annotate with multiple choice questions, a "cunning type" classification, and an explanation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: fallacy understanding
Phenomenon defined by authors: Yes
Phenomenon definition: "whether LLMs can understand cunning texts that may contain misleading, wrong premise, intentional ambiguity, and so forth" (1)
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The model is given a snippet of "cunning text" from the Chinese online forum Ruozhiba and told to either select an option answering the question or otherwise explain the fallacy at play in the text.
Task item definition: A text snippet might be a riddle like "Which one weighs more, a ton of iron or a ton of cotton?", with the correct multiple choice answer being ""A ton of iron" and "a ton of cotton" both weigh one ton and are the same weight."
Task definition detail: These are not fallacies. They're more like riddles and wordplay.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 834
Dataset metadata available: Yes
Dataset metadata detail: cunning type
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: Using LLM-as-a-judge to assess LLMs' abilities to comprehend puzzles does not pass face validity, especially when the scores are only correlated 0.57 with human judgements.

Also, for LLM-as-a-judge, the inter-rater agreement with human evaluators is not valid as humans rate on a different scale (1-5) from GPT-4 (1-10).
Metric aggregation: Geometric mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Selection, classification, and explanation scores
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: No
Metric face validity: No
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "our data come entirely from the real world and are all carefully constructed by netizens" (4)
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Other notes: Construct validity is extremely off for this paper, I suspect for language barrier reasons