Title: FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models
Benchmark: FRoG
Summary: The benchmark aims to measure the ability of LLMs to understand “general quantifiers”, words like “all”, “most”, “some”. The take GSM8K+MathQA questions and mask out occurrences of “x%”. Given the masked question and true answer the model has to decide which quantifier (out of a multiple choice selection) best entails the answer
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Fuzzy reasoning
Phenomenon defined by authors: No
Phenomenon definition: We aim to explore reasoning challenges associated with fuzzy events, which are mathematically ambiguous and articulated through fuzzy expressions of GQs, such as “most in the price of gas”. Specifically we focus on mathematical reasoning problems that incorporate GQs.
Definition scope: Subset
Definition integrity: Authors' description is unclear
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: GQs widely accepted. I think consensus is that fuzzy reasoning incorporates more than just GQs.
Task definition: They take GSM8K+MathQA questions and mask out occurrences of “x%”. Given the masked question and true answer the model has to decide which quantifier (out of a multiple choice selection) best entails the answer
Task item definition: Masked question: “There has been successive increases of 20% and then <masked percentage value> in the price of gas from the previous month. By what percentage should a drive reduce gas consumption so that the expenditure does not change?” 
Original answer: 24%
Choices for GQ: A. none B. small amount. C. few D. tiny amount
Task definition detail: The example is the best they give but is a bit confusing. The model should identify that C. few is the quantifier that would best replace the <masked percentage word> to give the original answer.
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: Sourced from GSM8K, MathQA 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2044
Dataset metadata available: Yes
Dataset metadata detail: There are a variety of difficulty levels based on how similar the multiple choice values provided are. 
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Variety of difficulty levels, created by adjusting how similar the multiple choices given are
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: No
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Author validity detail: Claim an "inverse scaling law" but this is not clearly present in results
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning