Title: Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
Benchmark: ConsisEval
Summary: The paper introduces ConsisEval, a benchmark to evaluate the "hard-to-easy" inconsistency in large language models (LLMs), i.e., when they can correctly solve harder problems but fail on easier counterparts. The benchmark consists in pairs of questions with a strict order of difficulty, generated by an LLM and annotated by humans. They propose an evaluation metric rooted in probability theory.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications), Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: consistency in problem solving
Phenomenon defined by authors: Yes
Phenomenon definition: "A counter-intuitive phenomenon occurs when an LLM, which can solve a harder problem, surprisingly goes wrong on an easier problem."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Solve two problems, one strictly harder than the other. The problems include code, math, and instruction-following.
Task item definition: Pair of questions with a strict order of difficulty.
Task source: Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Easy data comes from GSM8K (math), HumanEval (code) and IFEval (instruction-following). The corresponding harder questions are LLM-generated (GPT-4) and reviewed by humans.
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1,464 questions in total: 298 math + 164 code + 270 instruction-following = 732 pairs
Dataset metadata available: Yes
Dataset metadata detail: difficulty level (easy, hard) and topic (math, code, instruction-following)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation), Consistency Score, Relative Consistency Score
Metric definition detail: They create 2 new metrics: "We introduce a metric termed Consistency Score (CS), which is the conditional probability of a model correctly answering easy questions given that it has correctly answered harder ones. Based on these bounds [upper and lower bounds of CS], we propose Relative Consistency Score (RCS) to indicate the potential for improvement in consistency, and low RCS can reveal high potential for improvement in CS."
Metric aggregation: Weighted Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: By domain (math, code, instruction-following)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes, implicitly
Author validity detail: - Face: The authors motivate the benchmark with an intuitive example: “Unlike LLMs, humans are naturally consistent reasoners, and it is undisputed that an individual proficient in calculus can easily address simpler arithmetic problems.”
- Predictive/Concurrent: "The higher CS indicates the lower probability humans encounter inconsistency phenomena when using LLMs, so CS is almost equal to human perceptions of model consistency.", "We also observe a strong correlation between capability and consistency of LLMs".
- Content: All hard questions are derived from easy ones, with a strict difficulty hierarchy.  Moreover, the benchmark spans multiple domains (code, math, instruction-following), for coverage. The authors detail data generation and human verification process to ensure quality.
- Convergent/Discriminant: no comparison with other benchmarks
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability