Title: Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
Benchmark: Easy2Hard-Bench
Summary: Easy2Hard-Bench is a benchmark comprising six datasets across diverse domains like mathematics, programming, chess puzzles, and reasoning. Its main contribution is providing datasets with fine-grained, continuous numerical difficulty scores for each problem. The difficulty scores are systematically estimated using established difficulty ranking systems such as Item Response Theory (IRT) and Glicko-2 models, leveraging abundant performance data from humans and LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: reasoning, logic, code generation, natural language understanding, knowledge, instruction following, flexibility, data analysis
Phenomenon defined by authors: Yes
Phenomenon definition: "the ability to generalize across a broad spectrum of tasks, ranging from basic to complex problem-solving scenarios" (page 1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task involves LLMs solving problems across various domains (mathematics, programming, chess puzzles, and reasoning) that have been annotated with continuous difficulty scores.  The performance of LLMs is then evaluated and analyzed across these varying difficulty levels.
Task item definition: A single item is a problem (e.g., maths word problem, chess puzzle, programming task) with a known solution and a difficulty score
Task definition detail: Problems are filtered and standardised across domains. Difficulty is assigned using human and LLM performance data via IRT or Glicko-2 models
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Problems are drawn from real exams/competitions or widely-used benchmarks and adapted into a consistent format.
Dataset size: E2H-AMC: 2,975; E2H-Codeforces: 4,000; E2H-Lichess: 5,000; E2H-GSM8K: NA (not explicitly separated; dataset has 1,319 items total); E2H-ARC: NA (1,172 items total); E2H-Winogrande: NA (1,267 items total)
Dataset size detail: Existing datasets (GSM8K, ARC, Winogrande) are treated entirely as test/evaluation sets.
Newly curated datasets (AMC, Codeforces, Lichess) have both training and evaluation splits.
Distribution of difficulty is heavily skewed toward medium/hard items to better challenge LLMs​
Dataset size notes: E2H-AMC: Train: 1,000; E2H-Codeforces: Train: 3,663; E2H-Lichess: Train: 71,763; E2H-GSM8K, ARC, Winogrande: Not split (used as evaluation only)
Dataset metadata available: Yes
Dataset metadata detail: numerical difficulty scores (rating), difficulty uncertainty (rating_std), difficulty quantile, and various categorical tags. These tags cover areas such as competition origin (e.g., AMC8, HMMT-Nov), problem categories (e.g., Greedy, Math, Checkmate), and specific problem characteristics (e.g., Pin, Double Check). The original source platforms also provide metadata like contest index, year, and month for Codeforces. 
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: Mean, standard deviation, Spearman correlation for difficulty alignment (IRT vs human/GPT4)
Metric subscores: Yes
Metric subscores detail: Scores broken down by difficulty bands (e.g., easy/medium/hard quantiles), and in some cases by domain or tag​ (different categories within each dataset).
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors perform human difficulty ranking tasks and compare to IRT scores and include analysis of discrepancy and matching accuracy.
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Human difficulty rankings used for difficulty label verification, but no full model-like performance baseline.