cricketbench/README

CricketBench

A domain benchmark for cricket reasoning in large language models.

Cricket is a game of dense rules, sharp situations, deep records, precise terminology, and layered strategy. It's rewarding to model but easy to fake: most existing cricket "benchmarks" test surface trivia rather than reasoning. CricketBench targets the reasoning gap directly.

What's here

• Dataset — The canonical multiple-choice questions, versioned by release, organised into five reasoning dimensions:
  • laws_and_rule_precision — the 42 Laws of Cricket + ICC Playing Conditions
  • match_situation_reasoning — run rates, DLS, follow-on, powerplay
  • player_and_statistical_recall — records under reasoning, not trivia
  • commentary_generation_quality — terminology, shot/delivery identification
  • tactical_and_strategic_reasoning — captaincy, field settings, decisions
• Leaderboard — Per-model accuracy and per-dimension scores across the current release.
• Baselines — Per-model raw answers, kept alongside so anyone can reproduce discrimination scores from the same runs.

How it's built

Every question passes through a closed loop: hand-authored seeds → LLM-generated variants → red-team critique → answered by 8+ models across the capability gradient → per-question item-discrimination (point-biserial r against per-model ability) → human review. Only items that survive the human review land in the dataset.

Design decisions in one line each:

• Per-model item discrimination, not frontier-vs-open buckets — capability is measured from the data, not assigned by us.
• Unparseable model outputs are excluded from scoring, not counted wrong.
• Red-team major severity routes to human revise regardless of the statistical signal.
• Top-model-miss diagnostic flags items where the strongest legitimate model missed while ≥2 mid-tier models got it right — usually a sign the key is wrong.
• Model rankings require n ≥ 30 per dimension before we consider them publishable.

Where it comes from

CricketBench is a mindspace, while growing up in Chennai. The mindspace goes beyond winning and losing and primarily revolves around the intricacies and depth of the game. I am sure this is the case in any cricket-obsessed environment. I want to bring like minds to bring a strong bench that helps models compete for intelligence in one of the top sports in the world.

How to help

• Contribute questions — open an issue with a candidate item, or submit a PR against the dataset.
• Contribute baseline model results — submit a JSONL of predictions to the leaderboard Space.
• Contribute revisions — flag ambiguous items or wrong keys via HF Discussion.
• Contribute reviews — most valuable of all. The pipeline generates faster than a single person can review.

Maintainers

• Hariprasad Sudharshan — Founder and project lead. Senior product manager, based in Chennai. Responsible for taxonomy, scoring policies, and question review discipline.

Interested in becoming a maintainer? Contribute a batch of reviewed questions or a set of high-quality baseline model results and open a Discussion — sustained contributors are invited to write access on the org.

Citation

@misc{cricketbench2026,
  title        = {{CricketBench}: A domain benchmark for cricket reasoning
                  in large language models},
  author       = {Sudharshan, Hariprasad and {CricketBench contributors}},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/cricketbench/cricketbench},
  version      = {v0.0}
}

Contact

Discussions on the HF Community tab of any cricketbench/* repo.

---

Licensed CC-BY-SA 4.0. Every version is git-tagged; papers should cite a specific release.