Maicarons/WCA-Bench-baselines

WCA-Bench example results

This directory contains baseline results produced on the official WCA Results Export (format v2.0.2, export date 2026-09-21) over a sampled test window.

Everything here is regenerated by the commands in Reproducing below.

Dataset scale (full build)

Metric	Value
results	6,909,454
persons	298,551
competitions	18,708
train / validation / test	3,211,294 / 1,978,851 / 1,719,290
Global attempt-level DNF rate	3.18%
Frozen person–event statistics	612,224
result_attempts	31,847,257
scrambles	3,186,380

Evaluation setup

scripts/run_all_baselines.py --mode small samples a handful of competitions inside the test window and takes a representative subsample of the training split, so that the full baseline suite completes on a single machine. The numbers below are therefore indicative, not final — the public leaderboard must be re-run on the full dataset under the rolling-window protocol.

Setting	Value
Mode	small
Seed	42 (single run); 42, 43, 44 (multi-seed)
Device	CUDA — NVIDIA GeForce RTX 4060 Laptop GPU (8 GB), CUDA 13.0, torch 2.13.0+cu130
Baselines	21, across 6 method families

The device is recorded per baseline in every report: pure tabular baselines report device: "cpu", while the LSTM, the GNN and the boosting baselines report device: "cuda:NVIDIA GeForce RTX 4060 Laptop GPU" together with gpu_hours.

Files

File	Content
leaderboard.md / leaderboard.csv / leaderboard.json	Aggregated leaderboard, one section per task
multi_seed.md	Primary metric as mean ± standard deviation over seeds 42 / 43 / 44
<task>__<model>.json	One full report per baseline (21 files)
submission_template/	Skeleton of a valid leaderboard submission

Each report uses the same top-level schema:

{
  "task": "...", "model": "...",
  "overall": {...},
  "stratified": {"by_event": {...}, "by_skill_level": {...}, "by_time_slice": {...}, "by_continent": {...}},
  "hard_subset": {...},
  "significance": {"reference": "...", "metric": "...", "paired_unit": "...", "n_pairs": 0,
                   "mean_diff": 0.0, "ci95": [0.0, 0.0], "p_value": 1.0,
                   "effect_size": {"name": "cohens_d", "value": 0.0}, "test": "paired_t", "seed": 42},
  "cost": {"mode": "small", "baseline_kind": "...", "device": "cuda:...",
           "wall_clock_sec": 0.0, "cpu_hours": 0.0, "gpu_hours": 0.0, "gpu_model": "..."},
  "extras": {...}, "elapsed_sec": 0.0, "seed": 42, "data_summary": {...}
}

Leaderboard summary

Single run (seed 42). See leaderboard.md for the complete tables.

Task	Best baseline	Primary metric	n
T1 Result prediction	xgboost_log	MAE(log) ↓ 0.1639	2196
T2 Placement prediction	kde_simulation / plackett_luce / psych_sheet	Kendall τ ↑ 0.7673	2260
T3 DNF prediction	xgboost_dnf	AUC-PR ↑ 0.3731	2260
T4 Human limit estimation	gp_evt	leave-one-out stability ↓ 0.9836	—
T5 Skill transfer	spearman_correlation	413 identifiable event pairs	80000

Multi-seed view (see multi_seed.md for the full table):

Task	Best baseline (mean ± std)	Metric
T1	xgboost_log 0.1055 ± 0.0417	MAE(log) ↓
T2	psych_sheet / plackett_luce / kde_simulation 0.8132 ± 0.0342	Kendall τ ↑
T3	beta_binomial 0.2938 ± 0.0570	AUC-PR ↑
T4	gp_evt 0.9836 ± 0.0000	LOO stability ↓
T5	spearman_correlation 413	identifiable pairs

Two findings worth noting:

1. Domain-aware baselines beat generic deep and graph models. On T1 the LSTM (MAE(log) 0.9436 ± 0.0754) is far behind gradient-boosted trees (0.1055 ± 0.0417); on T2 the GNN (τ 0.7440 ± 0.1258) trails the rule-aware Psych Sheet baseline (0.8132 ± 0.0342).
2. Rank stability depends on the task. On T1–T2 the multi-seed ordering matches the single run. On T3 the single run favours xgboost_dnf (0.3731) while the multi-seed mean favours beta_binomial (0.2938 ± 0.0570 vs 0.2779 ± 0.0724) — the sampling variance of the evaluation window is large enough to reorder close contenders, which is exactly why multi-seed reporting is mandatory.

Reproducing

python -m venv .venv
.venv\Scripts\activate            # Windows
# source .venv/bin/activate       # macOS / Linux
pip install -e ".[dev,fast,boost]"
pip install -e ".[deep]"          # optional: torch for the LSTM / GNN baselines

python scripts/download_data.py                  # fetch the official TSV export
python scripts/build_dataset.py --source raw     # decode, featurize, split, freeze statistics
python scripts/run_all_baselines.py --mode small --device cuda
python scripts/build_leaderboard.py --report-dir outputs/reports --out-dir examples
python scripts/run_multi_seed.py --seeds 42 43 44 --mode small --device cuda \
    --out-dir outputs/multi_seed --examples-dir examples

Use --device cpu (the default) for a CPU-only, fully reproducible run; --device auto selects CUDA when it is available.

Offline / CI path, using synthetic data:

python scripts/generate_synthetic.py --small
python scripts/build_dataset.py --source synthetic
python scripts/run_all_baselines.py --mode small
python scripts/build_leaderboard.py