Post-hoc calibration methods for sklearn classifiers on tabular benchmarks
Domain: ml (machine-learning, calibration)
Topic: Evaluating probability calibration methods (Platt scaling, isotonic regression, temperature scaling) applied to scikit-learn classifiers (RandomForest, GBM, SVM-RBF) on UCI benchmarks
Venue: ARC-Bench 2026
Headline metric: ece (minimize)

Research briefing:
Many widely used scikit-learn classifiers optimize discrimination rather
than probability quality. Random forests, gradient boosting models, and
RBF-kernel SVMs can achieve strong accuracy while output probabilities are
miscalibrated, especially on small or moderately imbalanced tabular data.
This matters in decision settings where thresholds, risk ranking, or cost-
sensitive actions depend on reliable confidence estimates.

Post-hoc calibration methods are attractive because they can be layered onto
existing models without retraining the base learner. Platt scaling fits a
sigmoid map, isotonic regression fits a non-parametric monotone map, and
temperature scaling applies a single-parameter logit rescaling (implemented
for binary tasks via optimization on held-out logits/probabilities). These
methods differ in flexibility and overfitting risk, so their relative value
may depend on classifier family and dataset size.

A credible CPU-scale study should compare uncalibrated outputs versus at
least three calibrators across multiple sklearn tabular datasets, using a
proper train/calibration/test protocol and repeated seeds. Because this is a
calibration-centric question, expected calibration error (ECE) and log loss
should be primary metrics, with accuracy used as a guardrail to ensure that
calibration does not degrade classification utility.

The goal is not to reproduce a single paper result but to test whether any
one calibrator is consistently superior across heterogeneous models and
datasets under tight compute constraints. The key uncertainty is whether
method ranking is stable enough to justify a default choice.

*Which post-hoc calibration method (Platt, isotonic, temperature) most reliably improves ECE across RF/GBM/SVM-RBF on small tabular sklearn benchmarks without materially harming accuracy?*

Hypotheses:
statement: At least one post-hoc calibrator (Platt, isotonic, or temperature) reduces ECE by at least 10% relative to the uncalibrated model for at least 2 of 3 classifiers on at least 2 of 3 datasets, averaged over >=3 seeds., statement: Isotonic regression achieves lower mean ECE than Platt scaling on at least 2 of 3 datasets when calibration-set size is >=150 samples., statement: Applying post-hoc calibration changes test accuracy by no more than 1.0 absolute percentage point (mean over seeds) for each classifier-dataset pair.

Experiment design:
research_question: Which post-hoc calibration method (Platt, isotonic, temperature) most consistently improves probability calibration for RF/GBM/SVM-RBF on sklearn tabular datasets while preserving accuracy?
conditions: name: uncalibrated
description: Base classifier probabilities/decision scores used directly with no calibration., name: platt_scaling
description: Sigmoid calibration fit on a held-out calibration split (CalibratedClassifierCV with method='sigmoid' or equivalent)., name: isotonic_regression
description: Isotonic calibration fit on a held-out calibration split (CalibratedClassifierCV with method='isotonic' or equivalent)., name: temperature_scaling
description: Single temperature parameter optimized on calibration split to minimize NLL; applied to logits/scores before probability mapping.
baselines: uncalibrated outputs are the primary baseline, platt_scaling serves as a classic parametric calibration baseline
metrics: name: ece
direction: minimize
description: Expected Calibration Error (10-15 bins) on the held-out test split, averaged over seeds., name: log_loss
direction: minimize
description: Negative log-likelihood on test probabilities., name: test_accuracy
direction: maximize
description: Classification accuracy on held-out test split to verify discrimination is preserved.
datasets: name: breast_cancer
source: sklearn.datasets.load_breast_cancer, name: wine
source: sklearn.datasets.load_wine, name: digits_binary
source: sklearn.datasets.load_digits with target transformed to binary (e.g., digit<5 vs >=5)
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 480

Requirements:

Scoring rubric:
requirements: A credible experiment studying post-hoc probability calibration methods (Platt, isotonic, temperature, or equivalents) for sklearn classifiers: calibration conditions are implemented, execution covers multiple datasets/classifiers with repeated seeds, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated calibrator variants or alternative base classifiers that preserve the scientific question should be credited.
sub_tasks: requirements: Calibration methods and classifier setup are implemented correctly.
sub_tasks: requirements: The submission implements multiple calibration conditions — typically including uncalibrated, Platt, isotonic, and temperature scaling — as distinct code paths applied post-hoc to the same base model outputs.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The experiment includes multiple target classifiers (e.g., RandomForest, gradient boosting, SVM-RBF, or equivalents) with consistent train/calibration/test handling.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission uses multiple datasets (sklearn built-ins or comparable) and an explicit calibration split separate from train and test.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution reports calibration-focused metrics across conditions.
sub_tasks: requirements: Execution outputs numeric ECE, log loss, and test accuracy (or equivalents) for each implemented condition on at least one dataset-classifier pair in a machine-readable artifact.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Metrics are aggregated over multiple random seeds per evaluated cell with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Results quantitatively address H1/H2/H3 directionally with a clear narrative.
sub_tasks: requirements: The submission computes relative ECE change versus uncalibrated and conveys whether at least one calibrator yields a meaningful ECE reduction across most classifier/dataset pairs — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares isotonic vs Platt mean ECE per dataset and conveys the relative calibration quality plus any calibration-set size considerations (H2).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission reports accuracy deltas between calibrated and uncalibrated outputs per classifier-dataset pair and conveys whether accuracy changes are small (H3).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or report conveys per-hypothesis outcomes (supported / refuted / inconclusive), cites key ECE/log-loss/accuracy numbers, and discusses limitations (dataset scope, binning sensitivity, seed count, calibration split size). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation