Cross-validation strategy reliability for small-sample model selection
Domain: ml (machine-learning, model-selection)
Topic: Investigating the effectiveness of different cross-validation strategies (k-fold, stratified, repeated, leave-one-out) for model selection with small sample sizes
Venue: ARC-Bench 2026
Headline metric: estimation_bias (minimize)

Research briefing:
With small datasets, model selection can become highly sensitive to the
validation protocol. Practitioners often choose among k-fold,
stratified k-fold, repeated k-fold, and leave-one-out cross-validation
(LOOCV), but these strategies trade off bias, variance, and compute in
different ways. A method that appears best under one CV scheme may not be
best under another, especially when class balance is imperfect and sample
counts are low.

This topic studies CV strategy as the independent variable while keeping
candidate models fixed and lightweight. The key quantity is estimation bias:
the gap between CV-estimated score used for model selection and the realized
test score of the selected model on a held-out test set. In small-sample
settings, minimizing this gap is often more important than maximizing raw CV
score, because over-optimistic selection can produce brittle deployments.

A credible CPU-scale experiment should evaluate multiple sklearn datasets
reduced to small training sizes, run several random seeds, and compare at
least four CV strategies under the same model grid and scoring rule. The
study should report not only estimation bias but also variance across seeds
and model-selection stability. Baselines should include standard k-fold and
stratified k-fold; repeated stratified k-fold and LOOCV serve as contrasting
alternatives with different variance/compute characteristics.

The goal is not to prove one universally superior protocol, but to quantify
when more elaborate CV schemes improve reliability enough to justify their
cost under strict CPU budgets.

*Which cross-validation strategy yields the lowest and most stable model-selection estimation bias for small-sample classification tasks under a fixed candidate-model grid?*

Hypotheses:
statement: RepeatedStratifiedKFold (5 folds × 3 repeats) achieves lower mean absolute estimation bias than plain KFold (5 folds) on at least 2 of 3 evaluated small-sample datasets, averaged over ≥5 random seeds., statement: StratifiedKFold (5 folds) yields lower across-seed standard deviation of selected-model test accuracy than plain KFold (5 folds) on at least 2 of 3 datasets., statement: LOOCV does not outperform RepeatedStratifiedKFold in mean absolute estimation bias by more than 0.01 absolute score on any dataset, while requiring greater wall-clock time.

Experiment design:
research_question: For small-sample classification model selection, how do KFold, StratifiedKFold, RepeatedStratifiedKFold, and LOOCV compare in estimation bias, stability, and compute cost?
conditions: name: kfold_5
description: Model selection via 5-fold KFold (shuffle=True) on the training split., name: stratified_kfold_5
description: Model selection via 5-fold StratifiedKFold (shuffle=True) on the training split., name: repeated_stratified_5x3
description: Model selection via RepeatedStratifiedKFold with 5 folds and 3 repeats., name: loocv
description: Model selection via Leave-One-Out cross-validation on the training split.
baselines: kfold_5 as the non-stratified baseline, stratified_kfold_5 as the standard small-sample baseline
metrics: name: estimation_bias
direction: minimize
description: Absolute difference between best CV mean score and held-out test score of the selected model, aggregated over seeds., name: selected_model_test_accuracy
direction: maximize
description: Held-out test accuracy of the model selected by each CV strategy, mean over seeds., name: selection_stability
direction: maximize
description: Fraction of seeds selecting the modal hyperparameter configuration (higher = more stable)., name: wall_clock_sec
direction: minimize
description: Runtime per CV strategy over all seeds and datasets on CPU.
datasets: name: breast_cancer_small
source: sklearn.datasets.load_breast_cancer (subsample training set to n<=120), name: wine_small
source: sklearn.datasets.load_wine (subsample training set to n<=100), name: synthetic_imbalanced_small
source: sklearn.datasets.make_classification (n_samples=160, weights=[0.75,0.25], class_sep tuned for moderate difficulty)
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 600

Requirements:

Scoring rubric:
requirements: A credible experiment studying how cross-validation strategy choice (k-fold, stratified k-fold, repeated stratified k-fold, LOOCV) affects small-sample model-selection reliability: strategies are implemented consistently, runs cover multiple small-sample datasets with multiple seeds, and results address H1/H2/H3 directionally around estimation bias, stability, and compute.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative CV strategies or datasets that preserve the scientific question (small-sample bias/stability of CV) should be credited.
sub_tasks: requirements: Cross-validation strategy conditions and the model-selection pipeline are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct CV strategies — typically including plain k-fold, stratified k-fold, repeated stratified k-fold, and optionally LOOCV — as separate code paths used for model selection under a shared candidate-model grid.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission uses multiple small-sample datasets (e.g., reduced breast_cancer, wine, or synthetic imbalanced variants) with a held-out test split.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Code computes an estimation-bias style metric (e.g., |best_cv_score - selected_model_test_score|) or equivalent for each (dataset, seed, strategy) and aggregates it reproducibly.
task_category: Code Development
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution logs required metrics per strategy with multi-seed evaluation.
sub_tasks: requirements: Execution produces a machine-readable results artifact containing estimation bias (or equivalent) and selected-model test accuracy for each implemented strategy on at least one dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution uses multiple random seeds per (dataset, strategy) and records timing information sufficient to compare the more expensive strategies (e.g., LOOCV vs repeated stratified). Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Experimental Setup, requirements: Reported results address H1/H2/H3 directionally with interpretable evidence.
sub_tasks: requirements: The submission compares mean estimation bias of repeated-stratified CV vs plain k-fold per dataset and conveys whether repeated stratified is meaningfully better on most datasets — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports selection stability or across-seed variability for stratified k-fold vs plain k-fold (H2) and compares LOOCV versus repeated stratified CV on estimation bias and runtime (H3), conveying qualitative outcomes for both.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup describes the setup, reports per-strategy metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and discusses limitations (small dataset scope, candidate-model grid choice, metric sensitivity, compute budget). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation