Dropout regularization strategies on shallow tabular MLPs
Domain: ml (machine-learning, regularization)
Topic: Comparing dropout regularization strategies (standard, spatial, variational) for preventing overfitting in shallow MLPs on tabular classification benchmarks
Venue: ARC-Bench 2026
Headline metric: test_accuracy (maximize)

Research briefing:
Dropout-style regularization is a cornerstone of neural-net training for
overfitting control, but its impact on tabular classification with shallow
MLPs remains subtle. Standard element-wise dropout is the default; spatial
(feature-wise) dropout treats correlated input features as a block; MC /
variational dropout keeps dropout active at test time and averages over
samples. For tabular data with small feature counts and mild non-linearity,
these variants can produce equal test-set accuracy yet very different
probability calibration — a known gotcha that makes accuracy-only
evaluation misleading.

A credible CPU-scale study of this topic (a) compares at least three
dropout variants against a no-dropout control on multiple sklearn
classification benchmarks, (b) reports both accuracy and calibration
metrics (ECE, NLL, Brier), (c) averages over ≥5 seeds with statistical
tests, and (d) checks for ablation integrity (that different dropout
variants actually produce different output distributions, not just the
same network). The research question is: *does the choice of dropout
variant matter more for calibration than for accuracy on tabular data?*

Hypotheses:
statement: Variational / MC dropout produces lower Expected Calibration Error (ECE) than standard element-wise dropout on at least 2 of 3 tabular classification benchmarks, averaged over ≥5 seeds., statement: Test-set accuracy differs by <2 absolute percentage points between dropout variants on datasets where all methods exceed 95% accuracy — i.e. accuracy is non-discriminative and calibration is needed to distinguish methods., statement: The no-dropout control is strictly worse than at least one dropout variant in ECE on at least 2 of 3 datasets.

Experiment design:
research_question: Does the choice of dropout variant (standard / spatial / variational) matter more for probability calibration than for test accuracy on tabular classification with shallow MLPs?
conditions: name: no_dropout
description: Shallow MLP (2 hidden layers, 64 units, ReLU) trained with no dropout. Control condition., name: standard_dropout_p30
description: Same MLP with standard element-wise dropout at p=0.3 between hidden layers., name: standard_dropout_p50
description: Same MLP with standard element-wise dropout at p=0.5., name: spatial_dropout_p30
description: Same MLP with feature-blockwise dropout at p=0.3 applied to the input layer., name: mc_dropout_p30_T20
description: Same MLP with standard p=0.3 dropout left active at test time; prediction is the mean of T=20 stochastic forward passes.
baselines: no_dropout is the no-regularization baseline
metrics: name: test_accuracy
direction: maximize
description: Fraction correctly classified on held-out 20% test split, mean over 5 seeds., name: ece
direction: minimize
description: Expected Calibration Error (15 bins) on the test split, mean over 5 seeds., name: nll
direction: minimize
description: Mean per-example negative log-likelihood on the test split., name: brier
direction: minimize
description: Mean Brier score on the test split.
datasets: name: breast_cancer
source: sklearn.datasets.load_breast_cancer, name: wine
source: sklearn.datasets.load_wine, name: digits
source: sklearn.datasets.load_digits
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 240

Requirements:

Scoring rubric:
requirements: A credible experiment studying whether dropout-variant choice affects calibration more than accuracy on tabular classification: the main dropout variants are implemented, evaluation covers multiple sklearn benchmarks with reasonable seed coverage, and the writeup ties the numeric findings to the three hypotheses in direction and magnitude.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. Partial but well-motivated evidence deserves partial credit; rigid wording or name mismatches should not penalize a substantively correct experiment.
sub_tasks: requirements: The dropout conditions are implemented in a way that allows a meaningful calibration comparison.
sub_tasks: requirements: The submission implements the main dropout variants relevant to the hypotheses — typically a no-dropout baseline, a standard element-wise dropout at one or more rates, and at least one stochastic-test-time / MC-style dropout variant — as distinct code paths within a shared MLP architecture.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The model is a shallow tabular MLP (a small number of hidden layers at modest width) trained with a standard supervised classification objective, appropriate for the sklearn benchmarks used.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission loads multiple tabular sklearn datasets (e.g., from {breast_cancer, wine, digits} or comparable benchmarks) and uses a reasonable train/test split.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The MC-dropout-style condition performs several stochastic forward passes at test time and averages the predicted probabilities, rather than collapsing to a single deterministic forward pass.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: Execution produces accuracy and calibration numbers per condition that are adequate to evaluate the hypotheses.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact (e.g., results/metrics.json or stage-14/experiment_summary.json) with numeric accuracy and a calibration metric such as ECE, covering the implemented conditions on at least one dataset. Other calibration metrics (NLL, Brier) may substitute or supplement ECE.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are aggregated over multiple random seeds per (condition, dataset) cell with some form of dispersion reporting (std, stderr, CI, or min/max across seeds). More seeds are better, but a small-but-honest seed count with reported variance is preferable to a single deterministic run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The results analysis addresses the three hypotheses with quantitative evidence and a clear narrative.
sub_tasks: requirements: The submission compares calibration (ECE or analogous metric) between an MC/variational-style dropout and a standard dropout across the evaluated datasets, and conveys whether MC-style calibration tends to be better — judge whether the evidence is consistent with H1, refutes it, or is inconclusive, based on the reported numbers.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission discusses whether accuracy differences among dropout variants are small on datasets where all methods achieve high accuracy — i.e., whether calibration is the discriminative axis rather than accuracy. Exact percentage-point thresholds are not required; a qualitative comparison grounded in the reported numbers suffices.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares the no-dropout baseline against at least one dropout variant on calibration and conveys whether the baseline is clearly worse, comparable, or better.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes the method and setup, presents the key accuracy and calibration numbers, and conveys per-hypothesis outcomes (supported / refuted / inconclusive) with appropriate caveats on seed count, dataset scope, or calibration-metric choice. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation