Impact of imbalance-handling strategies on sklearn binary classifiers
Domain: ml (machine-learning, imbalanced-learning)
Topic: Analyzing the impact of class imbalance handling strategies (SMOTE, random oversampling, class weights, undersampling) on binary classification with scikit-learn
Venue: ARC-Bench 2026
Headline metric: balanced_accuracy (maximize)

Research briefing:
Class imbalance is one of the most common practical failure modes in binary
classification: a model can achieve high raw accuracy by predicting the
majority class while missing the minority class almost entirely. In
scikit-learn workflows, practitioners often choose among simple data-level
resampling (random oversampling, random undersampling), algorithm-level
weighting (class_weight="balanced"), or synthetic methods like SMOTE.
However, these strategies trade off minority recall, precision, and overall
discrimination differently, and their behavior varies with dataset geometry.

A compact CPU-friendly study can still be rigorous if it compares multiple
imbalance-handling strategies under the same base learner and preprocessing
pipeline, evaluates with metrics appropriate for skewed labels (balanced
accuracy, F1, PR-AUC), and averages over several random seeds. The focus is
not on maximizing one benchmark score but on understanding whether methods
that improve minority detection do so consistently across datasets, and at
what precision cost.

To keep implementation feasible in a short autonomous run, the experiment can
use sklearn-native datasets and synthetic imbalance generated from a standard
source dataset, with a shared train/test protocol and fixed model family
(e.g., logistic regression). SMOTE can be implemented in a lightweight way
using sklearn.neighbors to interpolate minority-neighbor pairs, avoiding any
external dependency.

The key analytical goal is to test whether weighted learning or synthetic
oversampling provides the most reliable gain in balanced accuracy over a
no-treatment baseline, and whether naive random oversampling tends to hurt
precision-oriented metrics relative to class weighting.

*Which imbalance-handling strategy (SMOTE, random oversampling, class weights, undersampling) gives the most consistent balanced-accuracy improvement for binary sklearn classification under a fixed model and protocol?*

Hypotheses:
statement: At least one of {class_weight_balanced, smote} achieves higher balanced_accuracy than no_handling by >=0.03 absolute on at least 2 of 3 datasets (mean over >=5 seeds)., statement: Random undersampling yields the highest minority recall among compared strategies on at least 2 of 3 datasets, but does not achieve the best PR-AUC on those same datasets., statement: Random oversampling does not outperform class_weight_balanced in balanced_accuracy on more than 1 of 3 datasets (seed-averaged comparison).

Experiment design:
research_question: Which imbalance-handling strategy provides the most consistent gains in balanced accuracy for binary classification with fixed sklearn models under controlled train/test splits?
conditions: name: no_handling
description: Baseline logistic regression trained on imbalanced data with no resampling and default class weights., name: class_weight_balanced
description: Same logistic regression with class_weight='balanced'., name: random_oversampling
description: Training-set minority class randomly oversampled with replacement to match majority count., name: random_undersampling
description: Training-set majority class randomly undersampled without replacement to match minority count., name: smote_k5
description: Training-set minority class augmented with synthetic samples via kNN interpolation (k=5) until class balance is reached.
baselines: no_handling is the primary baseline, class_weight_balanced serves as an algorithm-level baseline against data-level resampling
metrics: name: balanced_accuracy
direction: maximize
description: Mean of sensitivity and specificity on held-out test split, averaged over seeds., name: minority_recall
direction: maximize
description: Recall for the positive/minority class on test data., name: average_precision
direction: maximize
description: Area under precision-recall curve (PR-AUC / AP) on test data., name: f1
direction: maximize
description: Binary F1 score for the minority class at default threshold 0.5.
datasets: name: breast_cancer_imbalanced
source: sklearn.datasets.load_breast_cancer with induced imbalance by downsampling positive class in training folds, name: make_classification_90_10
source: sklearn.datasets.make_classification(n_samples=4000, n_features=20, weights=[0.9,0.1], random_state=seed), name: make_classification_95_05
source: sklearn.datasets.make_classification(n_samples=5000, n_features=25, weights=[0.95,0.05], class_sep=1.0, random_state=seed)
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 360

Requirements:

Scoring rubric:
requirements: A credible experiment studying imbalance-handling strategies (e.g., class weights, SMOTE, random over-/under-sampling) for binary classification: methods are implemented as distinct conditions, execution covers multiple datasets with multiple seeds, and results address H1/H2/H3 using balanced-accuracy-centered analysis.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. If SMOTE is implemented via a simplified interpolation recipe rather than the imblearn package, credit the scientific intent; alternative imbalance strategies that test the same question should also be credited.
sub_tasks: requirements: The imbalance-handling conditions are implemented in a way that supports a fair comparison under a shared classifier.
sub_tasks: requirements: The submission implements multiple imbalance-handling strategies — typically including a no-handling baseline, class-weight balancing, an oversampling variant (random or SMOTE-like), and an undersampling variant — as distinct code paths under a common binary classifier.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: If a SMOTE-style condition is claimed, synthetic minority samples are generated by interpolation between minority neighbors (not mere duplication) and applied only to training data. A simplified SMOTE-style implementation is acceptable if the intent is clear.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission uses multiple datasets (loaded or synthesized via sklearn utilities) and enforces an imbalanced binary class distribution in training data.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution reports imbalance-aware metrics per condition with reasonable seed coverage.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact with numeric balanced-accuracy and at least one of {minority recall, average precision, F1} per implemented condition on at least one dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are aggregated over multiple random seeds per (dataset, condition) with some dispersion measure. More seeds are better, but honest small-seed runs with variance reported are preferable to single deterministic runs.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The analysis addresses H1/H2/H3 directionally with a clear narrative.
sub_tasks: requirements: The submission compares no-handling against class-weighting and/or SMOTE-style oversampling on balanced-accuracy and conveys whether imbalance handling yields a meaningful improvement — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports the minority-recall / average-precision comparisons needed for H2 and the random-oversampling vs class-weighting comparison needed for H3, conveying qualitative outcomes for both.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes setup, reports per-dataset metric outcomes, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations such as synthetic-dataset realism, threshold dependence, seed count, or SMOTE simplifications. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation