Adaptive learning-rate schedules for logistic-regression convergence on binary classification
Domain: ml (optimization, machine-learning)
Topic: Comparing adaptive learning rate schedules (cosine annealing, step decay, exponential decay, warm restarts) for logistic regression convergence on binary classification
Venue: ARC-Bench 2026
Headline metric: convergence_epochs (minimize)

Research briefing:
Logistic regression is often treated as a solved baseline, yet its practical
training behavior still depends strongly on optimization details. In
first-order methods, the learning-rate schedule can dominate time-to-quality:
a constant step size may converge slowly or oscillate, while adaptive
schedules can accelerate early progress or stabilize late optimization.
Because logistic regression has a smooth convex objective, it offers a clean
setting to compare schedules without confounds from deep-network
non-convexity.

This topic studies four schedule families in a unified mini-batch gradient
descent implementation: step decay, exponential decay, cosine annealing, and
cosine warm restarts. The focus is not raw final accuracy alone, but
optimization efficiency measured by epochs needed to reach a predefined
validation-loss threshold, plus robustness across datasets and random seeds.
A fixed-learning-rate baseline is required so any claimed speedup is
attributable to schedule design.

A credible CPU-scale experiment should run on multiple binary classification
datasets from sklearn, with feature standardization, identical model
parameterization, and synchronized stopping criteria. Reporting should include
convergence epochs, final validation log loss, and test ROC-AUC so that faster
convergence is not achieved at the expense of generalization. Since warm
restarts can re-increase the learning rate, trajectory logging is important to
verify that restart behavior is actually implemented.

The key decision criterion is whether adaptive schedules reduce optimization
effort in a consistent, measurable way. The study should conclude with clear
per-hypothesis verdicts and practical limitations (dataset size, threshold
choice, and sensitivity to initial learning rate).

*Do adaptive learning-rate schedules (especially cosine variants) reduce logistic-regression convergence epochs versus fixed/monotone decay schedules while maintaining comparable binary-classification performance?*

Hypotheses:
statement: Cosine annealing or cosine warm restarts achieves at least 20% lower mean convergence_epochs than fixed learning rate on at least 2 of 3 evaluated binary datasets, averaged over >=5 seeds., statement: Among adaptive schedules (step decay, exponential decay, cosine annealing, warm restarts), the best schedule's final validation log loss is no worse than 0.01 absolute compared with the worst schedule on each dataset (i.e., speed differences exceed quality differences)., statement: Warm restarts reaches convergence in fewer epochs than monotone exponential decay on at least 2 of 3 datasets, without reducing test ROC-AUC by more than 0.01 absolute on those datasets.

Experiment design:
research_question: Which learning-rate schedule yields the fastest convergence for mini-batch logistic regression on binary sklearn tasks, and do faster schedules preserve validation/test performance?
conditions: name: fixed_lr
description: Mini-batch logistic regression trained with constant learning rate (baseline)., name: step_decay
description: Learning rate multiplied by gamma at fixed epoch milestones (e.g., every 20 epochs)., name: exponential_decay
description: Learning rate updated each epoch as lr_t = lr0 * exp(-k * t)., name: cosine_annealing
description: Learning rate follows cosine decay from lr0 to lr_min over total epochs without restarts., name: cosine_warm_restarts
description: Cosine schedule with periodic restarts (SGDR-style) resetting to higher learning rate at restart boundaries.
baselines: fixed_lr is the primary optimization baseline, exponential_decay is a monotone adaptive baseline for comparison to warm restarts
metrics: name: convergence_epochs
direction: minimize
description: Epoch index at which validation log loss first drops below a preset threshold (or max_epochs if never reached), averaged over seeds., name: val_log_loss
direction: minimize
description: Final validation logistic loss after training, mean over seeds., name: test_roc_auc
direction: maximize
description: ROC-AUC on held-out test set, mean over seeds.
datasets: name: breast_cancer
source: sklearn.datasets.load_breast_cancer, name: synthetic_binary_medium
source: sklearn.datasets.make_classification (n_samples=2000, n_features=30, n_informative=12, class_sep=1.0), name: synthetic_binary_noisy
source: sklearn.datasets.make_classification (n_samples=2500, n_features=40, n_informative=10, flip_y=0.08, class_sep=0.7)
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 420

Requirements:

Scoring rubric:
requirements: A credible experiment studying whether adaptive learning-rate schedules improve logistic-regression convergence on binary classification: schedule conditions are implemented as distinct update rules, runs are executed on multiple datasets with multiple seeds, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. Equivalent schedules (e.g., a different decay parameterization) should be credited when the underlying mechanism is clearly the intended one.
sub_tasks: requirements: The logistic-regression training framework and learning-rate schedule variants are implemented in a way that allows a meaningful convergence comparison.
sub_tasks: requirements: The submission implements a fixed-learning-rate baseline and multiple adaptive schedules (e.g., step-decay, exponential-decay, cosine-annealing, cosine-warm-restarts, or comparable alternatives) as distinct update rules, not duplicated code paths.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: A binary logistic-regression objective (sigmoid + log-loss) is trained via iterative gradient-based optimization with per-epoch control, enabling convergence-epoch tracking.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission uses multiple datasets (including at least one sklearn built-in or synthetic classification set), performs train/validation/test splitting, and applies feature scaling consistently.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution produces convergence and quality metrics with reasonable seed coverage.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact with numeric convergence-epochs (or equivalent convergence measure) and validation log-loss for each implemented (dataset, condition) cell.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Reported metrics are aggregated over multiple random seeds per (dataset, condition) with some dispersion measure. More seeds are better, but honest small-seed runs with variance reported are preferable to single deterministic runs.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution also reports a final predictive-quality metric (e.g., test ROC-AUC) for each condition on at least one dataset, so readers can check that convergence-speed gains do not collapse classification quality.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The analysis evaluates all three hypotheses and discusses the speed-vs-quality trade-off.
sub_tasks: requirements: The submission compares cosine-style schedules against the fixed-rate baseline on convergence-epochs across the evaluated datasets and conveys whether cosine schedules converge meaningfully faster — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports the spread of final validation log-loss among the adaptive schedules and conveys whether the best-vs-worst gap is small (i.e., final-quality differences are minor once converged).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares cosine-warm-restarts against exponential-decay (or equivalent schedules) on convergence speed and final ROC-AUC and conveys the qualitative outcome for H3.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes setup and key metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations such as convergence-threshold choice, seed count, or dataset scope. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation