Online binary classification under concept drift: SGD, PA, NB, and FTRL
Domain: ml (online-learning, machine-learning)
Topic: Benchmarking online learning algorithms (SGD logistic, Passive-Aggressive, online Naive Bayes, Follow-the-Regularized-Leader) on streaming binary classification with concept drift
Venue: ARC-Bench 2026
Headline metric: prequential_accuracy (maximize)

Research briefing:
In streaming classification, data arrive sequentially and model updates must
be cheap and immediate. Under concept drift, the relationship between
features and labels changes over time, so static train/test evaluation can
be misleading. Prequential (test-then-train) evaluation better reflects
deployment: each incoming point is first predicted, then used for an update.
This setting creates practical trade-offs among stability, adaptability, and
computational cost.

Several lightweight online learners in sklearn-style workflows represent
different adaptation biases. SGD logistic regression can track drift via
gradient updates but may require careful learning-rate choices. Passive-
Aggressive updates can react strongly to mistakes and often adapt quickly to
abrupt shifts. Online Naive Bayes is extremely cheap and robust but may lag
when feature dependence structure changes. A simple FTRL-style proximal
logistic update (implemented with diagonal accumulators) offers adaptive
per-feature learning rates and implicit regularization.

A credible CPU-only benchmark should compare these methods on at least one
synthetic abrupt-drift stream and one real sklearn dataset converted to a
stream with induced drift (e.g., blockwise class-prior or feature transform
shift). It should report prequential accuracy as the primary metric, plus at
least one drift-sensitive secondary metric such as post-drift recovery and
cumulative log loss. Multi-seed runs are important because stream order and
drift-point randomness can materially change outcomes.

The study should also verify that drift actually hurts models by comparing a
no-drift control stream against drifted streams, and should discuss which
algorithms recover fastest after drift while maintaining overall accuracy.
This makes the benchmark more diagnostic than a single aggregate score.

*Which lightweight online learner provides the best trade-off between overall prequential accuracy and adaptation speed after concept drift in binary streams?*

Hypotheses:
statement: On drifted streams, Passive-Aggressive or FTRL achieves higher prequential_accuracy than online Naive Bayes by at least 0.02 absolute on at least 2 of 3 datasets, averaged over >=5 seeds., statement: For at least 2 of 3 drifted datasets, the best post-drift recovery accuracy in a fixed window of 200 samples after each drift point is achieved by either Passive-Aggressive or FTRL., statement: For SGD logistic and Passive-Aggressive, prequential_accuracy on a drifted version of each stream is at least 0.03 lower than on its matched no-drift control, on at least 2 of 3 datasets.

Experiment design:
research_question: Which online method among SGD logistic, Passive-Aggressive, online Naive Bayes, and FTRL-style logistic best balances prequential accuracy and post-drift recovery on binary data streams with concept drift?
conditions: name: sgd_logistic
description: SGDClassifier with log_loss and partial_fit in prequential test-then-train loop., name: passive_aggressive
description: PassiveAggressiveClassifier with partial_fit in the same loop., name: online_naive_bayes
description: GaussianNB updated incrementally via partial_fit., name: ftrl_prox_logistic
description: Custom numpy FTRL-proximal logistic regression with diagonal accumulator and L1/L2 regularization.
baselines: online_naive_bayes as a simple low-cost baseline, sgd_logistic as a standard linear online-learning baseline
metrics: name: prequential_accuracy
direction: maximize
description: Mean test-then-train accuracy over the full stream, averaged over 5 seeds., name: post_drift_recovery_accuracy
direction: maximize
description: Accuracy in the first 200 samples after each drift point, averaged across drift events and seeds., name: prequential_log_loss
direction: minimize
description: Cumulative mean log loss computed prequentially when probabilistic outputs are available (or clipped score-to-prob mapping).
datasets: name: synthetic_abrupt_drift
source: numpy synthesis (piecewise make_classification with coefficient/sign flips at fixed indices), name: breast_cancer_stream
source: sklearn.datasets.load_breast_cancer transformed into a repeated/shuffled stream with blockwise feature scaling shift, name: spambase_like_synthetic
source: numpy synthesis with sparse informative features and class-prior shift across blocks
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 420

Requirements:

Scoring rubric:
requirements: A credible experiment studying online learners (SGD logistic, Passive-Aggressive, online Naive Bayes, FTRL-style logistic, or equivalents) under concept drift: streaming prequential setup is implemented, execution covers multiple datasets with multiple seeds, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated online-learner substitutes or drift-stream variants that test the same scientific question should be credited.
sub_tasks: requirements: Online-learning conditions and drifted streaming setup are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct online methods, each as a real incremental update path (e.g., partial_fit or equivalent), not batch retraining.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: A prequential test-then-train loop is implemented: each sample (or mini-batch) is predicted before being used for model update, with ordered stream processing.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission uses multiple streams including at least one drifted stream and one no-drift or matched-control stream, generated from sklearn datasets or numpy synthesis.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution reports prequential metrics for each condition.
sub_tasks: requirements: Execution produces a metrics artifact containing numeric prequential accuracy (or equivalent) for each implemented method on at least one dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution includes a post-drift recovery measure (e.g., fixed-window post-drift accuracy) with a documented computation referencing known drift points.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Each reported (dataset, method) metric is averaged over multiple random seeds, with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Hyperparameter Tuning, requirements: Quantitative analysis addresses H1/H2/H3 directionally.
sub_tasks: requirements: The submission compares prequential accuracy of margin-based online methods (Passive-Aggressive, FTRL) against online Naive Bayes across drifted datasets and conveys whether margin-based methods are meaningfully better — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports post-drift recovery comparisons and conveys whether Passive-Aggressive or FTRL tends to lead on most datasets (H2).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The submission compares drifted vs no-drift control performance for the implemented methods and conveys whether drift produces a meaningful accuracy degradation (H3).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The README or writeup describes setup, key metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (stream realism, drift design choices, seed count, metric assumptions). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation