Active learning query strategies for logistic regression on small tabular pools
Domain: ml (machine-learning, active-learning)
Topic: Evaluating active learning query strategies (uncertainty sampling, margin sampling, query-by-committee, expected-error reduction) for logistic regression on small UCI pools
Venue: ARC-Bench 2026
Headline metric: accuracy_at_budget (maximize)

Research briefing:
Pool-based active learning is attractive when labels are expensive but a
modest unlabeled pool is available. For linear probabilistic models such as
logistic regression, classic query policies include uncertainty sampling,
margin sampling, query-by-committee (QBC), and expected-error reduction
style lookahead approximations. These methods are often taught as broadly
useful, but on small tabular datasets their gains can be inconsistent due to
model misspecification, class imbalance, and high variance from tiny initial
labeled sets.

A useful CPU-bounded benchmark should compare these query strategies under a
fixed annotation budget and shared learner, rather than mixing in model
architecture changes. The core signal is label efficiency: how quickly test
performance improves as labels are acquired. Because final accuracy alone can
hide early-budget differences, the study should include both
accuracy-at-budget and a budget-curve summary metric such as area under the
learning curve.

The experiment should therefore run multiple seeds, use at least two
sklearn-resident tabular classification datasets, and enforce identical
initialization, batch size, and stopping budget across strategies. A random
query baseline is essential to determine whether sophisticated policies
provide real value beyond chance selection. A passive full-data reference is
also useful for contextualizing attainable performance ceilings under the
same logistic-regression family.

Practical constraints matter: expected-error reduction can be expensive if
naively recomputing retrains for every candidate. A tractable variant can
evaluate a capped candidate subset per round and use one-step hypothetical
label outcomes, preserving the spirit of expected future loss minimization
while staying within single-core runtime limits.

*Do uncertainty, margin, QBC, and approximate expected-error reduction deliver better label efficiency than random sampling for logistic regression on small tabular pools under a fixed annotation budget?*

Hypotheses:
statement: At 30% labeling budget, at least 2 of {uncertainty_sampling, margin_sampling, qbc, expected_error_reduction} achieve higher mean test accuracy than random_sampling by ≥1.5 percentage points, averaged over ≥5 seeds., statement: expected_error_reduction attains the highest mean area_under_learning_curve (AULC) on at least 1 of 3 datasets, and its AULC is not lower than random_sampling on any evaluated dataset., statement: qbc outperforms uncertainty_sampling in mean test accuracy at early budget (10% labels) on at least 2 of 3 datasets.

Experiment design:
research_question: Which active learning query strategy provides the best label efficiency for logistic regression on small tabular pool-based classification tasks under fixed budgets?
conditions: name: random_sampling
description: Baseline pool-based active learning that queries unlabeled points uniformly at random each round., name: uncertainty_sampling
description: Queries samples with highest predictive entropy from current logistic regression model., name: margin_sampling
description: Queries samples with smallest top-2 class probability margin (binary: |p-0.5|)., name: qbc
description: Query-by-committee with 5 bootstrapped logistic regression committee members; selects by vote entropy/disagreement., name: expected_error_reduction
description: Approximate one-step expected-error reduction over a capped candidate subset per round using hypothetical labels and expected future log-loss reduction.
baselines: random_sampling is the primary active-learning baseline, passive_full_data logistic regression reference trained once on all labels for context
metrics: name: accuracy_at_budget
direction: maximize
description: Test accuracy at 30% labeled budget, averaged over seeds., name: aulc
direction: maximize
description: Area under the test-accuracy-vs-labeled-fraction curve from initial seed set to 30% budget., name: early_accuracy_10pct
direction: maximize
description: Test accuracy when 10% of pool labels have been acquired.
datasets: name: breast_cancer
source: sklearn.datasets.load_breast_cancer, name: wine
source: sklearn.datasets.load_wine, name: digits
source: sklearn.datasets.load_digits
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 600

Requirements:

Scoring rubric:
requirements: A credible experiment studying active-learning query strategies (random, uncertainty, margin, QBC, expected error reduction, or equivalents) for logistic regression: strategies are implemented as distinct code paths, execution covers multiple datasets with repeated seeds under a fixed budget, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative acquisition functions or base classifiers that preserve the scientific question should be credited.
sub_tasks: requirements: Active-learning strategies and shared logistic-regression pipeline are implemented correctly.
sub_tasks: requirements: The submission implements multiple distinct query conditions — typically random plus several of {uncertainty, margin, QBC, expected error reduction} — with genuinely different acquisition logic.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: There is a pool-based active-learning loop with an initial labeled seed set, iterative querying, model retraining/update, and stopping at a defined label budget.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: The submission uses multiple datasets (sklearn built-ins or comparable) with consistent train/pool/test handling for all strategies.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: Execution emits budget-aware performance metrics.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact containing numeric accuracy-at-budget and area-under-learning-curve (or equivalents) for each implemented strategy on at least one dataset.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Each reported (strategy, dataset) result is aggregated over multiple random seeds, with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The run logs performance across multiple budget checkpoints so early-budget accuracy and AULC are computable from recorded traces.
task_category: Code Execution
finegrained_task_category: Logging, Analysis & Presentation, requirements: Quantitative analysis addresses H1/H2/H3 directionally.
sub_tasks: requirements: The submission compares non-random strategies to random sampling at the final budget and conveys whether informative strategies meaningfully outperform random — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports per-dataset AULC rankings and conveys whether expected-error-reduction (or a comparable principled strategy) is at least competitive, never falling below random (H2).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares QBC vs uncertainty sampling at an early-budget checkpoint per dataset and conveys a qualitative verdict (H3).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or report describes setup, reports key numeric outcomes, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (runtime approximations, seed count, dataset scope, strategy sensitivity). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation