Text feature extraction trade-offs: TF-IDF vs count vs hashing with NB/SVM
Domain: ml (natural-language-processing, machine-learning)
Topic: Evaluating text feature extraction methods (TF-IDF, count vectors, hashing) combined with Naive Bayes and SVM for document classification on 20newsgroups
Venue: ARC-Bench 2026
Headline metric: f1_score (maximize)

Research briefing:
Sparse bag-of-words pipelines remain strong baselines for document
classification, especially under strict CPU and latency constraints.
Three common feature extractors — CountVectorizer, TfidfVectorizer, and
HashingVectorizer — define different bias/variance and efficiency trade-offs.
Count vectors preserve raw frequency signals useful for generative models;
TF-IDF often improves linear margin classifiers by downweighting common
tokens; hashing avoids vocabulary construction and can reduce memory and
fit-time at the cost of collisions.

Classifier choice interacts strongly with the representation. Multinomial
Naive Bayes (MNB) is typically paired with nonnegative count-like features,
while linear SVM variants often benefit from TF-IDF normalization. In
practical workflows, teams frequently need to choose between slightly better
macro-F1 and much faster runtime. This makes a pure "best score" benchmark
incomplete unless paired with timing and robustness checks.

A credible CPU-scale study should compare multiple extractor+classifier
combinations on at least two document datasets available directly through
sklearn (or trivially synthesized text), use fixed train/test splits with
repeated seeds where relevant, and report both predictive quality and
efficiency metrics. The experiment should explicitly test whether TF-IDF +
linear SVM provides the strongest macro-F1 while hashing offers meaningful
speed advantages with limited degradation.

*Does TF-IDF with linear SVM deliver the best macro-F1 on sklearn text benchmarks, and is HashingVectorizer a worthwhile speed/accuracy trade-off versus vocabulary-based features?*

Hypotheses:
statement: TF-IDF + linear SVM achieves the highest macro-F1 among implemented conditions on at least 2 of 3 evaluated text datasets., statement: HashingVectorizer-based pipelines reduce vectorization+fit wall-clock time by at least 20% versus the corresponding TF-IDF pipeline on at least 2 of 3 datasets., statement: Multinomial Naive Bayes with count vectors attains macro-F1 within 0.05 absolute of TF-IDF + linear SVM on at least 1 of 3 datasets.

Experiment design:
research_question: Across sklearn-accessible document datasets, what are the accuracy-efficiency trade-offs between count, TF-IDF, and hashing features when paired with Multinomial Naive Bayes and linear SVM classifiers?
conditions: name: count_mnb
description: CountVectorizer (word unigrams, min_df=2) + MultinomialNB(alpha=1.0)., name: tfidf_mnb
description: TfidfVectorizer (word unigrams, min_df=2, norm=l2) + MultinomialNB(alpha=1.0)., name: tfidf_linear_svm
description: TfidfVectorizer (word unigrams, min_df=2, norm=l2) + LinearSVC(C=1.0)., name: hashing_linear_svm
description: HashingVectorizer (word unigrams, n_features=2^18, alternate_sign=False) + LinearSVC(C=1.0).
baselines: count_mnb as the classic sparse-text baseline, tfidf_mnb as a same-classifier feature-ablation baseline
metrics: name: macro_f1
direction: maximize
description: Macro-averaged F1 score on held-out test split., name: accuracy
direction: maximize
description: Overall test accuracy for comparability with prior text benchmarks., name: fit_predict_time_sec
direction: minimize
description: End-to-end wall-clock time for vectorization, model fit, and test prediction.
datasets: name: 20newsgroups_4class
source: sklearn.datasets.fetch_20newsgroups with 4 categories (subset=train/test, remove=("headers","footers","quotes")), name: 20newsgroups_8class
source: sklearn.datasets.fetch_20newsgroups with 8 categories (subset=train/test, remove=("headers","footers","quotes")), name: synthetic_topic_docs
source: Trivially synthesized corpus via templated sentence generation for 4 classes, 2000 documents total
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 600

Requirements:

Scoring rubric:
requirements: A credible experiment studying text-feature-extraction trade-offs (count, TF-IDF, hashing) with Naive Bayes and linear SVM for document classification: multiple vectorizer+classifier pipelines are implemented, runs execute on multiple datasets, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. Equivalent vectorizer/classifier substitutes (e.g., SGDClassifier in place of LinearSVC) should be credited when the scientific question is preserved.
sub_tasks: requirements: Feature extractors and classifiers are implemented as distinct pipeline code paths enabling fair comparison.
sub_tasks: requirements: The submission implements multiple text-classification pipelines combining a vectorizer (count, TF-IDF, or hashing) with a classifier (Multinomial Naive Bayes or LinearSVC-style linear model) as distinct code paths rather than a single reused model without feature changes.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission uses multiple document datasets (including at least one 20newsgroups subset or comparable) with explicit train/test usage and preprocessing applied consistently across conditions.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The experiment keeps comparable tokenization / n-gram choices across vectorizers where applicable and uses reasonable, shared hyperparameters (e.g., C for LinearSVC, alpha for MNB) so condition comparisons are fair.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: Execution logs predictive-quality and efficiency metrics per condition.
sub_tasks: requirements: Execution produces a machine-readable metrics artifact with numeric macro-F1 (or equivalent) and a wall-clock fit+predict time measure for each implemented (condition, dataset) cell.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution repeats evaluation where randomness exists (e.g., synthetic-data generation or classifier random_state) across multiple seeds and reports dispersion for macro-F1. Deterministic pipelines may report a single run if no stochasticity is involved.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Results directly address H1/H2/H3 with an interpretable narrative of accuracy-efficiency trade-offs.
sub_tasks: requirements: The submission ranks conditions by macro-F1 per dataset and conveys whether TF-IDF + linear SVM (or equivalent) tends to be the top pipeline — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares the runtime of hashing-vectorizer-based pipelines vs TF-IDF-based ones and conveys whether hashing yields a meaningful runtime reduction.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares count + Multinomial NB against TF-IDF + linear SVM on macro-F1 and conveys whether count-MNB is approximately competitive on at least one dataset — judge directionally against H3.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup describes methods and datasets, reports the key metric numbers, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations such as dataset scope, preprocessing choices, timing noise, and hyperparameter coverage. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation