Topic-model comparison on small 20newsgroups subsets: LDA vs NMF vs LSA
Domain: ml (natural-language-processing, topic-modeling)
Topic: Comparing topic models (LDA, NMF, LSA) on a small subset of 20newsgroups by topic coherence (c_v) and document-cluster adjusted rand score
Venue: ARC-Bench 2026
Headline metric: coherence_cv (maximize)

Research briefing:
Topic modeling methods are often compared using either intrinsic quality
measures (such as topic coherence) or extrinsic utility (such as how well
document representations align with known labels), but the two views can
disagree. Latent Dirichlet Allocation (LDA), Non-negative Matrix
Factorization (NMF), and Latent Semantic Analysis (LSA/SVD) each induce
different assumptions over term-document structure and may therefore rank
differently depending on metric choice.

On CPU-constrained benchmarks, a practical study should use a compact text
corpus and a controlled preprocessing pipeline so that runtime stays short
while still yielding meaningful distinctions. A small subset of
20newsgroups is ideal because it has accessible labels for external
clustering evaluation and enough lexical diversity for coherence analysis.
Using multiple subset difficulties (well-separated vs more confusable
categories) helps test robustness of conclusions.

A credible experiment compares LDA, NMF, and LSA under matched topic counts
and vectorization settings, then reports both c_v-like coherence and
adjusted rand index (ARI) from document-cluster assignments against true
newsgroup labels. Because exact c_v implementations are not native in
sklearn, an explicit approximation based on sliding-window co-occurrence
and normalized PMI should be documented and applied consistently across
methods. Repeated runs for stochastic models are needed to avoid
over-interpreting single-seed noise.

The core question is whether better intrinsic coherence implies better
label alignment on small real-world corpora, and which model offers the
best trade-off under strict CPU budgets.

*Do LDA, NMF, and LSA produce different rankings on coherence versus ARI on small 20newsgroups subsets, and does NMF provide the strongest coherence-structure trade-off under CPU limits?*

Hypotheses:
statement: NMF achieves higher mean coherence_cv than LSA on at least 2 of 3 evaluated 20newsgroups subsets when using the same number of topics and preprocessing pipeline., statement: Across the three methods (LDA, NMF, LSA), the method with the highest coherence_cv is also the highest-ARI method on no more than 1 of the 3 subsets, indicating weak metric agreement., statement: LDA achieves mean ARI at least 0.03 higher than LSA on at least 2 of 3 subsets when document clusters are obtained by argmax topic assignment.

Experiment design:
research_question: How do LDA, NMF, and LSA compare on coherence_cv versus document-cluster ARI over small 20newsgroups subsets, and do intrinsic/extrinsic rankings diverge?
conditions: name: lda_k4
description: LatentDirichletAllocation with n_components=4, max_iter=20, learning_method='batch', CountVectorizer features, repeated over 3 seeds., name: nmf_k4
description: NMF with n_components=4, init='nndsvda', solver='cd', max_iter=300 on TF-IDF features, repeated over 3 seeds., name: lsa_k4
description: TruncatedSVD (LSA) with n_components=4 on TF-IDF features; document-topic embeddings clustered via argmax after non-negative shift or via KMeans(k=4) with fixed seed., name: nmf_k6_sensitivity
description: NMF sensitivity condition with n_components=6 to test topic-count robustness for coherence and ARI trends.
baselines: lsa_k4 as a linear-algebra baseline without probabilistic topic assumptions, lda_k4 as a probabilistic-topic baseline
metrics: name: coherence_cv
direction: maximize
description: Approximate c_v coherence computed from top words per topic using sliding-window co-occurrence and NPMI aggregation, averaged across topics and seeds., name: ari
direction: maximize
description: Adjusted Rand Index between predicted document clusters (topic argmax or equivalent) and true newsgroup labels., name: runtime_sec
direction: minimize
description: Wall-clock training + inference time per condition/dataset cell on single CPU core.
datasets: name: 20ng_easy4
source: sklearn.datasets.fetch_20newsgroups with categories=['sci.space','rec.autos','comp.graphics','talk.politics.misc'], name: 20ng_related4
source: sklearn.datasets.fetch_20newsgroups with categories=['comp.graphics','comp.os.ms-windows.misc','comp.sys.ibm.pc.hardware','comp.sys.mac.hardware'], name: 20ng_science4
source: sklearn.datasets.fetch_20newsgroups with categories=['sci.space','sci.med','sci.electronics','sci.crypt']
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 720

Requirements:

Scoring rubric:
requirements: A credible experiment comparing LDA, NMF, and LSA topic models on small 20newsgroups subsets: conditions are implemented, execution reports coherence and ARI on multiple subsets with repeated seeds, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative topic-model implementations or coherence approximations that test the same scientific question should be credited.
sub_tasks: requirements: Topic-model conditions and dataset pipeline are implemented correctly.
sub_tasks: requirements: The submission implements LDA, NMF, and LSA (or equivalents such as a truncated SVD for LSA) as distinct modeling code paths with a matched topic-count setting.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission loads multiple 20newsgroups subsets (or comparable document corpora) with consistent text preprocessing/vectorization documented in code.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: The code computes a coherence-style metric (e.g., c_v or a documented approximation) and ARI from document-cluster assignments for each method.
task_category: Code Development
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution produces comparable numeric outputs for all implemented methods.
sub_tasks: requirements: Execution evaluates the core methods on multiple datasets and writes per-method, per-dataset coherence and ARI values to a machine-readable artifact.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution repeats stochastic methods with multiple seeds per dataset, reports dispersion, and logs a wall-clock timing measure per condition. Honest small-seed runs with variance reported are preferable to a single run.
task_category: Code Execution
finegrained_task_category: Experimental Setup, requirements: Quantitative analysis addresses H1/H2/H3 directionally.
sub_tasks: requirements: The submission compares NMF vs LSA on coherence for each evaluated dataset and conveys whether NMF tends to yield better coherence — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission checks ranking agreement between coherence and ARI across methods per dataset and conveys whether coherence-ranking and ARI-ranking tend to diverge (H2).
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission reports LDA vs LSA ARI deltas per dataset and conveys whether LDA tends to achieve meaningfully higher document-cluster ARI (H3).
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup describes methods and preprocessing, reports coherence/ARI/runtime results, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (subset choice, coherence approximation validity, seed count, clustering-assignment assumptions). No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation