bert-base-agnews-topic-classification

Full fine-tuning of bert-base-uncased for 4-class news topic classification, delivered as part of assignment U2T01 (Adapting BERT for NLP tasks — Trends in Data Science, Unit 2, Universidad Politécnica de Yucatán).

Model description

bert-base-uncased with a linear classification head on top of the [CLS] token's last hidden state, mapping to 4 topic classes: World, Sports, Business, Sci/Tech.

Training data

• Dataset: AG News (fancyzhx/ag_news)
• 10% of the official train split was held out as validation; the official test split was left untouched.
• 4 balanced classes.

Training procedure

Two adaptation methods were trained and compared; full fine-tuning is the delivered model, since it beat the feature-based baseline by a margin well above the run-to-run noise floor (±1–3 points from seed variance).

Hyperparameter	Value
Base model	bert-base-uncased (110M params)
Method	Full fine-tuning (BERT body + head, jointly)
Learning rate (head)	1e-3
Learning rate (BERT body)	2e-5
Epochs	2
Batch size	32 (train) / 64 (eval)
Seed	42
Trainable parameters	109,485,316
Training time	76.4 min (single T4 GPU)

Compared alternative (not delivered): feature-based adaptation — BERT body fully frozen, with a scikit-learn logistic regression trained on the frozen [CLS] embeddings (0 trainable BERT parameters, 17.7 min).

Evaluation results

Method	Val Accuracy	Val F1	Test Accuracy	Test F1
Feature-based (LogReg on frozen [CLS])	90.60%	90.56%	90.24%	90.23%
Full fine-tuning (delivered)	94.92%	94.89%	94.59%	94.60%

The 4.36-point F1 gap on test is above the ±1–3 point noise margin observed for this setup, so it is treated as a real effect of the adaptation method rather than seed variance.

Intended uses & limitations

• Intended use: topic classification of short English news text into the 4 AG News categories, for coursework/research and as a fine-tuning reference example.
• Limitations: trained and evaluated only on AG News; it will not generalize to topics or writing styles outside that domain (e.g. non-English text, other news taxonomies, social media text). Trained for 2 epochs on a single seed — not hyperparameter-tuned for production use. Inherits any biases present in the AG News corpus and in bert-base-uncased's pretraining data.

References

• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
• Hugging Face. Fine-tune a pretrained model. https://huggingface.co/docs/transformers/training
• Dataset: fancyzhx/ag_news