BERT adapted to topic classification (4 classes) — full fine-tuning

abelcetina/u2t01-bert-agnews — bert-base-uncased adapted to topic classification (4 classes) with full fine-tuning.

Produced for the university assignment U2T01: Adapting BERT for NLP tasks. The companion repository contains the training code, the raw results file behind every number below, and the comparison against the alternative adaptation methods.

Model description

Base encoder: bert-base-uncased (109,485,316 parameters in total).
Task: topic classification (4 classes).
Adaptation method: full fine-tuning. Every parameter of the encoder and of the task head received gradients, with a small learning rate on the body and a larger one on the randomly initialised head.
Head: a linear classification head over the pooled sentence representation.
Label set: 4 topics: World, Sports, Business, Sci/Tech
Trained by: U2T01 team · generated 2026-09-25.

Intended use

Research and teaching: reproducing the feature-based versus fine-tuning comparison of BERT section 5.3 across four task types.
Inference on English text that resembles the training domain (short English news headlines and lead sentences).
Usage:

from transformers import pipeline
clf = pipeline("text-classification", model="abelcetina/u2t01-bert-agnews")
print(clf("Stocks rallied as the central bank held rates steady."))

Out-of-scope use

Any decision affecting a person (hiring, moderation with consequences, credit, legal or medical decisions). This is a coursework model validated on one academic benchmark with a single seed.
Languages other than English, and domains far from short English news headlines and lead sentences — performance is not characterised there and is expected to drop.
Extracting facts about the world: for the QA model, the answer is a span copied from the context passage it is given, never knowledge the model holds.
Commercial redistribution without checking the training-data licence below.

Training data


Dataset	AG News
Identifier used in code	`fancyzhx/ag_news`
Licence	the Hub card carries the terms of the original AG news corpus used by Zhang, Zhao & LeCun (2015): free for non-commercial research use
Splits	train 120,000 · test 7,600 (a subsample was used for training, see below)
Training examples actually used	19,000
Evaluation split reported below	`validation`

Only a subsample of the training split was used, so absolute accuracy is below the published state of the art; the point of the experiment is the comparison between adaptation methods at a fixed data size.

Adaptation method and what was frozen

Method flag: --method full
Trainable components: the whole BERT encoder (embeddings + 12 layers) and the task head
Frozen components: nothing

Parameters	Value
Total	109,485,316
Trainable	109,485,316
Trainable share	100 %

Hyper-parameters

Hyper-parameter	Value
Epochs	2
Batch size	32
Head learning rate	0.001
Body learning rate	0.00002
Weight decay	0.01
Max sequence length	128
Top layers unfrozen	all 12 encoder layers (full fine-tuning)
Seed	42

Evaluation results

Metrics computed by src/metrics.py; the raw values live in agnews__full__seed42.json.

Evaluation split (validation)

Metric	Value
`loss`	0.2608
`accuracy`	0.9190
`macro_f1`	0.9194

Test split

Metric	Value
`loss`	0.2277
`accuracy`	0.9293
`macro_f1`	0.9293

Headline number: accuracy = 0.919 (macro_f1 = 0.9194).

The comparison against the other adaptation methods of this task, with the gap in percentage points and the noise flag, is in report/results_tables.md of the companion code repository (Table 1, Table 2.agnews and Table 3). A gap below one point there is reported as noise, not as an improvement.

Limitations and bias

Dataset bias. AG News is a news-wire corpus: the four topics reflect an editorial taxonomy from the mid-2000s, and entities, events and phrasing are those of that period and of English-language news. The model inherits both that domain skew and the social biases of the bert-base-uncased pre-training corpus (BookCorpus and English Wikipedia), which is known to encode gender, ethnic and occupational stereotypes. Nothing here mitigates or measures them.
Single seed. Every number on this card comes from one run with seed 42. No variance estimate exists, so a difference of roughly one point against another configuration must be read as noise rather than as an improvement.
Subsampling. Training used 19,000 examples (hyperparams.train_size). Only a subsample of the training split was used, so absolute accuracy is below the published state of the art; the point of the experiment is the comparison between adaptation methods at a fixed data size.
Sequence truncation. Inputs were truncated to 128 tokens; longer inputs lose their tail at inference too.
Frozen-encoder caveat (feature-based models only). Not applicable: this model was fine-tuned, so the encoder weights moved.
Only a validation (and where available test) split was measured. No robustness, adversarial, out-of-domain or fairness evaluation was performed.

Environmental and compute footprint


Hardware	1 × Tesla T4 (Google Colab), CUDA 12.8
Training wall time	213.8 s (~3.6 min) over 1,188 steps
Seconds per step	0.18
Peak GPU memory	3104.5 MB
Software	transformers 5.16.1 · torch 2.11.0+cu128 · datasets 4.8.5
Run timestamp	2026-09-25T07:05:06

A single short fine-tuning run on one T4 is a small footprint in absolute terms, and that is the point of this assignment: the cost-versus-quality table in the companion report shows how much of the quality survives when only a fraction of the parameters is trained, which is the part that scales. No CO₂e figure is claimed here, because the energy mix of the Colab region is unknown.

References and citation

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT. arXiv:1810.04805. Section 5.3 is the feature-based versus fine-tuning comparison this model is part of.
Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level Convolutional Networks for Text Classification. NeurIPS. arXiv:1509.01626.
Tunstall, L., von Werra, L., & Wolf, T. (2022). Natural Language Processing with Transformers, chapters 1–3. O'Reilly.
Hugging Face, Fine-tuning a pretrained model: https://huggingface.co/docs/transformers/training

@inproceedings{devlin2019bert,
  title     = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {Proceedings of NAACL-HLT},
  pages     = {4171--4186},
  year      = {2019},
  url       = {https://arxiv.org/abs/1810.04805}
}

Cite this adapted model as:

@misc{u2t01_agnews_full,
  title  = {BERT adapted to topic classification (4 classes) — full fine-tuning},
  author = {U2T01 team},
  year   = {2026},
  note   = {U2T01: Adapting BERT for NLP tasks. Adaptation method: full. Seed 42.},
  url    = {https://huggingface.co/abelcetina/u2t01-bert-agnews}
}