abelcetina/u2t01-bert-ud-ewt-pos

BERT adapted to part-of-speech tagging (Universal Dependencies, 17 UPOS tags) — full fine-tuning

abelcetina/u2t01-bert-ud-ewt-pos — bert-base-uncased adapted to part-of-speech tagging (Universal Dependencies, 17 UPOS tags) with full fine-tuning.

Produced for the university assignment U2T01: Adapting BERT for NLP tasks. The companion repository contains the training code, the raw results file behind every number below, and the comparison against the alternative adaptation methods.

Model description

• Base encoder: bert-base-uncased (108,904,721 parameters in total).
• Task: part-of-speech tagging (Universal Dependencies, 17 UPOS tags).
• Adaptation method: full fine-tuning. Every parameter of the encoder and of the task head received gradients, with a small learning rate on the body and a larger one on the randomly initialised head.
• Head: a linear token-classification head over every token representation.
• Label set: 17 UPOS tags: ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, SYM, VERB, X
• Trained by: U2T01 team · generated 2026-09-25.

Intended use

• Research and teaching: reproducing the feature-based versus fine-tuning comparison of BERT section 5.3 across four task types.
• Inference on English text that resembles the training domain (English web text: weblogs, newsgroups, e-mail, reviews and question answers).
• Usage:

from transformers import pipeline
tagger = pipeline("token-classification", model="abelcetina/u2t01-bert-ud-ewt-pos", aggregation_strategy="simple")
print(tagger("Angela Merkel visited the European Central Bank in Frankfurt."))

Out-of-scope use

• Any decision affecting a person (hiring, moderation with consequences, credit, legal or medical decisions). This is a coursework model validated on one academic benchmark with a single seed.
• Languages other than English, and domains far from English web text: weblogs, newsgroups, e-mail, reviews and question answers — performance is not characterised there and is expected to drop.
• Extracting facts about the world: for the QA model, the answer is a span copied from the context passage it is given, never knowledge the model holds.
• Commercial redistribution without checking the training-data licence below.

Training data

Dataset	UD English-EWT (official CoNLL-U files)
Identifier used in code	universal_dependencies/UD_English-EWT
Licence	CC BY-SA 4.0
Splits	the official en_ewt-ud-{train,dev,test}.conllu splits
Training examples actually used	12,544
Evaluation split reported below	validation

Parsed directly from the official CoNLL-U files of UD English-EWT instead of the universal_dependencies dataset, which is a loading script and therefore no longer loads under datasets >= 4.0. Same treebank, same tag set, same splits.

Adaptation method and what was frozen

• Method flag: --method full
• Trainable components: the whole BERT encoder (embeddings + 12 layers) and the task head
• Frozen components: nothing

Parameters	Value
Total	108,904,721
Trainable	108,904,721
Trainable share	100 %

Hyper-parameters

Hyper-parameter	Value
Epochs	3
Batch size	32
Head learning rate	0.001
Body learning rate	0.00002
Weight decay	0.01
Max sequence length	128
Top layers unfrozen	all 12 encoder layers (full fine-tuning)
Seed	42

Evaluation results

Metrics computed by src/metrics.py; the raw values live in pos__full__seed42.json.

Evaluation split (validation)

Metric	Value
loss	0.1058
accuracy	0.9727
macro_f1	0.9226

Test split

Metric	Value
loss	0.0956
accuracy	0.9742
macro_f1	0.9324

Headline number: accuracy = 0.9727 (macro_f1 = 0.9226).

The comparison against the other adaptation methods of this task, with the gap in percentage points and the noise flag, is in report/results_tables.md of the companion code repository (Table 1, Table 2.pos and Table 3). A gap below one point there is reported as noise, not as an improvement.

Limitations and bias

• Dataset bias. English Web Treebank: informal web genres, so punctuation, emoticons and non-standard spelling are frequent, while formal edited prose is under-represented. The model inherits both that domain skew and the social biases of the bert-base-uncased pre-training corpus (BookCorpus and English Wikipedia), which is known to encode gender, ethnic and occupational stereotypes. Nothing here mitigates or measures them.
• Single seed. Every number on this card comes from one run with seed 42. No variance estimate exists, so a difference of roughly one point against another configuration must be read as noise rather than as an improvement.
• Subsampling. Training used 12,544 examples (hyperparams.train_size). Parsed directly from the official CoNLL-U files of UD English-EWT instead of the universal_dependencies dataset, which is a loading script and therefore no longer loads under datasets >= 4.0. Same treebank, same tag set, same splits.
• Sequence truncation. Inputs were truncated to 128 tokens; longer inputs lose their tail at inference too.
• Frozen-encoder caveat (feature-based models only). Not applicable: this model was fine-tuned, so the encoder weights moved.
• Only a validation (and where available test) split was measured. No robustness, adversarial, out-of-domain or fairness evaluation was performed.

Environmental and compute footprint

Hardware	1 × Tesla T4 (Google Colab), CUDA 12.8
Training wall time	149.6 s (~2.5 min) over 1,176 steps
Seconds per step	0.127
Peak GPU memory	3073.1 MB
Software	transformers 5.16.1 · torch 2.11.0+cu128 · datasets 4.8.5
Run timestamp	2026-09-25T07:59:36

A single short fine-tuning run on one T4 is a small footprint in absolute terms, and that is the point of this assignment: the cost-versus-quality table in the companion report shows how much of the quality survives when only a fraction of the parameters is trained, which is the part that scales. No CO₂e figure is claimed here, because the energy mix of the Colab region is unknown.

References and citation

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT. arXiv:1810.04805. Section 5.3 is the feature-based versus fine-tuning comparison this model is part of.
• Silveira, N. et al. (2014). A Gold Standard Dependency Corpus for English. LREC. (English Web Treebank / UD English-EWT.)
• Tunstall, L., von Werra, L., & Wolf, T. (2022). Natural Language Processing with Transformers, chapters 1–3. O'Reilly.
• Hugging Face, Fine-tuning a pretrained model: https://huggingface.co/docs/transformers/training

@inproceedings{devlin2019bert,
  title     = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author    = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {Proceedings of NAACL-HLT},
  pages     = {4171--4186},
  year      = {2019},
  url       = {https://arxiv.org/abs/1810.04805}
}

Cite this adapted model as:

@misc{u2t01_pos_full,
  title  = {BERT adapted to part-of-speech tagging (Universal Dependencies, 17 UPOS tags) — full fine-tuning},
  author = {U2T01 team},
  year   = {2026},
  note   = {U2T01: Adapting BERT for NLP tasks. Adaptation method: full. Seed 42.},
  url    = {https://huggingface.co/abelcetina/u2t01-bert-ud-ewt-pos}
}