bert-base-conll03-ner

Fine-tuning of bert-base-uncased for Named Entity Recognition (NER), delivered as part of assignment U2T01 (Adapting BERT for NLP tasks — Trends in Data Science, Unit 2, Universidad Politécnica de Yucatán).

Model description

bert-base-uncased with a token-level linear classification head over the last hidden state of every token, predicting one of 9 BIO-scheme entity tags (person, organization, location, miscellaneous, or none).

Training data

• Dataset: CoNLL-2003 (lhoestq/conll2003), using its own official train/validation/test splits.
• Subword alignment: since the dataset labels whole words but the tokenizer can split a word into multiple subwords, the label is placed on the first subword of each word only; all other positions (continuation subwords, [CLS], [SEP], padding) get label -100, which CrossEntropyLoss ignores.

Training procedure

Two adaptation methods were trained and compared; full fine-tuning is the delivered model.

Hyperparameter	Value
Base model	bert-base-uncased (110M params)
Method	Full fine-tuning (BERT body + head, jointly)
Learning rate (head)	1e-3
Learning rate (BERT body)	2e-5
Epochs	4
Batch size	32 (train) / 64 (eval)
Seed	42
Trainable parameters	108,898,569
Training time	4.4 min (single T4 GPU)

Compared alternative (not delivered): partial fine-tuning, freezing the entire BERT body except its last 2 encoder layers (14,182,665 trainable params, 1.9 min).

Evaluation results

Metric: seqeval F1 (evaluated at the entity level, e.g. "New York" counts as one LOC entity rather than two tokens — important because the dominant "O" class would make per-token accuracy misleadingly high).

Method	Val F1	Test F1	Test Precision	Test Recall
Partial fine-tuning (last 2 layers + head)	88.75%	85.37%	83.58%	87.23%
Full fine-tuning (delivered)	94.27%	90.16%	89.36%	90.97%

The 4.79-point F1 gap on test is above the ±1–3 point noise margin. Note the consistent drop from validation to test in both methods — a known property of the CoNLL-2003 test split being harder than its validation split, not a sign of overfitting.

Intended uses & limitations

• Intended use: named entity recognition (person, organization, location, misc) on English news-style text similar to CoNLL-2003, for coursework/research.
• Limitations: trained only on CoNLL-2003 (Reuters newswire from the 1990s); may not generalize well to informal text, social media, or domains with different entity distributions (e.g. biomedical, legal). Single seed, 4 epochs, no extensive hyperparameter search. Inherits biases from the source corpus and from bert-base-uncased's pretraining data.

References

• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
• Hugging Face. Fine-tune a pretrained model. https://huggingface.co/docs/transformers/training
• Dataset: lhoestq/conll2003