bert-base-squad-qa

Fine-tuning of bert-base-uncased for extractive Question Answering, delivered as part of assignment U2T01 (Adapting BERT for NLP tasks — Trends in Data Science, Unit 2, Universidad Politécnica de Yucatán).

Model description

bert-base-uncased with a qa_outputs head that predicts, per token, the probability of being the start and the end of the answer span within the given context.

Training data

• Dataset: SQuAD v1.1 (rajpurkar/squad), subsampled to 15,000 examples from the official train split (as required by the assignment), with a 90/10 split used for training/ validation. SQuAD's official validation set (10,570 questions, never seen during training) was reserved as the final test set.
• Preprocessing uses a sliding window (stride) since contexts can exceed the maximum input length; each window is labeled with the start/end position of the answer inside that window, or (0, 0) if the answer does not fit.

Training procedure

Two adaptation methods were trained and compared; full fine-tuning is the delivered model, since it is necessary (not optional) for this task — partial fine-tuning underperforms by a wide margin.

Hyperparameter	Value
Base model	bert-base-uncased (110M params)
Method	Full fine-tuning (BERT body + head, jointly)
Learning rate (head)	1e-3
Learning rate (BERT body)	2e-5
Epochs	3
Batch size	16 (train) / 32 (eval)
Seed	42
Trainable parameters	108,893,186
Training time	13.5 min (single T4 GPU)

Compared alternative (not delivered): partial fine-tuning, freezing the entire BERT body except its last 2 encoder layers (14,177,282 trainable params, 5.9 min).

Evaluation results

Metrics: official SQuAD Exact Match (EM) and word-level F1.

Method	Val EM	Val F1	Test EM	Test F1
Partial fine-tuning (last 2 layers + head)	39.53%	55.05%	45.91%	59.45%
Full fine-tuning (delivered)	59.27%	73.79%	71.16%	81.06%

The 21.61-point F1 gap on test is by far the largest gap of the four tasks in this project (roughly 5x the gap seen in classification, NER, or POS), reflecting that extractive QA requires jointly reasoning over the question and locating a span in the context — something only 2 unfrozen layers cannot capture well.

Note on scale: this model was trained on 15k examples rather than the full 87.6k SQuAD train set, so its 81.06 test F1 is below the ~88 F1 reported in the literature for BERT-base trained on the full dataset — a result consistent with the reduced amount of training data.

Intended uses & limitations

• Intended use: extractive question answering over short English passages, in the style of SQuAD v1.1, for coursework/research.
• Limitations: trained on only 15k (of 87.6k) SQuAD examples, so accuracy is meaningfully below full-dataset BERT-base benchmarks. Only answers questions whose answer is a contiguous span present in the given context (no "no answer" case, as in SQuAD v1.1, and no closed-book/generative QA). Single seed, 3 epochs, no extensive hyperparameter search.

References

• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
• Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text.
• Hugging Face. Fine-tune a pretrained model. https://huggingface.co/docs/transformers/training
• Dataset: rajpurkar/squad