license: apache-2.0
tags:
- generated_from_trainer
- course
- fine-tuned
- bert
datasets:
- squad
model-index:
- name: bert-finetuned-squad
results: []
bert-finetuned-squad
This model is a fine-tuned version of bert-base-cased on the squad dataset.
Model description
The model is based on a pretrained model where fine-tuning was done to setup the model for the actual task (use case).
Intended uses & limitations
Intended use here is to create a working model to teach myself and others about how to use transformers for Questions/Answering tasks. The limitation is that this approach expects input data to always have context, question, answer and to use the syntax [MASK] for this use case.
Training and evaluation data
Training took more time than expected given use case.
Training procedure
Setup files with the expected format from pretrained model (context, question, answer)
display("Context: ", raw_datasets["train"][0]["context"])
display("Question: ", raw_datasets["train"][0]["question"])
display("Answer: ", raw_datasets["train"][0]["answers"])
output for code above
__Context :__ Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.
__Question :__ To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?
__Answer :__ {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]}
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
Framework versions
- Transformers 4.25.1
- Pytorch 1.13.1+cu116
- Datasets 2.8.0
- Tokenizers 0.13.2