--- library_name: transformers tags: [] --- # `google-bert/bert-base-uncased` Fine-Tuned on SQuAD # bert_squad Pretrained model on context-based Question Answering using the SQuAD dataset. This model is fine-tuned from the BERT architecture for extracting answers from passages. ### Model Description bert_squad is a transformer-based model trained for context-based question answering tasks. It leverages the pretrained BERT architecture and adapts it for extracting precise answers given a question and a related context. This model uses the Stanford Question Answering Dataset (SQuAD), available via Hugging Face datasets, for training and fine-tuning. The model was trained using free computational resources, demonstrating its accessibility for educational and small-scale research purposes. Fine-tuned by: SADAT PARVEJ, RAFIFA BINTE JAHIR Shared by: SADAT PARVEJ Language(s) (NLP): ENGLISH Finetuned from model: https://huggingface.co/google-bert/bert-base-uncased ## Training Objective The model predicts the most relevant span of text in a given passage that answers a specific question. It fine-tunes BERT's ability to analyze context using supervised data from SQuAD. ### Performance Benchmarks Training Loss: 0.477800 Validation Loss: 0.465936 Exact Match (EM): 87.568590% ## Intended Uses & Limitations This model is designed for tasks such as: Extractive Question Answering Reading comprehension applications Known Limitations: As BERT is inherently a masked language model (MLM), its original pretraining limits its ability for generative tasks or handling queries outside the SQuAD-style question-answering setup. The model's predictions may be biased or overly reliant on the training dataset, as SQuAD comprises structured and fact-based question-answer pairs. ## How to Get Started with the Model Use the code below to get started with the model. ```python import torch from transformers import AutoTokenizer, AutoModelForQuestionAnswering # Load the model and tokenizer model_name = "Sadat07/bert_squad" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForQuestionAnswering.from_pretrained(model_name) device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device) context = """ The person who invented light was Thomas Edison.He was born in 1879. """ question = "When did Thomas Edison invent?" inputs = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=512) input_ids = inputs["input_ids"].to(device) attention_mask = inputs["attention_mask"].to(device) print("Tokenized Input:", tokenizer.decode(input_ids[0])) # Perform inference with torch.no_grad(): outputs = model(input_ids=input_ids, attention_mask=attention_mask) start_scores = outputs.start_logits end_scores = outputs.end_logits # Logits print("Start logits:", start_scores) print("End logits:", end_scores) # Get start and end indices start_idx = torch.argmax(start_scores) end_idx = torch.argmax(end_scores) + 1 # Decode the answer if start_idx >= end_idx: print("Model did not predict a valid answer. Please check context and question.") else: answer = tokenizer.convert_tokens_to_string( tokenizer.convert_ids_to_tokens(input_ids[0][start_idx:end_idx]) ) print(f"Question: {question}") print(f"Answer: {answer}") ``` ## Training Details | Step | Training Loss | Validation Loss | Exact Match | Squad F1 | Start Accuracy | End Accuracy | |------|---------------|-----------------|-------------|----------|----------------|--------------| | 100 | 0.632200 | 0.811809 | 84.749290 | 84.749290| 0.847493 | 0.899243 | | 200 | 0.751500 | 0.627198 | 84.768212 | 84.768212| 0.847682 | 0.899243 | | 300 | 0.662600 | 0.557515 | 86.244087 | 86.244087| 0.862441 | 0.899243 | | 400 | 0.600400 | 0.567693 | 86.177862 | 86.177862| 0.861779 | 0.899243 | | 500 | 0.613200 | 0.523546 | 86.499527 | 86.499527| 0.864995 | 0.899243 | | 600 | 0.495200 | 0.539225 | 86.565752 | 86.565752| 0.865658 | 0.899243 | | 700 | 0.645300 | 0.552358 | 85.354778 | 85.354778| 0.853548 | 0.899243 | | 800 | 0.499100 | 0.562317 | 86.338694 | 86.338694| 0.863387 | 0.899243 | | 900 | 0.482800 | 0.499747 | 86.811731 | 86.811731| 0.868117 | 0.899243 | | 1000 | 0.372800 | 0.543513 | 86.972564 | 86.972564| 0.869726 | 0.900000 | | 1100 | 0.554000 | 0.502747 | 85.969726 | 85.969726| 0.859697 | 0.894797 | | 1200 | 0.459800 | 0.484941 | 87.019868 | 87.019868| 0.870199 | 0.900662 | | 1300 | 0.463600 | 0.477527 | 87.407758 | 87.407758| 0.874078 | 0.899905 | | 1400 | 0.356800 | 0.499119 | 87.549669 | 87.549669| 0.875497 | 0.901608 | | 1500 | 0.494200 | 0.485287 | 87.549669 | 87.549669| 0.875497 | 0.901703 | | 1600 | 0.521100 | 0.466062 | 87.284768 | 87.284768| 0.872848 | 0.899243 | | 1700 | 0.461200 | 0.462704 | 87.540208 | 87.540208| 0.875402 | 0.901419 | | 1800 | 0.415700 | 0.474295 | 87.691580 | 87.691580| 0.876916 | 0.901892 | | 1900 | 0.622900 | 0.462900 | 87.417219 | 87.417219| 0.874172 | 0.901987 | | 2000 | 0.477800 | 0.465936 | 87.568590 | 87.568590| 0.875686 | 0.901892 | ### Training Data The model was trained on the [SQuAD](https://huggingface.co/datasets/squad) dataset, a widely used benchmark for context-based question-answering tasks. It consists of passages from Wikipedia and corresponding questions, with human-annotated answers. During training, the dataset was processed to extract contexts, questions, and answers, ensuring compatibility with the BERT architecture for QA. The training utilized free resources to minimize costs and focus on model efficiency. ### Training Procedure **Training Objective** The model was trained with the objective of performing context-based question answering using the SQuAD dataset. The fine-tuning process adapts BERT's masked language model (MLM) architecture for QA tasks by leveraging its ability to encode contextual relationships between the passage, question, and answer. **Optimization** The training utilized the AdamW optimizer with a linear learning rate scheduler and warm-up steps to ensure effective weight updates and prevent overfitting. The training was run for 2000 steps, with early stopping applied based on the validation loss and exact match score. **Hardware and Resources** Training was conducted on free resources, such as Google Colab or equivalent free GPU resources. While this limited the scale, adjustments in batch size and learning rate were optimized to make the training efficient within these constraints. **Unique Features** The model fine-tuning procedure emphasizes efficient learning, leveraging BERT's pre-trained knowledge while adapting it specifically to QA tasks in a resource-constrained environment. #### Metrics Performance was evaluated using the following metrics: - **Exact Match (EM)**: Measures the percentage of predictions that match the ground-truth answers exactly. - **F1 Score**: Assesses the overlap between the predicted and true answers at a token level, balancing precision and recall. - **Start and End Accuracy**: Tracks the model’s ability to correctly identify the start and end indices of answers within the context. ### Results The model trained on the SQuAD dataset achieved the following key performance metrics: Exact Match (EM): Up to 87.69% F1 Score: Up to 87.69% Validation Loss: Reduced to 0.46 Start Accuracy: Peaked at 87.69% End Accuracy: Peaked at 90.19% #### Summary The model, **bert_squad**, was fine-tuned for context-based question answering using the SQuAD dataset from Hugging Face. Key metrics include an Exact Match (EM) and F1 score of up to **87.69%**, demonstrating strong accuracy. Performance benchmarks show consistent improvement in loss and accuracy over 2000 steps, with validation loss reaching as low as **0.46**. The training utilized free resources, leveraging BERT’s robust pretraining, although BERT’s limitation as a Masked Language Model (MLM) remains a consideration. This work highlights the potential for effective question-answering systems built on pre-existing datasets and infrastructure. ### Model Architecture and Objective The model uses BERT, a pre-trained Transformer-based architecture, fine-tuned for context-based question answering tasks. It aims to predict answers based on the given input text and context. ### Compute Infrastructure #### Hardware GPU: Tesla P100, NVIDIA T4 #### Software Framework: Hugging Face Transformers Dataset: SQuAD (from Hugging Face) Other tools: Python, PyTorch **BibTeX:** ```bibtex @misc{bert_squad_finetune, title = {BERT Fine-tuned for SQuAD}, author = {Your Name or Team Name}, year = {2024}, url = {https://huggingface.co/your-model-repository} } ``` ## Glossary Exact Match (EM): A metric measuring the percentage of predictions that match the ground truth exactly. Masked Language Model (MLM): Pre-training objective for BERT, predicting masked words in input sentences.