AbdelrahmanAkl/distilbert-squad-qa

DistilBERT SQuAD Extractive Question Answering

A DistilBERT model fine-tuned on the SQuAD v1.1 dataset for extractive Question Answering.

Given a question and a context passage, the model predicts the start and end positions of the answer span within the supplied context.

Model Details

• Base model: distilbert-base-uncased
• Architecture: DistilBertForQuestionAnswering
• Task: Extractive Question Answering
• Dataset: SQuAD v1.1
• Parameters: 66,364,418
• Maximum sequence length: 384
• Document stride: 128
• Maximum answer length: 30 tokens

Intended Use

This model is intended for English extractive Question Answering tasks where the answer is expected to appear explicitly within the supplied context.

Potential applications include:

• Reading comprehension
• Document Question Answering
• Information extraction
• Knowledge-based interfaces
• NLP research and experimentation
• Educational Question Answering systems

The model is not intended to generate answers that are not explicitly present in the supplied context.

Training Data

The model was fine-tuned on SQuAD v1.1.

Split	Examples
Training	87,599
Validation	10,570

SQuAD v1.1 consists of question-context pairs with answer spans annotated directly in the corresponding context.

Dataset:

https://rajpurkar.github.io/SQuAD-explorer/

Reference:

Rajpurkar et al., "SQuAD: 100,000+ Questions for Machine Comprehension of Text"

Training Configuration

Base Model: distilbert-base-uncased
Dataset: SQuAD v1.1
Epochs: 2
Batch Size: 8
Learning Rate: 3e-5
Weight Decay: 0.01
Warmup Steps: 500
Maximum Sequence Length: 384
Document Stride: 128
Precision: FP16
Random Seed: 42
Hardware: NVIDIA Tesla T4

Training completed successfully on an NVIDIA Tesla T4 GPU.

Long-Context Processing

The model uses a sliding-window strategy for contexts longer than the maximum sequence length.

The context is divided into overlapping windows using:

Maximum sequence length: 384
Document stride: 128

Each window is evaluated independently, and candidate answer spans are generated only from valid context tokens.

Character offsets are then used to reconstruct the final answer span from the original context.

This allows the inference system to process contexts that exceed the model's maximum input length.

Evaluation

Evaluation was performed on the SQuAD v1.1 validation set.

Metric	Score
Exact Match (EM)	77.31
Token-level F1	85.54

Exact Match

Exact Match measures whether the normalized predicted answer exactly matches the reference answer.

Token-level F1

Token-level F1 measures token overlap between the predicted and reference answers using precision and recall.

These results were measured on the SQuAD v1.1 validation set after fine-tuning.

Usage

The model can be loaded directly with the Hugging Face Transformers library.

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

model_id = "AbdelrahmanAkl/distilbert-squad-qa"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForQuestionAnswering.from_pretrained(model_id)

question = "Where is the University of Notre Dame located?"

context = (
    "The University of Notre Dame is a private Catholic research "
    "university located in Notre Dame, Indiana, United States."
)

inputs = tokenizer(
    question,
    context,
    return_tensors="pt",
    truncation=True,
    max_length=384,
)

with torch.no_grad():
    outputs = model(**inputs)

start_position = torch.argmax(outputs.start_logits, dim=-1).item()
end_position = torch.argmax(outputs.end_logits, dim=-1).item()

answer = tokenizer.decode(
    inputs["input_ids"][0][start_position:end_position + 1],
    skip_special_tokens=True,
)

print(answer)

For long contexts, the production inference implementation in the project repository adds sliding-window processing, candidate generation, answer-span validation, and character-offset reconstruction.

Production Inference

A complete production-oriented inference implementation is available in the project repository:

https://github.com/AbdelrhmanAkl/Question-Answering-with-Transformers

The custom inference engine provides:

• Sliding-window processing
• Dynamic padding
• Start/end candidate generation
• Context-only span filtering
• Maximum answer length control
• Character offset reconstruction
• Candidate ranking
• Inference latency measurement

Live Demo

A Streamlit application using this model is available here:

https://question-answering-with-transformers.streamlit.app/

Limitations

• English language only.
• Extractive Question Answering only.
• Answers must be present in the supplied context.
• The model may return incorrect spans when multiple context passages contain semantically related information.
• Long contexts require multiple sliding-window inference passes.
• The raw span score is based on start and end logits and is not a calibrated probability.
• Performance may vary on domains that differ significantly from SQuAD.

Ethical and Safety Considerations

This model is a research and portfolio implementation based on SQuAD v1.1.

It should not be treated as a general-purpose factual knowledge system. Predictions depend on the supplied context and may be incorrect.

For high-stakes applications, model outputs should be independently verified before being used for decisions or actions.

License

This model repository uses the Apache-2.0 license.

Author

Abdelrahman Akl

GitHub:

https://github.com/AbdelrhmanAkl

LinkedIn:

https://www.linkedin.com/in/abdelrahmanakl/