Fine-tuned distilbert-base-uncased on SQuAD - Best F1: 84.5885

README.md

@@ -0,0 +1,88 @@
+---
+language: en
+tags:
+- question-answering
+- squad
+- transformers
+datasets:
+- squad
+metrics:
+- exact_match
+- f1
+model-index:
+- name: HariomSahu/distilbert-base-uncased-squadv1-adam-lin-e526
+  results:
+  - task:
+      type: question-answering
+      name: Question Answering
+    dataset:
+      name: SQuAD
+      type: squad
+    metrics:
+    - type: exact_match
+      value: 76.35761589403974
+    - type: f1
+      value: N/A
+---
+
+# distilbert-base-uncased fine-tuned on SQuAD
+
+This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the SQuAD dataset.
+
+## Training Details
+
+### Training Hyperparameters
+
+- **Model**: distilbert-base-uncased
+- **Dataset**: SQuAD
+- **Optimizer**: adamw
+- **Learning Rate Scheduler**: linear
+- **Learning Rate**: 2e-05
+- **Batch Size**: 16 per device
+- **Total Batch Size**: 64
+- **Epochs**: 5 (with early stopping)
+- **Weight Decay**: 0.01
+- **Warmup Ratio**: 0.1
+- **Max Gradient Norm**: 1.0
+
+### Early Stopping
+
+- **Patience**: 3
+- **Metric**: exact_match
+- **Best Epoch**: 3
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering
+
+tokenizer = AutoTokenizer.from_pretrained("HariomSahu/distilbert-base-uncased-squadv1-adam-lin-e526")
+model = AutoModelForQuestionAnswering.from_pretrained("HariomSahu/distilbert-base-uncased-squadv1-adam-lin-e526")
+
+# Example usage
+question = "What is the capital of France?"
+context = "France is a country in Europe. Its capital city is Paris."
+
+inputs = tokenizer(question, context, return_tensors="pt")
+outputs = model(**inputs)
+
+# Get answer
+start_scores, end_scores = outputs.start_logits, outputs.end_logits
+start_index = start_scores.argmax()
+end_index = end_scores.argmax()
+answer = tokenizer.decode(inputs["input_ids"][0][start_index:end_index+1])
+print(f"Answer: {answer}")
+```
+
+## Evaluation Results
+
+The model achieved the following results on the evaluation set:
+
+- **Exact Match**: 76.3103
+- **F1 Score**: 84.5885
+
+## Training Configuration Hash
+
+Config Hash: e5265f15
+
+This hash can be used to reproduce the exact training configuration.

config.json

@@ -0,0 +1,23 @@
+{
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForQuestionAnswering"
+  ],
+  "attention_dropout": 0.1,
+  "dim": 768,
+  "dropout": 0.1,
+  "hidden_dim": 3072,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.54.0",
+  "vocab_size": 30522
+}

eval_results.json

@@ -0,0 +1,4 @@
+{
+  "exact_match": 76.35761589403974,
+  "f1": 84.47050870302236
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee5fbbb8aa6f57483725bc64cbd4e3d466479e6ae4441a270d14720e6e0812d7
+size 265470032

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,56 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "DistilBertTokenizer",
+  "unk_token": "[UNK]"
+}

training_config.json

@@ -0,0 +1,46 @@
+{
+  "squad_v2": false,
+  "model_checkpoint": "distilbert-base-uncased",
+  "max_length": 384,
+  "doc_stride": 128,
+  "n_best_size": 20,
+  "max_answer_length": 30,
+  "batch_size": 16,
+  "num_epochs": 5,
+  "learning_rate": 2e-05,
+  "weight_decay": 0.01,
+  "warmup_ratio": 0.1,
+  "gradient_accumulation_steps": 1,
+  "max_grad_norm": 1.0,
+  "optimizer_type": "adamw",
+  "optimizer_betas": [
+    0.9,
+    0.999
+  ],
+  "optimizer_eps": 1e-08,
+  "scheduler_type": "linear",
+  "scheduler_power": 1.0,
+  "scheduler_eta_min": 0.0,
+  "early_stopping_patience": 3,
+  "early_stopping_threshold": 0.001,
+  "early_stopping_metric": "exact_match",
+  "log_interval": 50,
+  "eval_steps": null,
+  "save_steps": null,
+  "save_total_limit": 3,
+  "wandb_project": "question-answering-enhanced",
+  "wandb_entity": null,
+  "use_wandb": true,
+  "wandb_tags": [
+    "question-answering",
+    "squad",
+    "multi-gpu"
+  ],
+  "push_to_hub": true,
+  "hub_model_id": null,
+  "hub_private": false,
+  "hub_model_name_max_length": 50,
+  "seed": 42,
+  "dataloader_num_workers": 0,
+  "dataloader_pin_memory": true
+}