--- datasets: - esnli license: apache-2.0 language: - en metrics: - accuracy - f1 - precision - recall model-index: - name: backpack-gpt2-nli results: - task: name: Natural Language Inference type: text-classification dataset: name: e-SNLI type: esnli split: validation metrics: - name: Accuracy type: accuracy value: 0.9006299532615322 - name: F1 type: f1 value: 0.9004261302857443 - name: Precision type: precision value: 0.9004584180714215 - name: Recall type: recall value: 0.9004554220756779 - task: name: Natural Language Inference type: text-classification dataset: name: e-SNLI type: esnli split: test metrics: - name: Accuracy type: accuracy value: 0.8957654723127035 - name: F1 type: f1 value: 0.8954702227331482 - name: Precision type: precision value: 0.8954036872157838 - name: Recall type: recall value: 0.8955997285576146 pipeline_tag: text-classification tags: - Natural Language Inference - Sequence Classification - GPT2 - Backpack - ESNLI --- # Model Card for Backpack-GPT2-NLI This is a fine-tuned version of [backpack-gpt2](https://huggingface.co/stanfordnlp/backpack-gpt2) with a NLI classification head on the [esnli](https://huggingface.co/datasets/esnli) dataset. Results: - On Validation Set: - CrossEntropyLoss: 0.3168 - Accuracy: 0.9006 - F1: 0.9004 - On Test Set: - CrossEntropyLoss: 0.3277 - Accuracy: 0.8958 - F1: 0.8955 ### Model Description - **Developed by:** [Erfan Moosavi Monazzah](https://huggingface.co/ErfanMoosaviMonazzah) - **Model type:** Sequence Classifier - **Language(s) (NLP):** English - **License:** apache-2.0 - **Finetuned from model [optional]:** [Backpack-GPT2](https://huggingface.co/stanfordnlp/backpack-gpt2) ## How to Get Started with the Model ```python from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained('gpt2') tokenizer.pad_token = tokenizer.eos_token def tokenize_function(examples): concatenated_sentences = [f'{premise.strip(".")}. ^ {hypothesis.strip(".")}.' for premise, hypothesis in zip(examples['premise'], examples['hypothesis'])] tokenized_inputs = tokenizer( concatenated_sentences, padding="max_length", truncation=True, max_length=512, return_tensors="pt", ) return tokenized_inputs model = AutoModelForSequenceClassification.from_pretrained('ErfanMoosaviMonazzah/backpack-gpt2-nli', trust_remote_code=True) model.eval() tokenized_sent = tokenize_function({ 'premise':['A boy is jumping on skateboard in the middle of a red bridge.', 'Two women who just had lunch hugging and saying goodbye.', 'Children smiling and waving at camera'], 'hypothesis':['The boy does a skateboarding trick.', 'The friends have just met for the first time in 20 years, and have had a great time catching up.', 'The kids are frowning'] }) model.predict(input_ids=tokenized_sent['input_ids'], attention_mask=tokenized_sent['attention_mask']) ``` ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-5 - train_batch_size: 64 - eval_batch_size: 64 - seed: 2023 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0 - num_epochs: 3 ### Training results |Step |Training Loss|Validation Loss|Precision|Recall |F1 |Accuracy| |------------|-------------|---------------|---------|--------|--------|--------| |512 |0.614900 |0.463713 |0.826792 |0.824639|0.825133|0.824731| |1024 |0.503300 |0.431796 |0.844831 |0.839414|0.839980|0.839565| |1536 |0.475600 |0.400771 |0.848741 |0.847009|0.846287|0.847795| |2048 |0.455900 |0.375981 |0.859064 |0.857357|0.857749|0.857448| |2560 |0.440400 |0.365537 |0.862000 |0.862078|0.861917|0.862426| |3072 |0.433100 |0.365180 |0.864717 |0.859693|0.860237|0.859785| |3584 |0.425100 |0.346340 |0.872312 |0.870635|0.870865|0.870961| |4096 |0.413300 |0.343761 |0.873606 |0.873046|0.873174|0.873298| |4608 |0.412000 |0.344890 |0.882609 |0.882120|0.882255|0.882341| |5120 |0.402600 |0.336744 |0.876463 |0.875629|0.875827|0.875737| |5632 |0.390600 |0.323248 |0.882598 |0.880779|0.881129|0.880817| |6144 |0.388300 |0.338029 |0.877255 |0.877041|0.877126|0.877261| |6656 |0.390800 |0.333301 |0.876357 |0.876362|0.875965|0.876753| |7168 |0.383800 |0.328297 |0.883593 |0.883675|0.883629|0.883967| |7680 |0.380800 |0.331854 |0.882362 |0.880373|0.880764|0.880512| |8192 |0.368400 |0.323076 |0.881730 |0.881378|0.881419|0.881528| |8704 |0.367000 |0.313959 |0.889204 |0.889047|0.889053|0.889352| |9216 |0.315600 |0.333637 |0.885518 |0.883965|0.884266|0.883967| |9728 |0.303100 |0.319416 |0.888667 |0.888092|0.888256|0.888234| |10240 |0.307200 |0.317827 |0.887575 |0.887647|0.887418|0.888031| |10752 |0.300100 |0.311810 |0.890908 |0.890827|0.890747|0.891181| |11264 |0.303400 |0.311010 |0.889871 |0.887939|0.888309|0.887929| |11776 |0.300500 |0.309282 |0.891041 |0.889819|0.890077|0.889860| |12288 |0.303600 |0.326918 |0.891272 |0.891250|0.890942|0.891689| |12800 |0.300300 |0.301688 |0.894516 |0.894619|0.894481|0.894940| |13312 |0.302200 |0.302173 |0.896441 |0.896527|0.896462|0.896769| |13824 |0.299800 |0.293489 |0.895047 |0.895172|0.895084|0.895448| |14336 |0.294600 |0.297645 |0.895865 |0.896012|0.895886|0.896261| |14848 |0.296700 |0.300751 |0.895277 |0.895401|0.895304|0.895651| |15360 |0.293100 |0.293049 |0.896855 |0.896705|0.896757|0.896871| |15872 |0.293600 |0.294201 |0.895933 |0.895557|0.895624|0.895651| |16384 |0.290100 |0.289367 |0.897847 |0.897889|0.897840|0.898090| |16896 |0.293600 |0.283990 |0.898889 |0.898724|0.898789|0.898903| |17408 |0.285800 |0.308257 |0.898250 |0.898102|0.898162|0.898293| |17920 |0.252400 |0.327164 |0.898860 |0.898807|0.898831|0.899004| |18432 |0.219500 |0.315286 |0.898877 |0.898835|0.898831|0.899004| |18944 |0.217900 |0.312738 |0.898857 |0.898958|0.898886|0.899207| |19456 |0.186400 |0.320669 |0.899252 |0.899166|0.899194|0.899411| |19968 |0.199000 |0.316840 |0.900458 |0.900455|0.900426|0.900630| ## Model Card Authors [Erfan Moosavi Monazzah](https://huggingface.co/ErfanMoosaviMonazzah)