|
--- |
|
datasets: |
|
- snli |
|
- multi_nli |
|
metrics: |
|
- accuracy |
|
- f1 |
|
- precision |
|
- recall |
|
inference: false |
|
model-index: |
|
- name: distilroberta-nli |
|
results: |
|
- task: |
|
type: text-classification |
|
name: Text Classification |
|
metrics: |
|
- type: loss |
|
value: 0.438475 |
|
- type: accuracy |
|
value: 0.829536 |
|
name: Accuracy |
|
- type: f1 |
|
value: 0.828703 |
|
name: F1 |
|
- type: precision |
|
value: 0.828907 |
|
name: Precision |
|
- type: recall |
|
value: 0.828617 |
|
name: Recall |
|
language: |
|
- en |
|
--- |
|
## DistilRoBERTa-NLI |
|
|
|
This model utilizes the [Distilroberta base](https://huggingface.co/distilroberta-base) architecture, which has been fine-tuned for NLI tasks on the [MultiNLI](https://huggingface.co/datasets/multi_nli) and [SNLI](https://huggingface.co/datasets/snli) datasets. |
|
It achieves the following results on the evaluation set: |
|
* Loss: 0.4384 |
|
* Accuracy: 0.8295 |
|
|
|
|
|
## Model description |
|
|
|
The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE). |
|
|
|
The Multi-Genre Natural Language Inference (MultiNLI) corpus is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information. The corpus is modeled on the SNLI corpus, but differs in that covers a range of genres of spoken and written text, and supports a distinctive cross-genre generalization evaluation. |
|
|
|
## Usage |
|
Inference API has been disabled as it is not suitable for this kind of task. |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
import torch |
|
|
|
# Load model and tokenizer |
|
model_checkpoint = 'AdamCodd/distilroberta-NLI' |
|
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint) |
|
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint) |
|
|
|
# Set device |
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
model.to(device) |
|
|
|
# Sample premise and hypothesis |
|
premise = "The cat is sleeping under the sun." |
|
hypothesis = "It's raining, and the cat is getting wet." |
|
|
|
# Tokenize and predict |
|
input = tokenizer(premise, hypothesis, truncation=True, padding=True, return_tensors="pt").to(device) |
|
with torch.no_grad(): |
|
output = model(**input) |
|
probabilities = torch.softmax(output.logits, dim=-1)[0].tolist() |
|
|
|
# Output prediction |
|
label_names = ["Entailment", "Neutral", "Contradiction"] |
|
prediction = {name: round(prob * 100, 1) for name, prob in zip(label_names, probabilities)} |
|
print(prediction) |
|
# {'Entailment': 1.3, 'Neutral': 8.2, 'Contradiction': 90.5} |
|
``` |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 3e-05 |
|
- train_batch_size: 32 |
|
- eval_batch_size: 32 |
|
- seed: 42 |
|
- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 150 |
|
- num_epochs: 1 |
|
- weight_decay: 0.01 |
|
|
|
### Training results |
|
|
|
Metrics: Accuracy, F1, Precision, Recall |
|
|
|
``` |
|
'eval_loss': 0.438475, |
|
'eval_accuracy': 0.829536, |
|
'eval_f1': 0.828703, |
|
'eval_precision': 0.828907, |
|
'eval_recall': 0.828617 |
|
``` |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.36.0 |
|
- Datasets 2.15.0 |
|
- Tokenizers 0.15.0 |
|
|
|
If you want to support me, you can [here](https://ko-fi.com/adamcodd). |
