|
--- |
|
license: apache-2.0 |
|
tags: |
|
- transformers |
|
- pytorch |
|
datasets: |
|
- conv_ai_2 |
|
model-index: |
|
- name: distillbert_conv_quality_score |
|
results: [] |
|
language: |
|
- en |
|
--- |
|
|
|
|
|
# distillbert_conv_quality_score |
|
|
|
This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the conv_ai_2 dataset. |
|
It was trained to generate a score (in the [0, 1] range) from a conversation |
|
|
|
|
|
It achieves the following results on the evaluation set: |
|
- training/loss: 0.0165 |
|
- validation/loss: 0.0149 |
|
|
|
|
|
## Usage |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
|
model_name = "alespalla/distillbert_conv_quality_score" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForSequenceClassification.from_pretrained(model_name) |
|
|
|
conversation = ''' |
|
Q: Begin |
|
A: lol ! do you think it is strange to feel like you have been through life before ? |
|
Q: Hellow |
|
A: I don't understand you 🙈. Also, try to guess: i like to ... |
|
Q: How are you? |
|
A: make time stop, funny you :) |
|
Q: What is your name? |
|
A: jessie. hows your day going ? 😃 |
|
''' |
|
|
|
score = model(**tokenizer(conversation, return_tensors='pt')).logits.item() |
|
print(f"Score: {score}") |
|
``` |
|
|
|
## Training and evaluation data |
|
|
|
The training data was generated from `conv_ai_2` using the following function |
|
|
|
```python |
|
|
|
from datasets import load_dataset |
|
|
|
def get_dataset(regression=False): |
|
|
|
db = load_dataset("conv_ai_2") |
|
|
|
def generate_converation(elem): |
|
text = "" |
|
for idx, txt in enumerate(elem["dialog"]): |
|
if idx % 2: |
|
text += f"A: {txt['text']}\n" |
|
else: |
|
text += f"Q: {txt['text']}\n" |
|
if regression: |
|
return {'text': text, "labels": (elem['eval_score'] - 1)/4} |
|
return {'text': text, "labels": elem['eval_score'] - 1} |
|
|
|
db = db.filter(lambda example: example["eval_score"] > 0) |
|
db = db.map(generate_converation, remove_columns=db['train'].column_names) |
|
db = db['train'].train_test_split(test_size=0.2).shuffle(42) |
|
|
|
return db |
|
|
|
``` |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- epochs: 40 |
|
- batch_size: 16 |
|
- learning_rate: 0.0002 |
|
- eval_steps: 82 |
|
- log_steps: 82 |
|
- save_steps: 41 |
|
- gradient_accumulation_steps: 1 |
|
- warmup_steps: 0 |
|
|
|
### Training results |
|
|
|
| step | training/loss | validation/loss | |
|
|:----:|:-------------:|:---------------:| |
|
| 81 | 0.1020 | 0.0794 | |
|
| 163 | 0.0800 | 0.0713 | |
|
| 245 | 0.0553 | 0.0491 | |
|
| 327 | 0.0362 | 0.0440 | |
|
| 409 | 0.0282 | 0.0352 | |
|
| 491 | 0.0282 | 0.0412 | |
|
| 573 | 0.0256 | 0.0293 | |
|
| 655 | 0.0238 | 0.0252 | |
|
| 737 | 0.0175 | 0.0226 | |
|
| 819 | 0.0154 | 0.0228 | |
|
| 901 | 0.0116 | 0.0205 | |
|
| 983 | 0.0160 | 0.0202 | |
|
| 1065 | 0.0146 | 0.0240 | |
|
| 1147 | 0.0182 | 0.0180 | |
|
| 1229 | 0.0171 | 0.0192 | |
|
| 1311 | 0.0091 | 0.0174 | |
|
| 1393 | 0.0171 | 0.0158 | |
|
| 1475 | 0.0137 | 0.0158 | |
|
| 1557 | 0.0158 | 0.0148 | |
|
| 1639 | 0.0165 | 0.0149 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.26.1 |
|
- Datasets 2.10.1 |
|
- Tokenizers 0.13.2 |
