---
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- AdamCodd/emotion-balanced
metrics:
- accuracy
- f1
- recall
- precision
base_model: prajjwal1/bert-tiny
model-index:
- name: AdamCodd/tinybert-emotion-balanced
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: emotion
      type: emotion
      args: default
    metrics:
    - type: accuracy
      value: 0.9354
      name: Accuracy
    - type: loss
      value: 0.1809
      name: Loss
    - type: f1
      value: 0.9354946613311768
      name: F1
---

# tinybert-emotion

This model is a fine-tuned version of [bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) on the [emotion balanced dataset](https://huggingface.co/datasets/AdamCodd/emotion-balanced).
It achieves the following results on the evaluation set:
- Loss: 0.1809
- Accuracy: 0.9354

## Model description

TinyBERT is 7.5 times smaller and 9.4 times faster on inference compared to its teacher BERT model (while DistilBERT is 40% smaller and 1.6 times faster than BERT). 
The model has been trained on 89_754 examples split into train, validation and test. Each label was perfectly balanced in each split.

## Intended uses & limitations

This model is not as accurate as the [distilbert-emotion-balanced](https://huggingface.co/AdamCodd/distilbert-base-uncased-finetuned-emotion-balanced) one because the focus was on speed, which can lead to misinterpretation of complex sentences. Despite this, its performance is quite good and should be more than sufficient for most use cases.

Usage:
```python
from transformers import pipeline

# Create the pipeline
emotion_classifier = pipeline('text-classification', model='AdamCodd/tinybert-emotion-balanced')

# Now you can use the pipeline to classify emotions
result = emotion_classifier("We are delighted that you will be coming to visit us. It will be so nice to have you here.")
print(result)
#[{'label': 'joy', 'score': 0.9895486831665039}]
```
This model faces challenges in accurately categorizing negative sentences, as well as those containing elements of sarcasm or irony. These limitations are largely attributable to TinyBERT's constrained capabilities in semantic understanding. Although the model is generally proficient in emotion detection tasks, it may lack the nuance necessary for interpreting complex emotional nuances.

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 64
- seed: 1270
- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 150
- num_epochs: 10
- weight_decay: 0.01

### Training results

              precision    recall  f1-score   support

     sadness     0.9733    0.9245    0.9482      1496
         joy     0.9651    0.8864    0.9240      1496
        love     0.9127    0.9786    0.9445      1496
       anger     0.9479    0.9365    0.9422      1496
        fear     0.9213    0.9004    0.9108      1496
    surprise     0.9016    0.9866    0.9422      1496

    accuracy                         0.9355      8976
    macro avg    0.9370    0.9355    0.9353      8976
    weighted avg 0.9370    0.9355    0.9353      8976
    
    test_acc:     0.9354946613311768
    test_loss:    0.1809326708316803

### Framework versions

- Transformers 4.33.0
- Pytorch lightning 2.0.8
- Tokenizers 0.13.3

If you want to support me, you can [here](https://ko-fi.com/adamcodd).