Edit model card


This model is a fine-tuned version of bert-tiny on amazon-polarity dataset. It achieves the following results on the evaluation set:

  • Loss: 0.153
  • Accuracy: 0.942
  • F1_score: 0.940

Model description

TinyBERT is 7.5 times smaller and 9.4 times faster on inference compared to its teacher BERT model (while DistilBERT is 40% smaller and 1.6 times faster than BERT). This model was trained using the entire dataset (3.6M of samples) in constrast to the distilbert model which was trained on only 10% of the dataset.

Intended uses & limitations

While this model may not be as accurate as the distilbert model, its performance should be enough for most use cases.

from transformers import pipeline

# Create the pipeline
sentiment_classifier = pipeline('text-classification', model='AdamCodd/tinybert-sentiment-amazon')

# Now you can use the pipeline to classify emotions
result = sentiment_classifier("This product doesn't fit me at all.")
#[{'label': 'negative', 'score': 0.9969743490219116}]

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 1270
  • optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 150
  • num_epochs: 1
  • weight_decay: 0.01

Framework versions

  • Transformers 4.35.0
  • Pytorch lightning 2.1.0
  • Tokenizers 0.14.1

If you want to support me, you can here.

Downloads last month
Model size
4.39M params
Tensor type
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Finetuned from

Dataset used to train AdamCodd/tinybert-sentiment-amazon

Collection including AdamCodd/tinybert-sentiment-amazon

Evaluation results