Edit model card

xlm-r-base-amazon-massive-intent

This model is a fine-tuned version of xlm-roberta-base on Amazon Massive dataset (only en-US subset). It achieves the following results on the evaluation set:

  • Loss: 0.5439
  • Accuracy: 0.8775
  • F1: 0.8775

Results

domain train-accuracy test-accuracy
alarm 0.967 0.9846
audio 0.7458 0.659
calendar 0.9797 0.3181
cooking 0.9714 0.9571
datetime 0.9777 0.9402
email 0.9727 0.9296
general 0.8952 0.5949
iot 0.9329 0.9122
list 0.9792 0.9538
music 0.9355 0.8837
news 0.9607 0.8764
play 0.9419 0.874
qa 0.9677 0.8591
recommendation 0.9515 0.8764
social 0.9671 0.8932
takeaway 0.9192 0.8478
transport 0.9425 0.9193
weather 0.9895 0.93

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5

Training results

Training Loss Epoch Step Validation Loss Accuracy F1
2.734 1.0 720 1.1883 0.7196 0.7196
1.2774 2.0 1440 0.7162 0.8342 0.8342
0.6301 3.0 2160 0.5817 0.8672 0.8672
0.4901 4.0 2880 0.5555 0.8770 0.8770
0.3398 5.0 3600 0.5439 0.8775 0.8775

Framework versions

  • Transformers 4.22.1
  • Pytorch 1.12.1+cu113
  • Datasets 2.5.1
  • Tokenizers 0.12.1

Citation

@article{kubis2023back,
  title={Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors},
  author={Kubis, Marek and Sk{\'o}rzewski, Pawe{\l} and Sowa{\'n}ski, Marcin and Zi{\k{e}}tkiewicz, Tomasz},
  journal={arXiv preprint arXiv:2310.16609},
  year={2023}
  eprint={2310.16609},
  archivePrefix={arXiv},
}
Downloads last month
271
Safetensors
Model size
278M params
Tensor type
I64
·
F32
·

Finetuned from

Dataset used to train cartesinus/xlm-r-base-amazon-massive-intent

Evaluation results