--- language: en tags: - text-classification - onnx - bge-small-en-v1.5 - emotions - multi-class-classification - multi-label-classification datasets: - go_emotions models: - BAAI/bge-small-en-v1.5 license: mit inference: false widget: - text: ONNX is so much faster, its very handy! --- ### Overview This is a multi-label, multi-class linear classifer for emotions that works with [BGE-small-en-v1.5 embeddings](https://huggingface.co/BAAI/bge-small-en-v1.5), having been trained on the [go_emotions](https://huggingface.co/datasets/go_emotions) dataset. ### Labels The 28 labels from the [go_emotions](https://huggingface.co/datasets/go_emotions) dataset are: ``` ['admiration', 'amusement', 'anger', 'annoyance', 'approval', 'caring', 'confusion', 'curiosity', 'desire', 'disappointment', 'disapproval', 'disgust', 'embarrassment', 'excitement', 'fear', 'gratitude', 'grief', 'joy', 'love', 'nervousness', 'optimism', 'pride', 'realization', 'relief', 'remorse', 'sadness', 'surprise', 'neutral'] ``` ### Metrics (exact match of labels per item) This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification. Evaluating across all labels per item in the go_emotions test split the metrics are shown below. Using a fixed threshold of 0.5 to convert the scores to binary predictions for each label, the metrics (evaluated on the go_emotions test split) are: - Precision: 0.602 - Recall: 0.250 - F1: 0.303 Optimising the threshold per label to optimise the F1 metric, the metrics (evaluated on the go_emotions test split) are: - Precision: 0.445 - Recall: 0.476 - F1: 0.449 Weighted by the relative support of each label in the dataset, this is: - Precision: 0.472 - Recall: 0.582 - F1: 0.514 ### Metrics (per-label) This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification and metrics are better measured per label. Optimising the threshold per label to optimise the F1 metric, the metrics (evaluated on the go_emotions test split) are: | | f1 | precision | recall | support | threshold | | -------------- | ----- | --------- | ------ | ------- | --------- | | admiration | 0.583 | 0.574 | 0.593 | 504 | 0.30 | | amusement | 0.668 | 0.722 | 0.621 | 264 | 0.25 | | anger | 0.350 | 0.309 | 0.404 | 198 | 0.15 | | annoyance | 0.299 | 0.318 | 0.281 | 320 | 0.20 | | approval | 0.338 | 0.281 | 0.425 | 351 | 0.15 | | caring | 0.321 | 0.323 | 0.319 | 135 | 0.20 | | confusion | 0.384 | 0.313 | 0.497 | 153 | 0.15 | | curiosity | 0.467 | 0.432 | 0.507 | 284 | 0.20 | | desire | 0.426 | 0.381 | 0.482 | 83 | 0.20 | | disappointment | 0.210 | 0.147 | 0.364 | 151 | 0.10 | | disapproval | 0.366 | 0.288 | 0.502 | 267 | 0.15 | | disgust | 0.416 | 0.409 | 0.423 | 123 | 0.20 | | embarrassment | 0.370 | 0.341 | 0.405 | 37 | 0.30 | | excitement | 0.313 | 0.368 | 0.272 | 103 | 0.25 | | fear | 0.615 | 0.677 | 0.564 | 78 | 0.40 | | gratitude | 0.828 | 0.810 | 0.847 | 352 | 0.25 | | grief | 0.545 | 0.600 | 0.500 | 6 | 0.85 | | joy | 0.455 | 0.429 | 0.484 | 161 | 0.20 | | love | 0.642 | 0.673 | 0.613 | 238 | 0.30 | | nervousness | 0.350 | 0.412 | 0.304 | 23 | 0.60 | | optimism | 0.439 | 0.417 | 0.462 | 186 | 0.20 | | pride | 0.480 | 0.667 | 0.375 | 16 | 0.70 | | realization | 0.232 | 0.191 | 0.297 | 145 | 0.10 | | relief | 0.353 | 0.500 | 0.273 | 11 | 0.50 | | remorse | 0.643 | 0.529 | 0.821 | 56 | 0.20 | | sadness | 0.526 | 0.497 | 0.558 | 156 | 0.20 | | surprise | 0.329 | 0.318 | 0.340 | 141 | 0.15 | | neutral | 0.634 | 0.528 | 0.794 | 1787 | 0.30 | Using a fixed threshold of 0.5 to convert the scores to binary predictions for each label, the metrics (evaluated on the go_emotions test split) are: | | f1 | precision | recall | support | threshold | | -------------- | ----- | --------- | ------ | ------- | --------- | | admiration | 0.497 | 0.731 | 0.377 | 504 | 0.5 | | amusement | 0.484 | 0.793 | 0.348 | 264 | 0.5 | | anger | 0.162 | 0.528 | 0.096 | 198 | 0.5 | | annoyance | 0.042 | 0.636 | 0.022 | 320 | 0.5 | | approval | 0.106 | 0.769 | 0.057 | 351 | 0.5 | | caring | 0.182 | 0.500 | 0.111 | 135 | 0.5 | | confusion | 0.170 | 0.652 | 0.098 | 153 | 0.5 | | curiosity | 0.284 | 0.529 | 0.194 | 284 | 0.5 | | desire | 0.236 | 0.481 | 0.157 | 83 | 0.5 | | disappointment | 0.039 | 0.750 | 0.020 | 151 | 0.5 | | disapproval | 0.140 | 0.636 | 0.079 | 267 | 0.5 | | disgust | 0.273 | 0.677 | 0.171 | 123 | 0.5 | | embarrassment | 0.314 | 0.571 | 0.216 | 37 | 0.5 | | excitement | 0.130 | 0.400 | 0.078 | 103 | 0.5 | | fear | 0.527 | 0.667 | 0.436 | 78 | 0.5 | | gratitude | 0.792 | 0.908 | 0.702 | 352 | 0.5 | | grief | 0.385 | 0.250 | 0.833 | 6 | 0.5 | | joy | 0.276 | 0.771 | 0.168 | 161 | 0.5 | | love | 0.606 | 0.800 | 0.487 | 238 | 0.5 | | nervousness | 0.269 | 0.241 | 0.304 | 23 | 0.5 | | optimism | 0.305 | 0.720 | 0.194 | 186 | 0.5 | | pride | 0.375 | 0.375 | 0.375 | 16 | 0.5 | | realization | 0.013 | 0.250 | 0.007 | 145 | 0.5 | | relief | 0.353 | 0.500 | 0.273 | 11 | 0.5 | | remorse | 0.469 | 0.548 | 0.411 | 56 | 0.5 | | sadness | 0.365 | 0.731 | 0.244 | 156 | 0.5 | | surprise | 0.142 | 0.786 | 0.078 | 141 | 0.5 | | neutral | 0.547 | 0.644 | 0.475 | 1787 | 0.5 | ### Use with ONNXRuntime ```python pass ```