Hugging Face's logo

EuropeanParliament
/
eubert_embedding_v1

Feature Extraction
Transformers
Safetensors
roberta
Inference Endpoints
Model card Files Community
1
scampion commited on
Commit
963b222
•
1 Parent(s): addecd1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +56 -30
README.md CHANGED
@@ -23,36 +23,62 @@ vector = model.encode("Based on the masked language model EUBERT this sentence t
23
  Evaluation and benchmarking are welcome
24
 
25
 
26
- | Dataset | Name | Config | Metrics Type | Metric Value |
27
- |----------------------------------|------------------------------------|---------|--------------|--------------|
28
- | mteb/amazon_counterfactual | MTEB AmazonCounterfactualClassification (en) | en | accuracy | 65.46268656716417 |
29
- | mteb/amazon_counterfactual | MTEB AmazonCounterfactualClassification (en) | en | ap | 28.448646125211685 |
30
- | mteb/amazon_counterfactual | MTEB AmazonCounterfactualClassification (en) | en | f1 | 59.381505835828655 |
31
- | mteb/amazon_polarity | MTEB AmazonPolarityClassification | default | accuracy | 66.46035 |
32
- | mteb/amazon_polarity | MTEB AmazonPolarityClassification | default | ap | 61.29404861567824 |
33
- | mteb/amazon_polarity | MTEB AmazonPolarityClassification | default | f1 | 66.33660156778977 |
34
- | mteb/amazon_reviews_multi | MTEB AmazonReviewsClassification (en) | en | accuracy | 33.002 |
35
- | mteb/amazon_reviews_multi | MTEB AmazonReviewsClassification (en) | en | f1 | 32.703439998458286 |
36
- | mteb/banking77 | MTEB Banking77Classification | default | accuracy | 72.07792207792207 |
37
- | mteb/banking77 | MTEB Banking77Classification | default | f1 | 72.00698905672714 |
38
- | mteb/emotion | MTEB EmotionClassification | default | accuracy | 25.45 |
39
- | mteb/emotion | MTEB EmotionClassification | default | f1 | 22.489051015009604 |
40
- | mteb/imdb | MTEB ImdbClassification | default | accuracy | 61.0288 |
41
- | mteb/imdb | MTEB ImdbClassification | default | ap | 56.84210754735158 |
42
- | mteb/imdb | MTEB ImdbClassification | default | f1 | 60.72244426285243 |
43
- | mteb/mtop_domain | MTEB MTOPDomainClassification (en) | en | accuracy | 78.63657090743274 |
44
- | mteb/mtop_domain | MTEB MTOPDomainClassification (en) | en | f1 | 77.33756273016937 |
45
- | mteb/mtop_intent | MTEB MTOPIntentClassification (en) | en | accuracy | 53.504331965344285 |
46
- | mteb/mtop_intent | MTEB MTOPIntentClassification (en) | en | f1 | 37.650916452762054 |
47
- | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (en) | en | accuracy | 54.80833893745797 |
48
- | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (en) | en | f1 | 53.43307984316261 |
49
- | mteb/amazon_massive_scenario | MTEB MassiveScenarioClassification (en) | en | accuracy | 60.15803631472764 |
50
- | mteb/amazon_massive_scenario | MTEB MassiveScenarioClassification (en) | en | f1 | 58.1272976567113 |
51
- | mteb/toxic_conversations_50k | MTEB ToxicConversationsClassification | default | accuracy | 59.1906 |
52
- | mteb/toxic_conversations_50k | MTEB ToxicConversationsClassification | default | ap | 9.927869117155222 |
53
- | mteb/toxic_conversations_50k | MTEB ToxicConversationsClassification | default | f1 | 45.78592572294965 |
54
- | mteb/tweet_sentiment_extraction | MTEB TweetSentimentExtractionClassification | default | accuracy | 46.08658743633276 |
55
- | mteb/tweet_sentiment_extraction | MTEB TweetSentimentExtractionClassification | default | f1 | 46.17188224255574 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
 
57
 
58
  Author : sebastien.campion@europarl.europa.eu
 
23
  Evaluation and benchmarking are welcome
24
 
25
 
26
+
27
+ | task | dataset | name | config | split | revision | accuracy | ap | f1 | v_measure |
28
+ |-|-|-|-|-|-|-|-|-|-|
29
+ | Classification | mteb/amazon_counterfactual | MTEB AmazonCounterfactualClassification (en) | en | test | e8379541af4e31359cca9fbcf4b00f2671dba205 | 65.46268656716417 | 28.448646125211685 | 59.381505835828655 | |
30
+ | Classification | mteb/amazon_polarity | MTEB AmazonPolarityClassification | default | test | e2d317d38cd51312af73b3d32a06d1a08b442046 | 66.46035 | 61.29404861567824 | 66.33660156778977 | |
31
+ | Classification | mteb/amazon_reviews_multi | MTEB AmazonReviewsClassification (en) | en | test | 1399c76144fd37290681b995c656ef9b2e06e26d | 33.002 | | 32.703439998458286 | |
32
+ | Clustering | mteb/arxiv-clustering-p2p | MTEB ArxivClusteringP2P | default | test | a122ad7f3f0291bf49cc6f4d32aa80929df69d5d | | | | 26.726296122407874 |
33
+ | Classification | mteb/banking77 | MTEB Banking77Classification | default | test | 0fd18e25b25c072e09e0d92ab615fda904d66300 | 72.07792207792207 | | 72.00698905672714 | |
34
+ | Classification | mteb/emotion | MTEB EmotionClassification | default | test | 4f58c6b202a23cf9a4da393831edf4f9183cad37 | 25.45 | | 22.489051015009604 | |
35
+ | Classification | mteb/imdb | MTEB ImdbClassification | default | test | 3d86128a09e091d6018b6d26cad27f2739fc2db7 | 61.0288 | 56.84210754735158 | 60.72244426285243 | |
36
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (en) | en | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 78.63657090743274 | | 77.33756273016937 | |
37
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (de) | de | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 67.63313609467455 | | 65.31424834681424 | |
38
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (es) | es | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 72.03468979319545 | | 70.33858350063844 | |
39
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (fr) | fr | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 69.33604760413404 | | 67.2763398514464 | |
40
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (hi) | hi | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 19.336679813553243 | | 17.640206592911305 | |
41
+ | Classification | mteb/mtop_domain | MTEB MTOPDomainClassification (th) | th | test | d80d48c1eb48d3562165c59d59d0034df9fff0bf | 14.958408679927668 | | 12.200892995648038 | |
42
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (en) | en | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 53.504331965344285 | | 37.650916452762054 | |
43
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (de) | de | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 52.8007889546351 | | 35.18483837593346 | |
44
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (es) | es | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 53.268845897264846 | | 37.54041476398511 | |
45
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (fr) | fr | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 47.59160663952396 | | 33.779636915265606 | |
46
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (hi) | hi | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 4.180709931875224 | | 2.240473672484894 | |
47
+ | Classification | mteb/mtop_intent | MTEB MTOPIntentClassification (th) | th | test | ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba | 4.1482820976491865 | | 2.2953415174353546 | |
48
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (af) | af | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 43.843308675184936 | | 42.83274171307546 | |
49
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (am) | am | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 8.459986550100874 | | 8.56499841559428 | |
50
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ar) | ar | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 24.37457969065232 | | 23.648464353469087 | |
51
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (az) | az | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 43.61129791526564 | | 43.02872726206446 | |
52
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (bn) | bn | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 3.127101546738399 | | 1.7632874555194573 | |
53
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (cy) | cy | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 39.882313382649635 | | 39.09054995553107 | |
54
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (da) | da | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 49.05514458641561 | | 47.97657474719148 | |
55
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (de) | de | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 47.723604572965705 | | 46.266605736862424 | |
56
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (el) | el | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 49.2871553463349 | | 49.110660419740945 | |
57
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (en) | en | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 54.80833893745797 | | 53.43307984316261 | |
58
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (es) | es | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 48.73234700739745 | | 48.290537885757345 | |
59
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (fa) | fa | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 34.60322797579018 | | 33.21866171174647 | |
60
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (fi) | fi | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 47.09818426361803 | | 46.24034140543536 | |
61
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (fr) | fr | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 47.92871553463349 | | 47.2879827826325 | |
62
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (he) | he | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 23.429724277067923 | | 22.973698726459283 | |
63
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (hi) | hi | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 3.1909885675857437 | | 2.343483452751791 | |
64
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (hu) | hu | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 46.529926025554815 | | 45.585210075220026 | |
65
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (hy) | hy | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 33.00605245460659 | | 32.53906554922222 | |
66
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (id) | id | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 44.70073974445191 | | 44.63772874280639 | |
67
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (is) | is | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 42.56556825823806 | | 42.09519069412614 | |
68
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (it) | it | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 49.45191661062542 | | 49.73648735452711 | |
69
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ja) | ja | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 36.03227975790181 | | 34.81337003018146 | |
70
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (jv) | jv | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 39.85205110961668 | | 39.16645932365053 | |
71
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ka) | ka | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 29.84532616005381 | | 30.048107009813975 | |
72
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (km) | km | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 5.4942837928715536 | | 3.9402294020821236 | |
73
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (kn) | kn | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 3.5541358439811694 | | 2.3408708229868385 | |
74
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ko) | ko | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 31.055817081371888 | | 30.54791134524761 | |
75
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (lv) | lv | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 48.44989912575656 | | 47.46077758238515 | |
76
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ml) | ml | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 2.89172831203766 | | 1.1484871860887453 | |
77
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (mn) | mn | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 38.924008069939475 | | 38.953938082398274 | |
78
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (ms) | ms | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 43.25151311365165 | | 42.31124560201582 | |
79
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (my) | my | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 3.5137861466039007 | | 1.7087643302156377 | |
80
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (nb) | nb | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 45.34633490248823 | | 44.7188441016561 | |
81
+ | Classification | mteb/amazon_massive_intent | MTEB MassiveIntentClassification (nl) | nl | test | 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 | 47.25285810356422 | | 45.442034061197944 | |
82
 
83
 
84
  Author : sebastien.campion@europarl.europa.eu