diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -4,6 +4,6681 @@ tags: - sentence-transformers - feature-extraction - sentence-similarity +- mteb +model-index: +- name: sgpt-bloom-7b1-msmarco + results: + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (en) + config: en + split: test + revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 + metrics: + - type: accuracy + value: 68.05970149253731 + - type: ap + value: 31.640363460776193 + - type: f1 + value: 62.50025574145796 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (de) + config: de + split: test + revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 + metrics: + - type: accuracy + value: 61.34903640256959 + - type: ap + value: 75.18797161500426 + - type: f1 + value: 59.04772570730417 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (en-ext) + config: en-ext + split: test + revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 + metrics: + - type: accuracy + value: 67.78110944527737 + - type: ap + value: 19.218916023322706 + - type: f1 + value: 56.24477391445512 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (ja) + config: ja + split: test + revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 + metrics: + - type: accuracy + value: 58.23340471092078 + - type: ap + value: 13.20222967424681 + - type: f1 + value: 47.511718095460296 + - task: + type: Classification + dataset: + type: mteb/amazon_polarity + name: MTEB AmazonPolarityClassification + config: default + split: test + revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 + metrics: + - type: accuracy + value: 68.97232499999998 + - type: ap + value: 63.53632885535693 + - type: f1 + value: 68.62038513152868 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (en) + config: en + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 33.855999999999995 + - type: f1 + value: 33.43468222830134 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (de) + config: de + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 29.697999999999997 + - type: f1 + value: 29.39935388885501 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (es) + config: es + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 35.974000000000004 + - type: f1 + value: 35.25910820714383 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (fr) + config: fr + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 35.922 + - type: f1 + value: 35.38637028933444 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (ja) + config: ja + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 27.636 + - type: f1 + value: 27.178349955978266 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (zh) + config: zh + split: test + revision: c379a6705fec24a2493fa68e011692605f44e119 + metrics: + - type: accuracy + value: 32.632 + - type: f1 + value: 32.08014766494587 + - task: + type: Retrieval + dataset: + type: arguana + name: MTEB ArguAna + config: default + split: test + revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 + metrics: + - type: map_at_1 + value: 23.684 + - type: map_at_10 + value: 38.507999999999996 + - type: map_at_100 + value: 39.677 + - type: map_at_1000 + value: 39.690999999999995 + - type: map_at_3 + value: 33.369 + - type: map_at_5 + value: 36.15 + - type: mrr_at_1 + value: 24.04 + - type: mrr_at_10 + value: 38.664 + - type: mrr_at_100 + value: 39.833 + - type: mrr_at_1000 + value: 39.847 + - type: mrr_at_3 + value: 33.476 + - type: mrr_at_5 + value: 36.306 + - type: ndcg_at_1 + value: 23.684 + - type: ndcg_at_10 + value: 47.282000000000004 + - type: ndcg_at_100 + value: 52.215 + - type: ndcg_at_1000 + value: 52.551 + - type: ndcg_at_3 + value: 36.628 + - type: ndcg_at_5 + value: 41.653 + - type: precision_at_1 + value: 23.684 + - type: precision_at_10 + value: 7.553 + - type: precision_at_100 + value: 0.97 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_3 + value: 15.363 + - type: precision_at_5 + value: 11.664 + - type: recall_at_1 + value: 23.684 + - type: recall_at_10 + value: 75.533 + - type: recall_at_100 + value: 97.013 + - type: recall_at_1000 + value: 99.57300000000001 + - type: recall_at_3 + value: 46.088 + - type: recall_at_5 + value: 58.321 + - task: + type: Clustering + dataset: + type: mteb/arxiv-clustering-p2p + name: MTEB ArxivClusteringP2P + config: default + split: test + revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 + metrics: + - type: v_measure + value: 44.59375023881131 + - task: + type: Clustering + dataset: + type: mteb/arxiv-clustering-s2s + name: MTEB ArxivClusteringS2S + config: default + split: test + revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 + metrics: + - type: v_measure + value: 38.02921907752556 + - task: + type: Reranking + dataset: + type: mteb/askubuntudupquestions-reranking + name: MTEB AskUbuntuDupQuestions + config: default + split: test + revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c + metrics: + - type: map + value: 59.97321570342109 + - type: mrr + value: 73.18284746955106 + - task: + type: STS + dataset: + type: mteb/biosses-sts + name: MTEB BIOSSES + config: default + split: test + revision: 9ee918f184421b6bd48b78f6c714d86546106103 + metrics: + - type: cos_sim_pearson + value: 89.09091435741429 + - type: cos_sim_spearman + value: 85.31459455332202 + - type: euclidean_pearson + value: 79.3587681410798 + - type: euclidean_spearman + value: 76.8174129874685 + - type: manhattan_pearson + value: 79.57051762121769 + - type: manhattan_spearman + value: 76.75837549768094 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (de-en) + config: de-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 54.27974947807933 + - type: f1 + value: 54.00144411132214 + - type: precision + value: 53.87119374071357 + - type: recall + value: 54.27974947807933 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (fr-en) + config: fr-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 97.3365617433414 + - type: f1 + value: 97.06141316310809 + - type: precision + value: 96.92567319685965 + - type: recall + value: 97.3365617433414 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (ru-en) + config: ru-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 46.05472809144441 + - type: f1 + value: 45.30319274690595 + - type: precision + value: 45.00015469655234 + - type: recall + value: 46.05472809144441 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (zh-en) + config: zh-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 98.10426540284361 + - type: f1 + value: 97.96384061786905 + - type: precision + value: 97.89362822538178 + - type: recall + value: 98.10426540284361 + - task: + type: Classification + dataset: + type: mteb/banking77 + name: MTEB Banking77Classification + config: default + split: test + revision: 44fa15921b4c889113cc5df03dd4901b49161ab7 + metrics: + - type: accuracy + value: 84.33441558441558 + - type: f1 + value: 84.31653077470322 + - task: + type: Clustering + dataset: + type: mteb/biorxiv-clustering-p2p + name: MTEB BiorxivClusteringP2P + config: default + split: test + revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 + metrics: + - type: v_measure + value: 36.025318694698086 + - task: + type: Clustering + dataset: + type: mteb/biorxiv-clustering-s2s + name: MTEB BiorxivClusteringS2S + config: default + split: test + revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 + metrics: + - type: v_measure + value: 32.484889034590346 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackAndroidRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 30.203999999999997 + - type: map_at_10 + value: 41.314 + - type: map_at_100 + value: 42.66 + - type: map_at_1000 + value: 42.775999999999996 + - type: map_at_3 + value: 37.614999999999995 + - type: map_at_5 + value: 39.643 + - type: mrr_at_1 + value: 37.482 + - type: mrr_at_10 + value: 47.075 + - type: mrr_at_100 + value: 47.845 + - type: mrr_at_1000 + value: 47.887 + - type: mrr_at_3 + value: 44.635000000000005 + - type: mrr_at_5 + value: 45.966 + - type: ndcg_at_1 + value: 37.482 + - type: ndcg_at_10 + value: 47.676 + - type: ndcg_at_100 + value: 52.915 + - type: ndcg_at_1000 + value: 54.82900000000001 + - type: ndcg_at_3 + value: 42.562 + - type: ndcg_at_5 + value: 44.852 + - type: precision_at_1 + value: 37.482 + - type: precision_at_10 + value: 9.142 + - type: precision_at_100 + value: 1.436 + - type: precision_at_1000 + value: 0.189 + - type: precision_at_3 + value: 20.458000000000002 + - type: precision_at_5 + value: 14.821000000000002 + - type: recall_at_1 + value: 30.203999999999997 + - type: recall_at_10 + value: 60.343 + - type: recall_at_100 + value: 82.58 + - type: recall_at_1000 + value: 94.813 + - type: recall_at_3 + value: 45.389 + - type: recall_at_5 + value: 51.800999999999995 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackEnglishRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 30.889 + - type: map_at_10 + value: 40.949999999999996 + - type: map_at_100 + value: 42.131 + - type: map_at_1000 + value: 42.253 + - type: map_at_3 + value: 38.346999999999994 + - type: map_at_5 + value: 39.782000000000004 + - type: mrr_at_1 + value: 38.79 + - type: mrr_at_10 + value: 46.944 + - type: mrr_at_100 + value: 47.61 + - type: mrr_at_1000 + value: 47.650999999999996 + - type: mrr_at_3 + value: 45.053 + - type: mrr_at_5 + value: 46.101 + - type: ndcg_at_1 + value: 38.79 + - type: ndcg_at_10 + value: 46.286 + - type: ndcg_at_100 + value: 50.637 + - type: ndcg_at_1000 + value: 52.649 + - type: ndcg_at_3 + value: 42.851 + - type: ndcg_at_5 + value: 44.311 + - type: precision_at_1 + value: 38.79 + - type: precision_at_10 + value: 8.516 + - type: precision_at_100 + value: 1.3679999999999999 + - type: precision_at_1000 + value: 0.183 + - type: precision_at_3 + value: 20.637 + - type: precision_at_5 + value: 14.318 + - type: recall_at_1 + value: 30.889 + - type: recall_at_10 + value: 55.327000000000005 + - type: recall_at_100 + value: 74.091 + - type: recall_at_1000 + value: 86.75500000000001 + - type: recall_at_3 + value: 44.557 + - type: recall_at_5 + value: 49.064 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackGamingRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 39.105000000000004 + - type: map_at_10 + value: 50.928 + - type: map_at_100 + value: 51.958000000000006 + - type: map_at_1000 + value: 52.017 + - type: map_at_3 + value: 47.638999999999996 + - type: map_at_5 + value: 49.624 + - type: mrr_at_1 + value: 44.639 + - type: mrr_at_10 + value: 54.261 + - type: mrr_at_100 + value: 54.913999999999994 + - type: mrr_at_1000 + value: 54.945 + - type: mrr_at_3 + value: 51.681999999999995 + - type: mrr_at_5 + value: 53.290000000000006 + - type: ndcg_at_1 + value: 44.639 + - type: ndcg_at_10 + value: 56.678 + - type: ndcg_at_100 + value: 60.649 + - type: ndcg_at_1000 + value: 61.855000000000004 + - type: ndcg_at_3 + value: 51.092999999999996 + - type: ndcg_at_5 + value: 54.096999999999994 + - type: precision_at_1 + value: 44.639 + - type: precision_at_10 + value: 9.028 + - type: precision_at_100 + value: 1.194 + - type: precision_at_1000 + value: 0.135 + - type: precision_at_3 + value: 22.508 + - type: precision_at_5 + value: 15.661 + - type: recall_at_1 + value: 39.105000000000004 + - type: recall_at_10 + value: 70.367 + - type: recall_at_100 + value: 87.359 + - type: recall_at_1000 + value: 95.88 + - type: recall_at_3 + value: 55.581 + - type: recall_at_5 + value: 62.821000000000005 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackGisRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 23.777 + - type: map_at_10 + value: 32.297 + - type: map_at_100 + value: 33.516 + - type: map_at_1000 + value: 33.592 + - type: map_at_3 + value: 30.001 + - type: map_at_5 + value: 31.209999999999997 + - type: mrr_at_1 + value: 25.989 + - type: mrr_at_10 + value: 34.472 + - type: mrr_at_100 + value: 35.518 + - type: mrr_at_1000 + value: 35.577 + - type: mrr_at_3 + value: 32.185 + - type: mrr_at_5 + value: 33.399 + - type: ndcg_at_1 + value: 25.989 + - type: ndcg_at_10 + value: 37.037 + - type: ndcg_at_100 + value: 42.699 + - type: ndcg_at_1000 + value: 44.725 + - type: ndcg_at_3 + value: 32.485 + - type: ndcg_at_5 + value: 34.549 + - type: precision_at_1 + value: 25.989 + - type: precision_at_10 + value: 5.718 + - type: precision_at_100 + value: 0.89 + - type: precision_at_1000 + value: 0.11 + - type: precision_at_3 + value: 14.049 + - type: precision_at_5 + value: 9.672 + - type: recall_at_1 + value: 23.777 + - type: recall_at_10 + value: 49.472 + - type: recall_at_100 + value: 74.857 + - type: recall_at_1000 + value: 90.289 + - type: recall_at_3 + value: 37.086000000000006 + - type: recall_at_5 + value: 42.065999999999995 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackMathematicaRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 13.377 + - type: map_at_10 + value: 21.444 + - type: map_at_100 + value: 22.663 + - type: map_at_1000 + value: 22.8 + - type: map_at_3 + value: 18.857 + - type: map_at_5 + value: 20.426 + - type: mrr_at_1 + value: 16.542 + - type: mrr_at_10 + value: 25.326999999999998 + - type: mrr_at_100 + value: 26.323 + - type: mrr_at_1000 + value: 26.406000000000002 + - type: mrr_at_3 + value: 22.823 + - type: mrr_at_5 + value: 24.340999999999998 + - type: ndcg_at_1 + value: 16.542 + - type: ndcg_at_10 + value: 26.479000000000003 + - type: ndcg_at_100 + value: 32.29 + - type: ndcg_at_1000 + value: 35.504999999999995 + - type: ndcg_at_3 + value: 21.619 + - type: ndcg_at_5 + value: 24.19 + - type: precision_at_1 + value: 16.542 + - type: precision_at_10 + value: 5.075 + - type: precision_at_100 + value: 0.9339999999999999 + - type: precision_at_1000 + value: 0.135 + - type: precision_at_3 + value: 10.697 + - type: precision_at_5 + value: 8.134 + - type: recall_at_1 + value: 13.377 + - type: recall_at_10 + value: 38.027 + - type: recall_at_100 + value: 63.439 + - type: recall_at_1000 + value: 86.354 + - type: recall_at_3 + value: 25.0 + - type: recall_at_5 + value: 31.306 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackPhysicsRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 28.368 + - type: map_at_10 + value: 39.305 + - type: map_at_100 + value: 40.637 + - type: map_at_1000 + value: 40.753 + - type: map_at_3 + value: 36.077999999999996 + - type: map_at_5 + value: 37.829 + - type: mrr_at_1 + value: 34.937000000000005 + - type: mrr_at_10 + value: 45.03 + - type: mrr_at_100 + value: 45.78 + - type: mrr_at_1000 + value: 45.827 + - type: mrr_at_3 + value: 42.348 + - type: mrr_at_5 + value: 43.807 + - type: ndcg_at_1 + value: 34.937000000000005 + - type: ndcg_at_10 + value: 45.605000000000004 + - type: ndcg_at_100 + value: 50.941 + - type: ndcg_at_1000 + value: 52.983000000000004 + - type: ndcg_at_3 + value: 40.366 + - type: ndcg_at_5 + value: 42.759 + - type: precision_at_1 + value: 34.937000000000005 + - type: precision_at_10 + value: 8.402 + - type: precision_at_100 + value: 1.2959999999999998 + - type: precision_at_1000 + value: 0.164 + - type: precision_at_3 + value: 19.217000000000002 + - type: precision_at_5 + value: 13.725000000000001 + - type: recall_at_1 + value: 28.368 + - type: recall_at_10 + value: 58.5 + - type: recall_at_100 + value: 80.67999999999999 + - type: recall_at_1000 + value: 93.925 + - type: recall_at_3 + value: 43.956 + - type: recall_at_5 + value: 50.065000000000005 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackProgrammersRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 24.851 + - type: map_at_10 + value: 34.758 + - type: map_at_100 + value: 36.081 + - type: map_at_1000 + value: 36.205999999999996 + - type: map_at_3 + value: 31.678 + - type: map_at_5 + value: 33.398 + - type: mrr_at_1 + value: 31.279 + - type: mrr_at_10 + value: 40.138 + - type: mrr_at_100 + value: 41.005 + - type: mrr_at_1000 + value: 41.065000000000005 + - type: mrr_at_3 + value: 37.519000000000005 + - type: mrr_at_5 + value: 38.986 + - type: ndcg_at_1 + value: 31.279 + - type: ndcg_at_10 + value: 40.534 + - type: ndcg_at_100 + value: 46.093 + - type: ndcg_at_1000 + value: 48.59 + - type: ndcg_at_3 + value: 35.473 + - type: ndcg_at_5 + value: 37.801 + - type: precision_at_1 + value: 31.279 + - type: precision_at_10 + value: 7.477 + - type: precision_at_100 + value: 1.2 + - type: precision_at_1000 + value: 0.159 + - type: precision_at_3 + value: 17.047 + - type: precision_at_5 + value: 12.306000000000001 + - type: recall_at_1 + value: 24.851 + - type: recall_at_10 + value: 52.528 + - type: recall_at_100 + value: 76.198 + - type: recall_at_1000 + value: 93.12 + - type: recall_at_3 + value: 38.257999999999996 + - type: recall_at_5 + value: 44.440000000000005 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 25.289833333333334 + - type: map_at_10 + value: 34.379333333333335 + - type: map_at_100 + value: 35.56916666666666 + - type: map_at_1000 + value: 35.68633333333333 + - type: map_at_3 + value: 31.63916666666666 + - type: map_at_5 + value: 33.18383333333334 + - type: mrr_at_1 + value: 30.081749999999996 + - type: mrr_at_10 + value: 38.53658333333333 + - type: mrr_at_100 + value: 39.37825 + - type: mrr_at_1000 + value: 39.43866666666666 + - type: mrr_at_3 + value: 36.19025 + - type: mrr_at_5 + value: 37.519749999999995 + - type: ndcg_at_1 + value: 30.081749999999996 + - type: ndcg_at_10 + value: 39.62041666666667 + - type: ndcg_at_100 + value: 44.74825 + - type: ndcg_at_1000 + value: 47.11366666666667 + - type: ndcg_at_3 + value: 35.000499999999995 + - type: ndcg_at_5 + value: 37.19283333333333 + - type: precision_at_1 + value: 30.081749999999996 + - type: precision_at_10 + value: 6.940249999999999 + - type: precision_at_100 + value: 1.1164166666666668 + - type: precision_at_1000 + value: 0.15025000000000002 + - type: precision_at_3 + value: 16.110416666666666 + - type: precision_at_5 + value: 11.474416666666668 + - type: recall_at_1 + value: 25.289833333333334 + - type: recall_at_10 + value: 51.01591666666667 + - type: recall_at_100 + value: 73.55275000000002 + - type: recall_at_1000 + value: 90.02666666666667 + - type: recall_at_3 + value: 38.15208333333334 + - type: recall_at_5 + value: 43.78458333333334 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackStatsRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 23.479 + - type: map_at_10 + value: 31.2 + - type: map_at_100 + value: 32.11 + - type: map_at_1000 + value: 32.214 + - type: map_at_3 + value: 29.093999999999998 + - type: map_at_5 + value: 30.415 + - type: mrr_at_1 + value: 26.840000000000003 + - type: mrr_at_10 + value: 34.153 + - type: mrr_at_100 + value: 34.971000000000004 + - type: mrr_at_1000 + value: 35.047 + - type: mrr_at_3 + value: 32.285000000000004 + - type: mrr_at_5 + value: 33.443 + - type: ndcg_at_1 + value: 26.840000000000003 + - type: ndcg_at_10 + value: 35.441 + - type: ndcg_at_100 + value: 40.150000000000006 + - type: ndcg_at_1000 + value: 42.74 + - type: ndcg_at_3 + value: 31.723000000000003 + - type: ndcg_at_5 + value: 33.71 + - type: precision_at_1 + value: 26.840000000000003 + - type: precision_at_10 + value: 5.552 + - type: precision_at_100 + value: 0.859 + - type: precision_at_1000 + value: 0.11499999999999999 + - type: precision_at_3 + value: 13.804 + - type: precision_at_5 + value: 9.600999999999999 + - type: recall_at_1 + value: 23.479 + - type: recall_at_10 + value: 45.442 + - type: recall_at_100 + value: 67.465 + - type: recall_at_1000 + value: 86.53 + - type: recall_at_3 + value: 35.315999999999995 + - type: recall_at_5 + value: 40.253 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackTexRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 16.887 + - type: map_at_10 + value: 23.805 + - type: map_at_100 + value: 24.804000000000002 + - type: map_at_1000 + value: 24.932000000000002 + - type: map_at_3 + value: 21.632 + - type: map_at_5 + value: 22.845 + - type: mrr_at_1 + value: 20.75 + - type: mrr_at_10 + value: 27.686 + - type: mrr_at_100 + value: 28.522 + - type: mrr_at_1000 + value: 28.605000000000004 + - type: mrr_at_3 + value: 25.618999999999996 + - type: mrr_at_5 + value: 26.723999999999997 + - type: ndcg_at_1 + value: 20.75 + - type: ndcg_at_10 + value: 28.233000000000004 + - type: ndcg_at_100 + value: 33.065 + - type: ndcg_at_1000 + value: 36.138999999999996 + - type: ndcg_at_3 + value: 24.361 + - type: ndcg_at_5 + value: 26.111 + - type: precision_at_1 + value: 20.75 + - type: precision_at_10 + value: 5.124 + - type: precision_at_100 + value: 0.8750000000000001 + - type: precision_at_1000 + value: 0.131 + - type: precision_at_3 + value: 11.539000000000001 + - type: precision_at_5 + value: 8.273 + - type: recall_at_1 + value: 16.887 + - type: recall_at_10 + value: 37.774 + - type: recall_at_100 + value: 59.587 + - type: recall_at_1000 + value: 81.523 + - type: recall_at_3 + value: 26.837 + - type: recall_at_5 + value: 31.456 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackUnixRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 25.534000000000002 + - type: map_at_10 + value: 33.495999999999995 + - type: map_at_100 + value: 34.697 + - type: map_at_1000 + value: 34.805 + - type: map_at_3 + value: 31.22 + - type: map_at_5 + value: 32.277 + - type: mrr_at_1 + value: 29.944 + - type: mrr_at_10 + value: 37.723 + - type: mrr_at_100 + value: 38.645 + - type: mrr_at_1000 + value: 38.712999999999994 + - type: mrr_at_3 + value: 35.665 + - type: mrr_at_5 + value: 36.681999999999995 + - type: ndcg_at_1 + value: 29.944 + - type: ndcg_at_10 + value: 38.407000000000004 + - type: ndcg_at_100 + value: 43.877 + - type: ndcg_at_1000 + value: 46.312 + - type: ndcg_at_3 + value: 34.211000000000006 + - type: ndcg_at_5 + value: 35.760999999999996 + - type: precision_at_1 + value: 29.944 + - type: precision_at_10 + value: 6.343 + - type: precision_at_100 + value: 1.023 + - type: precision_at_1000 + value: 0.133 + - type: precision_at_3 + value: 15.360999999999999 + - type: precision_at_5 + value: 10.428999999999998 + - type: recall_at_1 + value: 25.534000000000002 + - type: recall_at_10 + value: 49.204 + - type: recall_at_100 + value: 72.878 + - type: recall_at_1000 + value: 89.95 + - type: recall_at_3 + value: 37.533 + - type: recall_at_5 + value: 41.611 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackWebmastersRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 26.291999999999998 + - type: map_at_10 + value: 35.245 + - type: map_at_100 + value: 36.762 + - type: map_at_1000 + value: 36.983 + - type: map_at_3 + value: 32.439 + - type: map_at_5 + value: 33.964 + - type: mrr_at_1 + value: 31.423000000000002 + - type: mrr_at_10 + value: 39.98 + - type: mrr_at_100 + value: 40.791 + - type: mrr_at_1000 + value: 40.854 + - type: mrr_at_3 + value: 37.451 + - type: mrr_at_5 + value: 38.854 + - type: ndcg_at_1 + value: 31.423000000000002 + - type: ndcg_at_10 + value: 40.848 + - type: ndcg_at_100 + value: 46.35 + - type: ndcg_at_1000 + value: 49.166 + - type: ndcg_at_3 + value: 36.344 + - type: ndcg_at_5 + value: 38.36 + - type: precision_at_1 + value: 31.423000000000002 + - type: precision_at_10 + value: 7.767 + - type: precision_at_100 + value: 1.498 + - type: precision_at_1000 + value: 0.23700000000000002 + - type: precision_at_3 + value: 16.733 + - type: precision_at_5 + value: 12.213000000000001 + - type: recall_at_1 + value: 26.291999999999998 + - type: recall_at_10 + value: 51.184 + - type: recall_at_100 + value: 76.041 + - type: recall_at_1000 + value: 94.11500000000001 + - type: recall_at_3 + value: 38.257000000000005 + - type: recall_at_5 + value: 43.68 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackWordpressRetrieval + config: default + split: test + revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db + metrics: + - type: map_at_1 + value: 20.715 + - type: map_at_10 + value: 27.810000000000002 + - type: map_at_100 + value: 28.810999999999996 + - type: map_at_1000 + value: 28.904999999999998 + - type: map_at_3 + value: 25.069999999999997 + - type: map_at_5 + value: 26.793 + - type: mrr_at_1 + value: 22.366 + - type: mrr_at_10 + value: 29.65 + - type: mrr_at_100 + value: 30.615 + - type: mrr_at_1000 + value: 30.686999999999998 + - type: mrr_at_3 + value: 27.017999999999997 + - type: mrr_at_5 + value: 28.644 + - type: ndcg_at_1 + value: 22.366 + - type: ndcg_at_10 + value: 32.221 + - type: ndcg_at_100 + value: 37.313 + - type: ndcg_at_1000 + value: 39.871 + - type: ndcg_at_3 + value: 26.918 + - type: ndcg_at_5 + value: 29.813000000000002 + - type: precision_at_1 + value: 22.366 + - type: precision_at_10 + value: 5.139 + - type: precision_at_100 + value: 0.8240000000000001 + - type: precision_at_1000 + value: 0.11199999999999999 + - type: precision_at_3 + value: 11.275 + - type: precision_at_5 + value: 8.540000000000001 + - type: recall_at_1 + value: 20.715 + - type: recall_at_10 + value: 44.023 + - type: recall_at_100 + value: 67.458 + - type: recall_at_1000 + value: 87.066 + - type: recall_at_3 + value: 30.055 + - type: recall_at_5 + value: 36.852000000000004 + - task: + type: Retrieval + dataset: + type: climate-fever + name: MTEB ClimateFEVER + config: default + split: test + revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce + metrics: + - type: map_at_1 + value: 11.859 + - type: map_at_10 + value: 20.625 + - type: map_at_100 + value: 22.5 + - type: map_at_1000 + value: 22.689 + - type: map_at_3 + value: 16.991 + - type: map_at_5 + value: 18.781 + - type: mrr_at_1 + value: 26.906000000000002 + - type: mrr_at_10 + value: 39.083 + - type: mrr_at_100 + value: 39.978 + - type: mrr_at_1000 + value: 40.014 + - type: mrr_at_3 + value: 35.44 + - type: mrr_at_5 + value: 37.619 + - type: ndcg_at_1 + value: 26.906000000000002 + - type: ndcg_at_10 + value: 29.386000000000003 + - type: ndcg_at_100 + value: 36.510999999999996 + - type: ndcg_at_1000 + value: 39.814 + - type: ndcg_at_3 + value: 23.558 + - type: ndcg_at_5 + value: 25.557999999999996 + - type: precision_at_1 + value: 26.906000000000002 + - type: precision_at_10 + value: 9.342 + - type: precision_at_100 + value: 1.6969999999999998 + - type: precision_at_1000 + value: 0.231 + - type: precision_at_3 + value: 17.503 + - type: precision_at_5 + value: 13.655000000000001 + - type: recall_at_1 + value: 11.859 + - type: recall_at_10 + value: 35.929 + - type: recall_at_100 + value: 60.21300000000001 + - type: recall_at_1000 + value: 78.606 + - type: recall_at_3 + value: 21.727 + - type: recall_at_5 + value: 27.349 + - task: + type: Retrieval + dataset: + type: dbpedia-entity + name: MTEB DBPedia + config: default + split: test + revision: f097057d03ed98220bc7309ddb10b71a54d667d6 + metrics: + - type: map_at_1 + value: 8.627 + - type: map_at_10 + value: 18.248 + - type: map_at_100 + value: 25.19 + - type: map_at_1000 + value: 26.741 + - type: map_at_3 + value: 13.286000000000001 + - type: map_at_5 + value: 15.126000000000001 + - type: mrr_at_1 + value: 64.75 + - type: mrr_at_10 + value: 71.865 + - type: mrr_at_100 + value: 72.247 + - type: mrr_at_1000 + value: 72.255 + - type: mrr_at_3 + value: 69.958 + - type: mrr_at_5 + value: 71.108 + - type: ndcg_at_1 + value: 53.25 + - type: ndcg_at_10 + value: 39.035 + - type: ndcg_at_100 + value: 42.735 + - type: ndcg_at_1000 + value: 50.166 + - type: ndcg_at_3 + value: 43.857 + - type: ndcg_at_5 + value: 40.579 + - type: precision_at_1 + value: 64.75 + - type: precision_at_10 + value: 30.75 + - type: precision_at_100 + value: 9.54 + - type: precision_at_1000 + value: 2.035 + - type: precision_at_3 + value: 47.333 + - type: precision_at_5 + value: 39.0 + - type: recall_at_1 + value: 8.627 + - type: recall_at_10 + value: 23.413 + - type: recall_at_100 + value: 48.037 + - type: recall_at_1000 + value: 71.428 + - type: recall_at_3 + value: 14.158999999999999 + - type: recall_at_5 + value: 17.002 + - task: + type: Classification + dataset: + type: mteb/emotion + name: MTEB EmotionClassification + config: default + split: test + revision: 829147f8f75a25f005913200eb5ed41fae320aa1 + metrics: + - type: accuracy + value: 44.865 + - type: f1 + value: 41.56625743266997 + - task: + type: Retrieval + dataset: + type: fever + name: MTEB FEVER + config: default + split: test + revision: 1429cf27e393599b8b359b9b72c666f96b2525f9 + metrics: + - type: map_at_1 + value: 57.335 + - type: map_at_10 + value: 68.29499999999999 + - type: map_at_100 + value: 68.69800000000001 + - type: map_at_1000 + value: 68.714 + - type: map_at_3 + value: 66.149 + - type: map_at_5 + value: 67.539 + - type: mrr_at_1 + value: 61.656 + - type: mrr_at_10 + value: 72.609 + - type: mrr_at_100 + value: 72.923 + - type: mrr_at_1000 + value: 72.928 + - type: mrr_at_3 + value: 70.645 + - type: mrr_at_5 + value: 71.938 + - type: ndcg_at_1 + value: 61.656 + - type: ndcg_at_10 + value: 73.966 + - type: ndcg_at_100 + value: 75.663 + - type: ndcg_at_1000 + value: 75.986 + - type: ndcg_at_3 + value: 69.959 + - type: ndcg_at_5 + value: 72.269 + - type: precision_at_1 + value: 61.656 + - type: precision_at_10 + value: 9.581000000000001 + - type: precision_at_100 + value: 1.054 + - type: precision_at_1000 + value: 0.11 + - type: precision_at_3 + value: 27.743000000000002 + - type: precision_at_5 + value: 17.939 + - type: recall_at_1 + value: 57.335 + - type: recall_at_10 + value: 87.24300000000001 + - type: recall_at_100 + value: 94.575 + - type: recall_at_1000 + value: 96.75399999999999 + - type: recall_at_3 + value: 76.44800000000001 + - type: recall_at_5 + value: 82.122 + - task: + type: Retrieval + dataset: + type: fiqa + name: MTEB FiQA2018 + config: default + split: test + revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be + metrics: + - type: map_at_1 + value: 17.014000000000003 + - type: map_at_10 + value: 28.469 + - type: map_at_100 + value: 30.178 + - type: map_at_1000 + value: 30.369 + - type: map_at_3 + value: 24.63 + - type: map_at_5 + value: 26.891 + - type: mrr_at_1 + value: 34.259 + - type: mrr_at_10 + value: 43.042 + - type: mrr_at_100 + value: 43.91 + - type: mrr_at_1000 + value: 43.963 + - type: mrr_at_3 + value: 40.483999999999995 + - type: mrr_at_5 + value: 42.135 + - type: ndcg_at_1 + value: 34.259 + - type: ndcg_at_10 + value: 35.836 + - type: ndcg_at_100 + value: 42.488 + - type: ndcg_at_1000 + value: 45.902 + - type: ndcg_at_3 + value: 32.131 + - type: ndcg_at_5 + value: 33.697 + - type: precision_at_1 + value: 34.259 + - type: precision_at_10 + value: 10.0 + - type: precision_at_100 + value: 1.699 + - type: precision_at_1000 + value: 0.22999999999999998 + - type: precision_at_3 + value: 21.502 + - type: precision_at_5 + value: 16.296 + - type: recall_at_1 + value: 17.014000000000003 + - type: recall_at_10 + value: 42.832 + - type: recall_at_100 + value: 67.619 + - type: recall_at_1000 + value: 88.453 + - type: recall_at_3 + value: 29.537000000000003 + - type: recall_at_5 + value: 35.886 + - task: + type: Retrieval + dataset: + type: hotpotqa + name: MTEB HotpotQA + config: default + split: test + revision: 766870b35a1b9ca65e67a0d1913899973551fc6c + metrics: + - type: map_at_1 + value: 34.558 + - type: map_at_10 + value: 48.039 + - type: map_at_100 + value: 48.867 + - type: map_at_1000 + value: 48.941 + - type: map_at_3 + value: 45.403 + - type: map_at_5 + value: 46.983999999999995 + - type: mrr_at_1 + value: 69.11500000000001 + - type: mrr_at_10 + value: 75.551 + - type: mrr_at_100 + value: 75.872 + - type: mrr_at_1000 + value: 75.887 + - type: mrr_at_3 + value: 74.447 + - type: mrr_at_5 + value: 75.113 + - type: ndcg_at_1 + value: 69.11500000000001 + - type: ndcg_at_10 + value: 57.25599999999999 + - type: ndcg_at_100 + value: 60.417 + - type: ndcg_at_1000 + value: 61.976 + - type: ndcg_at_3 + value: 53.258 + - type: ndcg_at_5 + value: 55.374 + - type: precision_at_1 + value: 69.11500000000001 + - type: precision_at_10 + value: 11.689 + - type: precision_at_100 + value: 1.418 + - type: precision_at_1000 + value: 0.163 + - type: precision_at_3 + value: 33.018 + - type: precision_at_5 + value: 21.488 + - type: recall_at_1 + value: 34.558 + - type: recall_at_10 + value: 58.447 + - type: recall_at_100 + value: 70.91199999999999 + - type: recall_at_1000 + value: 81.31 + - type: recall_at_3 + value: 49.527 + - type: recall_at_5 + value: 53.72 + - task: + type: Classification + dataset: + type: mteb/imdb + name: MTEB ImdbClassification + config: default + split: test + revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4 + metrics: + - type: accuracy + value: 61.772000000000006 + - type: ap + value: 57.48217702943605 + - type: f1 + value: 61.20495351356274 + - task: + type: Retrieval + dataset: + type: msmarco + name: MTEB MSMARCO + config: default + split: validation + revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849 + metrics: + - type: map_at_1 + value: 22.044 + - type: map_at_10 + value: 34.211000000000006 + - type: map_at_100 + value: 35.394 + - type: map_at_1000 + value: 35.443000000000005 + - type: map_at_3 + value: 30.318 + - type: map_at_5 + value: 32.535 + - type: mrr_at_1 + value: 22.722 + - type: mrr_at_10 + value: 34.842 + - type: mrr_at_100 + value: 35.954 + - type: mrr_at_1000 + value: 35.997 + - type: mrr_at_3 + value: 30.991000000000003 + - type: mrr_at_5 + value: 33.2 + - type: ndcg_at_1 + value: 22.722 + - type: ndcg_at_10 + value: 41.121 + - type: ndcg_at_100 + value: 46.841 + - type: ndcg_at_1000 + value: 48.049 + - type: ndcg_at_3 + value: 33.173 + - type: ndcg_at_5 + value: 37.145 + - type: precision_at_1 + value: 22.722 + - type: precision_at_10 + value: 6.516 + - type: precision_at_100 + value: 0.9400000000000001 + - type: precision_at_1000 + value: 0.104 + - type: precision_at_3 + value: 14.093 + - type: precision_at_5 + value: 10.473 + - type: recall_at_1 + value: 22.044 + - type: recall_at_10 + value: 62.382000000000005 + - type: recall_at_100 + value: 88.914 + - type: recall_at_1000 + value: 98.099 + - type: recall_at_3 + value: 40.782000000000004 + - type: recall_at_5 + value: 50.322 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (en) + config: en + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 93.68217054263563 + - type: f1 + value: 93.25810075739523 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (de) + config: de + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 82.05409974640745 + - type: f1 + value: 80.42814140324903 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (es) + config: es + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 93.54903268845896 + - type: f1 + value: 92.8909878077932 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (fr) + config: fr + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 90.98340119010334 + - type: f1 + value: 90.51522537281313 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (hi) + config: hi + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 89.33309429903191 + - type: f1 + value: 88.60371305209185 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (th) + config: th + split: test + revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 + metrics: + - type: accuracy + value: 60.4882459312839 + - type: f1 + value: 59.02590456131682 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (en) + config: en + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 71.34290925672595 + - type: f1 + value: 54.44803151449109 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (de) + config: de + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 61.92448577063963 + - type: f1 + value: 43.125939975781854 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (es) + config: es + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 74.48965977318213 + - type: f1 + value: 51.855353687466696 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (fr) + config: fr + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 69.11994989038521 + - type: f1 + value: 50.57872704171278 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (hi) + config: hi + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 64.84761563284331 + - type: f1 + value: 43.61322970761394 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (th) + config: th + split: test + revision: 6299947a7777084cc2d4b64235bf7190381ce755 + metrics: + - type: accuracy + value: 49.35623869801085 + - type: f1 + value: 33.48547326952042 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (af) + config: af + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 47.85474108944183 + - type: f1 + value: 46.50175016795915 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (am) + config: am + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 33.29858776059179 + - type: f1 + value: 31.803027601259082 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ar) + config: ar + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 59.24680564895763 + - type: f1 + value: 57.037691806846865 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (az) + config: az + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 45.23537323470073 + - type: f1 + value: 44.81126398428613 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (bn) + config: bn + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 61.590450571620714 + - type: f1 + value: 59.247442149977104 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (cy) + config: cy + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 44.9226630800269 + - type: f1 + value: 44.076183379991654 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (da) + config: da + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 51.23066577000672 + - type: f1 + value: 50.20719330417618 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (de) + config: de + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 56.0995292535306 + - type: f1 + value: 53.29421532133969 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (el) + config: el + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 46.12642905178211 + - type: f1 + value: 44.441530267639635 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (en) + config: en + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 69.67047747141896 + - type: f1 + value: 68.38493366054783 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (es) + config: es + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 66.3483523873571 + - type: f1 + value: 65.13046416817832 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fa) + config: fa + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 51.20040349697378 + - type: f1 + value: 49.02889836601541 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fi) + config: fi + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 45.33288500336248 + - type: f1 + value: 42.91893101970983 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fr) + config: fr + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 66.95359784801613 + - type: f1 + value: 64.98788914810562 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (he) + config: he + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 43.18090114324143 + - type: f1 + value: 41.31250407417542 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hi) + config: hi + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 63.54068594485541 + - type: f1 + value: 61.94829361488948 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hu) + config: hu + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 44.7343644922663 + - type: f1 + value: 43.23001702247849 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hy) + config: hy + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 38.1271015467384 + - type: f1 + value: 36.94700198241727 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (id) + config: id + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 64.05514458641561 + - type: f1 + value: 62.35033731674541 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (is) + config: is + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 44.351042367182245 + - type: f1 + value: 43.13370397574502 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (it) + config: it + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 60.77000672494955 + - type: f1 + value: 59.71546868957779 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ja) + config: ja + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 61.22057834566241 + - type: f1 + value: 59.447639306287044 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (jv) + config: jv + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 50.9448554135844 + - type: f1 + value: 48.524338247875214 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ka) + config: ka + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 33.8399462004035 + - type: f1 + value: 33.518999997305535 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (km) + config: km + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 37.34028244788165 + - type: f1 + value: 35.6156599064704 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (kn) + config: kn + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 53.544048419636844 + - type: f1 + value: 51.29299915455352 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ko) + config: ko + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 53.35574983187625 + - type: f1 + value: 51.463936565192945 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (lv) + config: lv + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 46.503026227303295 + - type: f1 + value: 46.049497734375514 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ml) + config: ml + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 58.268325487558826 + - type: f1 + value: 56.10849656896158 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (mn) + config: mn + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 40.27572293207801 + - type: f1 + value: 40.20097238549224 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ms) + config: ms + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 59.64694014794889 + - type: f1 + value: 58.39584148789066 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (my) + config: my + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 37.41761936785474 + - type: f1 + value: 35.04551731363685 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (nb) + config: nb + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 49.408204438466704 + - type: f1 + value: 48.39369057638714 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (nl) + config: nl + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 52.09482178883659 + - type: f1 + value: 49.91518031712698 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (pl) + config: pl + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 50.477471418964356 + - type: f1 + value: 48.429495257184705 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (pt) + config: pt + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 66.69468728984532 + - type: f1 + value: 65.40306868707009 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ro) + config: ro + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 50.52790854068594 + - type: f1 + value: 49.780400354514 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ru) + config: ru + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 58.31540013449899 + - type: f1 + value: 56.144142926685134 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sl) + config: sl + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 47.74041694687289 + - type: f1 + value: 46.16767322761359 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sq) + config: sq + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 48.94418291862811 + - type: f1 + value: 48.445352284756325 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sv) + config: sv + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 50.78681909885676 + - type: f1 + value: 49.64882295494536 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sw) + config: sw + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 49.811701412239415 + - type: f1 + value: 48.213234514449375 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ta) + config: ta + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 56.39542703429725 + - type: f1 + value: 54.031981085233795 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (te) + config: te + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 54.71082716879623 + - type: f1 + value: 52.513144113474596 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (th) + config: th + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 44.425016812373904 + - type: f1 + value: 43.96016300057656 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (tl) + config: tl + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 50.205110961667785 + - type: f1 + value: 48.86669996798709 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (tr) + config: tr + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 46.56355077336921 + - type: f1 + value: 45.18252022585022 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ur) + config: ur + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 56.748486886348346 + - type: f1 + value: 54.29884570375382 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (vi) + config: vi + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 64.52589105581708 + - type: f1 + value: 62.97947342861603 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (zh-CN) + config: zh-CN + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 67.06792199058508 + - type: f1 + value: 65.36025601634017 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (zh-TW) + config: zh-TW + split: test + revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea + metrics: + - type: accuracy + value: 62.89172831203766 + - type: f1 + value: 62.69803707054342 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (af) + config: af + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 51.47276395427035 + - type: f1 + value: 49.37463208130799 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (am) + config: am + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 34.86886348352387 + - type: f1 + value: 33.74178074349636 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ar) + config: ar + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 65.20511096166778 + - type: f1 + value: 65.85812500602437 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (az) + config: az + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 45.578345662407536 + - type: f1 + value: 44.44514917028003 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (bn) + config: bn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.29657027572293 + - type: f1 + value: 67.24477523937466 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (cy) + config: cy + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 46.29455279085407 + - type: f1 + value: 43.8563839951935 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (da) + config: da + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 53.52387357094821 + - type: f1 + value: 51.70977848027552 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (de) + config: de + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 61.741761936785466 + - type: f1 + value: 60.219169644792295 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (el) + config: el + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 48.957632817753876 + - type: f1 + value: 46.878428264460034 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (en) + config: en + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 75.33624747814393 + - type: f1 + value: 75.9143846211171 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (es) + config: es + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 73.34229993275049 + - type: f1 + value: 73.78165397558983 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fa) + config: fa + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 53.174176193678555 + - type: f1 + value: 51.709679227778985 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fi) + config: fi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 44.6906523201076 + - type: f1 + value: 41.54881682785664 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fr) + config: fr + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 72.9119031607263 + - type: f1 + value: 73.2742013056326 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (he) + config: he + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 43.10356422326832 + - type: f1 + value: 40.8859122581252 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hi) + config: hi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.27370544720914 + - type: f1 + value: 69.39544506405082 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hu) + config: hu + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 45.16476126429052 + - type: f1 + value: 42.74022531579054 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hy) + config: hy + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 38.73234700739744 + - type: f1 + value: 37.40546754951026 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (id) + config: id + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.12777404169468 + - type: f1 + value: 70.27219152812738 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (is) + config: is + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 44.21318090114325 + - type: f1 + value: 41.934593213829366 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (it) + config: it + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 65.57162071284466 + - type: f1 + value: 64.83341759045335 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ja) + config: ja + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 65.75991930060525 + - type: f1 + value: 65.16549875504951 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (jv) + config: jv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 54.79488903833223 + - type: f1 + value: 54.03616401426859 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ka) + config: ka + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 32.992602555480836 + - type: f1 + value: 31.820068470018846 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (km) + config: km + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 39.34431741761937 + - type: f1 + value: 36.436221665290105 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (kn) + config: kn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 60.501008742434436 + - type: f1 + value: 60.051013712579085 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ko) + config: ko + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 55.689307330195035 + - type: f1 + value: 53.94058032286942 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (lv) + config: lv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 44.351042367182245 + - type: f1 + value: 42.05421666771541 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ml) + config: ml + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 65.53127101546738 + - type: f1 + value: 65.98462024333497 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (mn) + config: mn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 38.71553463349025 + - type: f1 + value: 37.44327037149584 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ms) + config: ms + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 64.98991257565567 + - type: f1 + value: 63.87720198978004 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (my) + config: my + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 36.839273705447205 + - type: f1 + value: 35.233967279698376 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (nb) + config: nb + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 51.79892400806993 + - type: f1 + value: 49.66926632125972 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (nl) + config: nl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 56.31809011432415 + - type: f1 + value: 53.832185336179826 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (pl) + config: pl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 49.979825151311374 + - type: f1 + value: 48.83013175441888 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (pt) + config: pt + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 71.45595158036315 + - type: f1 + value: 72.08708814699702 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ro) + config: ro + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 53.68527236045729 + - type: f1 + value: 52.23278593929981 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ru) + config: ru + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 61.60390047074647 + - type: f1 + value: 60.50391482195116 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sl) + config: sl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 48.036314727639535 + - type: f1 + value: 46.43480413383716 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sq) + config: sq + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 50.05716207128445 + - type: f1 + value: 48.85821859948888 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sv) + config: sv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 51.728312037659705 + - type: f1 + value: 49.89292996950847 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sw) + config: sw + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 54.21990585070613 + - type: f1 + value: 52.8711542984193 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ta) + config: ta + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 62.770679219905844 + - type: f1 + value: 63.09441501491594 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (te) + config: te + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 62.58574310692671 + - type: f1 + value: 61.61370697612978 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (th) + config: th + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 45.17821116341628 + - type: f1 + value: 43.85143229183324 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (tl) + config: tl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 52.064559515803644 + - type: f1 + value: 50.94356892049626 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (tr) + config: tr + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 47.205783456624076 + - type: f1 + value: 47.04223644120489 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ur) + config: ur + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 64.25689307330195 + - type: f1 + value: 63.89944944984115 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (vi) + config: vi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.60524546065905 + - type: f1 + value: 71.5634157334358 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (zh-CN) + config: zh-CN + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 73.95427034297242 + - type: f1 + value: 74.39706882311063 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (zh-TW) + config: zh-TW + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.29926025554808 + - type: f1 + value: 71.32045932560297 + - task: + type: Clustering + dataset: + type: mteb/medrxiv-clustering-p2p + name: MTEB MedrxivClusteringP2P + config: default + split: test + revision: dcefc037ef84348e49b0d29109e891c01067226b + metrics: + - type: v_measure + value: 31.054474964883806 + - task: + type: Clustering + dataset: + type: mteb/medrxiv-clustering-s2s + name: MTEB MedrxivClusteringS2S + config: default + split: test + revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc + metrics: + - type: v_measure + value: 29.259725940477523 + - task: + type: Reranking + dataset: + type: mteb/mind_small + name: MTEB MindSmallReranking + config: default + split: test + revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 + metrics: + - type: map + value: 31.785007883256572 + - type: mrr + value: 32.983556622438456 + - task: + type: Retrieval + dataset: + type: nfcorpus + name: MTEB NFCorpus + config: default + split: test + revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610 + metrics: + - type: map_at_1 + value: 5.742 + - type: map_at_10 + value: 13.074 + - type: map_at_100 + value: 16.716 + - type: map_at_1000 + value: 18.238 + - type: map_at_3 + value: 9.600999999999999 + - type: map_at_5 + value: 11.129999999999999 + - type: mrr_at_1 + value: 47.988 + - type: mrr_at_10 + value: 55.958 + - type: mrr_at_100 + value: 56.58800000000001 + - type: mrr_at_1000 + value: 56.620000000000005 + - type: mrr_at_3 + value: 54.025 + - type: mrr_at_5 + value: 55.31 + - type: ndcg_at_1 + value: 46.44 + - type: ndcg_at_10 + value: 35.776 + - type: ndcg_at_100 + value: 32.891999999999996 + - type: ndcg_at_1000 + value: 41.835 + - type: ndcg_at_3 + value: 41.812 + - type: ndcg_at_5 + value: 39.249 + - type: precision_at_1 + value: 48.297000000000004 + - type: precision_at_10 + value: 26.687 + - type: precision_at_100 + value: 8.511000000000001 + - type: precision_at_1000 + value: 2.128 + - type: precision_at_3 + value: 39.009 + - type: precision_at_5 + value: 33.994 + - type: recall_at_1 + value: 5.742 + - type: recall_at_10 + value: 16.993 + - type: recall_at_100 + value: 33.69 + - type: recall_at_1000 + value: 66.75 + - type: recall_at_3 + value: 10.817 + - type: recall_at_5 + value: 13.256 + - task: + type: Retrieval + dataset: + type: nq + name: MTEB NQ + config: default + split: test + revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c + metrics: + - type: map_at_1 + value: 30.789 + - type: map_at_10 + value: 45.751999999999995 + - type: map_at_100 + value: 46.766000000000005 + - type: map_at_1000 + value: 46.798 + - type: map_at_3 + value: 41.746 + - type: map_at_5 + value: 44.046 + - type: mrr_at_1 + value: 34.618 + - type: mrr_at_10 + value: 48.288 + - type: mrr_at_100 + value: 49.071999999999996 + - type: mrr_at_1000 + value: 49.094 + - type: mrr_at_3 + value: 44.979 + - type: mrr_at_5 + value: 46.953 + - type: ndcg_at_1 + value: 34.589 + - type: ndcg_at_10 + value: 53.151 + - type: ndcg_at_100 + value: 57.537000000000006 + - type: ndcg_at_1000 + value: 58.321999999999996 + - type: ndcg_at_3 + value: 45.628 + - type: ndcg_at_5 + value: 49.474000000000004 + - type: precision_at_1 + value: 34.589 + - type: precision_at_10 + value: 8.731 + - type: precision_at_100 + value: 1.119 + - type: precision_at_1000 + value: 0.11900000000000001 + - type: precision_at_3 + value: 20.819 + - type: precision_at_5 + value: 14.728 + - type: recall_at_1 + value: 30.789 + - type: recall_at_10 + value: 73.066 + - type: recall_at_100 + value: 92.27 + - type: recall_at_1000 + value: 98.18 + - type: recall_at_3 + value: 53.632999999999996 + - type: recall_at_5 + value: 62.476 + - task: + type: Retrieval + dataset: + type: quora + name: MTEB QuoraRetrieval + config: default + split: test + revision: 6205996560df11e3a3da9ab4f926788fc30a7db4 + metrics: + - type: map_at_1 + value: 54.993 + - type: map_at_10 + value: 69.07600000000001 + - type: map_at_100 + value: 70.05799999999999 + - type: map_at_1000 + value: 70.09 + - type: map_at_3 + value: 65.456 + - type: map_at_5 + value: 67.622 + - type: mrr_at_1 + value: 63.07000000000001 + - type: mrr_at_10 + value: 72.637 + - type: mrr_at_100 + value: 73.029 + - type: mrr_at_1000 + value: 73.033 + - type: mrr_at_3 + value: 70.572 + - type: mrr_at_5 + value: 71.86399999999999 + - type: ndcg_at_1 + value: 63.07000000000001 + - type: ndcg_at_10 + value: 74.708 + - type: ndcg_at_100 + value: 77.579 + - type: ndcg_at_1000 + value: 77.897 + - type: ndcg_at_3 + value: 69.69999999999999 + - type: ndcg_at_5 + value: 72.321 + - type: precision_at_1 + value: 63.07000000000001 + - type: precision_at_10 + value: 11.851 + - type: precision_at_100 + value: 1.481 + - type: precision_at_1000 + value: 0.156 + - type: precision_at_3 + value: 30.747000000000003 + - type: precision_at_5 + value: 20.830000000000002 + - type: recall_at_1 + value: 54.993 + - type: recall_at_10 + value: 87.18900000000001 + - type: recall_at_100 + value: 98.137 + - type: recall_at_1000 + value: 99.833 + - type: recall_at_3 + value: 73.654 + - type: recall_at_5 + value: 80.36 + - task: + type: Clustering + dataset: + type: mteb/reddit-clustering + name: MTEB RedditClustering + config: default + split: test + revision: b2805658ae38990172679479369a78b86de8c390 + metrics: + - type: v_measure + value: 35.53178375429036 + - task: + type: Clustering + dataset: + type: mteb/reddit-clustering-p2p + name: MTEB RedditClusteringP2P + config: default + split: test + revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 + metrics: + - type: v_measure + value: 54.520782970558265 + - task: + type: Retrieval + dataset: + type: scidocs + name: MTEB SCIDOCS + config: default + split: test + revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 + metrics: + - type: map_at_1 + value: 4.3229999999999995 + - type: map_at_10 + value: 10.979999999999999 + - type: map_at_100 + value: 12.867 + - type: map_at_1000 + value: 13.147 + - type: map_at_3 + value: 7.973 + - type: map_at_5 + value: 9.513 + - type: mrr_at_1 + value: 21.3 + - type: mrr_at_10 + value: 32.34 + - type: mrr_at_100 + value: 33.428999999999995 + - type: mrr_at_1000 + value: 33.489999999999995 + - type: mrr_at_3 + value: 28.999999999999996 + - type: mrr_at_5 + value: 31.019999999999996 + - type: ndcg_at_1 + value: 21.3 + - type: ndcg_at_10 + value: 18.619 + - type: ndcg_at_100 + value: 26.108999999999998 + - type: ndcg_at_1000 + value: 31.253999999999998 + - type: ndcg_at_3 + value: 17.842 + - type: ndcg_at_5 + value: 15.673 + - type: precision_at_1 + value: 21.3 + - type: precision_at_10 + value: 9.55 + - type: precision_at_100 + value: 2.0340000000000003 + - type: precision_at_1000 + value: 0.327 + - type: precision_at_3 + value: 16.667 + - type: precision_at_5 + value: 13.76 + - type: recall_at_1 + value: 4.3229999999999995 + - type: recall_at_10 + value: 19.387 + - type: recall_at_100 + value: 41.307 + - type: recall_at_1000 + value: 66.475 + - type: recall_at_3 + value: 10.143 + - type: recall_at_5 + value: 14.007 + - task: + type: STS + dataset: + type: mteb/sickr-sts + name: MTEB SICK-R + config: default + split: test + revision: 20a6d6f312dd54037fe07a32d58e5e168867909d + metrics: + - type: cos_sim_pearson + value: 78.77975189382573 + - type: cos_sim_spearman + value: 69.81522686267631 + - type: euclidean_pearson + value: 71.37617936889518 + - type: euclidean_spearman + value: 65.71738481148611 + - type: manhattan_pearson + value: 71.58222165832424 + - type: manhattan_spearman + value: 65.86851365286654 + - task: + type: STS + dataset: + type: mteb/sts12-sts + name: MTEB STS12 + config: default + split: test + revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f + metrics: + - type: cos_sim_pearson + value: 77.75509450443367 + - type: cos_sim_spearman + value: 69.66180222442091 + - type: euclidean_pearson + value: 74.98512779786111 + - type: euclidean_spearman + value: 69.5997451409469 + - type: manhattan_pearson + value: 75.50135090962459 + - type: manhattan_spearman + value: 69.94984748475302 + - task: + type: STS + dataset: + type: mteb/sts13-sts + name: MTEB STS13 + config: default + split: test + revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 + metrics: + - type: cos_sim_pearson + value: 79.42363892383264 + - type: cos_sim_spearman + value: 79.66529244176742 + - type: euclidean_pearson + value: 79.50429208135942 + - type: euclidean_spearman + value: 80.44767586416276 + - type: manhattan_pearson + value: 79.58563944997708 + - type: manhattan_spearman + value: 80.51452267103 + - task: + type: STS + dataset: + type: mteb/sts14-sts + name: MTEB STS14 + config: default + split: test + revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b + metrics: + - type: cos_sim_pearson + value: 79.2749401478149 + - type: cos_sim_spearman + value: 74.6076920702392 + - type: euclidean_pearson + value: 73.3302002952881 + - type: euclidean_spearman + value: 70.67029803077013 + - type: manhattan_pearson + value: 73.52699344010296 + - type: manhattan_spearman + value: 70.8517556194297 + - task: + type: STS + dataset: + type: mteb/sts15-sts + name: MTEB STS15 + config: default + split: test + revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 + metrics: + - type: cos_sim_pearson + value: 83.20884740785921 + - type: cos_sim_spearman + value: 83.80600789090722 + - type: euclidean_pearson + value: 74.9154089816344 + - type: euclidean_spearman + value: 75.69243899592276 + - type: manhattan_pearson + value: 75.0312832634451 + - type: manhattan_spearman + value: 75.78324960357642 + - task: + type: STS + dataset: + type: mteb/sts16-sts + name: MTEB STS16 + config: default + split: test + revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd + metrics: + - type: cos_sim_pearson + value: 79.63194141000497 + - type: cos_sim_spearman + value: 80.40118418350866 + - type: euclidean_pearson + value: 72.07354384551088 + - type: euclidean_spearman + value: 72.28819150373845 + - type: manhattan_pearson + value: 72.08736119834145 + - type: manhattan_spearman + value: 72.28347083261288 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (ko-ko) + config: ko-ko + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 66.78512789499386 + - type: cos_sim_spearman + value: 66.89125587193288 + - type: euclidean_pearson + value: 58.74535708627959 + - type: euclidean_spearman + value: 59.62103716794647 + - type: manhattan_pearson + value: 59.00494529143961 + - type: manhattan_spearman + value: 59.832257846799806 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (ar-ar) + config: ar-ar + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 75.48960503523992 + - type: cos_sim_spearman + value: 76.4223037534204 + - type: euclidean_pearson + value: 64.93966381820944 + - type: euclidean_spearman + value: 62.39697395373789 + - type: manhattan_pearson + value: 65.54480770061505 + - type: manhattan_spearman + value: 62.944204863043105 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-ar) + config: en-ar + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 77.7331440643619 + - type: cos_sim_spearman + value: 78.0748413292835 + - type: euclidean_pearson + value: 38.533108233460304 + - type: euclidean_spearman + value: 35.37638615280026 + - type: manhattan_pearson + value: 41.0639726746513 + - type: manhattan_spearman + value: 37.688161243671765 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-de) + config: en-de + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 58.4628923720782 + - type: cos_sim_spearman + value: 59.10093128795948 + - type: euclidean_pearson + value: 30.422902393436836 + - type: euclidean_spearman + value: 27.837806030497457 + - type: manhattan_pearson + value: 32.51576984630963 + - type: manhattan_spearman + value: 29.181887010982514 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-en) + config: en-en + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 86.87447904613737 + - type: cos_sim_spearman + value: 87.06554974065622 + - type: euclidean_pearson + value: 76.82669047851108 + - type: euclidean_spearman + value: 75.45711985511991 + - type: manhattan_pearson + value: 77.46644556452847 + - type: manhattan_spearman + value: 76.0249120007112 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-tr) + config: en-tr + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 17.784495723497468 + - type: cos_sim_spearman + value: 11.79629537128697 + - type: euclidean_pearson + value: -4.354328445994008 + - type: euclidean_spearman + value: -6.984566116230058 + - type: manhattan_pearson + value: -4.166751901507852 + - type: manhattan_spearman + value: -6.984143198323786 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (es-en) + config: es-en + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 76.9009642643449 + - type: cos_sim_spearman + value: 78.21764726338341 + - type: euclidean_pearson + value: 50.578959144342925 + - type: euclidean_spearman + value: 51.664379260719606 + - type: manhattan_pearson + value: 53.95690880393329 + - type: manhattan_spearman + value: 54.910058464050785 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (es-es) + config: es-es + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 86.41638022270219 + - type: cos_sim_spearman + value: 86.00477030366811 + - type: euclidean_pearson + value: 79.7224037788285 + - type: euclidean_spearman + value: 79.21417626867616 + - type: manhattan_pearson + value: 80.29412412756984 + - type: manhattan_spearman + value: 79.49460867616206 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (fr-en) + config: fr-en + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 79.90432664091082 + - type: cos_sim_spearman + value: 80.46007940700204 + - type: euclidean_pearson + value: 49.25348015214428 + - type: euclidean_spearman + value: 47.13113020475859 + - type: manhattan_pearson + value: 54.57291204043908 + - type: manhattan_spearman + value: 51.98559736896087 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (it-en) + config: it-en + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 52.55164822309034 + - type: cos_sim_spearman + value: 51.57629192137736 + - type: euclidean_pearson + value: 16.63360593235354 + - type: euclidean_spearman + value: 14.479679923782912 + - type: manhattan_pearson + value: 18.524867185117472 + - type: manhattan_spearman + value: 16.65940056664755 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (nl-en) + config: nl-en + split: test + revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 + metrics: + - type: cos_sim_pearson + value: 46.83690919715875 + - type: cos_sim_spearman + value: 45.84993650002922 + - type: euclidean_pearson + value: 6.173128686815117 + - type: euclidean_spearman + value: 6.260781946306191 + - type: manhattan_pearson + value: 7.328440452367316 + - type: manhattan_spearman + value: 7.370842306497447 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (en) + config: en + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 64.97916914277232 + - type: cos_sim_spearman + value: 66.13392188807865 + - type: euclidean_pearson + value: 65.3921146908468 + - type: euclidean_spearman + value: 65.8381588635056 + - type: manhattan_pearson + value: 65.8866165769975 + - type: manhattan_spearman + value: 66.27774050472219 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de) + config: de + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 25.605130445111545 + - type: cos_sim_spearman + value: 30.054844562369254 + - type: euclidean_pearson + value: 23.890611005408196 + - type: euclidean_spearman + value: 29.07902600726761 + - type: manhattan_pearson + value: 24.239478426621833 + - type: manhattan_spearman + value: 29.48547576782375 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es) + config: es + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 61.6665616159781 + - type: cos_sim_spearman + value: 65.41310206289988 + - type: euclidean_pearson + value: 68.38805493215008 + - type: euclidean_spearman + value: 65.22777377603435 + - type: manhattan_pearson + value: 69.37445390454346 + - type: manhattan_spearman + value: 66.02437701858754 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (pl) + config: pl + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 15.302891825626372 + - type: cos_sim_spearman + value: 31.134517255070097 + - type: euclidean_pearson + value: 12.672592658843143 + - type: euclidean_spearman + value: 29.14881036784207 + - type: manhattan_pearson + value: 13.528545327757735 + - type: manhattan_spearman + value: 29.56217928148797 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (tr) + config: tr + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 28.79299114515319 + - type: cos_sim_spearman + value: 47.135864983626206 + - type: euclidean_pearson + value: 40.66410787594309 + - type: euclidean_spearman + value: 45.09585593138228 + - type: manhattan_pearson + value: 42.02561630700308 + - type: manhattan_spearman + value: 45.43979983670554 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (ar) + config: ar + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 46.00096625052943 + - type: cos_sim_spearman + value: 58.67147426715496 + - type: euclidean_pearson + value: 54.7154367422438 + - type: euclidean_spearman + value: 59.003235142442634 + - type: manhattan_pearson + value: 56.3116235357115 + - type: manhattan_spearman + value: 60.12956331404423 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (ru) + config: ru + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 29.3396354650316 + - type: cos_sim_spearman + value: 43.3632935734809 + - type: euclidean_pearson + value: 31.18506539466593 + - type: euclidean_spearman + value: 37.531745324803815 + - type: manhattan_pearson + value: 32.829038232529015 + - type: manhattan_spearman + value: 38.04574361589953 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (zh) + config: zh + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 62.9596148375188 + - type: cos_sim_spearman + value: 66.77653412402461 + - type: euclidean_pearson + value: 64.53156585980886 + - type: euclidean_spearman + value: 66.2884373036083 + - type: manhattan_pearson + value: 65.2831035495143 + - type: manhattan_spearman + value: 66.83641945244322 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (fr) + config: fr + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 79.9138821493919 + - type: cos_sim_spearman + value: 80.38097535004677 + - type: euclidean_pearson + value: 76.2401499094322 + - type: euclidean_spearman + value: 77.00897050735907 + - type: manhattan_pearson + value: 76.69531453728563 + - type: manhattan_spearman + value: 77.83189696428695 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-en) + config: de-en + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 51.27009640779202 + - type: cos_sim_spearman + value: 51.16120562029285 + - type: euclidean_pearson + value: 52.20594985566323 + - type: euclidean_spearman + value: 52.75331049709882 + - type: manhattan_pearson + value: 52.2725118792549 + - type: manhattan_spearman + value: 53.614847968995115 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es-en) + config: es-en + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 70.46044814118835 + - type: cos_sim_spearman + value: 75.05760236668672 + - type: euclidean_pearson + value: 72.80128921879461 + - type: euclidean_spearman + value: 73.81164755219257 + - type: manhattan_pearson + value: 72.7863795809044 + - type: manhattan_spearman + value: 73.65932033818906 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (it) + config: it + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 61.89276840435938 + - type: cos_sim_spearman + value: 65.65042955732055 + - type: euclidean_pearson + value: 61.22969491863841 + - type: euclidean_spearman + value: 63.451215637904724 + - type: manhattan_pearson + value: 61.16138956945465 + - type: manhattan_spearman + value: 63.34966179331079 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (pl-en) + config: pl-en + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 56.377577221753626 + - type: cos_sim_spearman + value: 53.31223653270353 + - type: euclidean_pearson + value: 26.488793041564307 + - type: euclidean_spearman + value: 19.524551741701472 + - type: manhattan_pearson + value: 24.322868054606474 + - type: manhattan_spearman + value: 19.50371443994939 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (zh-en) + config: zh-en + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 69.3634693673425 + - type: cos_sim_spearman + value: 68.45051245419702 + - type: euclidean_pearson + value: 56.1417414374769 + - type: euclidean_spearman + value: 55.89891749631458 + - type: manhattan_pearson + value: 57.266417430882925 + - type: manhattan_spearman + value: 56.57927102744128 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es-it) + config: es-it + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 60.04169437653179 + - type: cos_sim_spearman + value: 65.49531007553446 + - type: euclidean_pearson + value: 58.583860732586324 + - type: euclidean_spearman + value: 58.80034792537441 + - type: manhattan_pearson + value: 59.02513161664622 + - type: manhattan_spearman + value: 58.42942047904558 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-fr) + config: de-fr + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 48.81035211493999 + - type: cos_sim_spearman + value: 53.27599246786967 + - type: euclidean_pearson + value: 52.25710699032889 + - type: euclidean_spearman + value: 55.22995695529873 + - type: manhattan_pearson + value: 51.894901893217884 + - type: manhattan_spearman + value: 54.95919975149795 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-pl) + config: de-pl + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 36.75993101477816 + - type: cos_sim_spearman + value: 43.050156692479355 + - type: euclidean_pearson + value: 51.49021084746248 + - type: euclidean_spearman + value: 49.54771253090078 + - type: manhattan_pearson + value: 54.68410760796417 + - type: manhattan_spearman + value: 48.19277197691717 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (fr-pl) + config: fr-pl + split: test + revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 + metrics: + - type: cos_sim_pearson + value: 48.553763306386486 + - type: cos_sim_spearman + value: 28.17180849095055 + - type: euclidean_pearson + value: 17.50739087826514 + - type: euclidean_spearman + value: 16.903085094570333 + - type: manhattan_pearson + value: 20.750046512534112 + - type: manhattan_spearman + value: 5.634361698190111 + - task: + type: STS + dataset: + type: mteb/stsbenchmark-sts + name: MTEB STSBenchmark + config: default + split: test + revision: 8913289635987208e6e7c72789e4be2fe94b6abd + metrics: + - type: cos_sim_pearson + value: 82.17107190594417 + - type: cos_sim_spearman + value: 80.89611873505183 + - type: euclidean_pearson + value: 71.82491561814403 + - type: euclidean_spearman + value: 70.33608835403274 + - type: manhattan_pearson + value: 71.89538332420133 + - type: manhattan_spearman + value: 70.36082395775944 + - task: + type: Reranking + dataset: + type: mteb/scidocs-reranking + name: MTEB SciDocsRR + config: default + split: test + revision: 56a6d0140cf6356659e2a7c1413286a774468d44 + metrics: + - type: map + value: 79.77047154974562 + - type: mrr + value: 94.25887021475256 + - task: + type: Retrieval + dataset: + type: scifact + name: MTEB SciFact + config: default + split: test + revision: a75ae049398addde9b70f6b268875f5cbce99089 + metrics: + - type: map_at_1 + value: 56.328 + - type: map_at_10 + value: 67.167 + - type: map_at_100 + value: 67.721 + - type: map_at_1000 + value: 67.735 + - type: map_at_3 + value: 64.20400000000001 + - type: map_at_5 + value: 65.904 + - type: mrr_at_1 + value: 59.667 + - type: mrr_at_10 + value: 68.553 + - type: mrr_at_100 + value: 68.992 + - type: mrr_at_1000 + value: 69.004 + - type: mrr_at_3 + value: 66.22200000000001 + - type: mrr_at_5 + value: 67.739 + - type: ndcg_at_1 + value: 59.667 + - type: ndcg_at_10 + value: 72.111 + - type: ndcg_at_100 + value: 74.441 + - type: ndcg_at_1000 + value: 74.90599999999999 + - type: ndcg_at_3 + value: 67.11399999999999 + - type: ndcg_at_5 + value: 69.687 + - type: precision_at_1 + value: 59.667 + - type: precision_at_10 + value: 9.733 + - type: precision_at_100 + value: 1.09 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_3 + value: 26.444000000000003 + - type: precision_at_5 + value: 17.599999999999998 + - type: recall_at_1 + value: 56.328 + - type: recall_at_10 + value: 85.8 + - type: recall_at_100 + value: 96.167 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_3 + value: 72.433 + - type: recall_at_5 + value: 78.972 + - task: + type: PairClassification + dataset: + type: mteb/sprintduplicatequestions-pairclassification + name: MTEB SprintDuplicateQuestions + config: default + split: test + revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea + metrics: + - type: cos_sim_accuracy + value: 99.8019801980198 + - type: cos_sim_ap + value: 94.92527097094644 + - type: cos_sim_f1 + value: 89.91935483870968 + - type: cos_sim_precision + value: 90.65040650406505 + - type: cos_sim_recall + value: 89.2 + - type: dot_accuracy + value: 99.51782178217822 + - type: dot_ap + value: 81.30756869559929 + - type: dot_f1 + value: 75.88235294117648 + - type: dot_precision + value: 74.42307692307692 + - type: dot_recall + value: 77.4 + - type: euclidean_accuracy + value: 99.73069306930694 + - type: euclidean_ap + value: 91.05040371796932 + - type: euclidean_f1 + value: 85.7889237199582 + - type: euclidean_precision + value: 89.82494529540482 + - type: euclidean_recall + value: 82.1 + - type: manhattan_accuracy + value: 99.73762376237623 + - type: manhattan_ap + value: 91.4823412839869 + - type: manhattan_f1 + value: 86.39836984207845 + - type: manhattan_precision + value: 88.05815160955348 + - type: manhattan_recall + value: 84.8 + - type: max_accuracy + value: 99.8019801980198 + - type: max_ap + value: 94.92527097094644 + - type: max_f1 + value: 89.91935483870968 + - task: + type: Clustering + dataset: + type: mteb/stackexchange-clustering + name: MTEB StackExchangeClustering + config: default + split: test + revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235 + metrics: + - type: v_measure + value: 55.13046832022158 + - task: + type: Clustering + dataset: + type: mteb/stackexchange-clustering-p2p + name: MTEB StackExchangeClusteringP2P + config: default + split: test + revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0 + metrics: + - type: v_measure + value: 34.31252463546675 + - task: + type: Reranking + dataset: + type: mteb/stackoverflowdupquestions-reranking + name: MTEB StackOverflowDupQuestions + config: default + split: test + revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 + metrics: + - type: map + value: 51.06639688231414 + - type: mrr + value: 51.80205415499534 + - task: + type: Summarization + dataset: + type: mteb/summeval + name: MTEB SummEval + config: default + split: test + revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122 + metrics: + - type: cos_sim_pearson + value: 24.27681035619567 + - type: cos_sim_spearman + value: 24.989532429036974 + - type: dot_pearson + value: 26.468920496607705 + - type: dot_spearman + value: 27.254486462692025 + - task: + type: Retrieval + dataset: + type: trec-covid + name: MTEB TRECCOVID + config: default + split: test + revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 + metrics: + - type: map_at_1 + value: 0.23600000000000002 + - type: map_at_10 + value: 2.09 + - type: map_at_100 + value: 12.466000000000001 + - type: map_at_1000 + value: 29.852 + - type: map_at_3 + value: 0.6859999999999999 + - type: map_at_5 + value: 1.099 + - type: mrr_at_1 + value: 88.0 + - type: mrr_at_10 + value: 94.0 + - type: mrr_at_100 + value: 94.0 + - type: mrr_at_1000 + value: 94.0 + - type: mrr_at_3 + value: 94.0 + - type: mrr_at_5 + value: 94.0 + - type: ndcg_at_1 + value: 86.0 + - type: ndcg_at_10 + value: 81.368 + - type: ndcg_at_100 + value: 61.879 + - type: ndcg_at_1000 + value: 55.282 + - type: ndcg_at_3 + value: 84.816 + - type: ndcg_at_5 + value: 82.503 + - type: precision_at_1 + value: 88.0 + - type: precision_at_10 + value: 85.6 + - type: precision_at_100 + value: 63.85999999999999 + - type: precision_at_1000 + value: 24.682000000000002 + - type: precision_at_3 + value: 88.667 + - type: precision_at_5 + value: 86.0 + - type: recall_at_1 + value: 0.23600000000000002 + - type: recall_at_10 + value: 2.25 + - type: recall_at_100 + value: 15.488 + - type: recall_at_1000 + value: 52.196 + - type: recall_at_3 + value: 0.721 + - type: recall_at_5 + value: 1.159 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (sqi-eng) + config: sqi-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 12.7 + - type: f1 + value: 10.384182044950325 + - type: precision + value: 9.805277385275312 + - type: recall + value: 12.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fry-eng) + config: fry-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 30.63583815028902 + - type: f1 + value: 24.623726947426373 + - type: precision + value: 22.987809919828013 + - type: recall + value: 30.63583815028902 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kur-eng) + config: kur-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 10.487804878048781 + - type: f1 + value: 8.255945048627975 + - type: precision + value: 7.649047253615001 + - type: recall + value: 10.487804878048781 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tur-eng) + config: tur-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.5 + - type: f1 + value: 6.154428783776609 + - type: precision + value: 5.680727638128585 + - type: recall + value: 8.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (deu-eng) + config: deu-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 73.0 + - type: f1 + value: 70.10046605876393 + - type: precision + value: 69.0018253968254 + - type: recall + value: 73.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nld-eng) + config: nld-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 32.7 + - type: f1 + value: 29.7428583868239 + - type: precision + value: 28.81671359506905 + - type: recall + value: 32.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ron-eng) + config: ron-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 31.5 + - type: f1 + value: 27.228675552174003 + - type: precision + value: 25.950062299847747 + - type: recall + value: 31.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ang-eng) + config: ang-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 35.82089552238806 + - type: f1 + value: 28.75836980510979 + - type: precision + value: 26.971643613434658 + - type: recall + value: 35.82089552238806 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ido-eng) + config: ido-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 49.8 + - type: f1 + value: 43.909237401451776 + - type: precision + value: 41.944763440988936 + - type: recall + value: 49.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (jav-eng) + config: jav-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 18.536585365853657 + - type: f1 + value: 15.020182570246751 + - type: precision + value: 14.231108073213337 + - type: recall + value: 18.536585365853657 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (isl-eng) + config: isl-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.7 + - type: f1 + value: 6.2934784902885355 + - type: precision + value: 5.685926293425392 + - type: recall + value: 8.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (slv-eng) + config: slv-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 12.879708383961116 + - type: f1 + value: 10.136118341751114 + - type: precision + value: 9.571444036679436 + - type: recall + value: 12.879708383961116 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cym-eng) + config: cym-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 9.217391304347826 + - type: f1 + value: 6.965003297761793 + - type: precision + value: 6.476093529199119 + - type: recall + value: 9.217391304347826 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kaz-eng) + config: kaz-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 4.3478260869565215 + - type: f1 + value: 3.3186971707677397 + - type: precision + value: 3.198658632552104 + - type: recall + value: 4.3478260869565215 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (est-eng) + config: est-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 6.9 + - type: f1 + value: 4.760708297894056 + - type: precision + value: 4.28409511756074 + - type: recall + value: 6.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (heb-eng) + config: heb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 2.1999999999999997 + - type: f1 + value: 1.6862703878117107 + - type: precision + value: 1.6048118233915603 + - type: recall + value: 2.1999999999999997 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gla-eng) + config: gla-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 3.0156815440289506 + - type: f1 + value: 2.0913257250659134 + - type: precision + value: 1.9072775486461648 + - type: recall + value: 3.0156815440289506 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mar-eng) + config: mar-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 49.0 + - type: f1 + value: 45.5254456536713 + - type: precision + value: 44.134609250398725 + - type: recall + value: 49.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lat-eng) + config: lat-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 33.5 + - type: f1 + value: 28.759893973182564 + - type: precision + value: 27.401259116024836 + - type: recall + value: 33.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bel-eng) + config: bel-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 10.2 + - type: f1 + value: 8.030039981676275 + - type: precision + value: 7.548748077210127 + - type: recall + value: 10.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pms-eng) + config: pms-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 38.095238095238095 + - type: f1 + value: 31.944999250262406 + - type: precision + value: 30.04452690166976 + - type: recall + value: 38.095238095238095 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gle-eng) + config: gle-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 4.8 + - type: f1 + value: 3.2638960786708067 + - type: precision + value: 3.0495382950729644 + - type: recall + value: 4.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pes-eng) + config: pes-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 15.8 + - type: f1 + value: 12.131087470371275 + - type: precision + value: 11.141304011547815 + - type: recall + value: 15.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nob-eng) + config: nob-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 23.3 + - type: f1 + value: 21.073044636921384 + - type: precision + value: 20.374220568287285 + - type: recall + value: 23.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bul-eng) + config: bul-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 24.9 + - type: f1 + value: 20.091060685364987 + - type: precision + value: 18.899700591081224 + - type: recall + value: 24.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cbk-eng) + config: cbk-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 70.1 + - type: f1 + value: 64.62940836940835 + - type: precision + value: 62.46559523809524 + - type: recall + value: 70.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hun-eng) + config: hun-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.199999999999999 + - type: f1 + value: 5.06613460576115 + - type: precision + value: 4.625224463391809 + - type: recall + value: 7.199999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (uig-eng) + config: uig-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 1.7999999999999998 + - type: f1 + value: 1.2716249514772895 + - type: precision + value: 1.2107445914723798 + - type: recall + value: 1.7999999999999998 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (rus-eng) + config: rus-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 65.5 + - type: f1 + value: 59.84399711399712 + - type: precision + value: 57.86349567099567 + - type: recall + value: 65.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (spa-eng) + config: spa-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 95.7 + - type: f1 + value: 94.48333333333333 + - type: precision + value: 93.89999999999999 + - type: recall + value: 95.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hye-eng) + config: hye-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 0.8086253369272237 + - type: f1 + value: 0.4962046191492002 + - type: precision + value: 0.47272438578554393 + - type: recall + value: 0.8086253369272237 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tel-eng) + config: tel-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 69.23076923076923 + - type: f1 + value: 64.6227941099736 + - type: precision + value: 63.03795877325289 + - type: recall + value: 69.23076923076923 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (afr-eng) + config: afr-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 20.599999999999998 + - type: f1 + value: 16.62410040660465 + - type: precision + value: 15.598352437967069 + - type: recall + value: 20.599999999999998 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mon-eng) + config: mon-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 4.318181818181818 + - type: f1 + value: 2.846721192535661 + - type: precision + value: 2.6787861417537147 + - type: recall + value: 4.318181818181818 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (arz-eng) + config: arz-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 74.84276729559748 + - type: f1 + value: 70.6638714185884 + - type: precision + value: 68.86792452830188 + - type: recall + value: 74.84276729559748 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hrv-eng) + config: hrv-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 15.9 + - type: f1 + value: 12.793698974586706 + - type: precision + value: 12.088118017657736 + - type: recall + value: 15.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nov-eng) + config: nov-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 59.92217898832685 + - type: f1 + value: 52.23086900129701 + - type: precision + value: 49.25853869433636 + - type: recall + value: 59.92217898832685 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gsw-eng) + config: gsw-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 27.350427350427353 + - type: f1 + value: 21.033781033781032 + - type: precision + value: 19.337955491801644 + - type: recall + value: 27.350427350427353 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nds-eng) + config: nds-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 29.299999999999997 + - type: f1 + value: 23.91597452425777 + - type: precision + value: 22.36696598364942 + - type: recall + value: 29.299999999999997 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ukr-eng) + config: ukr-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 27.3 + - type: f1 + value: 22.059393517688886 + - type: precision + value: 20.503235534170887 + - type: recall + value: 27.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (uzb-eng) + config: uzb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.177570093457943 + - type: f1 + value: 4.714367017906037 + - type: precision + value: 4.163882933965758 + - type: recall + value: 8.177570093457943 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lit-eng) + config: lit-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 5.800000000000001 + - type: f1 + value: 4.4859357432293825 + - type: precision + value: 4.247814465614043 + - type: recall + value: 5.800000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ina-eng) + config: ina-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 78.4 + - type: f1 + value: 73.67166666666667 + - type: precision + value: 71.83285714285714 + - type: recall + value: 78.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lfn-eng) + config: lfn-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 50.3 + - type: f1 + value: 44.85221545883311 + - type: precision + value: 43.04913026243909 + - type: recall + value: 50.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (zsm-eng) + config: zsm-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 83.5 + - type: f1 + value: 79.95151515151515 + - type: precision + value: 78.53611111111111 + - type: recall + value: 83.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ita-eng) + config: ita-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 69.89999999999999 + - type: f1 + value: 65.03756269256269 + - type: precision + value: 63.233519536019536 + - type: recall + value: 69.89999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cmn-eng) + config: cmn-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 93.2 + - type: f1 + value: 91.44666666666666 + - type: precision + value: 90.63333333333333 + - type: recall + value: 93.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lvs-eng) + config: lvs-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.3 + - type: f1 + value: 6.553388144729963 + - type: precision + value: 6.313497782829976 + - type: recall + value: 8.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (glg-eng) + config: glg-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 83.6 + - type: f1 + value: 79.86243107769424 + - type: precision + value: 78.32555555555555 + - type: recall + value: 83.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ceb-eng) + config: ceb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 9.166666666666666 + - type: f1 + value: 6.637753604420271 + - type: precision + value: 6.10568253585495 + - type: recall + value: 9.166666666666666 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bre-eng) + config: bre-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.3999999999999995 + - type: f1 + value: 4.6729483612322165 + - type: precision + value: 4.103844520292658 + - type: recall + value: 7.3999999999999995 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ben-eng) + config: ben-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 80.30000000000001 + - type: f1 + value: 75.97666666666667 + - type: precision + value: 74.16 + - type: recall + value: 80.30000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swg-eng) + config: swg-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 23.214285714285715 + - type: f1 + value: 16.88988095238095 + - type: precision + value: 15.364937641723353 + - type: recall + value: 23.214285714285715 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (arq-eng) + config: arq-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 33.15038419319429 + - type: f1 + value: 27.747873024072415 + - type: precision + value: 25.99320572578704 + - type: recall + value: 33.15038419319429 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kab-eng) + config: kab-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 2.6 + - type: f1 + value: 1.687059048752127 + - type: precision + value: 1.5384884521299 + - type: recall + value: 2.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fra-eng) + config: fra-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 93.30000000000001 + - type: f1 + value: 91.44000000000001 + - type: precision + value: 90.59166666666667 + - type: recall + value: 93.30000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (por-eng) + config: por-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 94.1 + - type: f1 + value: 92.61666666666667 + - type: precision + value: 91.88333333333333 + - type: recall + value: 94.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tat-eng) + config: tat-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 5.0 + - type: f1 + value: 3.589591971281927 + - type: precision + value: 3.3046491614532854 + - type: recall + value: 5.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (oci-eng) + config: oci-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 45.9 + - type: f1 + value: 40.171969141969136 + - type: precision + value: 38.30764368870302 + - type: recall + value: 45.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pol-eng) + config: pol-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 16.900000000000002 + - type: f1 + value: 14.094365204207351 + - type: precision + value: 13.276519841269844 + - type: recall + value: 16.900000000000002 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (war-eng) + config: war-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 12.8 + - type: f1 + value: 10.376574912567156 + - type: precision + value: 9.758423963284509 + - type: recall + value: 12.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (aze-eng) + config: aze-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.1 + - type: f1 + value: 6.319455355175778 + - type: precision + value: 5.849948830628881 + - type: recall + value: 8.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (vie-eng) + config: vie-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 95.5 + - type: f1 + value: 94.19666666666667 + - type: precision + value: 93.60000000000001 + - type: recall + value: 95.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nno-eng) + config: nno-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 19.1 + - type: f1 + value: 16.280080686081906 + - type: precision + value: 15.451573089395668 + - type: recall + value: 19.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cha-eng) + config: cha-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 30.656934306569344 + - type: f1 + value: 23.2568647897115 + - type: precision + value: 21.260309034031664 + - type: recall + value: 30.656934306569344 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mhr-eng) + config: mhr-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 2.1999999999999997 + - type: f1 + value: 1.556861047295521 + - type: precision + value: 1.4555993437238521 + - type: recall + value: 2.1999999999999997 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dan-eng) + config: dan-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 27.500000000000004 + - type: f1 + value: 23.521682636223492 + - type: precision + value: 22.345341306967683 + - type: recall + value: 27.500000000000004 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ell-eng) + config: ell-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.3999999999999995 + - type: f1 + value: 5.344253880846173 + - type: precision + value: 4.999794279068863 + - type: recall + value: 7.3999999999999995 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (amh-eng) + config: amh-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 0.5952380952380952 + - type: f1 + value: 0.026455026455026457 + - type: precision + value: 0.013528138528138528 + - type: recall + value: 0.5952380952380952 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pam-eng) + config: pam-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.3 + - type: f1 + value: 5.853140211779251 + - type: precision + value: 5.505563080945322 + - type: recall + value: 7.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hsb-eng) + config: hsb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 13.250517598343686 + - type: f1 + value: 9.676349506190704 + - type: precision + value: 8.930392053553216 + - type: recall + value: 13.250517598343686 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (srp-eng) + config: srp-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 14.499999999999998 + - type: f1 + value: 11.68912588067557 + - type: precision + value: 11.024716513105519 + - type: recall + value: 14.499999999999998 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (epo-eng) + config: epo-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 30.099999999999998 + - type: f1 + value: 26.196880936315146 + - type: precision + value: 25.271714086169478 + - type: recall + value: 30.099999999999998 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kzj-eng) + config: kzj-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 6.4 + - type: f1 + value: 5.1749445942023335 + - type: precision + value: 4.975338142029625 + - type: recall + value: 6.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (awa-eng) + config: awa-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 39.39393939393939 + - type: f1 + value: 35.005707393767096 + - type: precision + value: 33.64342032053631 + - type: recall + value: 39.39393939393939 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fao-eng) + config: fao-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 18.3206106870229 + - type: f1 + value: 12.610893447220345 + - type: precision + value: 11.079228765297467 + - type: recall + value: 18.3206106870229 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mal-eng) + config: mal-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 85.58951965065502 + - type: f1 + value: 83.30363944928548 + - type: precision + value: 82.40026591554977 + - type: recall + value: 85.58951965065502 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ile-eng) + config: ile-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 65.7 + - type: f1 + value: 59.589642857142856 + - type: precision + value: 57.392826797385624 + - type: recall + value: 65.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bos-eng) + config: bos-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 18.07909604519774 + - type: f1 + value: 13.65194306689995 + - type: precision + value: 12.567953943826327 + - type: recall + value: 18.07909604519774 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cor-eng) + config: cor-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 4.6 + - type: f1 + value: 2.8335386392505013 + - type: precision + value: 2.558444143575722 + - type: recall + value: 4.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cat-eng) + config: cat-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 90.7 + - type: f1 + value: 88.30666666666666 + - type: precision + value: 87.195 + - type: recall + value: 90.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (eus-eng) + config: eus-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 57.699999999999996 + - type: f1 + value: 53.38433067253876 + - type: precision + value: 51.815451335350346 + - type: recall + value: 57.699999999999996 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (yue-eng) + config: yue-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 80.60000000000001 + - type: f1 + value: 77.0290354090354 + - type: precision + value: 75.61685897435898 + - type: recall + value: 80.60000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swe-eng) + config: swe-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 24.6 + - type: f1 + value: 19.52814960069739 + - type: precision + value: 18.169084599880502 + - type: recall + value: 24.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dtp-eng) + config: dtp-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 5.0 + - type: f1 + value: 3.4078491753102376 + - type: precision + value: 3.1757682319102387 + - type: recall + value: 5.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kat-eng) + config: kat-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 1.2064343163538873 + - type: f1 + value: 0.4224313053283095 + - type: precision + value: 0.3360484946842894 + - type: recall + value: 1.2064343163538873 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (jpn-eng) + config: jpn-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 76.1 + - type: f1 + value: 71.36246031746032 + - type: precision + value: 69.5086544011544 + - type: recall + value: 76.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (csb-eng) + config: csb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 14.229249011857709 + - type: f1 + value: 10.026578603653704 + - type: precision + value: 9.09171178352764 + - type: recall + value: 14.229249011857709 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (xho-eng) + config: xho-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 8.450704225352112 + - type: f1 + value: 5.51214407186151 + - type: precision + value: 4.928281812084629 + - type: recall + value: 8.450704225352112 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (orv-eng) + config: orv-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.664670658682635 + - type: f1 + value: 5.786190079917295 + - type: precision + value: 5.3643643579244 + - type: recall + value: 7.664670658682635 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ind-eng) + config: ind-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 90.5 + - type: f1 + value: 88.03999999999999 + - type: precision + value: 86.94833333333334 + - type: recall + value: 90.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tuk-eng) + config: tuk-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 7.389162561576355 + - type: f1 + value: 5.482366349556517 + - type: precision + value: 5.156814449917898 + - type: recall + value: 7.389162561576355 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (max-eng) + config: max-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 41.54929577464789 + - type: f1 + value: 36.13520282534367 + - type: precision + value: 34.818226488560995 + - type: recall + value: 41.54929577464789 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swh-eng) + config: swh-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 20.76923076923077 + - type: f1 + value: 16.742497560177643 + - type: precision + value: 15.965759712090138 + - type: recall + value: 20.76923076923077 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hin-eng) + config: hin-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 88.1 + - type: f1 + value: 85.23176470588236 + - type: precision + value: 84.04458333333334 + - type: recall + value: 88.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dsb-eng) + config: dsb-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 11.899791231732777 + - type: f1 + value: 8.776706659565102 + - type: precision + value: 8.167815946521582 + - type: recall + value: 11.899791231732777 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ber-eng) + config: ber-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 6.1 + - type: f1 + value: 4.916589537178435 + - type: precision + value: 4.72523017415345 + - type: recall + value: 6.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tam-eng) + config: tam-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 76.54723127035831 + - type: f1 + value: 72.75787187839306 + - type: precision + value: 71.43338442869005 + - type: recall + value: 76.54723127035831 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (slk-eng) + config: slk-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 11.700000000000001 + - type: f1 + value: 9.975679190026007 + - type: precision + value: 9.569927715653522 + - type: recall + value: 11.700000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tgl-eng) + config: tgl-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 13.100000000000001 + - type: f1 + value: 10.697335850115408 + - type: precision + value: 10.113816082086341 + - type: recall + value: 13.100000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ast-eng) + config: ast-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 76.37795275590551 + - type: f1 + value: 71.12860892388451 + - type: precision + value: 68.89763779527559 + - type: recall + value: 76.37795275590551 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mkd-eng) + config: mkd-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 13.700000000000001 + - type: f1 + value: 10.471861684067568 + - type: precision + value: 9.602902567641697 + - type: recall + value: 13.700000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (khm-eng) + config: khm-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 0.554016620498615 + - type: f1 + value: 0.37034084643642423 + - type: precision + value: 0.34676040281208437 + - type: recall + value: 0.554016620498615 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ces-eng) + config: ces-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 12.4 + - type: f1 + value: 9.552607451092534 + - type: precision + value: 8.985175505050504 + - type: recall + value: 12.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tzl-eng) + config: tzl-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 33.65384615384615 + - type: f1 + value: 27.820512820512818 + - type: precision + value: 26.09432234432234 + - type: recall + value: 33.65384615384615 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (urd-eng) + config: urd-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 74.5 + - type: f1 + value: 70.09686507936507 + - type: precision + value: 68.3117857142857 + - type: recall + value: 74.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ara-eng) + config: ara-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 88.3 + - type: f1 + value: 85.37333333333333 + - type: precision + value: 84.05833333333334 + - type: recall + value: 88.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kor-eng) + config: kor-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 25.0 + - type: f1 + value: 22.393124632031995 + - type: precision + value: 21.58347686592367 + - type: recall + value: 25.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (yid-eng) + config: yid-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 0.589622641509434 + - type: f1 + value: 0.15804980033762941 + - type: precision + value: 0.1393275384872965 + - type: recall + value: 0.589622641509434 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fin-eng) + config: fin-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 4.1000000000000005 + - type: f1 + value: 3.4069011332551775 + - type: precision + value: 3.1784507042253516 + - type: recall + value: 4.1000000000000005 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tha-eng) + config: tha-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 3.102189781021898 + - type: f1 + value: 2.223851811694751 + - type: precision + value: 2.103465682299194 + - type: recall + value: 3.102189781021898 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (wuu-eng) + config: wuu-eng + split: test + revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 + metrics: + - type: accuracy + value: 83.1 + - type: f1 + value: 79.58255835667599 + - type: precision + value: 78.09708333333333 + - type: recall + value: 83.1 + - task: + type: Retrieval + dataset: + type: webis-touche2020 + name: MTEB Touche2020 + config: default + split: test + revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b + metrics: + - type: map_at_1 + value: 2.322 + - type: map_at_10 + value: 8.959999999999999 + - type: map_at_100 + value: 15.136 + - type: map_at_1000 + value: 16.694 + - type: map_at_3 + value: 4.837000000000001 + - type: map_at_5 + value: 6.196 + - type: mrr_at_1 + value: 28.571 + - type: mrr_at_10 + value: 47.589999999999996 + - type: mrr_at_100 + value: 48.166 + - type: mrr_at_1000 + value: 48.169000000000004 + - type: mrr_at_3 + value: 43.197 + - type: mrr_at_5 + value: 45.646 + - type: ndcg_at_1 + value: 26.531 + - type: ndcg_at_10 + value: 23.982 + - type: ndcg_at_100 + value: 35.519 + - type: ndcg_at_1000 + value: 46.878 + - type: ndcg_at_3 + value: 26.801000000000002 + - type: ndcg_at_5 + value: 24.879 + - type: precision_at_1 + value: 28.571 + - type: precision_at_10 + value: 22.041 + - type: precision_at_100 + value: 7.4079999999999995 + - type: precision_at_1000 + value: 1.492 + - type: precision_at_3 + value: 28.571 + - type: precision_at_5 + value: 25.306 + - type: recall_at_1 + value: 2.322 + - type: recall_at_10 + value: 15.443999999999999 + - type: recall_at_100 + value: 45.918 + - type: recall_at_1000 + value: 79.952 + - type: recall_at_3 + value: 6.143 + - type: recall_at_5 + value: 8.737 + - task: + type: Classification + dataset: + type: mteb/toxic_conversations_50k + name: MTEB ToxicConversationsClassification + config: default + split: test + revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de + metrics: + - type: accuracy + value: 66.5452 + - type: ap + value: 12.99191723223892 + - type: f1 + value: 51.667665096195734 + - task: + type: Classification + dataset: + type: mteb/tweet_sentiment_extraction + name: MTEB TweetSentimentExtractionClassification + config: default + split: test + revision: 62146448f05be9e52a36b8ee9936447ea787eede + metrics: + - type: accuracy + value: 55.854555744199196 + - type: f1 + value: 56.131766302254185 + - task: + type: Clustering + dataset: + type: mteb/twentynewsgroups-clustering + name: MTEB TwentyNewsgroupsClustering + config: default + split: test + revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 + metrics: + - type: v_measure + value: 37.27891385518074 + - task: + type: PairClassification + dataset: + type: mteb/twittersemeval2015-pairclassification + name: MTEB TwitterSemEval2015 + config: default + split: test + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + metrics: + - type: cos_sim_accuracy + value: 83.53102461703523 + - type: cos_sim_ap + value: 65.30753664579191 + - type: cos_sim_f1 + value: 61.739943872778305 + - type: cos_sim_precision + value: 55.438891222175556 + - type: cos_sim_recall + value: 69.65699208443272 + - type: dot_accuracy + value: 80.38981939560112 + - type: dot_ap + value: 53.52081118421347 + - type: dot_f1 + value: 54.232957844617346 + - type: dot_precision + value: 48.43393486828459 + - type: dot_recall + value: 61.60949868073878 + - type: euclidean_accuracy + value: 82.23758717291531 + - type: euclidean_ap + value: 60.361102792772535 + - type: euclidean_f1 + value: 57.50518791791561 + - type: euclidean_precision + value: 51.06470106470107 + - type: euclidean_recall + value: 65.8047493403694 + - type: manhattan_accuracy + value: 82.14221851344102 + - type: manhattan_ap + value: 60.341937223793366 + - type: manhattan_f1 + value: 57.53803596127247 + - type: manhattan_precision + value: 51.08473188702415 + - type: manhattan_recall + value: 65.85751978891821 + - type: max_accuracy + value: 83.53102461703523 + - type: max_ap + value: 65.30753664579191 + - type: max_f1 + value: 61.739943872778305 + - task: + type: PairClassification + dataset: + type: mteb/twitterurlcorpus-pairclassification + name: MTEB TwitterURLCorpus + config: default + split: test + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + metrics: + - type: cos_sim_accuracy + value: 88.75305623471883 + - type: cos_sim_ap + value: 85.46387153880272 + - type: cos_sim_f1 + value: 77.91527673159008 + - type: cos_sim_precision + value: 72.93667315828353 + - type: cos_sim_recall + value: 83.62334462580844 + - type: dot_accuracy + value: 85.08169363915086 + - type: dot_ap + value: 74.96808060965559 + - type: dot_f1 + value: 71.39685033990366 + - type: dot_precision + value: 64.16948111759288 + - type: dot_recall + value: 80.45888512473051 + - type: euclidean_accuracy + value: 85.84235650250321 + - type: euclidean_ap + value: 78.42045145247211 + - type: euclidean_f1 + value: 70.32669630775179 + - type: euclidean_precision + value: 70.6298050788227 + - type: euclidean_recall + value: 70.02617801047121 + - type: manhattan_accuracy + value: 85.86176116738464 + - type: manhattan_ap + value: 78.54012451558276 + - type: manhattan_f1 + value: 70.56508080693389 + - type: manhattan_precision + value: 69.39626293456413 + - type: manhattan_recall + value: 71.77394518016631 + - type: max_accuracy + value: 88.75305623471883 + - type: max_ap + value: 85.46387153880272 + - type: max_f1 + value: 77.91527673159008 --- ## Usage @@ -20,7 +6695,7 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/ms_marc `{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}` -See the evaluation folder for more results. +See the evaluation folder or [MTEB](https://huggingface.co/spaces/mteb/leaderboard) for more results. ## Training The model was trained with the parameters: