--- pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb model-index: - name: sgpt-bloom-7b1-msmarco results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 68.05970149253731 - type: ap value: 31.640363460776193 - type: f1 value: 62.50025574145796 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (de) config: de split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 61.34903640256959 - type: ap value: 75.18797161500426 - type: f1 value: 59.04772570730417 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en-ext) config: en-ext split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 67.78110944527737 - type: ap value: 19.218916023322706 - type: f1 value: 56.24477391445512 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (ja) config: ja split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 58.23340471092078 - type: ap value: 13.20222967424681 - type: f1 value: 47.511718095460296 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 metrics: - type: accuracy value: 68.97232499999998 - type: ap value: 63.53632885535693 - type: f1 value: 68.62038513152868 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 33.855999999999995 - type: f1 value: 33.43468222830134 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (de) config: de split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 29.697999999999997 - type: f1 value: 29.39935388885501 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (es) config: es split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 35.974000000000004 - type: f1 value: 35.25910820714383 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) config: fr split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 35.922 - type: f1 value: 35.38637028933444 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (ja) config: ja split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 27.636 - type: f1 value: 27.178349955978266 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) config: zh split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 32.632 - type: f1 value: 32.08014766494587 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna config: default split: test revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 metrics: - type: map_at_1 value: 23.684 - type: map_at_10 value: 38.507999999999996 - type: map_at_100 value: 39.677 - type: map_at_1000 value: 39.690999999999995 - type: map_at_3 value: 33.369 - type: map_at_5 value: 36.15 - type: mrr_at_1 value: 24.04 - type: mrr_at_10 value: 38.664 - type: mrr_at_100 value: 39.833 - type: mrr_at_1000 value: 39.847 - type: mrr_at_3 value: 33.476 - type: mrr_at_5 value: 36.306 - type: ndcg_at_1 value: 23.684 - type: ndcg_at_10 value: 47.282000000000004 - type: ndcg_at_100 value: 52.215 - type: ndcg_at_1000 value: 52.551 - type: ndcg_at_3 value: 36.628 - type: ndcg_at_5 value: 41.653 - type: precision_at_1 value: 23.684 - type: precision_at_10 value: 7.553 - type: precision_at_100 value: 0.97 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 15.363 - type: precision_at_5 value: 11.664 - type: recall_at_1 value: 23.684 - type: recall_at_10 value: 75.533 - type: recall_at_100 value: 97.013 - type: recall_at_1000 value: 99.57300000000001 - type: recall_at_3 value: 46.088 - type: recall_at_5 value: 58.321 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 metrics: - type: v_measure value: 44.59375023881131 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 metrics: - type: v_measure value: 38.02921907752556 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c metrics: - type: map value: 59.97321570342109 - type: mrr value: 73.18284746955106 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test revision: 9ee918f184421b6bd48b78f6c714d86546106103 metrics: - type: cos_sim_pearson value: 89.09091435741429 - type: cos_sim_spearman value: 85.31459455332202 - type: euclidean_pearson value: 79.3587681410798 - type: euclidean_spearman value: 76.8174129874685 - type: manhattan_pearson value: 79.57051762121769 - type: manhattan_spearman value: 76.75837549768094 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (de-en) config: de-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 54.27974947807933 - type: f1 value: 54.00144411132214 - type: precision value: 53.87119374071357 - type: recall value: 54.27974947807933 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (fr-en) config: fr-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 97.3365617433414 - type: f1 value: 97.06141316310809 - type: precision value: 96.92567319685965 - type: recall value: 97.3365617433414 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (ru-en) config: ru-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 46.05472809144441 - type: f1 value: 45.30319274690595 - type: precision value: 45.00015469655234 - type: recall value: 46.05472809144441 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (zh-en) config: zh-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 98.10426540284361 - type: f1 value: 97.96384061786905 - type: precision value: 97.89362822538178 - type: recall value: 98.10426540284361 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test revision: 44fa15921b4c889113cc5df03dd4901b49161ab7 metrics: - type: accuracy value: 84.33441558441558 - type: f1 value: 84.31653077470322 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 metrics: - type: v_measure value: 36.025318694698086 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 metrics: - type: v_measure value: 32.484889034590346 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 30.203999999999997 - type: map_at_10 value: 41.314 - type: map_at_100 value: 42.66 - type: map_at_1000 value: 42.775999999999996 - type: map_at_3 value: 37.614999999999995 - type: map_at_5 value: 39.643 - type: mrr_at_1 value: 37.482 - type: mrr_at_10 value: 47.075 - type: mrr_at_100 value: 47.845 - type: mrr_at_1000 value: 47.887 - type: mrr_at_3 value: 44.635000000000005 - type: mrr_at_5 value: 45.966 - type: ndcg_at_1 value: 37.482 - type: ndcg_at_10 value: 47.676 - type: ndcg_at_100 value: 52.915 - type: ndcg_at_1000 value: 54.82900000000001 - type: ndcg_at_3 value: 42.562 - type: ndcg_at_5 value: 44.852 - type: precision_at_1 value: 37.482 - type: precision_at_10 value: 9.142 - type: precision_at_100 value: 1.436 - type: precision_at_1000 value: 0.189 - type: precision_at_3 value: 20.458000000000002 - type: precision_at_5 value: 14.821000000000002 - type: recall_at_1 value: 30.203999999999997 - type: recall_at_10 value: 60.343 - type: recall_at_100 value: 82.58 - type: recall_at_1000 value: 94.813 - type: recall_at_3 value: 45.389 - type: recall_at_5 value: 51.800999999999995 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 30.889 - type: map_at_10 value: 40.949999999999996 - type: map_at_100 value: 42.131 - type: map_at_1000 value: 42.253 - type: map_at_3 value: 38.346999999999994 - type: map_at_5 value: 39.782000000000004 - type: mrr_at_1 value: 38.79 - type: mrr_at_10 value: 46.944 - type: mrr_at_100 value: 47.61 - type: mrr_at_1000 value: 47.650999999999996 - type: mrr_at_3 value: 45.053 - type: mrr_at_5 value: 46.101 - type: ndcg_at_1 value: 38.79 - type: ndcg_at_10 value: 46.286 - type: ndcg_at_100 value: 50.637 - type: ndcg_at_1000 value: 52.649 - type: ndcg_at_3 value: 42.851 - type: ndcg_at_5 value: 44.311 - type: precision_at_1 value: 38.79 - type: precision_at_10 value: 8.516 - type: precision_at_100 value: 1.3679999999999999 - type: precision_at_1000 value: 0.183 - type: precision_at_3 value: 20.637 - type: precision_at_5 value: 14.318 - type: recall_at_1 value: 30.889 - type: recall_at_10 value: 55.327000000000005 - type: recall_at_100 value: 74.091 - type: recall_at_1000 value: 86.75500000000001 - type: recall_at_3 value: 44.557 - type: recall_at_5 value: 49.064 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 39.105000000000004 - type: map_at_10 value: 50.928 - type: map_at_100 value: 51.958000000000006 - type: map_at_1000 value: 52.017 - type: map_at_3 value: 47.638999999999996 - type: map_at_5 value: 49.624 - type: mrr_at_1 value: 44.639 - type: mrr_at_10 value: 54.261 - type: mrr_at_100 value: 54.913999999999994 - type: mrr_at_1000 value: 54.945 - type: mrr_at_3 value: 51.681999999999995 - type: mrr_at_5 value: 53.290000000000006 - type: ndcg_at_1 value: 44.639 - type: ndcg_at_10 value: 56.678 - type: ndcg_at_100 value: 60.649 - type: ndcg_at_1000 value: 61.855000000000004 - type: ndcg_at_3 value: 51.092999999999996 - type: ndcg_at_5 value: 54.096999999999994 - type: precision_at_1 value: 44.639 - type: precision_at_10 value: 9.028 - type: precision_at_100 value: 1.194 - type: precision_at_1000 value: 0.135 - type: precision_at_3 value: 22.508 - type: precision_at_5 value: 15.661 - type: recall_at_1 value: 39.105000000000004 - type: recall_at_10 value: 70.367 - type: recall_at_100 value: 87.359 - type: recall_at_1000 value: 95.88 - type: recall_at_3 value: 55.581 - type: recall_at_5 value: 62.821000000000005 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 23.777 - type: map_at_10 value: 32.297 - type: map_at_100 value: 33.516 - type: map_at_1000 value: 33.592 - type: map_at_3 value: 30.001 - type: map_at_5 value: 31.209999999999997 - type: mrr_at_1 value: 25.989 - type: mrr_at_10 value: 34.472 - type: mrr_at_100 value: 35.518 - type: mrr_at_1000 value: 35.577 - type: mrr_at_3 value: 32.185 - type: mrr_at_5 value: 33.399 - type: ndcg_at_1 value: 25.989 - type: ndcg_at_10 value: 37.037 - type: ndcg_at_100 value: 42.699 - type: ndcg_at_1000 value: 44.725 - type: ndcg_at_3 value: 32.485 - type: ndcg_at_5 value: 34.549 - type: precision_at_1 value: 25.989 - type: precision_at_10 value: 5.718 - type: precision_at_100 value: 0.89 - type: precision_at_1000 value: 0.11 - type: precision_at_3 value: 14.049 - type: precision_at_5 value: 9.672 - type: recall_at_1 value: 23.777 - type: recall_at_10 value: 49.472 - type: recall_at_100 value: 74.857 - type: recall_at_1000 value: 90.289 - type: recall_at_3 value: 37.086000000000006 - type: recall_at_5 value: 42.065999999999995 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 13.377 - type: map_at_10 value: 21.444 - type: map_at_100 value: 22.663 - type: map_at_1000 value: 22.8 - type: map_at_3 value: 18.857 - type: map_at_5 value: 20.426 - type: mrr_at_1 value: 16.542 - type: mrr_at_10 value: 25.326999999999998 - type: mrr_at_100 value: 26.323 - type: mrr_at_1000 value: 26.406000000000002 - type: mrr_at_3 value: 22.823 - type: mrr_at_5 value: 24.340999999999998 - type: ndcg_at_1 value: 16.542 - type: ndcg_at_10 value: 26.479000000000003 - type: ndcg_at_100 value: 32.29 - type: ndcg_at_1000 value: 35.504999999999995 - type: ndcg_at_3 value: 21.619 - type: ndcg_at_5 value: 24.19 - type: precision_at_1 value: 16.542 - type: precision_at_10 value: 5.075 - type: precision_at_100 value: 0.9339999999999999 - type: precision_at_1000 value: 0.135 - type: precision_at_3 value: 10.697 - type: precision_at_5 value: 8.134 - type: recall_at_1 value: 13.377 - type: recall_at_10 value: 38.027 - type: recall_at_100 value: 63.439 - type: recall_at_1000 value: 86.354 - type: recall_at_3 value: 25.0 - type: recall_at_5 value: 31.306 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 28.368 - type: map_at_10 value: 39.305 - type: map_at_100 value: 40.637 - type: map_at_1000 value: 40.753 - type: map_at_3 value: 36.077999999999996 - type: map_at_5 value: 37.829 - type: mrr_at_1 value: 34.937000000000005 - type: mrr_at_10 value: 45.03 - type: mrr_at_100 value: 45.78 - type: mrr_at_1000 value: 45.827 - type: mrr_at_3 value: 42.348 - type: mrr_at_5 value: 43.807 - type: ndcg_at_1 value: 34.937000000000005 - type: ndcg_at_10 value: 45.605000000000004 - type: ndcg_at_100 value: 50.941 - type: ndcg_at_1000 value: 52.983000000000004 - type: ndcg_at_3 value: 40.366 - type: ndcg_at_5 value: 42.759 - type: precision_at_1 value: 34.937000000000005 - type: precision_at_10 value: 8.402 - type: precision_at_100 value: 1.2959999999999998 - type: precision_at_1000 value: 0.164 - type: precision_at_3 value: 19.217000000000002 - type: precision_at_5 value: 13.725000000000001 - type: recall_at_1 value: 28.368 - type: recall_at_10 value: 58.5 - type: recall_at_100 value: 80.67999999999999 - type: recall_at_1000 value: 93.925 - type: recall_at_3 value: 43.956 - type: recall_at_5 value: 50.065000000000005 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 24.851 - type: map_at_10 value: 34.758 - type: map_at_100 value: 36.081 - type: map_at_1000 value: 36.205999999999996 - type: map_at_3 value: 31.678 - type: map_at_5 value: 33.398 - type: mrr_at_1 value: 31.279 - type: mrr_at_10 value: 40.138 - type: mrr_at_100 value: 41.005 - type: mrr_at_1000 value: 41.065000000000005 - type: mrr_at_3 value: 37.519000000000005 - type: mrr_at_5 value: 38.986 - type: ndcg_at_1 value: 31.279 - type: ndcg_at_10 value: 40.534 - type: ndcg_at_100 value: 46.093 - type: ndcg_at_1000 value: 48.59 - type: ndcg_at_3 value: 35.473 - type: ndcg_at_5 value: 37.801 - type: precision_at_1 value: 31.279 - type: precision_at_10 value: 7.477 - type: precision_at_100 value: 1.2 - type: precision_at_1000 value: 0.159 - type: precision_at_3 value: 17.047 - type: precision_at_5 value: 12.306000000000001 - type: recall_at_1 value: 24.851 - type: recall_at_10 value: 52.528 - type: recall_at_100 value: 76.198 - type: recall_at_1000 value: 93.12 - type: recall_at_3 value: 38.257999999999996 - type: recall_at_5 value: 44.440000000000005 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 25.289833333333334 - type: map_at_10 value: 34.379333333333335 - type: map_at_100 value: 35.56916666666666 - type: map_at_1000 value: 35.68633333333333 - type: map_at_3 value: 31.63916666666666 - type: map_at_5 value: 33.18383333333334 - type: mrr_at_1 value: 30.081749999999996 - type: mrr_at_10 value: 38.53658333333333 - type: mrr_at_100 value: 39.37825 - type: mrr_at_1000 value: 39.43866666666666 - type: mrr_at_3 value: 36.19025 - type: mrr_at_5 value: 37.519749999999995 - type: ndcg_at_1 value: 30.081749999999996 - type: ndcg_at_10 value: 39.62041666666667 - type: ndcg_at_100 value: 44.74825 - type: ndcg_at_1000 value: 47.11366666666667 - type: ndcg_at_3 value: 35.000499999999995 - type: ndcg_at_5 value: 37.19283333333333 - type: precision_at_1 value: 30.081749999999996 - type: precision_at_10 value: 6.940249999999999 - type: precision_at_100 value: 1.1164166666666668 - type: precision_at_1000 value: 0.15025000000000002 - type: precision_at_3 value: 16.110416666666666 - type: precision_at_5 value: 11.474416666666668 - type: recall_at_1 value: 25.289833333333334 - type: recall_at_10 value: 51.01591666666667 - type: recall_at_100 value: 73.55275000000002 - type: recall_at_1000 value: 90.02666666666667 - type: recall_at_3 value: 38.15208333333334 - type: recall_at_5 value: 43.78458333333334 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 23.479 - type: map_at_10 value: 31.2 - type: map_at_100 value: 32.11 - type: map_at_1000 value: 32.214 - type: map_at_3 value: 29.093999999999998 - type: map_at_5 value: 30.415 - type: mrr_at_1 value: 26.840000000000003 - type: mrr_at_10 value: 34.153 - type: mrr_at_100 value: 34.971000000000004 - type: mrr_at_1000 value: 35.047 - type: mrr_at_3 value: 32.285000000000004 - type: mrr_at_5 value: 33.443 - type: ndcg_at_1 value: 26.840000000000003 - type: ndcg_at_10 value: 35.441 - type: ndcg_at_100 value: 40.150000000000006 - type: ndcg_at_1000 value: 42.74 - type: ndcg_at_3 value: 31.723000000000003 - type: ndcg_at_5 value: 33.71 - type: precision_at_1 value: 26.840000000000003 - type: precision_at_10 value: 5.552 - type: precision_at_100 value: 0.859 - type: precision_at_1000 value: 0.11499999999999999 - type: precision_at_3 value: 13.804 - type: precision_at_5 value: 9.600999999999999 - type: recall_at_1 value: 23.479 - type: recall_at_10 value: 45.442 - type: recall_at_100 value: 67.465 - type: recall_at_1000 value: 86.53 - type: recall_at_3 value: 35.315999999999995 - type: recall_at_5 value: 40.253 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 16.887 - type: map_at_10 value: 23.805 - type: map_at_100 value: 24.804000000000002 - type: map_at_1000 value: 24.932000000000002 - type: map_at_3 value: 21.632 - type: map_at_5 value: 22.845 - type: mrr_at_1 value: 20.75 - type: mrr_at_10 value: 27.686 - type: mrr_at_100 value: 28.522 - type: mrr_at_1000 value: 28.605000000000004 - type: mrr_at_3 value: 25.618999999999996 - type: mrr_at_5 value: 26.723999999999997 - type: ndcg_at_1 value: 20.75 - type: ndcg_at_10 value: 28.233000000000004 - type: ndcg_at_100 value: 33.065 - type: ndcg_at_1000 value: 36.138999999999996 - type: ndcg_at_3 value: 24.361 - type: ndcg_at_5 value: 26.111 - type: precision_at_1 value: 20.75 - type: precision_at_10 value: 5.124 - type: precision_at_100 value: 0.8750000000000001 - type: precision_at_1000 value: 0.131 - type: precision_at_3 value: 11.539000000000001 - type: precision_at_5 value: 8.273 - type: recall_at_1 value: 16.887 - type: recall_at_10 value: 37.774 - type: recall_at_100 value: 59.587 - type: recall_at_1000 value: 81.523 - type: recall_at_3 value: 26.837 - type: recall_at_5 value: 31.456 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 25.534000000000002 - type: map_at_10 value: 33.495999999999995 - type: map_at_100 value: 34.697 - type: map_at_1000 value: 34.805 - type: map_at_3 value: 31.22 - type: map_at_5 value: 32.277 - type: mrr_at_1 value: 29.944 - type: mrr_at_10 value: 37.723 - type: mrr_at_100 value: 38.645 - type: mrr_at_1000 value: 38.712999999999994 - type: mrr_at_3 value: 35.665 - type: mrr_at_5 value: 36.681999999999995 - type: ndcg_at_1 value: 29.944 - type: ndcg_at_10 value: 38.407000000000004 - type: ndcg_at_100 value: 43.877 - type: ndcg_at_1000 value: 46.312 - type: ndcg_at_3 value: 34.211000000000006 - type: ndcg_at_5 value: 35.760999999999996 - type: precision_at_1 value: 29.944 - type: precision_at_10 value: 6.343 - type: precision_at_100 value: 1.023 - type: precision_at_1000 value: 0.133 - type: precision_at_3 value: 15.360999999999999 - type: precision_at_5 value: 10.428999999999998 - type: recall_at_1 value: 25.534000000000002 - type: recall_at_10 value: 49.204 - type: recall_at_100 value: 72.878 - type: recall_at_1000 value: 89.95 - type: recall_at_3 value: 37.533 - type: recall_at_5 value: 41.611 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 26.291999999999998 - type: map_at_10 value: 35.245 - type: map_at_100 value: 36.762 - type: map_at_1000 value: 36.983 - type: map_at_3 value: 32.439 - type: map_at_5 value: 33.964 - type: mrr_at_1 value: 31.423000000000002 - type: mrr_at_10 value: 39.98 - type: mrr_at_100 value: 40.791 - type: mrr_at_1000 value: 40.854 - type: mrr_at_3 value: 37.451 - type: mrr_at_5 value: 38.854 - type: ndcg_at_1 value: 31.423000000000002 - type: ndcg_at_10 value: 40.848 - type: ndcg_at_100 value: 46.35 - type: ndcg_at_1000 value: 49.166 - type: ndcg_at_3 value: 36.344 - type: ndcg_at_5 value: 38.36 - type: precision_at_1 value: 31.423000000000002 - type: precision_at_10 value: 7.767 - type: precision_at_100 value: 1.498 - type: precision_at_1000 value: 0.23700000000000002 - type: precision_at_3 value: 16.733 - type: precision_at_5 value: 12.213000000000001 - type: recall_at_1 value: 26.291999999999998 - type: recall_at_10 value: 51.184 - type: recall_at_100 value: 76.041 - type: recall_at_1000 value: 94.11500000000001 - type: recall_at_3 value: 38.257000000000005 - type: recall_at_5 value: 43.68 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 20.715 - type: map_at_10 value: 27.810000000000002 - type: map_at_100 value: 28.810999999999996 - type: map_at_1000 value: 28.904999999999998 - type: map_at_3 value: 25.069999999999997 - type: map_at_5 value: 26.793 - type: mrr_at_1 value: 22.366 - type: mrr_at_10 value: 29.65 - type: mrr_at_100 value: 30.615 - type: mrr_at_1000 value: 30.686999999999998 - type: mrr_at_3 value: 27.017999999999997 - type: mrr_at_5 value: 28.644 - type: ndcg_at_1 value: 22.366 - type: ndcg_at_10 value: 32.221 - type: ndcg_at_100 value: 37.313 - type: ndcg_at_1000 value: 39.871 - type: ndcg_at_3 value: 26.918 - type: ndcg_at_5 value: 29.813000000000002 - type: precision_at_1 value: 22.366 - type: precision_at_10 value: 5.139 - type: precision_at_100 value: 0.8240000000000001 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_3 value: 11.275 - type: precision_at_5 value: 8.540000000000001 - type: recall_at_1 value: 20.715 - type: recall_at_10 value: 44.023 - type: recall_at_100 value: 67.458 - type: recall_at_1000 value: 87.066 - type: recall_at_3 value: 30.055 - type: recall_at_5 value: 36.852000000000004 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER config: default split: test revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce metrics: - type: map_at_1 value: 11.859 - type: map_at_10 value: 20.625 - type: map_at_100 value: 22.5 - type: map_at_1000 value: 22.689 - type: map_at_3 value: 16.991 - type: map_at_5 value: 18.781 - type: mrr_at_1 value: 26.906000000000002 - type: mrr_at_10 value: 39.083 - type: mrr_at_100 value: 39.978 - type: mrr_at_1000 value: 40.014 - type: mrr_at_3 value: 35.44 - type: mrr_at_5 value: 37.619 - type: ndcg_at_1 value: 26.906000000000002 - type: ndcg_at_10 value: 29.386000000000003 - type: ndcg_at_100 value: 36.510999999999996 - type: ndcg_at_1000 value: 39.814 - type: ndcg_at_3 value: 23.558 - type: ndcg_at_5 value: 25.557999999999996 - type: precision_at_1 value: 26.906000000000002 - type: precision_at_10 value: 9.342 - type: precision_at_100 value: 1.6969999999999998 - type: precision_at_1000 value: 0.231 - type: precision_at_3 value: 17.503 - type: precision_at_5 value: 13.655000000000001 - type: recall_at_1 value: 11.859 - type: recall_at_10 value: 35.929 - type: recall_at_100 value: 60.21300000000001 - type: recall_at_1000 value: 78.606 - type: recall_at_3 value: 21.727 - type: recall_at_5 value: 27.349 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia config: default split: test revision: f097057d03ed98220bc7309ddb10b71a54d667d6 metrics: - type: map_at_1 value: 8.627 - type: map_at_10 value: 18.248 - type: map_at_100 value: 25.19 - type: map_at_1000 value: 26.741 - type: map_at_3 value: 13.286000000000001 - type: map_at_5 value: 15.126000000000001 - type: mrr_at_1 value: 64.75 - type: mrr_at_10 value: 71.865 - type: mrr_at_100 value: 72.247 - type: mrr_at_1000 value: 72.255 - type: mrr_at_3 value: 69.958 - type: mrr_at_5 value: 71.108 - type: ndcg_at_1 value: 53.25 - type: ndcg_at_10 value: 39.035 - type: ndcg_at_100 value: 42.735 - type: ndcg_at_1000 value: 50.166 - type: ndcg_at_3 value: 43.857 - type: ndcg_at_5 value: 40.579 - type: precision_at_1 value: 64.75 - type: precision_at_10 value: 30.75 - type: precision_at_100 value: 9.54 - type: precision_at_1000 value: 2.035 - type: precision_at_3 value: 47.333 - type: precision_at_5 value: 39.0 - type: recall_at_1 value: 8.627 - type: recall_at_10 value: 23.413 - type: recall_at_100 value: 48.037 - type: recall_at_1000 value: 71.428 - type: recall_at_3 value: 14.158999999999999 - type: recall_at_5 value: 17.002 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test revision: 829147f8f75a25f005913200eb5ed41fae320aa1 metrics: - type: accuracy value: 44.865 - type: f1 value: 41.56625743266997 - task: type: Retrieval dataset: type: fever name: MTEB FEVER config: default split: test revision: 1429cf27e393599b8b359b9b72c666f96b2525f9 metrics: - type: map_at_1 value: 57.335 - type: map_at_10 value: 68.29499999999999 - type: map_at_100 value: 68.69800000000001 - type: map_at_1000 value: 68.714 - type: map_at_3 value: 66.149 - type: map_at_5 value: 67.539 - type: mrr_at_1 value: 61.656 - type: mrr_at_10 value: 72.609 - type: mrr_at_100 value: 72.923 - type: mrr_at_1000 value: 72.928 - type: mrr_at_3 value: 70.645 - type: mrr_at_5 value: 71.938 - type: ndcg_at_1 value: 61.656 - type: ndcg_at_10 value: 73.966 - type: ndcg_at_100 value: 75.663 - type: ndcg_at_1000 value: 75.986 - type: ndcg_at_3 value: 69.959 - type: ndcg_at_5 value: 72.269 - type: precision_at_1 value: 61.656 - type: precision_at_10 value: 9.581000000000001 - type: precision_at_100 value: 1.054 - type: precision_at_1000 value: 0.11 - type: precision_at_3 value: 27.743000000000002 - type: precision_at_5 value: 17.939 - type: recall_at_1 value: 57.335 - type: recall_at_10 value: 87.24300000000001 - type: recall_at_100 value: 94.575 - type: recall_at_1000 value: 96.75399999999999 - type: recall_at_3 value: 76.44800000000001 - type: recall_at_5 value: 82.122 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 config: default split: test revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be metrics: - type: map_at_1 value: 17.014000000000003 - type: map_at_10 value: 28.469 - type: map_at_100 value: 30.178 - type: map_at_1000 value: 30.369 - type: map_at_3 value: 24.63 - type: map_at_5 value: 26.891 - type: mrr_at_1 value: 34.259 - type: mrr_at_10 value: 43.042 - type: mrr_at_100 value: 43.91 - type: mrr_at_1000 value: 43.963 - type: mrr_at_3 value: 40.483999999999995 - type: mrr_at_5 value: 42.135 - type: ndcg_at_1 value: 34.259 - type: ndcg_at_10 value: 35.836 - type: ndcg_at_100 value: 42.488 - type: ndcg_at_1000 value: 45.902 - type: ndcg_at_3 value: 32.131 - type: ndcg_at_5 value: 33.697 - type: precision_at_1 value: 34.259 - type: precision_at_10 value: 10.0 - type: precision_at_100 value: 1.699 - type: precision_at_1000 value: 0.22999999999999998 - type: precision_at_3 value: 21.502 - type: precision_at_5 value: 16.296 - type: recall_at_1 value: 17.014000000000003 - type: recall_at_10 value: 42.832 - type: recall_at_100 value: 67.619 - type: recall_at_1000 value: 88.453 - type: recall_at_3 value: 29.537000000000003 - type: recall_at_5 value: 35.886 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA config: default split: test revision: 766870b35a1b9ca65e67a0d1913899973551fc6c metrics: - type: map_at_1 value: 34.558 - type: map_at_10 value: 48.039 - type: map_at_100 value: 48.867 - type: map_at_1000 value: 48.941 - type: map_at_3 value: 45.403 - type: map_at_5 value: 46.983999999999995 - type: mrr_at_1 value: 69.11500000000001 - type: mrr_at_10 value: 75.551 - type: mrr_at_100 value: 75.872 - type: mrr_at_1000 value: 75.887 - type: mrr_at_3 value: 74.447 - type: mrr_at_5 value: 75.113 - type: ndcg_at_1 value: 69.11500000000001 - type: ndcg_at_10 value: 57.25599999999999 - type: ndcg_at_100 value: 60.417 - type: ndcg_at_1000 value: 61.976 - type: ndcg_at_3 value: 53.258 - type: ndcg_at_5 value: 55.374 - type: precision_at_1 value: 69.11500000000001 - type: precision_at_10 value: 11.689 - type: precision_at_100 value: 1.418 - type: precision_at_1000 value: 0.163 - type: precision_at_3 value: 33.018 - type: precision_at_5 value: 21.488 - type: recall_at_1 value: 34.558 - type: recall_at_10 value: 58.447 - type: recall_at_100 value: 70.91199999999999 - type: recall_at_1000 value: 81.31 - type: recall_at_3 value: 49.527 - type: recall_at_5 value: 53.72 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4 metrics: - type: accuracy value: 61.772000000000006 - type: ap value: 57.48217702943605 - type: f1 value: 61.20495351356274 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO config: default split: validation revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849 metrics: - type: map_at_1 value: 22.044 - type: map_at_10 value: 34.211000000000006 - type: map_at_100 value: 35.394 - type: map_at_1000 value: 35.443000000000005 - type: map_at_3 value: 30.318 - type: map_at_5 value: 32.535 - type: mrr_at_1 value: 22.722 - type: mrr_at_10 value: 34.842 - type: mrr_at_100 value: 35.954 - type: mrr_at_1000 value: 35.997 - type: mrr_at_3 value: 30.991000000000003 - type: mrr_at_5 value: 33.2 - type: ndcg_at_1 value: 22.722 - type: ndcg_at_10 value: 41.121 - type: ndcg_at_100 value: 46.841 - type: ndcg_at_1000 value: 48.049 - type: ndcg_at_3 value: 33.173 - type: ndcg_at_5 value: 37.145 - type: precision_at_1 value: 22.722 - type: precision_at_10 value: 6.516 - type: precision_at_100 value: 0.9400000000000001 - type: precision_at_1000 value: 0.104 - type: precision_at_3 value: 14.093 - type: precision_at_5 value: 10.473 - type: recall_at_1 value: 22.044 - type: recall_at_10 value: 62.382000000000005 - type: recall_at_100 value: 88.914 - type: recall_at_1000 value: 98.099 - type: recall_at_3 value: 40.782000000000004 - type: recall_at_5 value: 50.322 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 93.68217054263563 - type: f1 value: 93.25810075739523 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (de) config: de split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 82.05409974640745 - type: f1 value: 80.42814140324903 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (es) config: es split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 93.54903268845896 - type: f1 value: 92.8909878077932 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) config: fr split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 90.98340119010334 - type: f1 value: 90.51522537281313 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (hi) config: hi split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 89.33309429903191 - type: f1 value: 88.60371305209185 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (th) config: th split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 60.4882459312839 - type: f1 value: 59.02590456131682 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 71.34290925672595 - type: f1 value: 54.44803151449109 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (de) config: de split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 61.92448577063963 - type: f1 value: 43.125939975781854 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (es) config: es split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 74.48965977318213 - type: f1 value: 51.855353687466696 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) config: fr split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 69.11994989038521 - type: f1 value: 50.57872704171278 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (hi) config: hi split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 64.84761563284331 - type: f1 value: 43.61322970761394 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (th) config: th split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 49.35623869801085 - type: f1 value: 33.48547326952042 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (af) config: af split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 47.85474108944183 - type: f1 value: 46.50175016795915 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (am) config: am split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 33.29858776059179 - type: f1 value: 31.803027601259082 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ar) config: ar split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 59.24680564895763 - type: f1 value: 57.037691806846865 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (az) config: az split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 45.23537323470073 - type: f1 value: 44.81126398428613 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (bn) config: bn split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 61.590450571620714 - type: f1 value: 59.247442149977104 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (cy) config: cy split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 44.9226630800269 - type: f1 value: 44.076183379991654 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (da) config: da split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 51.23066577000672 - type: f1 value: 50.20719330417618 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (de) config: de split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 56.0995292535306 - type: f1 value: 53.29421532133969 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (el) config: el split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 46.12642905178211 - type: f1 value: 44.441530267639635 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 69.67047747141896 - type: f1 value: 68.38493366054783 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (es) config: es split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 66.3483523873571 - type: f1 value: 65.13046416817832 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fa) config: fa split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 51.20040349697378 - type: f1 value: 49.02889836601541 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fi) config: fi split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 45.33288500336248 - type: f1 value: 42.91893101970983 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) config: fr split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 66.95359784801613 - type: f1 value: 64.98788914810562 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (he) config: he split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 43.18090114324143 - type: f1 value: 41.31250407417542 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hi) config: hi split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 63.54068594485541 - type: f1 value: 61.94829361488948 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hu) config: hu split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 44.7343644922663 - type: f1 value: 43.23001702247849 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hy) config: hy split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 38.1271015467384 - type: f1 value: 36.94700198241727 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (id) config: id split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 64.05514458641561 - type: f1 value: 62.35033731674541 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (is) config: is split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 44.351042367182245 - type: f1 value: 43.13370397574502 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (it) config: it split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 60.77000672494955 - type: f1 value: 59.71546868957779 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ja) config: ja split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 61.22057834566241 - type: f1 value: 59.447639306287044 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (jv) config: jv split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 50.9448554135844 - type: f1 value: 48.524338247875214 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ka) config: ka split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 33.8399462004035 - type: f1 value: 33.518999997305535 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (km) config: km split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 37.34028244788165 - type: f1 value: 35.6156599064704 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (kn) config: kn split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 53.544048419636844 - type: f1 value: 51.29299915455352 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ko) config: ko split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 53.35574983187625 - type: f1 value: 51.463936565192945 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (lv) config: lv split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 46.503026227303295 - type: f1 value: 46.049497734375514 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ml) config: ml split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 58.268325487558826 - type: f1 value: 56.10849656896158 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (mn) config: mn split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 40.27572293207801 - type: f1 value: 40.20097238549224 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ms) config: ms split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 59.64694014794889 - type: f1 value: 58.39584148789066 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (my) config: my split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 37.41761936785474 - type: f1 value: 35.04551731363685 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nb) config: nb split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 49.408204438466704 - type: f1 value: 48.39369057638714 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nl) config: nl split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 52.09482178883659 - type: f1 value: 49.91518031712698 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) config: pl split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 50.477471418964356 - type: f1 value: 48.429495257184705 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pt) config: pt split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 66.69468728984532 - type: f1 value: 65.40306868707009 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ro) config: ro split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 50.52790854068594 - type: f1 value: 49.780400354514 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ru) config: ru split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 58.31540013449899 - type: f1 value: 56.144142926685134 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sl) config: sl split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 47.74041694687289 - type: f1 value: 46.16767322761359 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sq) config: sq split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 48.94418291862811 - type: f1 value: 48.445352284756325 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sv) config: sv split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 50.78681909885676 - type: f1 value: 49.64882295494536 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sw) config: sw split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 49.811701412239415 - type: f1 value: 48.213234514449375 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ta) config: ta split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 56.39542703429725 - type: f1 value: 54.031981085233795 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (te) config: te split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 54.71082716879623 - type: f1 value: 52.513144113474596 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (th) config: th split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 44.425016812373904 - type: f1 value: 43.96016300057656 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tl) config: tl split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 50.205110961667785 - type: f1 value: 48.86669996798709 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tr) config: tr split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 46.56355077336921 - type: f1 value: 45.18252022585022 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ur) config: ur split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 56.748486886348346 - type: f1 value: 54.29884570375382 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (vi) config: vi split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 64.52589105581708 - type: f1 value: 62.97947342861603 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) config: zh-CN split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 67.06792199058508 - type: f1 value: 65.36025601634017 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-TW) config: zh-TW split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 62.89172831203766 - type: f1 value: 62.69803707054342 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (af) config: af split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 51.47276395427035 - type: f1 value: 49.37463208130799 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (am) config: am split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 34.86886348352387 - type: f1 value: 33.74178074349636 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ar) config: ar split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.20511096166778 - type: f1 value: 65.85812500602437 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (az) config: az split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 45.578345662407536 - type: f1 value: 44.44514917028003 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (bn) config: bn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 67.29657027572293 - type: f1 value: 67.24477523937466 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (cy) config: cy split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 46.29455279085407 - type: f1 value: 43.8563839951935 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (da) config: da split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 53.52387357094821 - type: f1 value: 51.70977848027552 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (de) config: de split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 61.741761936785466 - type: f1 value: 60.219169644792295 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (el) config: el split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 48.957632817753876 - type: f1 value: 46.878428264460034 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 75.33624747814393 - type: f1 value: 75.9143846211171 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (es) config: es split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 73.34229993275049 - type: f1 value: 73.78165397558983 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fa) config: fa split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 53.174176193678555 - type: f1 value: 51.709679227778985 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fi) config: fi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 44.6906523201076 - type: f1 value: 41.54881682785664 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) config: fr split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 72.9119031607263 - type: f1 value: 73.2742013056326 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (he) config: he split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 43.10356422326832 - type: f1 value: 40.8859122581252 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hi) config: hi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 69.27370544720914 - type: f1 value: 69.39544506405082 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hu) config: hu split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 45.16476126429052 - type: f1 value: 42.74022531579054 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hy) config: hy split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 38.73234700739744 - type: f1 value: 37.40546754951026 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (id) config: id split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 70.12777404169468 - type: f1 value: 70.27219152812738 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (is) config: is split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 44.21318090114325 - type: f1 value: 41.934593213829366 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (it) config: it split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.57162071284466 - type: f1 value: 64.83341759045335 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ja) config: ja split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.75991930060525 - type: f1 value: 65.16549875504951 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (jv) config: jv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 54.79488903833223 - type: f1 value: 54.03616401426859 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ka) config: ka split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 32.992602555480836 - type: f1 value: 31.820068470018846 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (km) config: km split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 39.34431741761937 - type: f1 value: 36.436221665290105 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (kn) config: kn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 60.501008742434436 - type: f1 value: 60.051013712579085 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ko) config: ko split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 55.689307330195035 - type: f1 value: 53.94058032286942 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (lv) config: lv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 44.351042367182245 - type: f1 value: 42.05421666771541 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ml) config: ml split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.53127101546738 - type: f1 value: 65.98462024333497 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (mn) config: mn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 38.71553463349025 - type: f1 value: 37.44327037149584 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ms) config: ms split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.98991257565567 - type: f1 value: 63.87720198978004 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (my) config: my split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 36.839273705447205 - type: f1 value: 35.233967279698376 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nb) config: nb split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 51.79892400806993 - type: f1 value: 49.66926632125972 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nl) config: nl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 56.31809011432415 - type: f1 value: 53.832185336179826 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) config: pl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 49.979825151311374 - type: f1 value: 48.83013175441888 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pt) config: pt split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 71.45595158036315 - type: f1 value: 72.08708814699702 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ro) config: ro split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 53.68527236045729 - type: f1 value: 52.23278593929981 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ru) config: ru split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 61.60390047074647 - type: f1 value: 60.50391482195116 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sl) config: sl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 48.036314727639535 - type: f1 value: 46.43480413383716 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sq) config: sq split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 50.05716207128445 - type: f1 value: 48.85821859948888 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sv) config: sv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 51.728312037659705 - type: f1 value: 49.89292996950847 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sw) config: sw split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 54.21990585070613 - type: f1 value: 52.8711542984193 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ta) config: ta split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.770679219905844 - type: f1 value: 63.09441501491594 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (te) config: te split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.58574310692671 - type: f1 value: 61.61370697612978 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (th) config: th split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 45.17821116341628 - type: f1 value: 43.85143229183324 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tl) config: tl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 52.064559515803644 - type: f1 value: 50.94356892049626 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tr) config: tr split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 47.205783456624076 - type: f1 value: 47.04223644120489 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ur) config: ur split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.25689307330195 - type: f1 value: 63.89944944984115 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (vi) config: vi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 70.60524546065905 - type: f1 value: 71.5634157334358 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) config: zh-CN split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 73.95427034297242 - type: f1 value: 74.39706882311063 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-TW) config: zh-TW split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 70.29926025554808 - type: f1 value: 71.32045932560297 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test revision: dcefc037ef84348e49b0d29109e891c01067226b metrics: - type: v_measure value: 31.054474964883806 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc metrics: - type: v_measure value: 29.259725940477523 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 metrics: - type: map value: 31.785007883256572 - type: mrr value: 32.983556622438456 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus config: default split: test revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610 metrics: - type: map_at_1 value: 5.742 - type: map_at_10 value: 13.074 - type: map_at_100 value: 16.716 - type: map_at_1000 value: 18.238 - type: map_at_3 value: 9.600999999999999 - type: map_at_5 value: 11.129999999999999 - type: mrr_at_1 value: 47.988 - type: mrr_at_10 value: 55.958 - type: mrr_at_100 value: 56.58800000000001 - type: mrr_at_1000 value: 56.620000000000005 - type: mrr_at_3 value: 54.025 - type: mrr_at_5 value: 55.31 - type: ndcg_at_1 value: 46.44 - type: ndcg_at_10 value: 35.776 - type: ndcg_at_100 value: 32.891999999999996 - type: ndcg_at_1000 value: 41.835 - type: ndcg_at_3 value: 41.812 - type: ndcg_at_5 value: 39.249 - type: precision_at_1 value: 48.297000000000004 - type: precision_at_10 value: 26.687 - type: precision_at_100 value: 8.511000000000001 - type: precision_at_1000 value: 2.128 - type: precision_at_3 value: 39.009 - type: precision_at_5 value: 33.994 - type: recall_at_1 value: 5.742 - type: recall_at_10 value: 16.993 - type: recall_at_100 value: 33.69 - type: recall_at_1000 value: 66.75 - type: recall_at_3 value: 10.817 - type: recall_at_5 value: 13.256 - task: type: Retrieval dataset: type: nq name: MTEB NQ config: default split: test revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c metrics: - type: map_at_1 value: 30.789 - type: map_at_10 value: 45.751999999999995 - type: map_at_100 value: 46.766000000000005 - type: map_at_1000 value: 46.798 - type: map_at_3 value: 41.746 - type: map_at_5 value: 44.046 - type: mrr_at_1 value: 34.618 - type: mrr_at_10 value: 48.288 - type: mrr_at_100 value: 49.071999999999996 - type: mrr_at_1000 value: 49.094 - type: mrr_at_3 value: 44.979 - type: mrr_at_5 value: 46.953 - type: ndcg_at_1 value: 34.589 - type: ndcg_at_10 value: 53.151 - type: ndcg_at_100 value: 57.537000000000006 - type: ndcg_at_1000 value: 58.321999999999996 - type: ndcg_at_3 value: 45.628 - type: ndcg_at_5 value: 49.474000000000004 - type: precision_at_1 value: 34.589 - type: precision_at_10 value: 8.731 - type: precision_at_100 value: 1.119 - type: precision_at_1000 value: 0.11900000000000001 - type: precision_at_3 value: 20.819 - type: precision_at_5 value: 14.728 - type: recall_at_1 value: 30.789 - type: recall_at_10 value: 73.066 - type: recall_at_100 value: 92.27 - type: recall_at_1000 value: 98.18 - type: recall_at_3 value: 53.632999999999996 - type: recall_at_5 value: 62.476 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval config: default split: test revision: 6205996560df11e3a3da9ab4f926788fc30a7db4 metrics: - type: map_at_1 value: 54.993 - type: map_at_10 value: 69.07600000000001 - type: map_at_100 value: 70.05799999999999 - type: map_at_1000 value: 70.09 - type: map_at_3 value: 65.456 - type: map_at_5 value: 67.622 - type: mrr_at_1 value: 63.07000000000001 - type: mrr_at_10 value: 72.637 - type: mrr_at_100 value: 73.029 - type: mrr_at_1000 value: 73.033 - type: mrr_at_3 value: 70.572 - type: mrr_at_5 value: 71.86399999999999 - type: ndcg_at_1 value: 63.07000000000001 - type: ndcg_at_10 value: 74.708 - type: ndcg_at_100 value: 77.579 - type: ndcg_at_1000 value: 77.897 - type: ndcg_at_3 value: 69.69999999999999 - type: ndcg_at_5 value: 72.321 - type: precision_at_1 value: 63.07000000000001 - type: precision_at_10 value: 11.851 - type: precision_at_100 value: 1.481 - type: precision_at_1000 value: 0.156 - type: precision_at_3 value: 30.747000000000003 - type: precision_at_5 value: 20.830000000000002 - type: recall_at_1 value: 54.993 - type: recall_at_10 value: 87.18900000000001 - type: recall_at_100 value: 98.137 - type: recall_at_1000 value: 99.833 - type: recall_at_3 value: 73.654 - type: recall_at_5 value: 80.36 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test revision: b2805658ae38990172679479369a78b86de8c390 metrics: - type: v_measure value: 35.53178375429036 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 metrics: - type: v_measure value: 54.520782970558265 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS config: default split: test revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 metrics: - type: map_at_1 value: 4.3229999999999995 - type: map_at_10 value: 10.979999999999999 - type: map_at_100 value: 12.867 - type: map_at_1000 value: 13.147 - type: map_at_3 value: 7.973 - type: map_at_5 value: 9.513 - type: mrr_at_1 value: 21.3 - type: mrr_at_10 value: 32.34 - type: mrr_at_100 value: 33.428999999999995 - type: mrr_at_1000 value: 33.489999999999995 - type: mrr_at_3 value: 28.999999999999996 - type: mrr_at_5 value: 31.019999999999996 - type: ndcg_at_1 value: 21.3 - type: ndcg_at_10 value: 18.619 - type: ndcg_at_100 value: 26.108999999999998 - type: ndcg_at_1000 value: 31.253999999999998 - type: ndcg_at_3 value: 17.842 - type: ndcg_at_5 value: 15.673 - type: precision_at_1 value: 21.3 - type: precision_at_10 value: 9.55 - type: precision_at_100 value: 2.0340000000000003 - type: precision_at_1000 value: 0.327 - type: precision_at_3 value: 16.667 - type: precision_at_5 value: 13.76 - type: recall_at_1 value: 4.3229999999999995 - type: recall_at_10 value: 19.387 - type: recall_at_100 value: 41.307 - type: recall_at_1000 value: 66.475 - type: recall_at_3 value: 10.143 - type: recall_at_5 value: 14.007 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test revision: 20a6d6f312dd54037fe07a32d58e5e168867909d metrics: - type: cos_sim_pearson value: 78.77975189382573 - type: cos_sim_spearman value: 69.81522686267631 - type: euclidean_pearson value: 71.37617936889518 - type: euclidean_spearman value: 65.71738481148611 - type: manhattan_pearson value: 71.58222165832424 - type: manhattan_spearman value: 65.86851365286654 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f metrics: - type: cos_sim_pearson value: 77.75509450443367 - type: cos_sim_spearman value: 69.66180222442091 - type: euclidean_pearson value: 74.98512779786111 - type: euclidean_spearman value: 69.5997451409469 - type: manhattan_pearson value: 75.50135090962459 - type: manhattan_spearman value: 69.94984748475302 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 metrics: - type: cos_sim_pearson value: 79.42363892383264 - type: cos_sim_spearman value: 79.66529244176742 - type: euclidean_pearson value: 79.50429208135942 - type: euclidean_spearman value: 80.44767586416276 - type: manhattan_pearson value: 79.58563944997708 - type: manhattan_spearman value: 80.51452267103 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b metrics: - type: cos_sim_pearson value: 79.2749401478149 - type: cos_sim_spearman value: 74.6076920702392 - type: euclidean_pearson value: 73.3302002952881 - type: euclidean_spearman value: 70.67029803077013 - type: manhattan_pearson value: 73.52699344010296 - type: manhattan_spearman value: 70.8517556194297 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 metrics: - type: cos_sim_pearson value: 83.20884740785921 - type: cos_sim_spearman value: 83.80600789090722 - type: euclidean_pearson value: 74.9154089816344 - type: euclidean_spearman value: 75.69243899592276 - type: manhattan_pearson value: 75.0312832634451 - type: manhattan_spearman value: 75.78324960357642 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd metrics: - type: cos_sim_pearson value: 79.63194141000497 - type: cos_sim_spearman value: 80.40118418350866 - type: euclidean_pearson value: 72.07354384551088 - type: euclidean_spearman value: 72.28819150373845 - type: manhattan_pearson value: 72.08736119834145 - type: manhattan_spearman value: 72.28347083261288 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ko-ko) config: ko-ko split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 66.78512789499386 - type: cos_sim_spearman value: 66.89125587193288 - type: euclidean_pearson value: 58.74535708627959 - type: euclidean_spearman value: 59.62103716794647 - type: manhattan_pearson value: 59.00494529143961 - type: manhattan_spearman value: 59.832257846799806 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ar-ar) config: ar-ar split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 75.48960503523992 - type: cos_sim_spearman value: 76.4223037534204 - type: euclidean_pearson value: 64.93966381820944 - type: euclidean_spearman value: 62.39697395373789 - type: manhattan_pearson value: 65.54480770061505 - type: manhattan_spearman value: 62.944204863043105 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-ar) config: en-ar split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 77.7331440643619 - type: cos_sim_spearman value: 78.0748413292835 - type: euclidean_pearson value: 38.533108233460304 - type: euclidean_spearman value: 35.37638615280026 - type: manhattan_pearson value: 41.0639726746513 - type: manhattan_spearman value: 37.688161243671765 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-de) config: en-de split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 58.4628923720782 - type: cos_sim_spearman value: 59.10093128795948 - type: euclidean_pearson value: 30.422902393436836 - type: euclidean_spearman value: 27.837806030497457 - type: manhattan_pearson value: 32.51576984630963 - type: manhattan_spearman value: 29.181887010982514 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 86.87447904613737 - type: cos_sim_spearman value: 87.06554974065622 - type: euclidean_pearson value: 76.82669047851108 - type: euclidean_spearman value: 75.45711985511991 - type: manhattan_pearson value: 77.46644556452847 - type: manhattan_spearman value: 76.0249120007112 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-tr) config: en-tr split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 17.784495723497468 - type: cos_sim_spearman value: 11.79629537128697 - type: euclidean_pearson value: -4.354328445994008 - type: euclidean_spearman value: -6.984566116230058 - type: manhattan_pearson value: -4.166751901507852 - type: manhattan_spearman value: -6.984143198323786 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-en) config: es-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 76.9009642643449 - type: cos_sim_spearman value: 78.21764726338341 - type: euclidean_pearson value: 50.578959144342925 - type: euclidean_spearman value: 51.664379260719606 - type: manhattan_pearson value: 53.95690880393329 - type: manhattan_spearman value: 54.910058464050785 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-es) config: es-es split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 86.41638022270219 - type: cos_sim_spearman value: 86.00477030366811 - type: euclidean_pearson value: 79.7224037788285 - type: euclidean_spearman value: 79.21417626867616 - type: manhattan_pearson value: 80.29412412756984 - type: manhattan_spearman value: 79.49460867616206 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (fr-en) config: fr-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 79.90432664091082 - type: cos_sim_spearman value: 80.46007940700204 - type: euclidean_pearson value: 49.25348015214428 - type: euclidean_spearman value: 47.13113020475859 - type: manhattan_pearson value: 54.57291204043908 - type: manhattan_spearman value: 51.98559736896087 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (it-en) config: it-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 52.55164822309034 - type: cos_sim_spearman value: 51.57629192137736 - type: euclidean_pearson value: 16.63360593235354 - type: euclidean_spearman value: 14.479679923782912 - type: manhattan_pearson value: 18.524867185117472 - type: manhattan_spearman value: 16.65940056664755 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (nl-en) config: nl-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 46.83690919715875 - type: cos_sim_spearman value: 45.84993650002922 - type: euclidean_pearson value: 6.173128686815117 - type: euclidean_spearman value: 6.260781946306191 - type: manhattan_pearson value: 7.328440452367316 - type: manhattan_spearman value: 7.370842306497447 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 64.97916914277232 - type: cos_sim_spearman value: 66.13392188807865 - type: euclidean_pearson value: 65.3921146908468 - type: euclidean_spearman value: 65.8381588635056 - type: manhattan_pearson value: 65.8866165769975 - type: manhattan_spearman value: 66.27774050472219 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de) config: de split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 25.605130445111545 - type: cos_sim_spearman value: 30.054844562369254 - type: euclidean_pearson value: 23.890611005408196 - type: euclidean_spearman value: 29.07902600726761 - type: manhattan_pearson value: 24.239478426621833 - type: manhattan_spearman value: 29.48547576782375 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es) config: es split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 61.6665616159781 - type: cos_sim_spearman value: 65.41310206289988 - type: euclidean_pearson value: 68.38805493215008 - type: euclidean_spearman value: 65.22777377603435 - type: manhattan_pearson value: 69.37445390454346 - type: manhattan_spearman value: 66.02437701858754 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl) config: pl split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 15.302891825626372 - type: cos_sim_spearman value: 31.134517255070097 - type: euclidean_pearson value: 12.672592658843143 - type: euclidean_spearman value: 29.14881036784207 - type: manhattan_pearson value: 13.528545327757735 - type: manhattan_spearman value: 29.56217928148797 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (tr) config: tr split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 28.79299114515319 - type: cos_sim_spearman value: 47.135864983626206 - type: euclidean_pearson value: 40.66410787594309 - type: euclidean_spearman value: 45.09585593138228 - type: manhattan_pearson value: 42.02561630700308 - type: manhattan_spearman value: 45.43979983670554 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ar) config: ar split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 46.00096625052943 - type: cos_sim_spearman value: 58.67147426715496 - type: euclidean_pearson value: 54.7154367422438 - type: euclidean_spearman value: 59.003235142442634 - type: manhattan_pearson value: 56.3116235357115 - type: manhattan_spearman value: 60.12956331404423 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ru) config: ru split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 29.3396354650316 - type: cos_sim_spearman value: 43.3632935734809 - type: euclidean_pearson value: 31.18506539466593 - type: euclidean_spearman value: 37.531745324803815 - type: manhattan_pearson value: 32.829038232529015 - type: manhattan_spearman value: 38.04574361589953 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh) config: zh split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 62.9596148375188 - type: cos_sim_spearman value: 66.77653412402461 - type: euclidean_pearson value: 64.53156585980886 - type: euclidean_spearman value: 66.2884373036083 - type: manhattan_pearson value: 65.2831035495143 - type: manhattan_spearman value: 66.83641945244322 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr) config: fr split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 79.9138821493919 - type: cos_sim_spearman value: 80.38097535004677 - type: euclidean_pearson value: 76.2401499094322 - type: euclidean_spearman value: 77.00897050735907 - type: manhattan_pearson value: 76.69531453728563 - type: manhattan_spearman value: 77.83189696428695 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-en) config: de-en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 51.27009640779202 - type: cos_sim_spearman value: 51.16120562029285 - type: euclidean_pearson value: 52.20594985566323 - type: euclidean_spearman value: 52.75331049709882 - type: manhattan_pearson value: 52.2725118792549 - type: manhattan_spearman value: 53.614847968995115 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-en) config: es-en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 70.46044814118835 - type: cos_sim_spearman value: 75.05760236668672 - type: euclidean_pearson value: 72.80128921879461 - type: euclidean_spearman value: 73.81164755219257 - type: manhattan_pearson value: 72.7863795809044 - type: manhattan_spearman value: 73.65932033818906 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (it) config: it split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 61.89276840435938 - type: cos_sim_spearman value: 65.65042955732055 - type: euclidean_pearson value: 61.22969491863841 - type: euclidean_spearman value: 63.451215637904724 - type: manhattan_pearson value: 61.16138956945465 - type: manhattan_spearman value: 63.34966179331079 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl-en) config: pl-en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 56.377577221753626 - type: cos_sim_spearman value: 53.31223653270353 - type: euclidean_pearson value: 26.488793041564307 - type: euclidean_spearman value: 19.524551741701472 - type: manhattan_pearson value: 24.322868054606474 - type: manhattan_spearman value: 19.50371443994939 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh-en) config: zh-en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 69.3634693673425 - type: cos_sim_spearman value: 68.45051245419702 - type: euclidean_pearson value: 56.1417414374769 - type: euclidean_spearman value: 55.89891749631458 - type: manhattan_pearson value: 57.266417430882925 - type: manhattan_spearman value: 56.57927102744128 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-it) config: es-it split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 60.04169437653179 - type: cos_sim_spearman value: 65.49531007553446 - type: euclidean_pearson value: 58.583860732586324 - type: euclidean_spearman value: 58.80034792537441 - type: manhattan_pearson value: 59.02513161664622 - type: manhattan_spearman value: 58.42942047904558 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-fr) config: de-fr split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 48.81035211493999 - type: cos_sim_spearman value: 53.27599246786967 - type: euclidean_pearson value: 52.25710699032889 - type: euclidean_spearman value: 55.22995695529873 - type: manhattan_pearson value: 51.894901893217884 - type: manhattan_spearman value: 54.95919975149795 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-pl) config: de-pl split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 36.75993101477816 - type: cos_sim_spearman value: 43.050156692479355 - type: euclidean_pearson value: 51.49021084746248 - type: euclidean_spearman value: 49.54771253090078 - type: manhattan_pearson value: 54.68410760796417 - type: manhattan_spearman value: 48.19277197691717 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr-pl) config: fr-pl split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 48.553763306386486 - type: cos_sim_spearman value: 28.17180849095055 - type: euclidean_pearson value: 17.50739087826514 - type: euclidean_spearman value: 16.903085094570333 - type: manhattan_pearson value: 20.750046512534112 - type: manhattan_spearman value: 5.634361698190111 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test revision: 8913289635987208e6e7c72789e4be2fe94b6abd metrics: - type: cos_sim_pearson value: 82.17107190594417 - type: cos_sim_spearman value: 80.89611873505183 - type: euclidean_pearson value: 71.82491561814403 - type: euclidean_spearman value: 70.33608835403274 - type: manhattan_pearson value: 71.89538332420133 - type: manhattan_spearman value: 70.36082395775944 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test revision: 56a6d0140cf6356659e2a7c1413286a774468d44 metrics: - type: map value: 79.77047154974562 - type: mrr value: 94.25887021475256 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact config: default split: test revision: a75ae049398addde9b70f6b268875f5cbce99089 metrics: - type: map_at_1 value: 56.328 - type: map_at_10 value: 67.167 - type: map_at_100 value: 67.721 - type: map_at_1000 value: 67.735 - type: map_at_3 value: 64.20400000000001 - type: map_at_5 value: 65.904 - type: mrr_at_1 value: 59.667 - type: mrr_at_10 value: 68.553 - type: mrr_at_100 value: 68.992 - type: mrr_at_1000 value: 69.004 - type: mrr_at_3 value: 66.22200000000001 - type: mrr_at_5 value: 67.739 - type: ndcg_at_1 value: 59.667 - type: ndcg_at_10 value: 72.111 - type: ndcg_at_100 value: 74.441 - type: ndcg_at_1000 value: 74.90599999999999 - type: ndcg_at_3 value: 67.11399999999999 - type: ndcg_at_5 value: 69.687 - type: precision_at_1 value: 59.667 - type: precision_at_10 value: 9.733 - type: precision_at_100 value: 1.09 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_3 value: 26.444000000000003 - type: precision_at_5 value: 17.599999999999998 - type: recall_at_1 value: 56.328 - type: recall_at_10 value: 85.8 - type: recall_at_100 value: 96.167 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 72.433 - type: recall_at_5 value: 78.972 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea metrics: - type: cos_sim_accuracy value: 99.8019801980198 - type: cos_sim_ap value: 94.92527097094644 - type: cos_sim_f1 value: 89.91935483870968 - type: cos_sim_precision value: 90.65040650406505 - type: cos_sim_recall value: 89.2 - type: dot_accuracy value: 99.51782178217822 - type: dot_ap value: 81.30756869559929 - type: dot_f1 value: 75.88235294117648 - type: dot_precision value: 74.42307692307692 - type: dot_recall value: 77.4 - type: euclidean_accuracy value: 99.73069306930694 - type: euclidean_ap value: 91.05040371796932 - type: euclidean_f1 value: 85.7889237199582 - type: euclidean_precision value: 89.82494529540482 - type: euclidean_recall value: 82.1 - type: manhattan_accuracy value: 99.73762376237623 - type: manhattan_ap value: 91.4823412839869 - type: manhattan_f1 value: 86.39836984207845 - type: manhattan_precision value: 88.05815160955348 - type: manhattan_recall value: 84.8 - type: max_accuracy value: 99.8019801980198 - type: max_ap value: 94.92527097094644 - type: max_f1 value: 89.91935483870968 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235 metrics: - type: v_measure value: 55.13046832022158 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0 metrics: - type: v_measure value: 34.31252463546675 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 metrics: - type: map value: 51.06639688231414 - type: mrr value: 51.80205415499534 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122 metrics: - type: cos_sim_pearson value: 31.963331462886957 - type: cos_sim_spearman value: 33.59510652629926 - type: dot_pearson value: 29.033733540882123 - type: dot_spearman value: 31.550290638315504 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID config: default split: test revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 metrics: - type: map_at_1 value: 0.23600000000000002 - type: map_at_10 value: 2.09 - type: map_at_100 value: 12.466000000000001 - type: map_at_1000 value: 29.852 - type: map_at_3 value: 0.6859999999999999 - type: map_at_5 value: 1.099 - type: mrr_at_1 value: 88.0 - type: mrr_at_10 value: 94.0 - type: mrr_at_100 value: 94.0 - type: mrr_at_1000 value: 94.0 - type: mrr_at_3 value: 94.0 - type: mrr_at_5 value: 94.0 - type: ndcg_at_1 value: 86.0 - type: ndcg_at_10 value: 81.368 - type: ndcg_at_100 value: 61.879 - type: ndcg_at_1000 value: 55.282 - type: ndcg_at_3 value: 84.816 - type: ndcg_at_5 value: 82.503 - type: precision_at_1 value: 88.0 - type: precision_at_10 value: 85.6 - type: precision_at_100 value: 63.85999999999999 - type: precision_at_1000 value: 24.682000000000002 - type: precision_at_3 value: 88.667 - type: precision_at_5 value: 86.0 - type: recall_at_1 value: 0.23600000000000002 - type: recall_at_10 value: 2.25 - type: recall_at_100 value: 15.488 - type: recall_at_1000 value: 52.196 - type: recall_at_3 value: 0.721 - type: recall_at_5 value: 1.159 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (sqi-eng) config: sqi-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 12.7 - type: f1 value: 10.384182044950325 - type: precision value: 9.805277385275312 - type: recall value: 12.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fry-eng) config: fry-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 30.63583815028902 - type: f1 value: 24.623726947426373 - type: precision value: 22.987809919828013 - type: recall value: 30.63583815028902 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kur-eng) config: kur-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 10.487804878048781 - type: f1 value: 8.255945048627975 - type: precision value: 7.649047253615001 - type: recall value: 10.487804878048781 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tur-eng) config: tur-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.5 - type: f1 value: 6.154428783776609 - type: precision value: 5.680727638128585 - type: recall value: 8.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (deu-eng) config: deu-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 73.0 - type: f1 value: 70.10046605876393 - type: precision value: 69.0018253968254 - type: recall value: 73.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nld-eng) config: nld-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 32.7 - type: f1 value: 29.7428583868239 - type: precision value: 28.81671359506905 - type: recall value: 32.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ron-eng) config: ron-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 31.5 - type: f1 value: 27.228675552174003 - type: precision value: 25.950062299847747 - type: recall value: 31.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ang-eng) config: ang-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 35.82089552238806 - type: f1 value: 28.75836980510979 - type: precision value: 26.971643613434658 - type: recall value: 35.82089552238806 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ido-eng) config: ido-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 49.8 - type: f1 value: 43.909237401451776 - type: precision value: 41.944763440988936 - type: recall value: 49.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (jav-eng) config: jav-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 18.536585365853657 - type: f1 value: 15.020182570246751 - type: precision value: 14.231108073213337 - type: recall value: 18.536585365853657 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (isl-eng) config: isl-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.7 - type: f1 value: 6.2934784902885355 - type: precision value: 5.685926293425392 - type: recall value: 8.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (slv-eng) config: slv-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 12.879708383961116 - type: f1 value: 10.136118341751114 - type: precision value: 9.571444036679436 - type: recall value: 12.879708383961116 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cym-eng) config: cym-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 9.217391304347826 - type: f1 value: 6.965003297761793 - type: precision value: 6.476093529199119 - type: recall value: 9.217391304347826 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kaz-eng) config: kaz-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 4.3478260869565215 - type: f1 value: 3.3186971707677397 - type: precision value: 3.198658632552104 - type: recall value: 4.3478260869565215 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (est-eng) config: est-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 6.9 - type: f1 value: 4.760708297894056 - type: precision value: 4.28409511756074 - type: recall value: 6.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (heb-eng) config: heb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 2.1999999999999997 - type: f1 value: 1.6862703878117107 - type: precision value: 1.6048118233915603 - type: recall value: 2.1999999999999997 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gla-eng) config: gla-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 3.0156815440289506 - type: f1 value: 2.0913257250659134 - type: precision value: 1.9072775486461648 - type: recall value: 3.0156815440289506 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mar-eng) config: mar-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 49.0 - type: f1 value: 45.5254456536713 - type: precision value: 44.134609250398725 - type: recall value: 49.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lat-eng) config: lat-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 33.5 - type: f1 value: 28.759893973182564 - type: precision value: 27.401259116024836 - type: recall value: 33.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bel-eng) config: bel-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 10.2 - type: f1 value: 8.030039981676275 - type: precision value: 7.548748077210127 - type: recall value: 10.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pms-eng) config: pms-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 38.095238095238095 - type: f1 value: 31.944999250262406 - type: precision value: 30.04452690166976 - type: recall value: 38.095238095238095 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gle-eng) config: gle-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 4.8 - type: f1 value: 3.2638960786708067 - type: precision value: 3.0495382950729644 - type: recall value: 4.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pes-eng) config: pes-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 15.8 - type: f1 value: 12.131087470371275 - type: precision value: 11.141304011547815 - type: recall value: 15.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nob-eng) config: nob-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 23.3 - type: f1 value: 21.073044636921384 - type: precision value: 20.374220568287285 - type: recall value: 23.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bul-eng) config: bul-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 24.9 - type: f1 value: 20.091060685364987 - type: precision value: 18.899700591081224 - type: recall value: 24.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cbk-eng) config: cbk-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 70.1 - type: f1 value: 64.62940836940835 - type: precision value: 62.46559523809524 - type: recall value: 70.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hun-eng) config: hun-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.199999999999999 - type: f1 value: 5.06613460576115 - type: precision value: 4.625224463391809 - type: recall value: 7.199999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (uig-eng) config: uig-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 1.7999999999999998 - type: f1 value: 1.2716249514772895 - type: precision value: 1.2107445914723798 - type: recall value: 1.7999999999999998 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (rus-eng) config: rus-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 65.5 - type: f1 value: 59.84399711399712 - type: precision value: 57.86349567099567 - type: recall value: 65.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (spa-eng) config: spa-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 95.7 - type: f1 value: 94.48333333333333 - type: precision value: 93.89999999999999 - type: recall value: 95.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hye-eng) config: hye-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 0.8086253369272237 - type: f1 value: 0.4962046191492002 - type: precision value: 0.47272438578554393 - type: recall value: 0.8086253369272237 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tel-eng) config: tel-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 69.23076923076923 - type: f1 value: 64.6227941099736 - type: precision value: 63.03795877325289 - type: recall value: 69.23076923076923 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (afr-eng) config: afr-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 20.599999999999998 - type: f1 value: 16.62410040660465 - type: precision value: 15.598352437967069 - type: recall value: 20.599999999999998 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mon-eng) config: mon-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 4.318181818181818 - type: f1 value: 2.846721192535661 - type: precision value: 2.6787861417537147 - type: recall value: 4.318181818181818 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (arz-eng) config: arz-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 74.84276729559748 - type: f1 value: 70.6638714185884 - type: precision value: 68.86792452830188 - type: recall value: 74.84276729559748 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hrv-eng) config: hrv-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 15.9 - type: f1 value: 12.793698974586706 - type: precision value: 12.088118017657736 - type: recall value: 15.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nov-eng) config: nov-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 59.92217898832685 - type: f1 value: 52.23086900129701 - type: precision value: 49.25853869433636 - type: recall value: 59.92217898832685 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gsw-eng) config: gsw-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 27.350427350427353 - type: f1 value: 21.033781033781032 - type: precision value: 19.337955491801644 - type: recall value: 27.350427350427353 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nds-eng) config: nds-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 29.299999999999997 - type: f1 value: 23.91597452425777 - type: precision value: 22.36696598364942 - type: recall value: 29.299999999999997 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ukr-eng) config: ukr-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 27.3 - type: f1 value: 22.059393517688886 - type: precision value: 20.503235534170887 - type: recall value: 27.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (uzb-eng) config: uzb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.177570093457943 - type: f1 value: 4.714367017906037 - type: precision value: 4.163882933965758 - type: recall value: 8.177570093457943 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lit-eng) config: lit-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 5.800000000000001 - type: f1 value: 4.4859357432293825 - type: precision value: 4.247814465614043 - type: recall value: 5.800000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ina-eng) config: ina-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 78.4 - type: f1 value: 73.67166666666667 - type: precision value: 71.83285714285714 - type: recall value: 78.4 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lfn-eng) config: lfn-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 50.3 - type: f1 value: 44.85221545883311 - type: precision value: 43.04913026243909 - type: recall value: 50.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (zsm-eng) config: zsm-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 83.5 - type: f1 value: 79.95151515151515 - type: precision value: 78.53611111111111 - type: recall value: 83.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ita-eng) config: ita-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 69.89999999999999 - type: f1 value: 65.03756269256269 - type: precision value: 63.233519536019536 - type: recall value: 69.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cmn-eng) config: cmn-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 93.2 - type: f1 value: 91.44666666666666 - type: precision value: 90.63333333333333 - type: recall value: 93.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lvs-eng) config: lvs-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.3 - type: f1 value: 6.553388144729963 - type: precision value: 6.313497782829976 - type: recall value: 8.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (glg-eng) config: glg-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 83.6 - type: f1 value: 79.86243107769424 - type: precision value: 78.32555555555555 - type: recall value: 83.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ceb-eng) config: ceb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 9.166666666666666 - type: f1 value: 6.637753604420271 - type: precision value: 6.10568253585495 - type: recall value: 9.166666666666666 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bre-eng) config: bre-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.3999999999999995 - type: f1 value: 4.6729483612322165 - type: precision value: 4.103844520292658 - type: recall value: 7.3999999999999995 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ben-eng) config: ben-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 80.30000000000001 - type: f1 value: 75.97666666666667 - type: precision value: 74.16 - type: recall value: 80.30000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swg-eng) config: swg-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 23.214285714285715 - type: f1 value: 16.88988095238095 - type: precision value: 15.364937641723353 - type: recall value: 23.214285714285715 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (arq-eng) config: arq-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 33.15038419319429 - type: f1 value: 27.747873024072415 - type: precision value: 25.99320572578704 - type: recall value: 33.15038419319429 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kab-eng) config: kab-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 2.6 - type: f1 value: 1.687059048752127 - type: precision value: 1.5384884521299 - type: recall value: 2.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fra-eng) config: fra-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 93.30000000000001 - type: f1 value: 91.44000000000001 - type: precision value: 90.59166666666667 - type: recall value: 93.30000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (por-eng) config: por-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 94.1 - type: f1 value: 92.61666666666667 - type: precision value: 91.88333333333333 - type: recall value: 94.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tat-eng) config: tat-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 5.0 - type: f1 value: 3.589591971281927 - type: precision value: 3.3046491614532854 - type: recall value: 5.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (oci-eng) config: oci-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 45.9 - type: f1 value: 40.171969141969136 - type: precision value: 38.30764368870302 - type: recall value: 45.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pol-eng) config: pol-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 16.900000000000002 - type: f1 value: 14.094365204207351 - type: precision value: 13.276519841269844 - type: recall value: 16.900000000000002 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (war-eng) config: war-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 12.8 - type: f1 value: 10.376574912567156 - type: precision value: 9.758423963284509 - type: recall value: 12.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (aze-eng) config: aze-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.1 - type: f1 value: 6.319455355175778 - type: precision value: 5.849948830628881 - type: recall value: 8.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (vie-eng) config: vie-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 95.5 - type: f1 value: 94.19666666666667 - type: precision value: 93.60000000000001 - type: recall value: 95.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nno-eng) config: nno-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 19.1 - type: f1 value: 16.280080686081906 - type: precision value: 15.451573089395668 - type: recall value: 19.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cha-eng) config: cha-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 30.656934306569344 - type: f1 value: 23.2568647897115 - type: precision value: 21.260309034031664 - type: recall value: 30.656934306569344 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mhr-eng) config: mhr-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 2.1999999999999997 - type: f1 value: 1.556861047295521 - type: precision value: 1.4555993437238521 - type: recall value: 2.1999999999999997 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dan-eng) config: dan-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 27.500000000000004 - type: f1 value: 23.521682636223492 - type: precision value: 22.345341306967683 - type: recall value: 27.500000000000004 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ell-eng) config: ell-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.3999999999999995 - type: f1 value: 5.344253880846173 - type: precision value: 4.999794279068863 - type: recall value: 7.3999999999999995 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (amh-eng) config: amh-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 0.5952380952380952 - type: f1 value: 0.026455026455026457 - type: precision value: 0.013528138528138528 - type: recall value: 0.5952380952380952 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pam-eng) config: pam-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.3 - type: f1 value: 5.853140211779251 - type: precision value: 5.505563080945322 - type: recall value: 7.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hsb-eng) config: hsb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 13.250517598343686 - type: f1 value: 9.676349506190704 - type: precision value: 8.930392053553216 - type: recall value: 13.250517598343686 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (srp-eng) config: srp-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 14.499999999999998 - type: f1 value: 11.68912588067557 - type: precision value: 11.024716513105519 - type: recall value: 14.499999999999998 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (epo-eng) config: epo-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 30.099999999999998 - type: f1 value: 26.196880936315146 - type: precision value: 25.271714086169478 - type: recall value: 30.099999999999998 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kzj-eng) config: kzj-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 6.4 - type: f1 value: 5.1749445942023335 - type: precision value: 4.975338142029625 - type: recall value: 6.4 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (awa-eng) config: awa-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 39.39393939393939 - type: f1 value: 35.005707393767096 - type: precision value: 33.64342032053631 - type: recall value: 39.39393939393939 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fao-eng) config: fao-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 18.3206106870229 - type: f1 value: 12.610893447220345 - type: precision value: 11.079228765297467 - type: recall value: 18.3206106870229 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mal-eng) config: mal-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 85.58951965065502 - type: f1 value: 83.30363944928548 - type: precision value: 82.40026591554977 - type: recall value: 85.58951965065502 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ile-eng) config: ile-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 65.7 - type: f1 value: 59.589642857142856 - type: precision value: 57.392826797385624 - type: recall value: 65.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bos-eng) config: bos-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 18.07909604519774 - type: f1 value: 13.65194306689995 - type: precision value: 12.567953943826327 - type: recall value: 18.07909604519774 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cor-eng) config: cor-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 4.6 - type: f1 value: 2.8335386392505013 - type: precision value: 2.558444143575722 - type: recall value: 4.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cat-eng) config: cat-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 90.7 - type: f1 value: 88.30666666666666 - type: precision value: 87.195 - type: recall value: 90.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (eus-eng) config: eus-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 57.699999999999996 - type: f1 value: 53.38433067253876 - type: precision value: 51.815451335350346 - type: recall value: 57.699999999999996 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (yue-eng) config: yue-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 80.60000000000001 - type: f1 value: 77.0290354090354 - type: precision value: 75.61685897435898 - type: recall value: 80.60000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swe-eng) config: swe-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 24.6 - type: f1 value: 19.52814960069739 - type: precision value: 18.169084599880502 - type: recall value: 24.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dtp-eng) config: dtp-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 5.0 - type: f1 value: 3.4078491753102376 - type: precision value: 3.1757682319102387 - type: recall value: 5.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kat-eng) config: kat-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 1.2064343163538873 - type: f1 value: 0.4224313053283095 - type: precision value: 0.3360484946842894 - type: recall value: 1.2064343163538873 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (jpn-eng) config: jpn-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 76.1 - type: f1 value: 71.36246031746032 - type: precision value: 69.5086544011544 - type: recall value: 76.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (csb-eng) config: csb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 14.229249011857709 - type: f1 value: 10.026578603653704 - type: precision value: 9.09171178352764 - type: recall value: 14.229249011857709 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (xho-eng) config: xho-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 8.450704225352112 - type: f1 value: 5.51214407186151 - type: precision value: 4.928281812084629 - type: recall value: 8.450704225352112 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (orv-eng) config: orv-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.664670658682635 - type: f1 value: 5.786190079917295 - type: precision value: 5.3643643579244 - type: recall value: 7.664670658682635 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ind-eng) config: ind-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 90.5 - type: f1 value: 88.03999999999999 - type: precision value: 86.94833333333334 - type: recall value: 90.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tuk-eng) config: tuk-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 7.389162561576355 - type: f1 value: 5.482366349556517 - type: precision value: 5.156814449917898 - type: recall value: 7.389162561576355 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (max-eng) config: max-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 41.54929577464789 - type: f1 value: 36.13520282534367 - type: precision value: 34.818226488560995 - type: recall value: 41.54929577464789 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swh-eng) config: swh-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 20.76923076923077 - type: f1 value: 16.742497560177643 - type: precision value: 15.965759712090138 - type: recall value: 20.76923076923077 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hin-eng) config: hin-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 88.1 - type: f1 value: 85.23176470588236 - type: precision value: 84.04458333333334 - type: recall value: 88.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dsb-eng) config: dsb-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 11.899791231732777 - type: f1 value: 8.776706659565102 - type: precision value: 8.167815946521582 - type: recall value: 11.899791231732777 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ber-eng) config: ber-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 6.1 - type: f1 value: 4.916589537178435 - type: precision value: 4.72523017415345 - type: recall value: 6.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tam-eng) config: tam-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 76.54723127035831 - type: f1 value: 72.75787187839306 - type: precision value: 71.43338442869005 - type: recall value: 76.54723127035831 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (slk-eng) config: slk-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 11.700000000000001 - type: f1 value: 9.975679190026007 - type: precision value: 9.569927715653522 - type: recall value: 11.700000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tgl-eng) config: tgl-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 13.100000000000001 - type: f1 value: 10.697335850115408 - type: precision value: 10.113816082086341 - type: recall value: 13.100000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ast-eng) config: ast-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 76.37795275590551 - type: f1 value: 71.12860892388451 - type: precision value: 68.89763779527559 - type: recall value: 76.37795275590551 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mkd-eng) config: mkd-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 13.700000000000001 - type: f1 value: 10.471861684067568 - type: precision value: 9.602902567641697 - type: recall value: 13.700000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (khm-eng) config: khm-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 0.554016620498615 - type: f1 value: 0.37034084643642423 - type: precision value: 0.34676040281208437 - type: recall value: 0.554016620498615 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ces-eng) config: ces-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 12.4 - type: f1 value: 9.552607451092534 - type: precision value: 8.985175505050504 - type: recall value: 12.4 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tzl-eng) config: tzl-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 33.65384615384615 - type: f1 value: 27.820512820512818 - type: precision value: 26.09432234432234 - type: recall value: 33.65384615384615 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (urd-eng) config: urd-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 74.5 - type: f1 value: 70.09686507936507 - type: precision value: 68.3117857142857 - type: recall value: 74.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ara-eng) config: ara-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 88.3 - type: f1 value: 85.37333333333333 - type: precision value: 84.05833333333334 - type: recall value: 88.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kor-eng) config: kor-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 25.0 - type: f1 value: 22.393124632031995 - type: precision value: 21.58347686592367 - type: recall value: 25.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (yid-eng) config: yid-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 0.589622641509434 - type: f1 value: 0.15804980033762941 - type: precision value: 0.1393275384872965 - type: recall value: 0.589622641509434 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fin-eng) config: fin-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 4.1000000000000005 - type: f1 value: 3.4069011332551775 - type: precision value: 3.1784507042253516 - type: recall value: 4.1000000000000005 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tha-eng) config: tha-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 3.102189781021898 - type: f1 value: 2.223851811694751 - type: precision value: 2.103465682299194 - type: recall value: 3.102189781021898 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (wuu-eng) config: wuu-eng split: test revision: ed9e4a974f867fd9736efcf222fc3a26487387a5 metrics: - type: accuracy value: 83.1 - type: f1 value: 79.58255835667599 - type: precision value: 78.09708333333333 - type: recall value: 83.1 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 config: default split: test revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b metrics: - type: map_at_1 value: 2.322 - type: map_at_10 value: 8.959999999999999 - type: map_at_100 value: 15.136 - type: map_at_1000 value: 16.694 - type: map_at_3 value: 4.837000000000001 - type: map_at_5 value: 6.196 - type: mrr_at_1 value: 28.571 - type: mrr_at_10 value: 47.589999999999996 - type: mrr_at_100 value: 48.166 - type: mrr_at_1000 value: 48.169000000000004 - type: mrr_at_3 value: 43.197 - type: mrr_at_5 value: 45.646 - type: ndcg_at_1 value: 26.531 - type: ndcg_at_10 value: 23.982 - type: ndcg_at_100 value: 35.519 - type: ndcg_at_1000 value: 46.878 - type: ndcg_at_3 value: 26.801000000000002 - type: ndcg_at_5 value: 24.879 - type: precision_at_1 value: 28.571 - type: precision_at_10 value: 22.041 - type: precision_at_100 value: 7.4079999999999995 - type: precision_at_1000 value: 1.492 - type: precision_at_3 value: 28.571 - type: precision_at_5 value: 25.306 - type: recall_at_1 value: 2.322 - type: recall_at_10 value: 15.443999999999999 - type: recall_at_100 value: 45.918 - type: recall_at_1000 value: 79.952 - type: recall_at_3 value: 6.143 - type: recall_at_5 value: 8.737 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de metrics: - type: accuracy value: 66.5452 - type: ap value: 12.99191723223892 - type: f1 value: 51.667665096195734 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test revision: 62146448f05be9e52a36b8ee9936447ea787eede metrics: - type: accuracy value: 55.854555744199196 - type: f1 value: 56.131766302254185 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 metrics: - type: v_measure value: 37.27891385518074 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 metrics: - type: cos_sim_accuracy value: 83.53102461703523 - type: cos_sim_ap value: 65.30753664579191 - type: cos_sim_f1 value: 61.739943872778305 - type: cos_sim_precision value: 55.438891222175556 - type: cos_sim_recall value: 69.65699208443272 - type: dot_accuracy value: 80.38981939560112 - type: dot_ap value: 53.52081118421347 - type: dot_f1 value: 54.232957844617346 - type: dot_precision value: 48.43393486828459 - type: dot_recall value: 61.60949868073878 - type: euclidean_accuracy value: 82.23758717291531 - type: euclidean_ap value: 60.361102792772535 - type: euclidean_f1 value: 57.50518791791561 - type: euclidean_precision value: 51.06470106470107 - type: euclidean_recall value: 65.8047493403694 - type: manhattan_accuracy value: 82.14221851344102 - type: manhattan_ap value: 60.341937223793366 - type: manhattan_f1 value: 57.53803596127247 - type: manhattan_precision value: 51.08473188702415 - type: manhattan_recall value: 65.85751978891821 - type: max_accuracy value: 83.53102461703523 - type: max_ap value: 65.30753664579191 - type: max_f1 value: 61.739943872778305 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf metrics: - type: cos_sim_accuracy value: 88.75305623471883 - type: cos_sim_ap value: 85.46387153880272 - type: cos_sim_f1 value: 77.91527673159008 - type: cos_sim_precision value: 72.93667315828353 - type: cos_sim_recall value: 83.62334462580844 - type: dot_accuracy value: 85.08169363915086 - type: dot_ap value: 74.96808060965559 - type: dot_f1 value: 71.39685033990366 - type: dot_precision value: 64.16948111759288 - type: dot_recall value: 80.45888512473051 - type: euclidean_accuracy value: 85.84235650250321 - type: euclidean_ap value: 78.42045145247211 - type: euclidean_f1 value: 70.32669630775179 - type: euclidean_precision value: 70.6298050788227 - type: euclidean_recall value: 70.02617801047121 - type: manhattan_accuracy value: 85.86176116738464 - type: manhattan_ap value: 78.54012451558276 - type: manhattan_f1 value: 70.56508080693389 - type: manhattan_precision value: 69.39626293456413 - type: manhattan_recall value: 71.77394518016631 - type: max_accuracy value: 88.75305623471883 - type: max_ap value: 85.46387153880272 - type: max_f1 value: 77.91527673159008 --- ## Usage For usage instructions, refer to: https://github.com/Muennighoff/sgpt#asymmetric-semantic-search-be The model was trained with the command ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/ms_marco/train_bi-encoder_mnrl.py --model_name bigscience/bloom-7b1 --train_batch_size 32 --eval_batch_size 16 --freezenonbias --specb --lr 4e-4 --wandb --wandbwatchlog gradients --pooling weightedmean --gradcache --chunksize 8 ``` ## Evaluation Results `{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}` See the evaluation folder or [MTEB](https://huggingface.co/spaces/mteb/leaderboard) for more results. ## Training The model was trained with the parameters: **DataLoader**: `torch.utils.data.dataloader.DataLoader` of length 15600 with parameters: ``` {'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} ``` The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904 **Loss**: `sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache` Parameters of the fit()-Method: ``` { "epochs": 10, "evaluation_steps": 0, "evaluator": "NoneType", "max_grad_norm": 1, "optimizer_class": "", "optimizer_params": { "lr": 0.0004 }, "scheduler": "WarmupLinear", "steps_per_epoch": null, "warmup_steps": 1000, "weight_decay": 0.01 } ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: BloomModel (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False}) ) ``` ## Citing & Authors ```bibtex @article{muennighoff2022sgpt, title={SGPT: GPT Sentence Embeddings for Semantic Search}, author={Muennighoff, Niklas}, journal={arXiv preprint arXiv:2202.08904}, year={2022} } ```