--- pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb model-index: - name: SGPT-5.8B-weightedmean-nli-bitfit results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 74.07462686567165 - type: ap value: 37.44692407529112 - type: f1 value: 68.28971003916419 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (de) config: de split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 66.63811563169165 - type: ap value: 78.57252079915924 - type: f1 value: 64.5543087846584 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en-ext) config: en-ext split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 77.21889055472263 - type: ap value: 25.663426367826712 - type: f1 value: 64.26265688503176 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (ja) config: ja split: test revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996 metrics: - type: accuracy value: 58.06209850107067 - type: ap value: 14.028219107023915 - type: f1 value: 48.10387189660778 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1 metrics: - type: accuracy value: 82.30920000000002 - type: ap value: 76.88786578621213 - type: f1 value: 82.15455656065011 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 41.584 - type: f1 value: 41.203137944390114 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (de) config: de split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 35.288000000000004 - type: f1 value: 34.672995558518096 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (es) config: es split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 38.34 - type: f1 value: 37.608755629529455 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) config: fr split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 37.839999999999996 - type: f1 value: 36.86898201563507 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (ja) config: ja split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 30.936000000000003 - type: f1 value: 30.49401738527071 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) config: zh split: test revision: c379a6705fec24a2493fa68e011692605f44e119 metrics: - type: accuracy value: 33.75 - type: f1 value: 33.38338946025617 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna config: default split: test revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3 metrics: - type: map_at_1 value: 13.727 - type: map_at_10 value: 26.740000000000002 - type: map_at_100 value: 28.218 - type: map_at_1000 value: 28.246 - type: map_at_3 value: 21.728 - type: map_at_5 value: 24.371000000000002 - type: ndcg_at_1 value: 13.727 - type: ndcg_at_10 value: 35.07 - type: ndcg_at_100 value: 41.947 - type: ndcg_at_1000 value: 42.649 - type: ndcg_at_3 value: 24.484 - type: ndcg_at_5 value: 29.282999999999998 - type: precision_at_1 value: 13.727 - type: precision_at_10 value: 6.223 - type: precision_at_100 value: 0.9369999999999999 - type: precision_at_1000 value: 0.099 - type: precision_at_3 value: 10.835 - type: precision_at_5 value: 8.848 - type: recall_at_1 value: 13.727 - type: recall_at_10 value: 62.233000000000004 - type: recall_at_100 value: 93.67 - type: recall_at_1000 value: 99.14699999999999 - type: recall_at_3 value: 32.504 - type: recall_at_5 value: 44.239 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 metrics: - type: v_measure value: 40.553923271901695 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 metrics: - type: v_measure value: 32.49323183712211 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c metrics: - type: map value: 55.89811361443445 - type: mrr value: 70.16235764850724 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test revision: 9ee918f184421b6bd48b78f6c714d86546106103 metrics: - type: cos_sim_pearson value: 82.50506557805856 - type: cos_sim_spearman value: 79.50000423261176 - type: euclidean_pearson value: 75.76190885392926 - type: euclidean_spearman value: 76.7330737163434 - type: manhattan_pearson value: 75.825318036112 - type: manhattan_spearman value: 76.7415076434559 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (de-en) config: de-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 75.49060542797494 - type: f1 value: 75.15379262352123 - type: precision value: 74.99391092553932 - type: recall value: 75.49060542797494 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (fr-en) config: fr-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 0.4182258419546555 - type: f1 value: 0.4182258419546555 - type: precision value: 0.4182258419546555 - type: recall value: 0.4182258419546555 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (ru-en) config: ru-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 0.013855213023900243 - type: f1 value: 0.0115460108532502 - type: precision value: 0.010391409767925183 - type: recall value: 0.013855213023900243 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (zh-en) config: zh-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 0.315955766192733 - type: f1 value: 0.315955766192733 - type: precision value: 0.315955766192733 - type: recall value: 0.315955766192733 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test revision: 44fa15921b4c889113cc5df03dd4901b49161ab7 metrics: - type: accuracy value: 81.74025974025973 - type: f1 value: 81.66568824876 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55 metrics: - type: v_measure value: 33.59451202614059 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1 metrics: - type: v_measure value: 29.128241446157165 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 26.715 - type: map_at_10 value: 35.007 - type: map_at_100 value: 36.352000000000004 - type: map_at_1000 value: 36.51 - type: map_at_3 value: 32.257999999999996 - type: map_at_5 value: 33.595000000000006 - type: ndcg_at_1 value: 33.906 - type: ndcg_at_10 value: 40.353 - type: ndcg_at_100 value: 45.562999999999995 - type: ndcg_at_1000 value: 48.454 - type: ndcg_at_3 value: 36.349 - type: ndcg_at_5 value: 37.856 - type: precision_at_1 value: 33.906 - type: precision_at_10 value: 7.854 - type: precision_at_100 value: 1.29 - type: precision_at_1000 value: 0.188 - type: precision_at_3 value: 17.549 - type: precision_at_5 value: 12.561 - type: recall_at_1 value: 26.715 - type: recall_at_10 value: 49.508 - type: recall_at_100 value: 71.76599999999999 - type: recall_at_1000 value: 91.118 - type: recall_at_3 value: 37.356 - type: recall_at_5 value: 41.836 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 19.663 - type: map_at_10 value: 27.086 - type: map_at_100 value: 28.066999999999997 - type: map_at_1000 value: 28.18 - type: map_at_3 value: 24.819 - type: map_at_5 value: 26.332 - type: ndcg_at_1 value: 25.732 - type: ndcg_at_10 value: 31.613999999999997 - type: ndcg_at_100 value: 35.757 - type: ndcg_at_1000 value: 38.21 - type: ndcg_at_3 value: 28.332 - type: ndcg_at_5 value: 30.264000000000003 - type: precision_at_1 value: 25.732 - type: precision_at_10 value: 6.038 - type: precision_at_100 value: 1.034 - type: precision_at_1000 value: 0.149 - type: precision_at_3 value: 13.864 - type: precision_at_5 value: 10.241999999999999 - type: recall_at_1 value: 19.663 - type: recall_at_10 value: 39.585 - type: recall_at_100 value: 57.718 - type: recall_at_1000 value: 74.26700000000001 - type: recall_at_3 value: 29.845 - type: recall_at_5 value: 35.105 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 30.125 - type: map_at_10 value: 39.824 - type: map_at_100 value: 40.935 - type: map_at_1000 value: 41.019 - type: map_at_3 value: 37.144 - type: map_at_5 value: 38.647999999999996 - type: ndcg_at_1 value: 34.922 - type: ndcg_at_10 value: 45.072 - type: ndcg_at_100 value: 50.046 - type: ndcg_at_1000 value: 51.895 - type: ndcg_at_3 value: 40.251 - type: ndcg_at_5 value: 42.581 - type: precision_at_1 value: 34.922 - type: precision_at_10 value: 7.303999999999999 - type: precision_at_100 value: 1.0739999999999998 - type: precision_at_1000 value: 0.13 - type: precision_at_3 value: 17.994 - type: precision_at_5 value: 12.475999999999999 - type: recall_at_1 value: 30.125 - type: recall_at_10 value: 57.253 - type: recall_at_100 value: 79.35799999999999 - type: recall_at_1000 value: 92.523 - type: recall_at_3 value: 44.088 - type: recall_at_5 value: 49.893 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 16.298000000000002 - type: map_at_10 value: 21.479 - type: map_at_100 value: 22.387 - type: map_at_1000 value: 22.483 - type: map_at_3 value: 19.743 - type: map_at_5 value: 20.444000000000003 - type: ndcg_at_1 value: 17.740000000000002 - type: ndcg_at_10 value: 24.887 - type: ndcg_at_100 value: 29.544999999999998 - type: ndcg_at_1000 value: 32.417 - type: ndcg_at_3 value: 21.274 - type: ndcg_at_5 value: 22.399 - type: precision_at_1 value: 17.740000000000002 - type: precision_at_10 value: 3.932 - type: precision_at_100 value: 0.666 - type: precision_at_1000 value: 0.094 - type: precision_at_3 value: 8.927 - type: precision_at_5 value: 6.056 - type: recall_at_1 value: 16.298000000000002 - type: recall_at_10 value: 34.031 - type: recall_at_100 value: 55.769000000000005 - type: recall_at_1000 value: 78.19500000000001 - type: recall_at_3 value: 23.799999999999997 - type: recall_at_5 value: 26.562 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 10.958 - type: map_at_10 value: 16.999 - type: map_at_100 value: 17.979 - type: map_at_1000 value: 18.112000000000002 - type: map_at_3 value: 15.010000000000002 - type: map_at_5 value: 16.256999999999998 - type: ndcg_at_1 value: 14.179 - type: ndcg_at_10 value: 20.985 - type: ndcg_at_100 value: 26.216 - type: ndcg_at_1000 value: 29.675 - type: ndcg_at_3 value: 17.28 - type: ndcg_at_5 value: 19.301 - type: precision_at_1 value: 14.179 - type: precision_at_10 value: 3.968 - type: precision_at_100 value: 0.784 - type: precision_at_1000 value: 0.121 - type: precision_at_3 value: 8.541 - type: precision_at_5 value: 6.468 - type: recall_at_1 value: 10.958 - type: recall_at_10 value: 29.903000000000002 - type: recall_at_100 value: 53.413 - type: recall_at_1000 value: 78.74799999999999 - type: recall_at_3 value: 19.717000000000002 - type: recall_at_5 value: 24.817 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 21.217 - type: map_at_10 value: 29.677 - type: map_at_100 value: 30.928 - type: map_at_1000 value: 31.063000000000002 - type: map_at_3 value: 26.611 - type: map_at_5 value: 28.463 - type: ndcg_at_1 value: 26.083000000000002 - type: ndcg_at_10 value: 35.217 - type: ndcg_at_100 value: 40.715 - type: ndcg_at_1000 value: 43.559 - type: ndcg_at_3 value: 30.080000000000002 - type: ndcg_at_5 value: 32.701 - type: precision_at_1 value: 26.083000000000002 - type: precision_at_10 value: 6.622 - type: precision_at_100 value: 1.115 - type: precision_at_1000 value: 0.156 - type: precision_at_3 value: 14.629 - type: precision_at_5 value: 10.837 - type: recall_at_1 value: 21.217 - type: recall_at_10 value: 47.031 - type: recall_at_100 value: 70.378 - type: recall_at_1000 value: 89.704 - type: recall_at_3 value: 32.427 - type: recall_at_5 value: 39.31 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 19.274 - type: map_at_10 value: 26.398 - type: map_at_100 value: 27.711000000000002 - type: map_at_1000 value: 27.833000000000002 - type: map_at_3 value: 24.294 - type: map_at_5 value: 25.385 - type: ndcg_at_1 value: 24.886 - type: ndcg_at_10 value: 30.909 - type: ndcg_at_100 value: 36.941 - type: ndcg_at_1000 value: 39.838 - type: ndcg_at_3 value: 27.455000000000002 - type: ndcg_at_5 value: 28.828 - type: precision_at_1 value: 24.886 - type: precision_at_10 value: 5.6739999999999995 - type: precision_at_100 value: 1.0290000000000001 - type: precision_at_1000 value: 0.146 - type: precision_at_3 value: 13.242 - type: precision_at_5 value: 9.292 - type: recall_at_1 value: 19.274 - type: recall_at_10 value: 39.643 - type: recall_at_100 value: 66.091 - type: recall_at_1000 value: 86.547 - type: recall_at_3 value: 29.602 - type: recall_at_5 value: 33.561 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 18.653666666666666 - type: map_at_10 value: 25.606666666666666 - type: map_at_100 value: 26.669333333333334 - type: map_at_1000 value: 26.795833333333334 - type: map_at_3 value: 23.43433333333333 - type: map_at_5 value: 24.609666666666666 - type: ndcg_at_1 value: 22.742083333333333 - type: ndcg_at_10 value: 29.978333333333335 - type: ndcg_at_100 value: 34.89808333333333 - type: ndcg_at_1000 value: 37.806583333333336 - type: ndcg_at_3 value: 26.223666666666674 - type: ndcg_at_5 value: 27.91033333333333 - type: precision_at_1 value: 22.742083333333333 - type: precision_at_10 value: 5.397083333333334 - type: precision_at_100 value: 0.9340000000000002 - type: precision_at_1000 value: 0.13691666666666663 - type: precision_at_3 value: 12.331083333333332 - type: precision_at_5 value: 8.805499999999999 - type: recall_at_1 value: 18.653666666666666 - type: recall_at_10 value: 39.22625000000001 - type: recall_at_100 value: 61.31049999999999 - type: recall_at_1000 value: 82.19058333333334 - type: recall_at_3 value: 28.517333333333333 - type: recall_at_5 value: 32.9565 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 16.07 - type: map_at_10 value: 21.509 - type: map_at_100 value: 22.335 - type: map_at_1000 value: 22.437 - type: map_at_3 value: 19.717000000000002 - type: map_at_5 value: 20.574 - type: ndcg_at_1 value: 18.865000000000002 - type: ndcg_at_10 value: 25.135999999999996 - type: ndcg_at_100 value: 29.483999999999998 - type: ndcg_at_1000 value: 32.303 - type: ndcg_at_3 value: 21.719 - type: ndcg_at_5 value: 23.039 - type: precision_at_1 value: 18.865000000000002 - type: precision_at_10 value: 4.263999999999999 - type: precision_at_100 value: 0.696 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 9.866999999999999 - type: precision_at_5 value: 6.902 - type: recall_at_1 value: 16.07 - type: recall_at_10 value: 33.661 - type: recall_at_100 value: 54.001999999999995 - type: recall_at_1000 value: 75.564 - type: recall_at_3 value: 23.956 - type: recall_at_5 value: 27.264 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 10.847 - type: map_at_10 value: 15.518 - type: map_at_100 value: 16.384 - type: map_at_1000 value: 16.506 - type: map_at_3 value: 14.093 - type: map_at_5 value: 14.868 - type: ndcg_at_1 value: 13.764999999999999 - type: ndcg_at_10 value: 18.766 - type: ndcg_at_100 value: 23.076 - type: ndcg_at_1000 value: 26.344 - type: ndcg_at_3 value: 16.150000000000002 - type: ndcg_at_5 value: 17.373 - type: precision_at_1 value: 13.764999999999999 - type: precision_at_10 value: 3.572 - type: precision_at_100 value: 0.6779999999999999 - type: precision_at_1000 value: 0.11199999999999999 - type: precision_at_3 value: 7.88 - type: precision_at_5 value: 5.712 - type: recall_at_1 value: 10.847 - type: recall_at_10 value: 25.141999999999996 - type: recall_at_100 value: 44.847 - type: recall_at_1000 value: 68.92099999999999 - type: recall_at_3 value: 17.721999999999998 - type: recall_at_5 value: 20.968999999999998 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 18.377 - type: map_at_10 value: 26.005 - type: map_at_100 value: 26.996 - type: map_at_1000 value: 27.116 - type: map_at_3 value: 23.712 - type: map_at_5 value: 24.859 - type: ndcg_at_1 value: 22.201 - type: ndcg_at_10 value: 30.635 - type: ndcg_at_100 value: 35.623 - type: ndcg_at_1000 value: 38.551 - type: ndcg_at_3 value: 26.565 - type: ndcg_at_5 value: 28.28 - type: precision_at_1 value: 22.201 - type: precision_at_10 value: 5.41 - type: precision_at_100 value: 0.88 - type: precision_at_1000 value: 0.125 - type: precision_at_3 value: 12.531 - type: precision_at_5 value: 8.806 - type: recall_at_1 value: 18.377 - type: recall_at_10 value: 40.908 - type: recall_at_100 value: 63.563 - type: recall_at_1000 value: 84.503 - type: recall_at_3 value: 29.793999999999997 - type: recall_at_5 value: 34.144999999999996 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 20.246 - type: map_at_10 value: 27.528000000000002 - type: map_at_100 value: 28.78 - type: map_at_1000 value: 29.002 - type: map_at_3 value: 25.226 - type: map_at_5 value: 26.355 - type: ndcg_at_1 value: 25.099 - type: ndcg_at_10 value: 32.421 - type: ndcg_at_100 value: 37.2 - type: ndcg_at_1000 value: 40.693 - type: ndcg_at_3 value: 28.768 - type: ndcg_at_5 value: 30.23 - type: precision_at_1 value: 25.099 - type: precision_at_10 value: 6.245 - type: precision_at_100 value: 1.269 - type: precision_at_1000 value: 0.218 - type: precision_at_3 value: 13.767999999999999 - type: precision_at_5 value: 9.881 - type: recall_at_1 value: 20.246 - type: recall_at_10 value: 41.336 - type: recall_at_100 value: 63.098 - type: recall_at_1000 value: 86.473 - type: recall_at_3 value: 30.069000000000003 - type: recall_at_5 value: 34.262 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval config: default split: test revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db metrics: - type: map_at_1 value: 14.054 - type: map_at_10 value: 20.25 - type: map_at_100 value: 21.178 - type: map_at_1000 value: 21.288999999999998 - type: map_at_3 value: 18.584999999999997 - type: map_at_5 value: 19.536 - type: ndcg_at_1 value: 15.527 - type: ndcg_at_10 value: 23.745 - type: ndcg_at_100 value: 28.610999999999997 - type: ndcg_at_1000 value: 31.740000000000002 - type: ndcg_at_3 value: 20.461 - type: ndcg_at_5 value: 22.072 - type: precision_at_1 value: 15.527 - type: precision_at_10 value: 3.882 - type: precision_at_100 value: 0.6930000000000001 - type: precision_at_1000 value: 0.104 - type: precision_at_3 value: 9.181000000000001 - type: precision_at_5 value: 6.433 - type: recall_at_1 value: 14.054 - type: recall_at_10 value: 32.714 - type: recall_at_100 value: 55.723 - type: recall_at_1000 value: 79.72399999999999 - type: recall_at_3 value: 23.832 - type: recall_at_5 value: 27.754 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER config: default split: test revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce metrics: - type: map_at_1 value: 6.122 - type: map_at_10 value: 11.556 - type: map_at_100 value: 12.998000000000001 - type: map_at_1000 value: 13.202 - type: map_at_3 value: 9.657 - type: map_at_5 value: 10.585 - type: ndcg_at_1 value: 15.049000000000001 - type: ndcg_at_10 value: 17.574 - type: ndcg_at_100 value: 24.465999999999998 - type: ndcg_at_1000 value: 28.511999999999997 - type: ndcg_at_3 value: 13.931 - type: ndcg_at_5 value: 15.112 - type: precision_at_1 value: 15.049000000000001 - type: precision_at_10 value: 5.831 - type: precision_at_100 value: 1.322 - type: precision_at_1000 value: 0.20500000000000002 - type: precision_at_3 value: 10.749 - type: precision_at_5 value: 8.365 - type: recall_at_1 value: 6.122 - type: recall_at_10 value: 22.207 - type: recall_at_100 value: 47.08 - type: recall_at_1000 value: 70.182 - type: recall_at_3 value: 13.416 - type: recall_at_5 value: 16.672 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia config: default split: test revision: f097057d03ed98220bc7309ddb10b71a54d667d6 metrics: - type: map_at_1 value: 4.672 - type: map_at_10 value: 10.534 - type: map_at_100 value: 14.798 - type: map_at_1000 value: 15.927 - type: map_at_3 value: 7.317 - type: map_at_5 value: 8.726 - type: ndcg_at_1 value: 36.5 - type: ndcg_at_10 value: 26.098 - type: ndcg_at_100 value: 29.215999999999998 - type: ndcg_at_1000 value: 36.254999999999995 - type: ndcg_at_3 value: 29.247 - type: ndcg_at_5 value: 27.692 - type: precision_at_1 value: 47.25 - type: precision_at_10 value: 22.625 - type: precision_at_100 value: 7.042 - type: precision_at_1000 value: 1.6129999999999998 - type: precision_at_3 value: 34.083000000000006 - type: precision_at_5 value: 29.5 - type: recall_at_1 value: 4.672 - type: recall_at_10 value: 15.638 - type: recall_at_100 value: 36.228 - type: recall_at_1000 value: 58.831 - type: recall_at_3 value: 8.578 - type: recall_at_5 value: 11.18 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test revision: 829147f8f75a25f005913200eb5ed41fae320aa1 metrics: - type: accuracy value: 49.919999999999995 - type: f1 value: 45.37973678791632 - task: type: Retrieval dataset: type: fever name: MTEB FEVER config: default split: test revision: 1429cf27e393599b8b359b9b72c666f96b2525f9 metrics: - type: map_at_1 value: 25.801000000000002 - type: map_at_10 value: 33.941 - type: map_at_100 value: 34.73 - type: map_at_1000 value: 34.793 - type: map_at_3 value: 31.705 - type: map_at_5 value: 33.047 - type: ndcg_at_1 value: 27.933000000000003 - type: ndcg_at_10 value: 38.644 - type: ndcg_at_100 value: 42.594 - type: ndcg_at_1000 value: 44.352000000000004 - type: ndcg_at_3 value: 34.199 - type: ndcg_at_5 value: 36.573 - type: precision_at_1 value: 27.933000000000003 - type: precision_at_10 value: 5.603000000000001 - type: precision_at_100 value: 0.773 - type: precision_at_1000 value: 0.094 - type: precision_at_3 value: 14.171 - type: precision_at_5 value: 9.786999999999999 - type: recall_at_1 value: 25.801000000000002 - type: recall_at_10 value: 50.876 - type: recall_at_100 value: 69.253 - type: recall_at_1000 value: 82.907 - type: recall_at_3 value: 38.879000000000005 - type: recall_at_5 value: 44.651999999999994 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 config: default split: test revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be metrics: - type: map_at_1 value: 9.142 - type: map_at_10 value: 13.841999999999999 - type: map_at_100 value: 14.960999999999999 - type: map_at_1000 value: 15.187000000000001 - type: map_at_3 value: 11.966000000000001 - type: map_at_5 value: 12.921 - type: ndcg_at_1 value: 18.364 - type: ndcg_at_10 value: 18.590999999999998 - type: ndcg_at_100 value: 24.153 - type: ndcg_at_1000 value: 29.104000000000003 - type: ndcg_at_3 value: 16.323 - type: ndcg_at_5 value: 17.000999999999998 - type: precision_at_1 value: 18.364 - type: precision_at_10 value: 5.216 - type: precision_at_100 value: 1.09 - type: precision_at_1000 value: 0.193 - type: precision_at_3 value: 10.751 - type: precision_at_5 value: 7.932 - type: recall_at_1 value: 9.142 - type: recall_at_10 value: 22.747 - type: recall_at_100 value: 44.585 - type: recall_at_1000 value: 75.481 - type: recall_at_3 value: 14.602 - type: recall_at_5 value: 17.957 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA config: default split: test revision: 766870b35a1b9ca65e67a0d1913899973551fc6c metrics: - type: map_at_1 value: 18.677 - type: map_at_10 value: 26.616 - type: map_at_100 value: 27.605 - type: map_at_1000 value: 27.711999999999996 - type: map_at_3 value: 24.396 - type: map_at_5 value: 25.627 - type: ndcg_at_1 value: 37.352999999999994 - type: ndcg_at_10 value: 33.995 - type: ndcg_at_100 value: 38.423 - type: ndcg_at_1000 value: 40.947 - type: ndcg_at_3 value: 29.885 - type: ndcg_at_5 value: 31.874999999999996 - type: precision_at_1 value: 37.352999999999994 - type: precision_at_10 value: 7.539999999999999 - type: precision_at_100 value: 1.107 - type: precision_at_1000 value: 0.145 - type: precision_at_3 value: 18.938 - type: precision_at_5 value: 12.943 - type: recall_at_1 value: 18.677 - type: recall_at_10 value: 37.698 - type: recall_at_100 value: 55.354000000000006 - type: recall_at_1000 value: 72.255 - type: recall_at_3 value: 28.406 - type: recall_at_5 value: 32.357 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4 metrics: - type: accuracy value: 74.3292 - type: ap value: 68.30186110189658 - type: f1 value: 74.20709636944783 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO config: default split: validation revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849 metrics: - type: map_at_1 value: 6.889000000000001 - type: map_at_10 value: 12.321 - type: map_at_100 value: 13.416 - type: map_at_1000 value: 13.525 - type: map_at_3 value: 10.205 - type: map_at_5 value: 11.342 - type: ndcg_at_1 value: 7.092 - type: ndcg_at_10 value: 15.827 - type: ndcg_at_100 value: 21.72 - type: ndcg_at_1000 value: 24.836 - type: ndcg_at_3 value: 11.393 - type: ndcg_at_5 value: 13.462 - type: precision_at_1 value: 7.092 - type: precision_at_10 value: 2.7969999999999997 - type: precision_at_100 value: 0.583 - type: precision_at_1000 value: 0.08499999999999999 - type: precision_at_3 value: 5.019 - type: precision_at_5 value: 4.06 - type: recall_at_1 value: 6.889000000000001 - type: recall_at_10 value: 26.791999999999998 - type: recall_at_100 value: 55.371 - type: recall_at_1000 value: 80.12899999999999 - type: recall_at_3 value: 14.573 - type: recall_at_5 value: 19.557 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 89.6374829001368 - type: f1 value: 89.20878379358307 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (de) config: de split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 84.54212454212454 - type: f1 value: 82.81080100037023 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (es) config: es split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 86.46430953969313 - type: f1 value: 86.00019824223267 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) config: fr split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 81.31850923896022 - type: f1 value: 81.07860454762863 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (hi) config: hi split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 58.23234134098243 - type: f1 value: 56.63845098081841 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (th) config: th split: test revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3 metrics: - type: accuracy value: 72.28571428571429 - type: f1 value: 70.95796714592039 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 70.68171454628363 - type: f1 value: 52.57188062729139 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (de) config: de split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 60.521273598196665 - type: f1 value: 42.70492970339204 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (es) config: es split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 64.32288192128087 - type: f1 value: 45.97360620220273 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) config: fr split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 58.67209520826808 - type: f1 value: 42.82844991304579 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (hi) config: hi split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 41.95769092864826 - type: f1 value: 28.914127631431263 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (th) config: th split: test revision: 6299947a7777084cc2d4b64235bf7190381ce755 metrics: - type: accuracy value: 55.28390596745027 - type: f1 value: 38.33899250561289 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea metrics: - type: accuracy value: 70.00336247478144 - type: f1 value: 68.72041942191649 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 75.0268997982515 - type: f1 value: 75.29844481506652 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test revision: dcefc037ef84348e49b0d29109e891c01067226b metrics: - type: v_measure value: 30.327566856300813 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc metrics: - type: v_measure value: 28.01650210863619 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 metrics: - type: map value: 31.11041256752524 - type: mrr value: 32.14172939750204 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus config: default split: test revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610 metrics: - type: map_at_1 value: 3.527 - type: map_at_10 value: 9.283 - type: map_at_100 value: 11.995000000000001 - type: map_at_1000 value: 13.33 - type: map_at_3 value: 6.223 - type: map_at_5 value: 7.68 - type: ndcg_at_1 value: 36.223 - type: ndcg_at_10 value: 28.255999999999997 - type: ndcg_at_100 value: 26.355 - type: ndcg_at_1000 value: 35.536 - type: ndcg_at_3 value: 31.962000000000003 - type: ndcg_at_5 value: 30.61 - type: precision_at_1 value: 37.771 - type: precision_at_10 value: 21.889 - type: precision_at_100 value: 7.1080000000000005 - type: precision_at_1000 value: 1.989 - type: precision_at_3 value: 30.857 - type: precision_at_5 value: 27.307 - type: recall_at_1 value: 3.527 - type: recall_at_10 value: 14.015 - type: recall_at_100 value: 28.402 - type: recall_at_1000 value: 59.795 - type: recall_at_3 value: 7.5969999999999995 - type: recall_at_5 value: 10.641 - task: type: Retrieval dataset: type: nq name: MTEB NQ config: default split: test revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c metrics: - type: map_at_1 value: 11.631 - type: map_at_10 value: 19.532 - type: map_at_100 value: 20.821 - type: map_at_1000 value: 20.910999999999998 - type: map_at_3 value: 16.597 - type: map_at_5 value: 18.197 - type: ndcg_at_1 value: 13.413 - type: ndcg_at_10 value: 24.628 - type: ndcg_at_100 value: 30.883 - type: ndcg_at_1000 value: 33.216 - type: ndcg_at_3 value: 18.697 - type: ndcg_at_5 value: 21.501 - type: precision_at_1 value: 13.413 - type: precision_at_10 value: 4.571 - type: precision_at_100 value: 0.812 - type: precision_at_1000 value: 0.10300000000000001 - type: precision_at_3 value: 8.845 - type: precision_at_5 value: 6.889000000000001 - type: recall_at_1 value: 11.631 - type: recall_at_10 value: 38.429 - type: recall_at_100 value: 67.009 - type: recall_at_1000 value: 84.796 - type: recall_at_3 value: 22.74 - type: recall_at_5 value: 29.266 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval config: default split: test revision: 6205996560df11e3a3da9ab4f926788fc30a7db4 metrics: - type: map_at_1 value: 66.64 - type: map_at_10 value: 80.394 - type: map_at_100 value: 81.099 - type: map_at_1000 value: 81.122 - type: map_at_3 value: 77.289 - type: map_at_5 value: 79.25999999999999 - type: ndcg_at_1 value: 76.85 - type: ndcg_at_10 value: 84.68 - type: ndcg_at_100 value: 86.311 - type: ndcg_at_1000 value: 86.49900000000001 - type: ndcg_at_3 value: 81.295 - type: ndcg_at_5 value: 83.199 - type: precision_at_1 value: 76.85 - type: precision_at_10 value: 12.928999999999998 - type: precision_at_100 value: 1.51 - type: precision_at_1000 value: 0.156 - type: precision_at_3 value: 35.557 - type: precision_at_5 value: 23.576 - type: recall_at_1 value: 66.64 - type: recall_at_10 value: 93.059 - type: recall_at_100 value: 98.922 - type: recall_at_1000 value: 99.883 - type: recall_at_3 value: 83.49499999999999 - type: recall_at_5 value: 88.729 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test revision: b2805658ae38990172679479369a78b86de8c390 metrics: - type: v_measure value: 42.17131361041068 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 metrics: - type: v_measure value: 48.01815621479994 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS config: default split: test revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5 metrics: - type: map_at_1 value: 3.198 - type: map_at_10 value: 7.550999999999999 - type: map_at_100 value: 9.232 - type: map_at_1000 value: 9.51 - type: map_at_3 value: 5.2940000000000005 - type: map_at_5 value: 6.343999999999999 - type: ndcg_at_1 value: 15.8 - type: ndcg_at_10 value: 13.553999999999998 - type: ndcg_at_100 value: 20.776 - type: ndcg_at_1000 value: 26.204 - type: ndcg_at_3 value: 12.306000000000001 - type: ndcg_at_5 value: 10.952 - type: precision_at_1 value: 15.8 - type: precision_at_10 value: 7.180000000000001 - type: precision_at_100 value: 1.762 - type: precision_at_1000 value: 0.307 - type: precision_at_3 value: 11.333 - type: precision_at_5 value: 9.62 - type: recall_at_1 value: 3.198 - type: recall_at_10 value: 14.575 - type: recall_at_100 value: 35.758 - type: recall_at_1000 value: 62.317 - type: recall_at_3 value: 6.922000000000001 - type: recall_at_5 value: 9.767000000000001 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test revision: 20a6d6f312dd54037fe07a32d58e5e168867909d metrics: - type: cos_sim_pearson value: 84.5217161312271 - type: cos_sim_spearman value: 79.58562467776268 - type: euclidean_pearson value: 76.69364353942403 - type: euclidean_spearman value: 74.68959282070473 - type: manhattan_pearson value: 76.81159265133732 - type: manhattan_spearman value: 74.7519444048176 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f metrics: - type: cos_sim_pearson value: 83.70403706922605 - type: cos_sim_spearman value: 74.28502198729447 - type: euclidean_pearson value: 83.32719404608066 - type: euclidean_spearman value: 75.92189433460788 - type: manhattan_pearson value: 83.35841543005293 - type: manhattan_spearman value: 75.94458615451978 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9 metrics: - type: cos_sim_pearson value: 84.94127878986795 - type: cos_sim_spearman value: 85.35148434923192 - type: euclidean_pearson value: 81.71127467071571 - type: euclidean_spearman value: 82.88240481546771 - type: manhattan_pearson value: 81.72826221967252 - type: manhattan_spearman value: 82.90725064625128 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b metrics: - type: cos_sim_pearson value: 83.1474704168523 - type: cos_sim_spearman value: 79.20612995350827 - type: euclidean_pearson value: 78.85993329596555 - type: euclidean_spearman value: 78.91956572744715 - type: manhattan_pearson value: 78.89999720522347 - type: manhattan_spearman value: 78.93956842550107 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6 metrics: - type: cos_sim_pearson value: 84.81255514055894 - type: cos_sim_spearman value: 85.5217140762934 - type: euclidean_pearson value: 82.15024353784499 - type: euclidean_spearman value: 83.04155334389833 - type: manhattan_pearson value: 82.18598945053624 - type: manhattan_spearman value: 83.07248357693301 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd metrics: - type: cos_sim_pearson value: 80.63248465157822 - type: cos_sim_spearman value: 82.53853238521991 - type: euclidean_pearson value: 78.33936863828221 - type: euclidean_spearman value: 79.16305579487414 - type: manhattan_pearson value: 78.3888359870894 - type: manhattan_spearman value: 79.18504473136467 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0 metrics: - type: cos_sim_pearson value: 90.09066290639687 - type: cos_sim_spearman value: 90.43893699357069 - type: euclidean_pearson value: 82.39520777222396 - type: euclidean_spearman value: 81.23948185395952 - type: manhattan_pearson value: 82.35529784653383 - type: manhattan_spearman value: 81.12681522483975 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906 metrics: - type: cos_sim_pearson value: 63.52752323046846 - type: cos_sim_spearman value: 63.19719780439462 - type: euclidean_pearson value: 58.29085490641428 - type: euclidean_spearman value: 58.975178656335046 - type: manhattan_pearson value: 58.183542772416985 - type: manhattan_spearman value: 59.190630462178994 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test revision: 8913289635987208e6e7c72789e4be2fe94b6abd metrics: - type: cos_sim_pearson value: 85.45100366635687 - type: cos_sim_spearman value: 85.66816193002651 - type: euclidean_pearson value: 81.87976731329091 - type: euclidean_spearman value: 82.01382867690964 - type: manhattan_pearson value: 81.88260155706726 - type: manhattan_spearman value: 82.05258597906492 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test revision: 56a6d0140cf6356659e2a7c1413286a774468d44 metrics: - type: map value: 77.53549990038017 - type: mrr value: 93.37474163454556 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact config: default split: test revision: a75ae049398addde9b70f6b268875f5cbce99089 metrics: - type: map_at_1 value: 31.167 - type: map_at_10 value: 40.778 - type: map_at_100 value: 42.063 - type: map_at_1000 value: 42.103 - type: map_at_3 value: 37.12 - type: map_at_5 value: 39.205 - type: ndcg_at_1 value: 33.667 - type: ndcg_at_10 value: 46.662 - type: ndcg_at_100 value: 51.995999999999995 - type: ndcg_at_1000 value: 53.254999999999995 - type: ndcg_at_3 value: 39.397999999999996 - type: ndcg_at_5 value: 42.934 - type: precision_at_1 value: 33.667 - type: precision_at_10 value: 7.1 - type: precision_at_100 value: 0.993 - type: precision_at_1000 value: 0.11 - type: precision_at_3 value: 16.111 - type: precision_at_5 value: 11.600000000000001 - type: recall_at_1 value: 31.167 - type: recall_at_10 value: 63.744 - type: recall_at_100 value: 87.156 - type: recall_at_1000 value: 97.556 - type: recall_at_3 value: 44.0 - type: recall_at_5 value: 52.556000000000004 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea metrics: - type: cos_sim_accuracy value: 99.55148514851486 - type: cos_sim_ap value: 80.535236573428 - type: cos_sim_f1 value: 75.01331912626532 - type: cos_sim_precision value: 80.27366020524515 - type: cos_sim_recall value: 70.39999999999999 - type: dot_accuracy value: 99.04851485148515 - type: dot_ap value: 28.505358821499726 - type: dot_f1 value: 36.36363636363637 - type: dot_precision value: 37.160751565762006 - type: dot_recall value: 35.6 - type: euclidean_accuracy value: 99.4990099009901 - type: euclidean_ap value: 74.95819047075476 - type: euclidean_f1 value: 71.15489874110564 - type: euclidean_precision value: 78.59733978234583 - type: euclidean_recall value: 65.0 - type: manhattan_accuracy value: 99.50198019801981 - type: manhattan_ap value: 75.02070096015086 - type: manhattan_f1 value: 71.20535714285712 - type: manhattan_precision value: 80.55555555555556 - type: manhattan_recall value: 63.800000000000004 - type: max_accuracy value: 99.55148514851486 - type: max_ap value: 80.535236573428 - type: max_f1 value: 75.01331912626532 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235 metrics: - type: v_measure value: 54.13314692311623 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0 metrics: - type: v_measure value: 31.115181648287145 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9 metrics: - type: map value: 44.771112666694336 - type: mrr value: 45.30415764790765 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122 metrics: - type: cos_sim_pearson value: 30.849429597669374 - type: cos_sim_spearman value: 30.384175038360194 - type: dot_pearson value: 29.030383429536823 - type: dot_spearman value: 28.03273624951732 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID config: default split: test revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217 metrics: - type: map_at_1 value: 0.19499999999999998 - type: map_at_10 value: 1.0959999999999999 - type: map_at_100 value: 5.726 - type: map_at_1000 value: 13.611999999999998 - type: map_at_3 value: 0.45399999999999996 - type: map_at_5 value: 0.67 - type: ndcg_at_1 value: 71.0 - type: ndcg_at_10 value: 55.352999999999994 - type: ndcg_at_100 value: 40.797 - type: ndcg_at_1000 value: 35.955999999999996 - type: ndcg_at_3 value: 63.263000000000005 - type: ndcg_at_5 value: 60.14000000000001 - type: precision_at_1 value: 78.0 - type: precision_at_10 value: 56.99999999999999 - type: precision_at_100 value: 41.199999999999996 - type: precision_at_1000 value: 16.154 - type: precision_at_3 value: 66.667 - type: precision_at_5 value: 62.8 - type: recall_at_1 value: 0.19499999999999998 - type: recall_at_10 value: 1.3639999999999999 - type: recall_at_100 value: 9.317 - type: recall_at_1000 value: 33.629999999999995 - type: recall_at_3 value: 0.49300000000000005 - type: recall_at_5 value: 0.756 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 config: default split: test revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b metrics: - type: map_at_1 value: 1.335 - type: map_at_10 value: 6.293 - type: map_at_100 value: 10.928 - type: map_at_1000 value: 12.359 - type: map_at_3 value: 3.472 - type: map_at_5 value: 4.935 - type: ndcg_at_1 value: 19.387999999999998 - type: ndcg_at_10 value: 16.178 - type: ndcg_at_100 value: 28.149 - type: ndcg_at_1000 value: 39.845000000000006 - type: ndcg_at_3 value: 19.171 - type: ndcg_at_5 value: 17.864 - type: precision_at_1 value: 20.408 - type: precision_at_10 value: 14.49 - type: precision_at_100 value: 6.306000000000001 - type: precision_at_1000 value: 1.3860000000000001 - type: precision_at_3 value: 21.088 - type: precision_at_5 value: 18.367 - type: recall_at_1 value: 1.335 - type: recall_at_10 value: 10.825999999999999 - type: recall_at_100 value: 39.251000000000005 - type: recall_at_1000 value: 74.952 - type: recall_at_3 value: 4.9110000000000005 - type: recall_at_5 value: 7.312 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de metrics: - type: accuracy value: 69.93339999999999 - type: ap value: 13.87476602492533 - type: f1 value: 53.867357615848555 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test revision: 62146448f05be9e52a36b8ee9936447ea787eede metrics: - type: accuracy value: 62.43916242218449 - type: f1 value: 62.870386304954685 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4 metrics: - type: v_measure value: 37.202082549859796 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 metrics: - type: cos_sim_accuracy value: 83.65023544137807 - type: cos_sim_ap value: 65.99787692764193 - type: cos_sim_f1 value: 62.10650887573965 - type: cos_sim_precision value: 56.30901287553648 - type: cos_sim_recall value: 69.23482849604221 - type: dot_accuracy value: 79.10830303391549 - type: dot_ap value: 48.80109642320246 - type: dot_f1 value: 51.418744625967314 - type: dot_precision value: 40.30253107683091 - type: dot_recall value: 71.00263852242745 - type: euclidean_accuracy value: 82.45812719794957 - type: euclidean_ap value: 60.09969493259607 - type: euclidean_f1 value: 57.658573789246226 - type: euclidean_precision value: 55.62913907284768 - type: euclidean_recall value: 59.84168865435356 - type: manhattan_accuracy value: 82.46408773916671 - type: manhattan_ap value: 60.116199786815116 - type: manhattan_f1 value: 57.683903860160235 - type: manhattan_precision value: 53.41726618705036 - type: manhattan_recall value: 62.69129287598945 - type: max_accuracy value: 83.65023544137807 - type: max_ap value: 65.99787692764193 - type: max_f1 value: 62.10650887573965 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf metrics: - type: cos_sim_accuracy value: 88.34943920518494 - type: cos_sim_ap value: 84.5428891020442 - type: cos_sim_f1 value: 77.09709933923172 - type: cos_sim_precision value: 74.83150952967607 - type: cos_sim_recall value: 79.50415768401602 - type: dot_accuracy value: 84.53448208949432 - type: dot_ap value: 73.96328242371995 - type: dot_f1 value: 70.00553786515299 - type: dot_precision value: 63.58777665995976 - type: dot_recall value: 77.86418232214352 - type: euclidean_accuracy value: 86.87662514068381 - type: euclidean_ap value: 81.45499631520235 - type: euclidean_f1 value: 73.46567109816063 - type: euclidean_precision value: 69.71037533697381 - type: euclidean_recall value: 77.6485987064983 - type: manhattan_accuracy value: 86.88244654014825 - type: manhattan_ap value: 81.47180273946366 - type: manhattan_f1 value: 73.44624393136418 - type: manhattan_precision value: 70.80385852090032 - type: manhattan_recall value: 76.29350169387126 - type: max_accuracy value: 88.34943920518494 - type: max_ap value: 84.5428891020442 - type: max_f1 value: 77.09709933923172 --- # SGPT-5.8B-weightedmean-msmarco-specb-bitfit ## Usage For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt ## Evaluation Results For eval results, refer to our paper: https://arxiv.org/abs/2202.08904 ## Training The model was trained with the parameters: **DataLoader**: `torch.utils.data.dataloader.DataLoader` of length 249592 with parameters: ``` {'batch_size': 2, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} ``` **Loss**: `sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters: ``` {'scale': 20.0, 'similarity_fct': 'cos_sim'} ``` Parameters of the fit()-Method: ``` { "epochs": 10, "evaluation_steps": 0, "evaluator": "NoneType", "max_grad_norm": 1, "optimizer_class": "", "optimizer_params": { "lr": 5e-05 }, "scheduler": "WarmupLinear", "steps_per_epoch": null, "warmup_steps": 1000, "weight_decay": 0.01 } ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTJModel (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False}) ) ``` ## Citing & Authors ```bibtex @article{muennighoff2022sgpt, title={SGPT: GPT Sentence Embeddings for Semantic Search}, author={Muennighoff, Niklas}, journal={arXiv preprint arXiv:2202.08904}, year={2022} } ```