|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- mteb |
|
model-index: |
|
- name: cai-stellaris-text-embeddings |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 64.86567164179104 |
|
- type: ap |
|
value: 28.30760041689409 |
|
- type: f1 |
|
value: 59.08589995918376 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 65.168625 |
|
- type: ap |
|
value: 60.131922961382166 |
|
- type: f1 |
|
value: 65.02463910192814 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 31.016 |
|
- type: f1 |
|
value: 30.501226228002924 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.609 |
|
- type: map_at_10 |
|
value: 38.793 |
|
- type: map_at_100 |
|
value: 40.074 |
|
- type: map_at_1000 |
|
value: 40.083 |
|
- type: map_at_3 |
|
value: 33.736 |
|
- type: map_at_5 |
|
value: 36.642 |
|
- type: mrr_at_1 |
|
value: 25.533 |
|
- type: mrr_at_10 |
|
value: 39.129999999999995 |
|
- type: mrr_at_100 |
|
value: 40.411 |
|
- type: mrr_at_1000 |
|
value: 40.42 |
|
- type: mrr_at_3 |
|
value: 34.033 |
|
- type: mrr_at_5 |
|
value: 36.956 |
|
- type: ndcg_at_1 |
|
value: 24.609 |
|
- type: ndcg_at_10 |
|
value: 47.288000000000004 |
|
- type: ndcg_at_100 |
|
value: 52.654999999999994 |
|
- type: ndcg_at_1000 |
|
value: 52.88699999999999 |
|
- type: ndcg_at_3 |
|
value: 36.86 |
|
- type: ndcg_at_5 |
|
value: 42.085 |
|
- type: precision_at_1 |
|
value: 24.609 |
|
- type: precision_at_10 |
|
value: 7.468 |
|
- type: precision_at_100 |
|
value: 0.979 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 15.315000000000001 |
|
- type: precision_at_5 |
|
value: 11.721 |
|
- type: recall_at_1 |
|
value: 24.609 |
|
- type: recall_at_10 |
|
value: 74.68 |
|
- type: recall_at_100 |
|
value: 97.866 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 45.946 |
|
- type: recall_at_5 |
|
value: 58.606 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 42.014046191286525 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 31.406159641263052 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 60.35266033223575 |
|
- type: mrr |
|
value: 72.66796376907179 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 74.12337662337661 |
|
- type: f1 |
|
value: 73.12122145084057 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 34.72513663347855 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 29.280150859689826 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.787 |
|
- type: map_at_10 |
|
value: 30.409000000000002 |
|
- type: map_at_100 |
|
value: 31.947 |
|
- type: map_at_1000 |
|
value: 32.09 |
|
- type: map_at_3 |
|
value: 27.214 |
|
- type: map_at_5 |
|
value: 28.810999999999996 |
|
- type: mrr_at_1 |
|
value: 27.039 |
|
- type: mrr_at_10 |
|
value: 35.581 |
|
- type: mrr_at_100 |
|
value: 36.584 |
|
- type: mrr_at_1000 |
|
value: 36.645 |
|
- type: mrr_at_3 |
|
value: 32.713 |
|
- type: mrr_at_5 |
|
value: 34.272999999999996 |
|
- type: ndcg_at_1 |
|
value: 27.039 |
|
- type: ndcg_at_10 |
|
value: 36.157000000000004 |
|
- type: ndcg_at_100 |
|
value: 42.598 |
|
- type: ndcg_at_1000 |
|
value: 45.207 |
|
- type: ndcg_at_3 |
|
value: 30.907 |
|
- type: ndcg_at_5 |
|
value: 33.068 |
|
- type: precision_at_1 |
|
value: 27.039 |
|
- type: precision_at_10 |
|
value: 7.295999999999999 |
|
- type: precision_at_100 |
|
value: 1.303 |
|
- type: precision_at_1000 |
|
value: 0.186 |
|
- type: precision_at_3 |
|
value: 14.926 |
|
- type: precision_at_5 |
|
value: 11.044 |
|
- type: recall_at_1 |
|
value: 21.787 |
|
- type: recall_at_10 |
|
value: 47.693999999999996 |
|
- type: recall_at_100 |
|
value: 75.848 |
|
- type: recall_at_1000 |
|
value: 92.713 |
|
- type: recall_at_3 |
|
value: 32.92 |
|
- type: recall_at_5 |
|
value: 38.794000000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.560000000000002 |
|
- type: map_at_10 |
|
value: 34.756 |
|
- type: map_at_100 |
|
value: 36.169000000000004 |
|
- type: map_at_1000 |
|
value: 36.298 |
|
- type: map_at_3 |
|
value: 31.592 |
|
- type: map_at_5 |
|
value: 33.426 |
|
- type: mrr_at_1 |
|
value: 31.274 |
|
- type: mrr_at_10 |
|
value: 40.328 |
|
- type: mrr_at_100 |
|
value: 41.125 |
|
- type: mrr_at_1000 |
|
value: 41.171 |
|
- type: mrr_at_3 |
|
value: 37.866 |
|
- type: mrr_at_5 |
|
value: 39.299 |
|
- type: ndcg_at_1 |
|
value: 31.338 |
|
- type: ndcg_at_10 |
|
value: 40.696 |
|
- type: ndcg_at_100 |
|
value: 45.922000000000004 |
|
- type: ndcg_at_1000 |
|
value: 47.982 |
|
- type: ndcg_at_3 |
|
value: 36.116 |
|
- type: ndcg_at_5 |
|
value: 38.324000000000005 |
|
- type: precision_at_1 |
|
value: 31.338 |
|
- type: precision_at_10 |
|
value: 8.083 |
|
- type: precision_at_100 |
|
value: 1.4040000000000001 |
|
- type: precision_at_1000 |
|
value: 0.189 |
|
- type: precision_at_3 |
|
value: 18.089 |
|
- type: precision_at_5 |
|
value: 13.159 |
|
- type: recall_at_1 |
|
value: 24.560000000000002 |
|
- type: recall_at_10 |
|
value: 51.832 |
|
- type: recall_at_100 |
|
value: 74.26899999999999 |
|
- type: recall_at_1000 |
|
value: 87.331 |
|
- type: recall_at_3 |
|
value: 38.086999999999996 |
|
- type: recall_at_5 |
|
value: 44.294 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27.256999999999998 |
|
- type: map_at_10 |
|
value: 38.805 |
|
- type: map_at_100 |
|
value: 40.04 |
|
- type: map_at_1000 |
|
value: 40.117000000000004 |
|
- type: map_at_3 |
|
value: 35.425000000000004 |
|
- type: map_at_5 |
|
value: 37.317 |
|
- type: mrr_at_1 |
|
value: 31.912000000000003 |
|
- type: mrr_at_10 |
|
value: 42.045 |
|
- type: mrr_at_100 |
|
value: 42.956 |
|
- type: mrr_at_1000 |
|
value: 43.004 |
|
- type: mrr_at_3 |
|
value: 39.195 |
|
- type: mrr_at_5 |
|
value: 40.866 |
|
- type: ndcg_at_1 |
|
value: 31.912000000000003 |
|
- type: ndcg_at_10 |
|
value: 44.826 |
|
- type: ndcg_at_100 |
|
value: 49.85 |
|
- type: ndcg_at_1000 |
|
value: 51.562 |
|
- type: ndcg_at_3 |
|
value: 38.845 |
|
- type: ndcg_at_5 |
|
value: 41.719 |
|
- type: precision_at_1 |
|
value: 31.912000000000003 |
|
- type: precision_at_10 |
|
value: 7.768 |
|
- type: precision_at_100 |
|
value: 1.115 |
|
- type: precision_at_1000 |
|
value: 0.131 |
|
- type: precision_at_3 |
|
value: 18.015 |
|
- type: precision_at_5 |
|
value: 12.814999999999998 |
|
- type: recall_at_1 |
|
value: 27.256999999999998 |
|
- type: recall_at_10 |
|
value: 59.611999999999995 |
|
- type: recall_at_100 |
|
value: 81.324 |
|
- type: recall_at_1000 |
|
value: 93.801 |
|
- type: recall_at_3 |
|
value: 43.589 |
|
- type: recall_at_5 |
|
value: 50.589 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 15.588 |
|
- type: map_at_10 |
|
value: 22.936999999999998 |
|
- type: map_at_100 |
|
value: 24.015 |
|
- type: map_at_1000 |
|
value: 24.127000000000002 |
|
- type: map_at_3 |
|
value: 20.47 |
|
- type: map_at_5 |
|
value: 21.799 |
|
- type: mrr_at_1 |
|
value: 16.723 |
|
- type: mrr_at_10 |
|
value: 24.448 |
|
- type: mrr_at_100 |
|
value: 25.482 |
|
- type: mrr_at_1000 |
|
value: 25.568999999999996 |
|
- type: mrr_at_3 |
|
value: 21.94 |
|
- type: mrr_at_5 |
|
value: 23.386000000000003 |
|
- type: ndcg_at_1 |
|
value: 16.723 |
|
- type: ndcg_at_10 |
|
value: 27.451999999999998 |
|
- type: ndcg_at_100 |
|
value: 33.182 |
|
- type: ndcg_at_1000 |
|
value: 36.193999999999996 |
|
- type: ndcg_at_3 |
|
value: 22.545 |
|
- type: ndcg_at_5 |
|
value: 24.837 |
|
- type: precision_at_1 |
|
value: 16.723 |
|
- type: precision_at_10 |
|
value: 4.5760000000000005 |
|
- type: precision_at_100 |
|
value: 0.7929999999999999 |
|
- type: precision_at_1000 |
|
value: 0.11 |
|
- type: precision_at_3 |
|
value: 9.944 |
|
- type: precision_at_5 |
|
value: 7.321999999999999 |
|
- type: recall_at_1 |
|
value: 15.588 |
|
- type: recall_at_10 |
|
value: 40.039 |
|
- type: recall_at_100 |
|
value: 67.17699999999999 |
|
- type: recall_at_1000 |
|
value: 90.181 |
|
- type: recall_at_3 |
|
value: 26.663999999999998 |
|
- type: recall_at_5 |
|
value: 32.144 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 12.142999999999999 |
|
- type: map_at_10 |
|
value: 18.355 |
|
- type: map_at_100 |
|
value: 19.611 |
|
- type: map_at_1000 |
|
value: 19.750999999999998 |
|
- type: map_at_3 |
|
value: 16.073999999999998 |
|
- type: map_at_5 |
|
value: 17.187 |
|
- type: mrr_at_1 |
|
value: 15.547 |
|
- type: mrr_at_10 |
|
value: 22.615 |
|
- type: mrr_at_100 |
|
value: 23.671 |
|
- type: mrr_at_1000 |
|
value: 23.759 |
|
- type: mrr_at_3 |
|
value: 20.149 |
|
- type: mrr_at_5 |
|
value: 21.437 |
|
- type: ndcg_at_1 |
|
value: 15.547 |
|
- type: ndcg_at_10 |
|
value: 22.985 |
|
- type: ndcg_at_100 |
|
value: 29.192 |
|
- type: ndcg_at_1000 |
|
value: 32.448 |
|
- type: ndcg_at_3 |
|
value: 18.503 |
|
- type: ndcg_at_5 |
|
value: 20.322000000000003 |
|
- type: precision_at_1 |
|
value: 15.547 |
|
- type: precision_at_10 |
|
value: 4.49 |
|
- type: precision_at_100 |
|
value: 0.8840000000000001 |
|
- type: precision_at_1000 |
|
value: 0.129 |
|
- type: precision_at_3 |
|
value: 8.872 |
|
- type: precision_at_5 |
|
value: 6.741 |
|
- type: recall_at_1 |
|
value: 12.142999999999999 |
|
- type: recall_at_10 |
|
value: 33.271 |
|
- type: recall_at_100 |
|
value: 60.95399999999999 |
|
- type: recall_at_1000 |
|
value: 83.963 |
|
- type: recall_at_3 |
|
value: 20.645 |
|
- type: recall_at_5 |
|
value: 25.34 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.09 |
|
- type: map_at_10 |
|
value: 30.220000000000002 |
|
- type: map_at_100 |
|
value: 31.741999999999997 |
|
- type: map_at_1000 |
|
value: 31.878 |
|
- type: map_at_3 |
|
value: 27.455000000000002 |
|
- type: map_at_5 |
|
value: 28.808 |
|
- type: mrr_at_1 |
|
value: 27.718999999999998 |
|
- type: mrr_at_10 |
|
value: 35.476 |
|
- type: mrr_at_100 |
|
value: 36.53 |
|
- type: mrr_at_1000 |
|
value: 36.602000000000004 |
|
- type: mrr_at_3 |
|
value: 33.157 |
|
- type: mrr_at_5 |
|
value: 34.36 |
|
- type: ndcg_at_1 |
|
value: 27.718999999999998 |
|
- type: ndcg_at_10 |
|
value: 35.547000000000004 |
|
- type: ndcg_at_100 |
|
value: 42.079 |
|
- type: ndcg_at_1000 |
|
value: 44.861000000000004 |
|
- type: ndcg_at_3 |
|
value: 30.932 |
|
- type: ndcg_at_5 |
|
value: 32.748 |
|
- type: precision_at_1 |
|
value: 27.718999999999998 |
|
- type: precision_at_10 |
|
value: 6.795 |
|
- type: precision_at_100 |
|
value: 1.194 |
|
- type: precision_at_1000 |
|
value: 0.163 |
|
- type: precision_at_3 |
|
value: 14.758 |
|
- type: precision_at_5 |
|
value: 10.549 |
|
- type: recall_at_1 |
|
value: 22.09 |
|
- type: recall_at_10 |
|
value: 46.357 |
|
- type: recall_at_100 |
|
value: 74.002 |
|
- type: recall_at_1000 |
|
value: 92.99199999999999 |
|
- type: recall_at_3 |
|
value: 33.138 |
|
- type: recall_at_5 |
|
value: 38.034 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.904 |
|
- type: map_at_10 |
|
value: 25.075999999999997 |
|
- type: map_at_100 |
|
value: 26.400000000000002 |
|
- type: map_at_1000 |
|
value: 26.525 |
|
- type: map_at_3 |
|
value: 22.191 |
|
- type: map_at_5 |
|
value: 23.947 |
|
- type: mrr_at_1 |
|
value: 21.461 |
|
- type: mrr_at_10 |
|
value: 29.614 |
|
- type: mrr_at_100 |
|
value: 30.602 |
|
- type: mrr_at_1000 |
|
value: 30.677 |
|
- type: mrr_at_3 |
|
value: 27.017000000000003 |
|
- type: mrr_at_5 |
|
value: 28.626 |
|
- type: ndcg_at_1 |
|
value: 21.461 |
|
- type: ndcg_at_10 |
|
value: 30.304 |
|
- type: ndcg_at_100 |
|
value: 36.521 |
|
- type: ndcg_at_1000 |
|
value: 39.366 |
|
- type: ndcg_at_3 |
|
value: 25.267 |
|
- type: ndcg_at_5 |
|
value: 27.918 |
|
- type: precision_at_1 |
|
value: 21.461 |
|
- type: precision_at_10 |
|
value: 5.868 |
|
- type: precision_at_100 |
|
value: 1.072 |
|
- type: precision_at_1000 |
|
value: 0.151 |
|
- type: precision_at_3 |
|
value: 12.291 |
|
- type: precision_at_5 |
|
value: 9.429 |
|
- type: recall_at_1 |
|
value: 16.904 |
|
- type: recall_at_10 |
|
value: 41.521 |
|
- type: recall_at_100 |
|
value: 68.919 |
|
- type: recall_at_1000 |
|
value: 88.852 |
|
- type: recall_at_3 |
|
value: 27.733999999999998 |
|
- type: recall_at_5 |
|
value: 34.439 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 18.327916666666667 |
|
- type: map_at_10 |
|
value: 26.068 |
|
- type: map_at_100 |
|
value: 27.358833333333333 |
|
- type: map_at_1000 |
|
value: 27.491583333333335 |
|
- type: map_at_3 |
|
value: 23.45508333333333 |
|
- type: map_at_5 |
|
value: 24.857916666666664 |
|
- type: mrr_at_1 |
|
value: 22.05066666666667 |
|
- type: mrr_at_10 |
|
value: 29.805083333333332 |
|
- type: mrr_at_100 |
|
value: 30.80283333333333 |
|
- type: mrr_at_1000 |
|
value: 30.876166666666666 |
|
- type: mrr_at_3 |
|
value: 27.381083333333333 |
|
- type: mrr_at_5 |
|
value: 28.72441666666667 |
|
- type: ndcg_at_1 |
|
value: 22.056000000000004 |
|
- type: ndcg_at_10 |
|
value: 31.029416666666666 |
|
- type: ndcg_at_100 |
|
value: 36.90174999999999 |
|
- type: ndcg_at_1000 |
|
value: 39.716249999999995 |
|
- type: ndcg_at_3 |
|
value: 26.35533333333333 |
|
- type: ndcg_at_5 |
|
value: 28.471500000000006 |
|
- type: precision_at_1 |
|
value: 22.056000000000004 |
|
- type: precision_at_10 |
|
value: 5.7645833333333325 |
|
- type: precision_at_100 |
|
value: 1.0406666666666666 |
|
- type: precision_at_1000 |
|
value: 0.14850000000000002 |
|
- type: precision_at_3 |
|
value: 12.391416666666666 |
|
- type: precision_at_5 |
|
value: 9.112499999999999 |
|
- type: recall_at_1 |
|
value: 18.327916666666667 |
|
- type: recall_at_10 |
|
value: 42.15083333333333 |
|
- type: recall_at_100 |
|
value: 68.38666666666666 |
|
- type: recall_at_1000 |
|
value: 88.24183333333333 |
|
- type: recall_at_3 |
|
value: 29.094416666666667 |
|
- type: recall_at_5 |
|
value: 34.48716666666666 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 15.009 |
|
- type: map_at_10 |
|
value: 21.251 |
|
- type: map_at_100 |
|
value: 22.337 |
|
- type: map_at_1000 |
|
value: 22.455 |
|
- type: map_at_3 |
|
value: 19.241 |
|
- type: map_at_5 |
|
value: 20.381 |
|
- type: mrr_at_1 |
|
value: 17.638 |
|
- type: mrr_at_10 |
|
value: 24.184 |
|
- type: mrr_at_100 |
|
value: 25.156 |
|
- type: mrr_at_1000 |
|
value: 25.239 |
|
- type: mrr_at_3 |
|
value: 22.29 |
|
- type: mrr_at_5 |
|
value: 23.363999999999997 |
|
- type: ndcg_at_1 |
|
value: 17.638 |
|
- type: ndcg_at_10 |
|
value: 25.269000000000002 |
|
- type: ndcg_at_100 |
|
value: 30.781999999999996 |
|
- type: ndcg_at_1000 |
|
value: 33.757 |
|
- type: ndcg_at_3 |
|
value: 21.457 |
|
- type: ndcg_at_5 |
|
value: 23.293 |
|
- type: precision_at_1 |
|
value: 17.638 |
|
- type: precision_at_10 |
|
value: 4.294 |
|
- type: precision_at_100 |
|
value: 0.771 |
|
- type: precision_at_1000 |
|
value: 0.11100000000000002 |
|
- type: precision_at_3 |
|
value: 9.815999999999999 |
|
- type: precision_at_5 |
|
value: 7.086 |
|
- type: recall_at_1 |
|
value: 15.009 |
|
- type: recall_at_10 |
|
value: 35.014 |
|
- type: recall_at_100 |
|
value: 60.45399999999999 |
|
- type: recall_at_1000 |
|
value: 82.416 |
|
- type: recall_at_3 |
|
value: 24.131 |
|
- type: recall_at_5 |
|
value: 28.846 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 12.518 |
|
- type: map_at_10 |
|
value: 18.226 |
|
- type: map_at_100 |
|
value: 19.355 |
|
- type: map_at_1000 |
|
value: 19.496 |
|
- type: map_at_3 |
|
value: 16.243 |
|
- type: map_at_5 |
|
value: 17.288999999999998 |
|
- type: mrr_at_1 |
|
value: 15.382000000000001 |
|
- type: mrr_at_10 |
|
value: 21.559 |
|
- type: mrr_at_100 |
|
value: 22.587 |
|
- type: mrr_at_1000 |
|
value: 22.677 |
|
- type: mrr_at_3 |
|
value: 19.597 |
|
- type: mrr_at_5 |
|
value: 20.585 |
|
- type: ndcg_at_1 |
|
value: 15.382000000000001 |
|
- type: ndcg_at_10 |
|
value: 22.198 |
|
- type: ndcg_at_100 |
|
value: 27.860000000000003 |
|
- type: ndcg_at_1000 |
|
value: 31.302999999999997 |
|
- type: ndcg_at_3 |
|
value: 18.541 |
|
- type: ndcg_at_5 |
|
value: 20.089000000000002 |
|
- type: precision_at_1 |
|
value: 15.382000000000001 |
|
- type: precision_at_10 |
|
value: 4.178 |
|
- type: precision_at_100 |
|
value: 0.8380000000000001 |
|
- type: precision_at_1000 |
|
value: 0.132 |
|
- type: precision_at_3 |
|
value: 8.866999999999999 |
|
- type: precision_at_5 |
|
value: 6.476 |
|
- type: recall_at_1 |
|
value: 12.518 |
|
- type: recall_at_10 |
|
value: 31.036 |
|
- type: recall_at_100 |
|
value: 56.727000000000004 |
|
- type: recall_at_1000 |
|
value: 81.66799999999999 |
|
- type: recall_at_3 |
|
value: 20.610999999999997 |
|
- type: recall_at_5 |
|
value: 24.744 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 18.357 |
|
- type: map_at_10 |
|
value: 25.384 |
|
- type: map_at_100 |
|
value: 26.640000000000004 |
|
- type: map_at_1000 |
|
value: 26.762999999999998 |
|
- type: map_at_3 |
|
value: 22.863 |
|
- type: map_at_5 |
|
value: 24.197 |
|
- type: mrr_at_1 |
|
value: 21.735 |
|
- type: mrr_at_10 |
|
value: 29.069 |
|
- type: mrr_at_100 |
|
value: 30.119 |
|
- type: mrr_at_1000 |
|
value: 30.194 |
|
- type: mrr_at_3 |
|
value: 26.663999999999998 |
|
- type: mrr_at_5 |
|
value: 27.904 |
|
- type: ndcg_at_1 |
|
value: 21.735 |
|
- type: ndcg_at_10 |
|
value: 30.153999999999996 |
|
- type: ndcg_at_100 |
|
value: 36.262 |
|
- type: ndcg_at_1000 |
|
value: 39.206 |
|
- type: ndcg_at_3 |
|
value: 25.365 |
|
- type: ndcg_at_5 |
|
value: 27.403 |
|
- type: precision_at_1 |
|
value: 21.735 |
|
- type: precision_at_10 |
|
value: 5.354 |
|
- type: precision_at_100 |
|
value: 0.958 |
|
- type: precision_at_1000 |
|
value: 0.134 |
|
- type: precision_at_3 |
|
value: 11.567 |
|
- type: precision_at_5 |
|
value: 8.469999999999999 |
|
- type: recall_at_1 |
|
value: 18.357 |
|
- type: recall_at_10 |
|
value: 41.205000000000005 |
|
- type: recall_at_100 |
|
value: 68.30000000000001 |
|
- type: recall_at_1000 |
|
value: 89.294 |
|
- type: recall_at_3 |
|
value: 27.969 |
|
- type: recall_at_5 |
|
value: 32.989000000000004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 18.226 |
|
- type: map_at_10 |
|
value: 25.766 |
|
- type: map_at_100 |
|
value: 27.345000000000002 |
|
- type: map_at_1000 |
|
value: 27.575 |
|
- type: map_at_3 |
|
value: 22.945999999999998 |
|
- type: map_at_5 |
|
value: 24.383 |
|
- type: mrr_at_1 |
|
value: 21.542 |
|
- type: mrr_at_10 |
|
value: 29.448 |
|
- type: mrr_at_100 |
|
value: 30.509999999999998 |
|
- type: mrr_at_1000 |
|
value: 30.575000000000003 |
|
- type: mrr_at_3 |
|
value: 26.482 |
|
- type: mrr_at_5 |
|
value: 28.072999999999997 |
|
- type: ndcg_at_1 |
|
value: 21.542 |
|
- type: ndcg_at_10 |
|
value: 31.392999999999997 |
|
- type: ndcg_at_100 |
|
value: 37.589 |
|
- type: ndcg_at_1000 |
|
value: 40.717 |
|
- type: ndcg_at_3 |
|
value: 26.179000000000002 |
|
- type: ndcg_at_5 |
|
value: 28.557 |
|
- type: precision_at_1 |
|
value: 21.542 |
|
- type: precision_at_10 |
|
value: 6.462 |
|
- type: precision_at_100 |
|
value: 1.415 |
|
- type: precision_at_1000 |
|
value: 0.234 |
|
- type: precision_at_3 |
|
value: 12.187000000000001 |
|
- type: precision_at_5 |
|
value: 9.605 |
|
- type: recall_at_1 |
|
value: 18.226 |
|
- type: recall_at_10 |
|
value: 42.853 |
|
- type: recall_at_100 |
|
value: 70.97200000000001 |
|
- type: recall_at_1000 |
|
value: 91.662 |
|
- type: recall_at_3 |
|
value: 28.555999999999997 |
|
- type: recall_at_5 |
|
value: 34.203 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 15.495999999999999 |
|
- type: map_at_10 |
|
value: 21.631 |
|
- type: map_at_100 |
|
value: 22.705000000000002 |
|
- type: map_at_1000 |
|
value: 22.823999999999998 |
|
- type: map_at_3 |
|
value: 19.747 |
|
- type: map_at_5 |
|
value: 20.75 |
|
- type: mrr_at_1 |
|
value: 16.636 |
|
- type: mrr_at_10 |
|
value: 23.294 |
|
- type: mrr_at_100 |
|
value: 24.312 |
|
- type: mrr_at_1000 |
|
value: 24.401999999999997 |
|
- type: mrr_at_3 |
|
value: 21.503 |
|
- type: mrr_at_5 |
|
value: 22.52 |
|
- type: ndcg_at_1 |
|
value: 16.636 |
|
- type: ndcg_at_10 |
|
value: 25.372 |
|
- type: ndcg_at_100 |
|
value: 30.984 |
|
- type: ndcg_at_1000 |
|
value: 33.992 |
|
- type: ndcg_at_3 |
|
value: 21.607000000000003 |
|
- type: ndcg_at_5 |
|
value: 23.380000000000003 |
|
- type: precision_at_1 |
|
value: 16.636 |
|
- type: precision_at_10 |
|
value: 4.011 |
|
- type: precision_at_100 |
|
value: 0.741 |
|
- type: precision_at_1000 |
|
value: 0.11199999999999999 |
|
- type: precision_at_3 |
|
value: 9.365 |
|
- type: precision_at_5 |
|
value: 6.654 |
|
- type: recall_at_1 |
|
value: 15.495999999999999 |
|
- type: recall_at_10 |
|
value: 35.376000000000005 |
|
- type: recall_at_100 |
|
value: 61.694 |
|
- type: recall_at_1000 |
|
value: 84.029 |
|
- type: recall_at_3 |
|
value: 25.089 |
|
- type: recall_at_5 |
|
value: 29.43 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.662 |
|
- type: map_at_10 |
|
value: 8.638 |
|
- type: map_at_100 |
|
value: 9.86 |
|
- type: map_at_1000 |
|
value: 10.032 |
|
- type: map_at_3 |
|
value: 6.793 |
|
- type: map_at_5 |
|
value: 7.761 |
|
- type: mrr_at_1 |
|
value: 10.684000000000001 |
|
- type: mrr_at_10 |
|
value: 17.982 |
|
- type: mrr_at_100 |
|
value: 19.152 |
|
- type: mrr_at_1000 |
|
value: 19.231 |
|
- type: mrr_at_3 |
|
value: 15.113999999999999 |
|
- type: mrr_at_5 |
|
value: 16.658 |
|
- type: ndcg_at_1 |
|
value: 10.684000000000001 |
|
- type: ndcg_at_10 |
|
value: 13.483 |
|
- type: ndcg_at_100 |
|
value: 19.48 |
|
- type: ndcg_at_1000 |
|
value: 23.232 |
|
- type: ndcg_at_3 |
|
value: 9.75 |
|
- type: ndcg_at_5 |
|
value: 11.208 |
|
- type: precision_at_1 |
|
value: 10.684000000000001 |
|
- type: precision_at_10 |
|
value: 4.573 |
|
- type: precision_at_100 |
|
value: 1.085 |
|
- type: precision_at_1000 |
|
value: 0.17600000000000002 |
|
- type: precision_at_3 |
|
value: 7.514 |
|
- type: precision_at_5 |
|
value: 6.241 |
|
- type: recall_at_1 |
|
value: 4.662 |
|
- type: recall_at_10 |
|
value: 18.125 |
|
- type: recall_at_100 |
|
value: 39.675 |
|
- type: recall_at_1000 |
|
value: 61.332 |
|
- type: recall_at_3 |
|
value: 9.239 |
|
- type: recall_at_5 |
|
value: 12.863 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.869 |
|
- type: map_at_10 |
|
value: 8.701 |
|
- type: map_at_100 |
|
value: 11.806999999999999 |
|
- type: map_at_1000 |
|
value: 12.676000000000002 |
|
- type: map_at_3 |
|
value: 6.3100000000000005 |
|
- type: map_at_5 |
|
value: 7.471 |
|
- type: mrr_at_1 |
|
value: 38.5 |
|
- type: mrr_at_10 |
|
value: 48.754 |
|
- type: mrr_at_100 |
|
value: 49.544 |
|
- type: mrr_at_1000 |
|
value: 49.568 |
|
- type: mrr_at_3 |
|
value: 46.167 |
|
- type: mrr_at_5 |
|
value: 47.679 |
|
- type: ndcg_at_1 |
|
value: 30.5 |
|
- type: ndcg_at_10 |
|
value: 22.454 |
|
- type: ndcg_at_100 |
|
value: 25.380999999999997 |
|
- type: ndcg_at_1000 |
|
value: 31.582 |
|
- type: ndcg_at_3 |
|
value: 25.617 |
|
- type: ndcg_at_5 |
|
value: 24.254 |
|
- type: precision_at_1 |
|
value: 38.5 |
|
- type: precision_at_10 |
|
value: 18.4 |
|
- type: precision_at_100 |
|
value: 6.02 |
|
- type: precision_at_1000 |
|
value: 1.34 |
|
- type: precision_at_3 |
|
value: 29.083 |
|
- type: precision_at_5 |
|
value: 24.85 |
|
- type: recall_at_1 |
|
value: 3.869 |
|
- type: recall_at_10 |
|
value: 12.902 |
|
- type: recall_at_100 |
|
value: 30.496000000000002 |
|
- type: recall_at_1000 |
|
value: 51.066 |
|
- type: recall_at_3 |
|
value: 7.396 |
|
- type: recall_at_5 |
|
value: 9.852 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 36.705000000000005 |
|
- type: f1 |
|
value: 32.72625967901387 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 66.89840000000001 |
|
- type: ap |
|
value: 61.43175045563333 |
|
- type: f1 |
|
value: 66.67945656405962 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 89.12676698586411 |
|
- type: f1 |
|
value: 88.48426641357668 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 62.61513907888736 |
|
- type: f1 |
|
value: 40.96251281624023 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 61.95359784801614 |
|
- type: f1 |
|
value: 58.85654625260125 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 70.1983860121049 |
|
- type: f1 |
|
value: 68.73455379435487 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 31.772017072895846 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 30.944581802089044 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 30.977328237697133 |
|
- type: mrr |
|
value: 32.02612207306447 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 43.08588418858767 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 56.53785276450797 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 81.44882719207659 |
|
- type: mrr |
|
value: 94.71082022552609 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.77821782178218 |
|
- type: cos_sim_ap |
|
value: 93.22909989796688 |
|
- type: cos_sim_f1 |
|
value: 88.41778697001035 |
|
- type: cos_sim_precision |
|
value: 91.54175588865097 |
|
- type: cos_sim_recall |
|
value: 85.5 |
|
- type: dot_accuracy |
|
value: 99.77821782178218 |
|
- type: dot_ap |
|
value: 93.2290998979669 |
|
- type: dot_f1 |
|
value: 88.41778697001035 |
|
- type: dot_precision |
|
value: 91.54175588865097 |
|
- type: dot_recall |
|
value: 85.5 |
|
- type: euclidean_accuracy |
|
value: 99.77821782178218 |
|
- type: euclidean_ap |
|
value: 93.2290998979669 |
|
- type: euclidean_f1 |
|
value: 88.41778697001035 |
|
- type: euclidean_precision |
|
value: 91.54175588865097 |
|
- type: euclidean_recall |
|
value: 85.5 |
|
- type: manhattan_accuracy |
|
value: 99.77524752475247 |
|
- type: manhattan_ap |
|
value: 93.18492132451668 |
|
- type: manhattan_f1 |
|
value: 88.19552782111285 |
|
- type: manhattan_precision |
|
value: 91.87432286023835 |
|
- type: manhattan_recall |
|
value: 84.8 |
|
- type: max_accuracy |
|
value: 99.77821782178218 |
|
- type: max_ap |
|
value: 93.2290998979669 |
|
- type: max_f1 |
|
value: 88.41778697001035 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 48.225188905490285 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 34.76195959924048 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 48.16986372261003 |
|
- type: mrr |
|
value: 48.7718837535014 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 63.567200000000014 |
|
- type: ap |
|
value: 11.412292644030266 |
|
- type: f1 |
|
value: 49.102043399207716 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 51.04414261460101 |
|
- type: f1 |
|
value: 51.22880449155832 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 34.35595440606073 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 84.6754485307266 |
|
- type: cos_sim_ap |
|
value: 69.6007143804539 |
|
- type: cos_sim_f1 |
|
value: 65.99822312476202 |
|
- type: cos_sim_precision |
|
value: 63.58522866226461 |
|
- type: cos_sim_recall |
|
value: 68.60158311345647 |
|
- type: dot_accuracy |
|
value: 84.6754485307266 |
|
- type: dot_ap |
|
value: 69.60070881520775 |
|
- type: dot_f1 |
|
value: 65.99822312476202 |
|
- type: dot_precision |
|
value: 63.58522866226461 |
|
- type: dot_recall |
|
value: 68.60158311345647 |
|
- type: euclidean_accuracy |
|
value: 84.6754485307266 |
|
- type: euclidean_ap |
|
value: 69.60071394457518 |
|
- type: euclidean_f1 |
|
value: 65.99822312476202 |
|
- type: euclidean_precision |
|
value: 63.58522866226461 |
|
- type: euclidean_recall |
|
value: 68.60158311345647 |
|
- type: manhattan_accuracy |
|
value: 84.6754485307266 |
|
- type: manhattan_ap |
|
value: 69.57324451019119 |
|
- type: manhattan_f1 |
|
value: 65.7235045917101 |
|
- type: manhattan_precision |
|
value: 62.04311152764761 |
|
- type: manhattan_recall |
|
value: 69.86807387862797 |
|
- type: max_accuracy |
|
value: 84.6754485307266 |
|
- type: max_ap |
|
value: 69.6007143804539 |
|
- type: max_f1 |
|
value: 65.99822312476202 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 87.63922847052432 |
|
- type: cos_sim_ap |
|
value: 83.48934190421085 |
|
- type: cos_sim_f1 |
|
value: 75.42265503384861 |
|
- type: cos_sim_precision |
|
value: 71.17868124359413 |
|
- type: cos_sim_recall |
|
value: 80.20480443486295 |
|
- type: dot_accuracy |
|
value: 87.63922847052432 |
|
- type: dot_ap |
|
value: 83.4893468701264 |
|
- type: dot_f1 |
|
value: 75.42265503384861 |
|
- type: dot_precision |
|
value: 71.17868124359413 |
|
- type: dot_recall |
|
value: 80.20480443486295 |
|
- type: euclidean_accuracy |
|
value: 87.63922847052432 |
|
- type: euclidean_ap |
|
value: 83.48934073168017 |
|
- type: euclidean_f1 |
|
value: 75.42265503384861 |
|
- type: euclidean_precision |
|
value: 71.17868124359413 |
|
- type: euclidean_recall |
|
value: 80.20480443486295 |
|
- type: manhattan_accuracy |
|
value: 87.66251406838204 |
|
- type: manhattan_ap |
|
value: 83.46319621504654 |
|
- type: manhattan_f1 |
|
value: 75.41883304448297 |
|
- type: manhattan_precision |
|
value: 71.0089747076421 |
|
- type: manhattan_recall |
|
value: 80.41268863566368 |
|
- type: max_accuracy |
|
value: 87.66251406838204 |
|
- type: max_ap |
|
value: 83.4893468701264 |
|
- type: max_f1 |
|
value: 75.42265503384861 |
|
--- |
|
|
|
|
|
# {MODEL_NAME} |
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. |
|
|
|
<!--- Describe your model here --> |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences = ["This is an example sentence", "Each sentence is converted"] |
|
|
|
model = SentenceTransformer('{MODEL_NAME}') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
|
|
|
|
## Evaluation Results |
|
|
|
<!--- Describe how your model was evaluated --> |
|
|
|
For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME}) |
|
|
|
|
|
## Training |
|
The model was trained with the parameters: |
|
|
|
**DataLoader**: |
|
|
|
`torch.utils.data.dataloader.DataLoader` of length 15607 with parameters: |
|
``` |
|
{'batch_size': 48, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} |
|
``` |
|
|
|
**Loss**: |
|
|
|
`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters: |
|
``` |
|
{'scale': 20.0, 'similarity_fct': 'cos_sim'} |
|
``` |
|
|
|
Parameters of the fit()-Method: |
|
``` |
|
{ |
|
"epochs": 10, |
|
"evaluation_steps": 0, |
|
"evaluator": "NoneType", |
|
"max_grad_norm": 1, |
|
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>", |
|
"optimizer_params": { |
|
"lr": 2e-05 |
|
}, |
|
"scheduler": "WarmupLinear", |
|
"steps_per_epoch": null, |
|
"warmup_steps": 1000, |
|
"weight_decay": 0.01 |
|
} |
|
``` |
|
|
|
|
|
## Full Model Architecture |
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel |
|
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) |
|
(2): Normalize() |
|
) |
|
``` |
|
|
|
## Citing & Authors |
|
|
|
<!--- Describe where people can find more information --> |
