--- tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers - mteb license: lgpl language: - pl pipeline_tag: sentence-similarity model-index: - name: st-polish-kartonberta-base-alpha-v1 results: - task: type: Clustering dataset: type: PL-MTEB/8tags-clustering name: MTEB 8TagsClustering config: default split: test revision: None metrics: - type: v_measure value: 32.85180358455615 - task: type: Classification dataset: type: PL-MTEB/allegro-reviews name: MTEB AllegroReviews config: default split: test revision: None metrics: - type: accuracy value: 40.188866799204774 - type: f1 value: 34.71127012684797 - task: type: Retrieval dataset: type: arguana-pl name: MTEB ArguAna-PL config: default split: test revision: None metrics: - type: map_at_1 value: 30.939 - type: map_at_10 value: 47.467999999999996 - type: map_at_100 value: 48.303000000000004 - type: map_at_1000 value: 48.308 - type: map_at_3 value: 43.22 - type: map_at_5 value: 45.616 - type: mrr_at_1 value: 31.863000000000003 - type: mrr_at_10 value: 47.829 - type: mrr_at_100 value: 48.664 - type: mrr_at_1000 value: 48.67 - type: mrr_at_3 value: 43.492 - type: mrr_at_5 value: 46.006 - type: ndcg_at_1 value: 30.939 - type: ndcg_at_10 value: 56.058 - type: ndcg_at_100 value: 59.562000000000005 - type: ndcg_at_1000 value: 59.69799999999999 - type: ndcg_at_3 value: 47.260000000000005 - type: ndcg_at_5 value: 51.587 - type: precision_at_1 value: 30.939 - type: precision_at_10 value: 8.329 - type: precision_at_100 value: 0.984 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 19.654 - type: precision_at_5 value: 13.898 - type: recall_at_1 value: 30.939 - type: recall_at_10 value: 83.286 - type: recall_at_100 value: 98.43499999999999 - type: recall_at_1000 value: 99.502 - type: recall_at_3 value: 58.962 - type: recall_at_5 value: 69.488 - task: type: Classification dataset: type: PL-MTEB/cbd name: MTEB CBD config: default split: test revision: None metrics: - type: accuracy value: 67.69000000000001 - type: ap value: 21.078799692467182 - type: f1 value: 56.80107173953953 - task: type: PairClassification dataset: type: PL-MTEB/cdsce-pairclassification name: MTEB CDSC-E config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 89.2 - type: cos_sim_ap value: 79.11674608786898 - type: cos_sim_f1 value: 68.83468834688347 - type: cos_sim_precision value: 70.94972067039106 - type: cos_sim_recall value: 66.84210526315789 - type: dot_accuracy value: 89.2 - type: dot_ap value: 79.11674608786898 - type: dot_f1 value: 68.83468834688347 - type: dot_precision value: 70.94972067039106 - type: dot_recall value: 66.84210526315789 - type: euclidean_accuracy value: 89.2 - type: euclidean_ap value: 79.11674608786898 - type: euclidean_f1 value: 68.83468834688347 - type: euclidean_precision value: 70.94972067039106 - type: euclidean_recall value: 66.84210526315789 - type: manhattan_accuracy value: 89.1 - type: manhattan_ap value: 79.1220443374692 - type: manhattan_f1 value: 69.02173913043478 - type: manhattan_precision value: 71.34831460674157 - type: manhattan_recall value: 66.84210526315789 - type: max_accuracy value: 89.2 - type: max_ap value: 79.1220443374692 - type: max_f1 value: 69.02173913043478 - task: type: STS dataset: type: PL-MTEB/cdscr-sts name: MTEB CDSC-R config: default split: test revision: None metrics: - type: cos_sim_pearson value: 91.41534744278998 - type: cos_sim_spearman value: 92.12681551821147 - type: euclidean_pearson value: 91.74369794485992 - type: euclidean_spearman value: 92.12685848456046 - type: manhattan_pearson value: 91.66651938751657 - type: manhattan_spearman value: 92.057603126734 - task: type: Retrieval dataset: type: dbpedia-pl name: MTEB DBPedia-PL config: default split: test revision: None metrics: - type: map_at_1 value: 5.8709999999999996 - type: map_at_10 value: 12.486 - type: map_at_100 value: 16.897000000000002 - type: map_at_1000 value: 18.056 - type: map_at_3 value: 8.958 - type: map_at_5 value: 10.57 - type: mrr_at_1 value: 44.0 - type: mrr_at_10 value: 53.830999999999996 - type: mrr_at_100 value: 54.54 - type: mrr_at_1000 value: 54.568000000000005 - type: mrr_at_3 value: 51.87500000000001 - type: mrr_at_5 value: 53.113 - type: ndcg_at_1 value: 34.625 - type: ndcg_at_10 value: 26.996 - type: ndcg_at_100 value: 31.052999999999997 - type: ndcg_at_1000 value: 38.208 - type: ndcg_at_3 value: 29.471000000000004 - type: ndcg_at_5 value: 28.364 - type: precision_at_1 value: 44.0 - type: precision_at_10 value: 21.45 - type: precision_at_100 value: 6.837 - type: precision_at_1000 value: 1.6019999999999999 - type: precision_at_3 value: 32.333 - type: precision_at_5 value: 27.800000000000004 - type: recall_at_1 value: 5.8709999999999996 - type: recall_at_10 value: 17.318 - type: recall_at_100 value: 36.854 - type: recall_at_1000 value: 60.468999999999994 - type: recall_at_3 value: 10.213999999999999 - type: recall_at_5 value: 13.364 - task: type: Retrieval dataset: type: fiqa-pl name: MTEB FiQA-PL config: default split: test revision: None metrics: - type: map_at_1 value: 10.289 - type: map_at_10 value: 18.285999999999998 - type: map_at_100 value: 19.743 - type: map_at_1000 value: 19.964000000000002 - type: map_at_3 value: 15.193000000000001 - type: map_at_5 value: 16.962 - type: mrr_at_1 value: 21.914 - type: mrr_at_10 value: 30.653999999999996 - type: mrr_at_100 value: 31.623 - type: mrr_at_1000 value: 31.701 - type: mrr_at_3 value: 27.855 - type: mrr_at_5 value: 29.514000000000003 - type: ndcg_at_1 value: 21.914 - type: ndcg_at_10 value: 24.733 - type: ndcg_at_100 value: 31.253999999999998 - type: ndcg_at_1000 value: 35.617 - type: ndcg_at_3 value: 20.962 - type: ndcg_at_5 value: 22.553 - type: precision_at_1 value: 21.914 - type: precision_at_10 value: 7.346 - type: precision_at_100 value: 1.389 - type: precision_at_1000 value: 0.214 - type: precision_at_3 value: 14.352 - type: precision_at_5 value: 11.42 - type: recall_at_1 value: 10.289 - type: recall_at_10 value: 31.459 - type: recall_at_100 value: 56.854000000000006 - type: recall_at_1000 value: 83.722 - type: recall_at_3 value: 19.457 - type: recall_at_5 value: 24.767 - task: type: Retrieval dataset: type: hotpotqa-pl name: MTEB HotpotQA-PL config: default split: test revision: None metrics: - type: map_at_1 value: 29.669 - type: map_at_10 value: 41.615 - type: map_at_100 value: 42.571999999999996 - type: map_at_1000 value: 42.662 - type: map_at_3 value: 38.938 - type: map_at_5 value: 40.541 - type: mrr_at_1 value: 59.338 - type: mrr_at_10 value: 66.93900000000001 - type: mrr_at_100 value: 67.361 - type: mrr_at_1000 value: 67.38499999999999 - type: mrr_at_3 value: 65.384 - type: mrr_at_5 value: 66.345 - type: ndcg_at_1 value: 59.338 - type: ndcg_at_10 value: 50.607 - type: ndcg_at_100 value: 54.342999999999996 - type: ndcg_at_1000 value: 56.286 - type: ndcg_at_3 value: 46.289 - type: ndcg_at_5 value: 48.581 - type: precision_at_1 value: 59.338 - type: precision_at_10 value: 10.585 - type: precision_at_100 value: 1.353 - type: precision_at_1000 value: 0.161 - type: precision_at_3 value: 28.877000000000002 - type: precision_at_5 value: 19.133 - type: recall_at_1 value: 29.669 - type: recall_at_10 value: 52.92400000000001 - type: recall_at_100 value: 67.657 - type: recall_at_1000 value: 80.628 - type: recall_at_3 value: 43.315 - type: recall_at_5 value: 47.833 - task: type: Retrieval dataset: type: msmarco-pl name: MTEB MSMARCO-PL config: default split: test revision: None metrics: - type: map_at_1 value: 0.997 - type: map_at_10 value: 7.481999999999999 - type: map_at_100 value: 20.208000000000002 - type: map_at_1000 value: 25.601000000000003 - type: map_at_3 value: 3.055 - type: map_at_5 value: 4.853 - type: mrr_at_1 value: 55.814 - type: mrr_at_10 value: 64.651 - type: mrr_at_100 value: 65.003 - type: mrr_at_1000 value: 65.05199999999999 - type: mrr_at_3 value: 62.403 - type: mrr_at_5 value: 64.031 - type: ndcg_at_1 value: 44.186 - type: ndcg_at_10 value: 43.25 - type: ndcg_at_100 value: 40.515 - type: ndcg_at_1000 value: 48.345 - type: ndcg_at_3 value: 45.829 - type: ndcg_at_5 value: 46.477000000000004 - type: precision_at_1 value: 55.814 - type: precision_at_10 value: 50.465 - type: precision_at_100 value: 25.419000000000004 - type: precision_at_1000 value: 5.0840000000000005 - type: precision_at_3 value: 58.14 - type: precision_at_5 value: 57.67400000000001 - type: recall_at_1 value: 0.997 - type: recall_at_10 value: 8.985999999999999 - type: recall_at_100 value: 33.221000000000004 - type: recall_at_1000 value: 58.836999999999996 - type: recall_at_3 value: 3.472 - type: recall_at_5 value: 5.545 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) config: pl split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 68.19771351714861 - type: f1 value: 64.75039989217822 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) config: pl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 73.9677202420982 - type: f1 value: 73.72287107577753 - task: type: Retrieval dataset: type: nfcorpus-pl name: MTEB NFCorpus-PL config: default split: test revision: None metrics: - type: map_at_1 value: 5.167 - type: map_at_10 value: 10.791 - type: map_at_100 value: 14.072999999999999 - type: map_at_1000 value: 15.568000000000001 - type: map_at_3 value: 7.847999999999999 - type: map_at_5 value: 9.112 - type: mrr_at_1 value: 42.105 - type: mrr_at_10 value: 49.933 - type: mrr_at_100 value: 50.659 - type: mrr_at_1000 value: 50.705 - type: mrr_at_3 value: 47.988 - type: mrr_at_5 value: 49.056 - type: ndcg_at_1 value: 39.938 - type: ndcg_at_10 value: 31.147000000000002 - type: ndcg_at_100 value: 29.336000000000002 - type: ndcg_at_1000 value: 38.147 - type: ndcg_at_3 value: 35.607 - type: ndcg_at_5 value: 33.725 - type: precision_at_1 value: 41.486000000000004 - type: precision_at_10 value: 23.901 - type: precision_at_100 value: 7.960000000000001 - type: precision_at_1000 value: 2.086 - type: precision_at_3 value: 33.437 - type: precision_at_5 value: 29.598000000000003 - type: recall_at_1 value: 5.167 - type: recall_at_10 value: 14.244000000000002 - type: recall_at_100 value: 31.192999999999998 - type: recall_at_1000 value: 62.41799999999999 - type: recall_at_3 value: 8.697000000000001 - type: recall_at_5 value: 10.911 - task: type: Retrieval dataset: type: nq-pl name: MTEB NQ-PL config: default split: test revision: None metrics: - type: map_at_1 value: 14.417 - type: map_at_10 value: 23.330000000000002 - type: map_at_100 value: 24.521 - type: map_at_1000 value: 24.604 - type: map_at_3 value: 20.076 - type: map_at_5 value: 21.854000000000003 - type: mrr_at_1 value: 16.454 - type: mrr_at_10 value: 25.402 - type: mrr_at_100 value: 26.411 - type: mrr_at_1000 value: 26.479000000000003 - type: mrr_at_3 value: 22.369 - type: mrr_at_5 value: 24.047 - type: ndcg_at_1 value: 16.454 - type: ndcg_at_10 value: 28.886 - type: ndcg_at_100 value: 34.489999999999995 - type: ndcg_at_1000 value: 36.687999999999995 - type: ndcg_at_3 value: 22.421 - type: ndcg_at_5 value: 25.505 - type: precision_at_1 value: 16.454 - type: precision_at_10 value: 5.252 - type: precision_at_100 value: 0.8410000000000001 - type: precision_at_1000 value: 0.105 - type: precision_at_3 value: 10.428999999999998 - type: precision_at_5 value: 8.019 - type: recall_at_1 value: 14.417 - type: recall_at_10 value: 44.025 - type: recall_at_100 value: 69.404 - type: recall_at_1000 value: 86.18900000000001 - type: recall_at_3 value: 26.972 - type: recall_at_5 value: 34.132 - task: type: Classification dataset: type: laugustyniak/abusive-clauses-pl name: MTEB PAC config: default split: test revision: None metrics: - type: accuracy value: 66.55082536924412 - type: ap value: 76.44962281293184 - type: f1 value: 63.899803692180434 - task: type: PairClassification dataset: type: PL-MTEB/ppc-pairclassification name: MTEB PPC config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 86.5 - type: cos_sim_ap value: 92.65086645409387 - type: cos_sim_f1 value: 89.39157566302653 - type: cos_sim_precision value: 84.51327433628319 - type: cos_sim_recall value: 94.86754966887418 - type: dot_accuracy value: 86.5 - type: dot_ap value: 92.65086645409387 - type: dot_f1 value: 89.39157566302653 - type: dot_precision value: 84.51327433628319 - type: dot_recall value: 94.86754966887418 - type: euclidean_accuracy value: 86.5 - type: euclidean_ap value: 92.65086645409387 - type: euclidean_f1 value: 89.39157566302653 - type: euclidean_precision value: 84.51327433628319 - type: euclidean_recall value: 94.86754966887418 - type: manhattan_accuracy value: 86.5 - type: manhattan_ap value: 92.64975544736456 - type: manhattan_f1 value: 89.33852140077822 - type: manhattan_precision value: 84.28781204111601 - type: manhattan_recall value: 95.03311258278146 - type: max_accuracy value: 86.5 - type: max_ap value: 92.65086645409387 - type: max_f1 value: 89.39157566302653 - task: type: PairClassification dataset: type: PL-MTEB/psc-pairclassification name: MTEB PSC config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 95.64007421150278 - type: cos_sim_ap value: 98.42114841894346 - type: cos_sim_f1 value: 92.8895612708018 - type: cos_sim_precision value: 92.1921921921922 - type: cos_sim_recall value: 93.59756097560977 - type: dot_accuracy value: 95.64007421150278 - type: dot_ap value: 98.42114841894346 - type: dot_f1 value: 92.8895612708018 - type: dot_precision value: 92.1921921921922 - type: dot_recall value: 93.59756097560977 - type: euclidean_accuracy value: 95.64007421150278 - type: euclidean_ap value: 98.42114841894346 - type: euclidean_f1 value: 92.8895612708018 - type: euclidean_precision value: 92.1921921921922 - type: euclidean_recall value: 93.59756097560977 - type: manhattan_accuracy value: 95.82560296846012 - type: manhattan_ap value: 98.38712415914046 - type: manhattan_f1 value: 93.19213313161876 - type: manhattan_precision value: 92.49249249249249 - type: manhattan_recall value: 93.90243902439023 - type: max_accuracy value: 95.82560296846012 - type: max_ap value: 98.42114841894346 - type: max_f1 value: 93.19213313161876 - task: type: Classification dataset: type: PL-MTEB/polemo2_in name: MTEB PolEmo2.0-IN config: default split: test revision: None metrics: - type: accuracy value: 68.40720221606648 - type: f1 value: 67.09084289613526 - task: type: Classification dataset: type: PL-MTEB/polemo2_out name: MTEB PolEmo2.0-OUT config: default split: test revision: None metrics: - type: accuracy value: 38.056680161943326 - type: f1 value: 32.87731504372395 - task: type: Retrieval dataset: type: quora-pl name: MTEB Quora-PL config: default split: test revision: None metrics: - type: map_at_1 value: 65.422 - type: map_at_10 value: 79.259 - type: map_at_100 value: 80.0 - type: map_at_1000 value: 80.021 - type: map_at_3 value: 76.16199999999999 - type: map_at_5 value: 78.03999999999999 - type: mrr_at_1 value: 75.26 - type: mrr_at_10 value: 82.39699999999999 - type: mrr_at_100 value: 82.589 - type: mrr_at_1000 value: 82.593 - type: mrr_at_3 value: 81.08999999999999 - type: mrr_at_5 value: 81.952 - type: ndcg_at_1 value: 75.3 - type: ndcg_at_10 value: 83.588 - type: ndcg_at_100 value: 85.312 - type: ndcg_at_1000 value: 85.536 - type: ndcg_at_3 value: 80.128 - type: ndcg_at_5 value: 81.962 - type: precision_at_1 value: 75.3 - type: precision_at_10 value: 12.856000000000002 - type: precision_at_100 value: 1.508 - type: precision_at_1000 value: 0.156 - type: precision_at_3 value: 35.207 - type: precision_at_5 value: 23.316 - type: recall_at_1 value: 65.422 - type: recall_at_10 value: 92.381 - type: recall_at_100 value: 98.575 - type: recall_at_1000 value: 99.85300000000001 - type: recall_at_3 value: 82.59100000000001 - type: recall_at_5 value: 87.629 - task: type: Retrieval dataset: type: scidocs-pl name: MTEB SCIDOCS-PL config: default split: test revision: None metrics: - type: map_at_1 value: 2.52 - type: map_at_10 value: 6.814000000000001 - type: map_at_100 value: 8.267 - type: map_at_1000 value: 8.565000000000001 - type: map_at_3 value: 4.736 - type: map_at_5 value: 5.653 - type: mrr_at_1 value: 12.5 - type: mrr_at_10 value: 20.794999999999998 - type: mrr_at_100 value: 22.014 - type: mrr_at_1000 value: 22.109 - type: mrr_at_3 value: 17.8 - type: mrr_at_5 value: 19.42 - type: ndcg_at_1 value: 12.5 - type: ndcg_at_10 value: 12.209 - type: ndcg_at_100 value: 18.812 - type: ndcg_at_1000 value: 24.766 - type: ndcg_at_3 value: 10.847 - type: ndcg_at_5 value: 9.632 - type: precision_at_1 value: 12.5 - type: precision_at_10 value: 6.660000000000001 - type: precision_at_100 value: 1.6340000000000001 - type: precision_at_1000 value: 0.307 - type: precision_at_3 value: 10.299999999999999 - type: precision_at_5 value: 8.66 - type: recall_at_1 value: 2.52 - type: recall_at_10 value: 13.495 - type: recall_at_100 value: 33.188 - type: recall_at_1000 value: 62.34499999999999 - type: recall_at_3 value: 6.245 - type: recall_at_5 value: 8.76 - task: type: PairClassification dataset: type: PL-MTEB/sicke-pl-pairclassification name: MTEB SICK-E-PL config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 86.13942111699959 - type: cos_sim_ap value: 81.47480017120256 - type: cos_sim_f1 value: 74.79794268919912 - type: cos_sim_precision value: 77.2382397572079 - type: cos_sim_recall value: 72.50712250712252 - type: dot_accuracy value: 86.13942111699959 - type: dot_ap value: 81.47478531367476 - type: dot_f1 value: 74.79794268919912 - type: dot_precision value: 77.2382397572079 - type: dot_recall value: 72.50712250712252 - type: euclidean_accuracy value: 86.13942111699959 - type: euclidean_ap value: 81.47478531367476 - type: euclidean_f1 value: 74.79794268919912 - type: euclidean_precision value: 77.2382397572079 - type: euclidean_recall value: 72.50712250712252 - type: manhattan_accuracy value: 86.15980432123929 - type: manhattan_ap value: 81.40798042612397 - type: manhattan_f1 value: 74.86116253239543 - type: manhattan_precision value: 77.9491133384734 - type: manhattan_recall value: 72.00854700854701 - type: max_accuracy value: 86.15980432123929 - type: max_ap value: 81.47480017120256 - type: max_f1 value: 74.86116253239543 - task: type: STS dataset: type: PL-MTEB/sickr-pl-sts name: MTEB SICK-R-PL config: default split: test revision: None metrics: - type: cos_sim_pearson value: 84.27525342551935 - type: cos_sim_spearman value: 79.50631730805885 - type: euclidean_pearson value: 82.07169123942028 - type: euclidean_spearman value: 79.50631887406465 - type: manhattan_pearson value: 81.98288826317463 - type: manhattan_spearman value: 79.4244081650332 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl) config: pl split: test revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 metrics: - type: cos_sim_pearson value: 35.59400236598834 - type: cos_sim_spearman value: 36.782560207852846 - type: euclidean_pearson value: 28.546177668542942 - type: euclidean_spearman value: 36.68394223635756 - type: manhattan_pearson value: 28.45606963909248 - type: manhattan_spearman value: 36.475975118547524 - task: type: Retrieval dataset: type: scifact-pl name: MTEB SciFact-PL config: default split: test revision: None metrics: - type: map_at_1 value: 41.028 - type: map_at_10 value: 52.23799999999999 - type: map_at_100 value: 52.905 - type: map_at_1000 value: 52.945 - type: map_at_3 value: 49.102000000000004 - type: map_at_5 value: 50.992000000000004 - type: mrr_at_1 value: 43.333 - type: mrr_at_10 value: 53.551 - type: mrr_at_100 value: 54.138 - type: mrr_at_1000 value: 54.175 - type: mrr_at_3 value: 51.056000000000004 - type: mrr_at_5 value: 52.705999999999996 - type: ndcg_at_1 value: 43.333 - type: ndcg_at_10 value: 57.731 - type: ndcg_at_100 value: 61.18599999999999 - type: ndcg_at_1000 value: 62.261 - type: ndcg_at_3 value: 52.276999999999994 - type: ndcg_at_5 value: 55.245999999999995 - type: precision_at_1 value: 43.333 - type: precision_at_10 value: 8.267 - type: precision_at_100 value: 1.02 - type: precision_at_1000 value: 0.11100000000000002 - type: precision_at_3 value: 21.444 - type: precision_at_5 value: 14.533 - type: recall_at_1 value: 41.028 - type: recall_at_10 value: 73.111 - type: recall_at_100 value: 89.533 - type: recall_at_1000 value: 98.0 - type: recall_at_3 value: 58.744 - type: recall_at_5 value: 66.106 - task: type: Retrieval dataset: type: trec-covid-pl name: MTEB TRECCOVID-PL config: default split: test revision: None metrics: - type: map_at_1 value: 0.146 - type: map_at_10 value: 1.09 - type: map_at_100 value: 6.002 - type: map_at_1000 value: 15.479999999999999 - type: map_at_3 value: 0.41000000000000003 - type: map_at_5 value: 0.596 - type: mrr_at_1 value: 54.0 - type: mrr_at_10 value: 72.367 - type: mrr_at_100 value: 72.367 - type: mrr_at_1000 value: 72.367 - type: mrr_at_3 value: 70.333 - type: mrr_at_5 value: 72.033 - type: ndcg_at_1 value: 48.0 - type: ndcg_at_10 value: 48.827 - type: ndcg_at_100 value: 38.513999999999996 - type: ndcg_at_1000 value: 37.958 - type: ndcg_at_3 value: 52.614000000000004 - type: ndcg_at_5 value: 51.013 - type: precision_at_1 value: 54.0 - type: precision_at_10 value: 53.6 - type: precision_at_100 value: 40.300000000000004 - type: precision_at_1000 value: 17.276 - type: precision_at_3 value: 57.333 - type: precision_at_5 value: 55.60000000000001 - type: recall_at_1 value: 0.146 - type: recall_at_10 value: 1.438 - type: recall_at_100 value: 9.673 - type: recall_at_1000 value: 36.870999999999995 - type: recall_at_3 value: 0.47400000000000003 - type: recall_at_5 value: 0.721 --- # Model Card for st-polish-kartonberta-base-alpha-v1 This sentence transformer model is designed to convert text content into a 768-float vector space, ensuring an effective representation. It aims to be proficient in tasks involving sentence / document similarity. The model has been released in its alpha version. Numerous potential enhancements could boost its performance, such as adjusting training hyperparameters or extending the training duration (currently limited to only one epoch). The main reason is limited GPU. ## Model Description - **Developed by:** Bartłomiej Orlik, https://www.linkedin.com/in/bartłomiej-orlik/ - **Model type:** RoBERTa Sentence Transformer - **Language:** Polish - **License:** LGPL-3.0 - **Trained from model:** sdadas/polish-roberta-base-v2: https://huggingface.co/sdadas/polish-roberta-base-v2 ## How to Get Started with the Model Use the code below to get started with the model. ### Using Sentence-Transformers You can use the model with [sentence-transformers](https://www.SBERT.net): ``` pip install -U sentence-transformers ``` ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer('OrlikB/st-polish-kartonberta-base-alpha-v1') text_1 = 'Jestem wielkim fanem opakowań tekturowych' text_2 = 'Bardzo podobają mi się kartony' embeddings_1 = model.encode(text_1, normalize_embeddings=True) embeddings_2 = model.encode(text_2, normalize_embeddings=True) similarity = embeddings_1 @ embeddings_2.T print(similarity) ``` ### Using HuggingFace Transformers ```python from transformers import AutoTokenizer, AutoModel import torch import numpy as np def encode_text(text): encoded_input = tokenizer(text, padding=True, truncation=True, return_tensors='pt', max_length=512) with torch.no_grad(): model_output = model(**encoded_input) sentence_embeddings = model_output[0][:, 0] sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1) return sentence_embeddings.squeeze().numpy() cosine_similarity = lambda a, b: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) tokenizer = AutoTokenizer.from_pretrained('OrlikB/st-polish-kartonberta-base-alpha-v1') model = AutoModel.from_pretrained('OrlikB/st-polish-kartonberta-base-alpha-v1') model.eval() text_1 = 'Jestem wielkim fanem opakowań tekturowych' text_2 = 'Bardzo podobają mi się kartony' embeddings_1 = encode_text(text_1) embeddings_2 = encode_text(text_2) print(cosine_similarity(embeddings_1, embeddings_2)) ``` *Note: You can use the encode_text function for demonstration purposes. For the best experience, it's recommended to process text in batches. ## Evaluation #### [MTEB for Polish Language](https://huggingface.co/spaces/mteb/leaderboard) | Rank | Model | Model Size (GB) | Embedding Dimensions | Sequence Length | Average (26 datasets) | Classification Average (7 datasets) | Clustering Average (1 datasets) | Pair Classification Average (4 datasets) | Retrieval Average (11 datasets) | STS Average (3 datasets) | |-------:|:----------------------------------------|------------------:|-----------------------:|------------------:|------------------------:|--------------------------------------:|--------------------------------:|-----------------------------------------:|----------------------------------:|-------------------------:| | 1 | multilingual-e5-large | 2.24 | 1024 | 514 | 58.25 | 60.51 | 24.06 | 84.58 | 47.82 | 67.52 | | 2 | **st-polish-kartonberta-base-alpha-v1** | 0.5 | 768 | 514 | 56.92 | 60.44 | **32.85** | **87.92** | 42.19 | **69.47** | | 3 | multilingual-e5-base | 1.11 | 768 | 514 | 54.18 | 57.01 | 18.62 | 82.08 | 42.5 | 65.07 | | 4 | multilingual-e5-small | 0.47 | 384 | 512 | 53.15 | 54.35 | 19.64 | 81.67 | 41.52 | 66.08 | | 5 | st-polish-paraphrase-from-mpnet | 0.5 | 768 | 514 | 53.06 | 57.49 | 25.09 | 87.04 | 36.53 | 67.39 | | 6 | st-polish-paraphrase-from-distilroberta | 0.5 | 768 | 514 | 52.65 | 58.55 | 31.11 | 87 | 33.96 | 68.78 | ## More Information I developed this model as a personal scientific initiative. I plan to start the development on a new ST model. However, due to limited computational resources, I suspended further work to create a larger or enhanced version of current model.