Solon Embeddings — large 0.1
SOTA Open source french embedding model.
Instructions :
Add "query : " before the query to retrieve to increase performance of retrieval.
No instructions needed for passages.
| Model | Mean Score |
|---|---|
| OrdalieTech/Solon-embeddings-large-0.1 | 0.7490 |
| cohere/embed-multilingual-v3 | 0.7402 |
| OrdalieTech/Solon-embeddings-base-0.1 | 0.7306 |
| openai/ada-002 | 0.7290 |
| cohere/embed-multilingual-light-v3 | 0.6945 |
| antoinelouis/biencoder-camembert-base-mmarcoFR | 0.6826 |
| dangvantuan/sentence-camembert-large | 0.6756 |
| voyage/voyage-01 | 0.6753 |
| intfloat/multilingual-e5-large | 0.6660 |
| intfloat/multilingual-e5-base | 0.6597 |
| Sbert/paraphrase-multilingual-mpnet-base-v2 | 0.5975 |
| dangvantuan/sentence-camembert-base | 0.5456 |
| EuropeanParliament/eubert_embedding_v1 | 0.5063 |
These results have been obtained through 9 french benchmarks on a variety of text similarity tasks (classification, reranking, STS) :
- AmazonReviewsClassification (MTEB)
- MassiveIntentClassification (MTEB)
- MassiveScenarioClassification (MTEB)
- MTOPDomainClassification (MTEB)
- MTOPIntentClassification (MTEB)
- STS22 (MTEB)
- MiraclFRRerank (Miracl)
- OrdalieFRSTS (Ordalie)
- OrdalieFRReranking (Ordalie)
We created OrdalieFRSTS and OrdalieFRReranking to enhance the benchmarking capabilities of French STS and reranking assessments.
(evaluation script available here : github.com/OrdalieTech/mteb)
- Downloads last month
- 16,432
Space using OrdalieTech/Solon-embeddings-large-0.1 1
Evaluation results
- Recall@500 on mMARCO-frvalidation set self-reported92.700
- Recall@100 on mMARCO-frvalidation set self-reported82.700
- Recall@10 on mMARCO-frvalidation set self-reported55.500
- MAP@10 on mMARCO-frvalidation set self-reported29.400
- nDCG@10 on mMARCO-frvalidation set self-reported35.800
- MRR@10 on mMARCO-frvalidation set self-reported29.900
- v_measure on MTEB AlloProfClusteringP2Ptest set self-reported64.169
- v_measure on MTEB AlloProfClusteringS2Stest set self-reported38.171
- map on MTEB AlloprofRerankingtest set self-reported64.877
- mrr on MTEB AlloprofRerankingtest set self-reported66.391
- map_at_1 on MTEB AlloprofRetrievaltest set self-reported29.620
- map_at_10 on MTEB AlloprofRetrievaltest set self-reported40.963
- map_at_100 on MTEB AlloprofRetrievaltest set self-reported41.894
- map_at_1000 on MTEB AlloprofRetrievaltest set self-reported41.939
- map_at_3 on MTEB AlloprofRetrievaltest set self-reported37.709
- map_at_5 on MTEB AlloprofRetrievaltest set self-reported39.697
- mrr_at_1 on MTEB AlloprofRetrievaltest set self-reported29.620
- mrr_at_10 on MTEB AlloprofRetrievaltest set self-reported40.963
- mrr_at_100 on MTEB AlloprofRetrievaltest set self-reported41.894
- mrr_at_1000 on MTEB AlloprofRetrievaltest set self-reported41.939
- mrr_at_3 on MTEB AlloprofRetrievaltest set self-reported37.709
- mrr_at_5 on MTEB AlloprofRetrievaltest set self-reported39.697
- ndcg_at_1 on MTEB AlloprofRetrievaltest set self-reported29.620
- ndcg_at_10 on MTEB AlloprofRetrievaltest set self-reported46.942
- ndcg_at_100 on MTEB AlloprofRetrievaltest set self-reported51.630
- ndcg_at_1000 on MTEB AlloprofRetrievaltest set self-reported52.927
- ndcg_at_3 on MTEB AlloprofRetrievaltest set self-reported40.334
- ndcg_at_5 on MTEB AlloprofRetrievaltest set self-reported43.922
- precision_at_1 on MTEB AlloprofRetrievaltest set self-reported29.620
- precision_at_10 on MTEB AlloprofRetrievaltest set self-reported6.589
- precision_at_100 on MTEB AlloprofRetrievaltest set self-reported0.882
- precision_at_1000 on MTEB AlloprofRetrievaltest set self-reported0.099
- precision_at_3 on MTEB AlloprofRetrievaltest set self-reported15.976
- precision_at_5 on MTEB AlloprofRetrievaltest set self-reported11.330
- recall_at_1 on MTEB AlloprofRetrievaltest set self-reported29.620
- recall_at_10 on MTEB AlloprofRetrievaltest set self-reported65.889
- recall_at_100 on MTEB AlloprofRetrievaltest set self-reported88.212
- recall_at_1000 on MTEB AlloprofRetrievaltest set self-reported98.575
- recall_at_3 on MTEB AlloprofRetrievaltest set self-reported47.927
- recall_at_5 on MTEB AlloprofRetrievaltest set self-reported56.649
- accuracy on MTEB AmazonReviewsClassification (fr)test set self-reported42.078
- f1 on MTEB AmazonReviewsClassification (fr)test set self-reported40.645
- map_at_1 on MTEB BSARDRetrievaltest set self-reported0.901
- map_at_10 on MTEB BSARDRetrievaltest set self-reported1.524
- map_at_100 on MTEB BSARDRetrievaltest set self-reported1.833
- map_at_1000 on MTEB BSARDRetrievaltest set self-reported1.916
- map_at_3 on MTEB BSARDRetrievaltest set self-reported1.276
- map_at_5 on MTEB BSARDRetrievaltest set self-reported1.276
- mrr_at_1 on MTEB BSARDRetrievaltest set self-reported0.901
- mrr_at_10 on MTEB BSARDRetrievaltest set self-reported1.524
- mrr_at_100 on MTEB BSARDRetrievaltest set self-reported1.833
- mrr_at_1000 on MTEB BSARDRetrievaltest set self-reported1.916
- mrr_at_3 on MTEB BSARDRetrievaltest set self-reported1.276
- mrr_at_5 on MTEB BSARDRetrievaltest set self-reported1.276
- ndcg_at_1 on MTEB BSARDRetrievaltest set self-reported0.901
- ndcg_at_10 on MTEB BSARDRetrievaltest set self-reported2.085
- ndcg_at_100 on MTEB BSARDRetrievaltest set self-reported3.805
- ndcg_at_1000 on MTEB BSARDRetrievaltest set self-reported6.704
- ndcg_at_3 on MTEB BSARDRetrievaltest set self-reported1.410
- ndcg_at_5 on MTEB BSARDRetrievaltest set self-reported1.410
- precision_at_1 on MTEB BSARDRetrievaltest set self-reported0.901
- precision_at_10 on MTEB BSARDRetrievaltest set self-reported0.405
- precision_at_100 on MTEB BSARDRetrievaltest set self-reported0.126
- precision_at_1000 on MTEB BSARDRetrievaltest set self-reported0.037
- precision_at_3 on MTEB BSARDRetrievaltest set self-reported0.601
- precision_at_5 on MTEB BSARDRetrievaltest set self-reported0.360
- recall_at_1 on MTEB BSARDRetrievaltest set self-reported0.901
- recall_at_10 on MTEB BSARDRetrievaltest set self-reported4.054
- recall_at_100 on MTEB BSARDRetrievaltest set self-reported12.613
- recall_at_1000 on MTEB BSARDRetrievaltest set self-reported36.937
- recall_at_3 on MTEB BSARDRetrievaltest set self-reported1.802
- recall_at_5 on MTEB BSARDRetrievaltest set self-reported1.802
- accuracy on MTEB DiaBLaBitextMining (fr-en)test set self-reported88.900
- f1 on MTEB DiaBLaBitextMining (fr-en)test set self-reported86.950
- precision on MTEB DiaBLaBitextMining (fr-en)test set self-reported86.203
- recall on MTEB DiaBLaBitextMining (fr-en)test set self-reported88.900
- v_measure on MTEB HALClusteringS2Stest set self-reported24.088
- v_measure on MTEB MLSUMClusteringP2Ptest set self-reported43.796