Hugging Face's logo

antoinelouis
/
crossencoder-electra-base-mmarcoFR

Text Classification
sentence-transformers
Safetensors
French
electra
passage-reranking
Eval Results
Model card Files Community
1
antoinelouis commited on
Commit
4f9f21e
•
1 Parent(s): b196ebf

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +10 -28
README.md CHANGED
@@ -61,37 +61,19 @@ print(scores)
61
  ## Evaluation
62
  ***
63
 
64
- We evaluated our model on 500 random queries from the mMARCO-fr train set (which were excluded from training). Each of these queries has at least one relevant and up to 200 irrelevant passages.
65
 
66
- | r-precision | mrr@10 | recall@10 | recall@20 | recall@50 | recall@100 |
67
- |--------------:|---------:|------------:|------------:|------------:|-------------:|
68
- | 28.32 | 45.28 | 79.22 | 87.15 | 93.15 | 95.75 |
69
 
70
- Below, we compared its results with other cross-encoder models fine-tuned on the same dataset:
71
- | | model | r-precision | mrr@10 | recall@10 (↑) | recall@20 | recall@50 | recall@100 |
72
  |---:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------:|---------:|------------:|------------:|------------:|-------------:|
73
- | 1 | [crossencoder-camembert-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-camembert-base-mmarcoFR) | 35.65 | 50.44 | 82.95 | 91.5 | 96.8 | 98.8 |
74
- | 2 | [crossencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR) | 34.37 | 51.01 | 82.23 | 90.6 | 96.45 | 98.4 |
75
- | 3 | [crossencoder-mmarcoFR-mMiniLMv2-L12-H384-v1-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mmarcoFR-mMiniLMv2-L12-H384-v1-mmarcoFR) | 34.22 | 49.2 | 81.7 | 90.9 | 97.1 | 98.9 |
76
- | 4 | [crossencoder-mpnet-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mpnet-base-mmarcoFR) | 29.68 | 46.13 | 80.45 | 87.9 | 93.15 | 96.6 |
77
- | 5 | [crossencoder-distilcamembert-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-distilcamembert-base-mmarcoFR) | 27.28 | 43.71 | 80.3 | 89.1 | 95.55 | 98.6 |
78
- | 6 | [crossencoder-roberta-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-roberta-base-mmarcoFR) | 33.33 | 48.87 | 79.33 | 86.75 | 94.15 | 97.6 |
79
- | 7 | **crossencoder-electra-base-french-europeana-cased-discriminator-mmarcoFR** | 28.32 | 45.28 | 79.22 | 87.15 | 93.15 | 95.75 |
80
- | 8 | [crossencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR) | 33.92 | 49.33 | 79 | 88.35 | 94.8 | 98.2 |
81
- | 9 | [crossencoder-msmarco-electra-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-electra-base-mmarcoFR) | 25.52 | 42.46 | 78.73 | 88.85 | 96.55 | 98.85 |
82
- | 10 | [crossencoder-bert-base-uncased-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-bert-base-uncased-mmarcoFR) | 30.48 | 45.79 | 78.35 | 89.45 | 94.15 | 97.45 |
83
- | 11 | [crossencoder-msmarco-MiniLM-L-12-v2-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-MiniLM-L-12-v2-mmarcoFR) | 29.07 | 44.41 | 77.83 | 88.1 | 95.55 | 99 |
84
- | 12 | [crossencoder-msmarco-MiniLM-L-6-v2-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-MiniLM-L-6-v2-mmarcoFR) | 32.92 | 47.56 | 77.27 | 88.15 | 94.85 | 98.15 |
85
- | 13 | [crossencoder-msmarco-MiniLM-L-4-v2-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-MiniLM-L-4-v2-mmarcoFR) | 30.98 | 46.22 | 76.35 | 85.8 | 94.35 | 97.55 |
86
- | 14 | [crossencoder-MiniLM-L6-H384-uncased-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-MiniLM-L6-H384-uncased-mmarcoFR) | 29.23 | 45.12 | 76.08 | 83.7 | 92.65 | 97.45 |
87
- | 15 | [crossencoder-electra-base-discriminator-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-electra-base-discriminator-mmarcoFR) | 28.48 | 43.58 | 75.63 | 86.15 | 93.25 | 96.6 |
88
- | 16 | [crossencoder-electra-small-discriminator-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-electra-small-discriminator-mmarcoFR) | 31.83 | 45.97 | 75.13 | 84.95 | 94.55 | 98.15 |
89
- | 17 | [crossencoder-distilroberta-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-distilroberta-base-mmarcoFR) | 28.22 | 42.85 | 74.13 | 84.08 | 94.2 | 98.5 |
90
- | 18 | [crossencoder-msmarco-TinyBERT-L-6-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-TinyBERT-L-6-mmarcoFR) | 28.23 | 42.7 | 73.63 | 85.65 | 92.65 | 98.35 |
91
- | 19 | [crossencoder-msmarco-TinyBERT-L-4-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-TinyBERT-L-4-mmarcoFR) | 28.6 | 43.19 | 72.17 | 81.95 | 92.8 | 97.4 |
92
- | 20 | [crossencoder-msmarco-MiniLM-L-2-v2-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-MiniLM-L-2-v2-mmarcoFR) | 30.82 | 44.3 | 72.03 | 82.65 | 93.35 | 98.1 |
93
- | 21 | [crossencoder-distilbert-base-uncased-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-distilbert-base-uncased-mmarcoFR) | 25.47 | 40.11 | 71.37 | 85.6 | 93.85 | 97.95 |
94
- | 22 | [crossencoder-msmarco-TinyBERT-L-2-v2-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-msmarco-TinyBERT-L-2-v2-mmarcoFR) | 31.08 | 43.88 | 71.3 | 81.43 | 92.6 | 98.1 |
95
 
96
  ## Training
97
  ***
 
61
  ## Evaluation
62
  ***
63
 
64
+ We evaluated the model on 500 random queries from the mMARCO-fr train set (which were excluded from training). Each of these queries has at least one relevant and up to 200 irrelevant passages.
65
 
66
+ Below, we compare the model performance with other cross-encoder models fine-tuned on the same dataset. We report the R-precision (RP), mean reciprocal rank (MRR), and recall at various cut-offs (R@k).
 
 
67
 
68
+ | | model | RP | MRR@10 | R@10 (↑) | R@20 | R@50 | R@100 |
 
69
  |---:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------:|---------:|------------:|------------:|------------:|-------------:|
70
+ | 1 | [crossencoder-camembert-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-camembert-base-mmarcoFR) | 35.65 | 50.44 | 82.95 | 91.50 | 96.80 | 98.80 |
71
+ | 2 | [crossencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR) | 34.37 | 51.01 | 82.23 | 90.60 | 96.45 | 98.40 |
72
+ | 3 | [crossencoder-mmarcoFR-mMiniLMv2-L12-H384-v1-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mmarcoFR-mMiniLMv2-L12-H384-v1-mmarcoFR) | 34.22 | 49.20 | 81.70 | 90.90 | 97.10 | 98.90 |
73
+ | 4 | [crossencoder-mpnet-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mpnet-base-mmarcoFR) | 29.68 | 46.13 | 80.45 | 87.90 | 93.15 | 96.60 |
74
+ | 5 | [crossencoder-distilcamembert-base-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-distilcamembert-base-mmarcoFR) | 27.28 | 43.71 | 80.30 | 89.10 | 95.55 | 98.60 |
75
+ | 6 | **crossencoder-electra-base-french-europeana-cased-discriminator-mmarcoFR** | 28.32 | 45.28 | 79.22 | 87.15 | 93.15 | 95.75 |
76
+ | 7 | [crossencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/crossencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR) | 33.92 | 49.33 | 79.00 | 88.35 | 94.80 | 98.20 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77
 
78
  ## Training
79
  ***