Commit
•
389c149
1
Parent(s):
420a6a5
Update README.md
Browse files
README.md
CHANGED
|
@@ -38,8 +38,6 @@ embeddings = model.encode(sentences)
|
|
| 38 |
print(embeddings)
|
| 39 |
```
|
| 40 |
|
| 41 |
-
|
| 42 |
-
|
| 43 |
#### 🤗 Transformers
|
| 44 |
|
| 45 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
|
@@ -77,24 +75,21 @@ print("Sentence embeddings:")
|
|
| 77 |
print(sentence_embeddings)
|
| 78 |
```
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
## Evaluation
|
| 83 |
***
|
| 84 |
|
| 85 |
We evaluated our model on the smaller development set of mMARCO-fr, which consists of 6,980 queries for a corpus of 8.8M candidate passages. Below, we compared the model performance with other biencoder models fine-tuned on the same dataset. We report the mean reciprocal rank (MRR), normalized discounted cumulative gainand (NDCG), mean average precision (MAP), and recall at various cut-offs (R@k).
|
| 86 |
|
| 87 |
-
| | model
|
| 88 |
-
|
| 89 |
-
| 1 | **biencoder-camembert-base-mmarcoFR**
|
| 90 |
-
| 2 | [biencoder-
|
| 91 |
-
| 3 | [biencoder-
|
| 92 |
-
| 4 | [biencoder-
|
| 93 |
-
| 5 | [biencoder-mMiniLMv2-L12-
|
| 94 |
-
| 6 | [biencoder-camemberta-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-camemberta-base-mmarcoFR)
|
| 95 |
-
| 7 | [biencoder-electra-base-french-
|
| 96 |
-
| 8 | [biencoder-
|
| 97 |
-
| 9 | [biencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR) | 428MB | 22.29 | 26.57 | 21.8 | 41.25 | 66.78 | 79.83 |
|
| 98 |
|
| 99 |
## Training
|
| 100 |
***
|
|
@@ -116,8 +111,6 @@ We used the French version of the [mMARCO](https://huggingface.co/datasets/unica
|
|
| 116 |
- a smaller dev set of 6,980 queries (which is actually used for evaluation in most published works).
|
| 117 |
Link: [https://ir-datasets.com/mmarco.html#mmarco/v2/fr/](https://ir-datasets.com/mmarco.html#mmarco/v2/fr/)
|
| 118 |
|
| 119 |
-
|
| 120 |
-
|
| 121 |
## Citation
|
| 122 |
|
| 123 |
```bibtex
|
|
|
|
| 38 |
print(embeddings)
|
| 39 |
```
|
| 40 |
|
|
|
|
|
|
|
| 41 |
#### 🤗 Transformers
|
| 42 |
|
| 43 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
|
|
|
| 75 |
print(sentence_embeddings)
|
| 76 |
```
|
| 77 |
|
|
|
|
|
|
|
| 78 |
## Evaluation
|
| 79 |
***
|
| 80 |
|
| 81 |
We evaluated our model on the smaller development set of mMARCO-fr, which consists of 6,980 queries for a corpus of 8.8M candidate passages. Below, we compared the model performance with other biencoder models fine-tuned on the same dataset. We report the mean reciprocal rank (MRR), normalized discounted cumulative gainand (NDCG), mean average precision (MAP), and recall at various cut-offs (R@k).
|
| 82 |
|
| 83 |
+
| | model | Vocab. | #Param. | Size | MRR@10 | NDCG@10 | MAP@10 | R@10 | R@100(↑) | R@500 |
|
| 84 |
+
|---:|:------------------------------------------------------------------------------------------------------------------------|:-------|--------:|------:|---------:|----------:|---------:|-------:|-----------:|--------:|
|
| 85 |
+
| 1 | **biencoder-camembert-base-mmarcoFR** | 🇫🇷 | 110M | 443MB | 28.53 | 33.72 | 27.93 | 51.46 | 77.82 | 89.13 |
|
| 86 |
+
| 2 | [biencoder-mpnet-base-all-v2-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mpnet-base-all-v2-mmarcoFR) | 🇬🇧 | 109M | 438MB | 28.04 | 33.28 | 27.50 | 51.07 | 77.68 | 88.67 |
|
| 87 |
+
| 3 | [biencoder-distilcamembert-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-distilcamembert-mmarcoFR) | 🇫🇷 | 68M | 272MB | 26.80 | 31.87 | 26.23 | 49.20 | 76.44 | 87.87 |
|
| 88 |
+
| 4 | [biencoder-MiniLM-L6-all-v2-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-MiniLM-L6-all-v2-mmarcoFR) | 🇬🇧 | 23M | 91MB | 25.49 | 30.39 | 24.99 | 47.10 | 73.48 | 86.09 |
|
| 89 |
+
| 5 | [biencoder-mMiniLMv2-L12-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLMv2-L12-mmarcoFR) | 🇫🇷,99+ | 117M | 471MB | 24.74 | 29.41 | 24.23 | 45.40 | 71.52 | 84.42 |
|
| 90 |
+
| 6 | [biencoder-camemberta-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-camemberta-base-mmarcoFR) | 🇫🇷 | 112M | 447MB | 24.78 | 29.24 | 24.23 | 44.58 | 69.59 | 82.18 |
|
| 91 |
+
| 7 | [biencoder-electra-base-french-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-electra-base-french-mmarcoFR) | 🇫🇷 | 110M | 440MB | 23.38 | 27.97 | 22.91 | 43.50 | 68.96 | 81.61 |
|
| 92 |
+
| 8 | [biencoder-mMiniLMv2-L6-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLMv2-L6-mmarcoFR) | 🇫🇷,99+ | 107M | 428MB | 22.29 | 26.57 | 21.80 | 41.25 | 66.78 | 79.83 |
|
|
|
|
| 93 |
|
| 94 |
## Training
|
| 95 |
***
|
|
|
|
| 111 |
- a smaller dev set of 6,980 queries (which is actually used for evaluation in most published works).
|
| 112 |
Link: [https://ir-datasets.com/mmarco.html#mmarco/v2/fr/](https://ir-datasets.com/mmarco.html#mmarco/v2/fr/)
|
| 113 |
|
|
|
|
|
|
|
| 114 |
## Citation
|
| 115 |
|
| 116 |
```bibtex
|