|
--- |
|
pipeline_tag: sentence-similarity |
|
language: fr |
|
datasets: |
|
- stsb_multi_mt |
|
tags: |
|
- Text |
|
- Sentence Similarity |
|
- Sentence-Embedding |
|
- camembert-base |
|
license: apache-2.0 |
|
model-index: |
|
- name: sentence-camembert-base by Van Tuan DANG |
|
results: |
|
- task: |
|
name: Sentence-Embedding |
|
type: Text Similarity |
|
dataset: |
|
name: Text Similarity fr |
|
type: stsb_multi_mt |
|
args: fr |
|
metrics: |
|
- name: Test Pearson correlation coefficient |
|
type: Pearson_correlation_coefficient |
|
value: xx.xx |
|
--- |
|
|
|
## Pre-trained sentence embedding models are the state-of-the-art of Sentence Embeddings for French. |
|
Model is Fine-tuned using pre-trained [facebook/camembert-base](https://huggingface.co/camembert/camembert-base) and |
|
[Siamese BERT-Networks with 'sentences-transformers'](https://www.sbert.net/) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) |
|
|
|
|
|
## Usage |
|
The model can be used directly (without a language model) as follows: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
model = SentenceTransformer("dangvantuan/sentence-camembert-base") |
|
|
|
sentences = ["Un avion est en train de décoller.", |
|
"Un homme joue d'une grande flûte.", |
|
"Un homme étale du fromage râpé sur une pizza.", |
|
"Une personne jette un chat au plafond.", |
|
"Une personne est en train de plier un morceau de papier.", |
|
] |
|
|
|
embeddings = model.encode(sentences) |
|
``` |
|
|
|
## Evaluation |
|
The model can be evaluated as follows on the French test data of stsb. |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.readers import InputExample |
|
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator |
|
from datasets import load_dataset |
|
def convert_dataset(dataset): |
|
dataset_samples=[] |
|
for df in dataset: |
|
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1 |
|
inp_example = InputExample(texts=[df['sentence1'], |
|
df['sentence2']], label=score) |
|
dataset_samples.append(inp_example) |
|
return dataset_samples |
|
|
|
# Loading the dataset for evaluation |
|
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev") |
|
df_test = load_dataset("stsb_multi_mt", name="fr", split="test") |
|
|
|
# Convert the dataset for evaluation |
|
|
|
# For Dev set: |
|
dev_samples = convert_dataset(df_dev) |
|
val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev') |
|
val_evaluator(model, output_path="./") |
|
|
|
# For Test set: |
|
test_samples = convert_dataset(df_test) |
|
test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test') |
|
test_evaluator(model, output_path="./") |
|
``` |
|
|
|
**Test Result**: |
|
The performance is measured using Pearson and Spearman correlation: |
|
- On dev |
|
|
|
|
|
| Model | Pearson correlation | Spearman correlation | #params | |
|
| ------------- | ------------- | ------------- |------------- | |
|
| [dangvantuan/sentence-camembert-base](https://huggingface.co/dangvantuan/sentence-camembert-base)| 86.73 |86.54 | 110M | |
|
| [distiluse-base-multilingual-cased](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased) | 79.22 | 79.16|135M | |
|
- On test |
|
|
|
|
|
| Model | Pearson correlation | Spearman correlation | |
|
| ------------- | ------------- | ------------- | |
|
| [dangvantuan/sentence-camembert-base](https://huggingface.co/dangvantuan/sentence-camembert-base)| 82.36 | 81.64| |
|
| [distiluse-base-multilingual-cased](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased) | 78.62 | 77.48| |
|
|
|
|
|
## Citation |
|
|
|
|
|
@article{reimers2019sentence, |
|
title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, |
|
author={Nils Reimers, Iryna Gurevych}, |
|
journal={https://arxiv.org/abs/1908.10084}, |
|
year={2019} |
|
} |
|
|
|
|
|
@article{martin2020camembert, |
|
title={CamemBERT: a Tasty French Language Mode}, |
|
author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t}, |
|
journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, |
|
year={2020} |
|
} |
