Hugging Face's logo

antoinelouis
/
biencoder-camembert-L2-mmarcoFR

Sentence Similarity
sentence-transformers
PyTorch
Safetensors
French
camembert
passage-retrieval
Eval Results
Inference Endpoints
text-embeddings-inference
Model card Files Community
1
biencoder-camembert-L2-mmarcoFR / README.md
antoinelouis's picture
antoinelouis
Upload folder using huggingface_hub
df3c4b3 verified
preview code
|
raw history blame
No virus
14.4 kB
metadata
pipeline_tag: sentence-similarity
language: fr
license: mit
datasets:
  - unicamp-dl/mmarco
metrics:
  - recall
tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - transformers

biencoder-camembert-L2-mmarcoFR

This is a sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. The model was trained on the French portion of the mMARCO dataset.

Usage

Sentence-Transformers

Using this model becomes easy when you have sentence-transformers installed:

pip install -U sentence-transformers

Then you can use the model like this:

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('antoinelouis/biencoder-camembert-L2-mmarcoFR')
embeddings = model.encode(sentences)
print(embeddings)

🤗 Transformers

Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('antoinelouis/biencoder-camembert-L2-mmarcoFR')
model = AutoModel.from_pretrained('antoinelouis/biencoder-camembert-L2-mmarcoFR')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

Evaluation

Below, we compared its results with other biencoder models fine-tuned on the same dataset:

model MRR@10 NDCG@10 MAP@10 Recall@10 Recall@100 (↑) Recall@500
0 biencoder-camembert-base-mmarcoFR 28.53 33.72 27.93 51.46 77.82 89.13
1 biencoder-all-mpnet-base-v2-mmarcoFR 28.04 33.28 27.5 51.07 77.68 88.67
2 biencoder-multi-qa-mpnet-base-cos-v1-mmarcoFR 27.6 32.92 27.09 50.97 77.41 87.79
3 biencoder-sentence-camembert-base-mmarcoFR 27.63 32.7 27.01 50.1 76.85 88.73
4 biencoder-distilcamembert-base-mmarcoFR 26.8 31.87 26.23 49.2 76.44 87.87
5 biencoder-mpnet-base-mmarcoFR 27.2 32.22 26.63 49.41 75.71 86.88
6 biencoder-multi-qa-distilbert-cos-v1-mmarcoFR 26.36 31.26 25.82 47.93 75.42 86.78
7 biencoder-bert-base-uncased-mmarcoFR 26.3 31.14 25.74 47.67 74.57 86.33
8 biencoder-msmarco-distilbert-cos-v5-mmarcoFR 25.75 30.63 25.24 47.22 73.96 85.64
9 biencoder-all-distilroberta-v1-mmarcoFR 26.17 30.91 25.67 47.06 73.5 85.69
10 biencoder-all-MiniLM-L6-v2-mmarcoFR 25.49 30.39 24.99 47.1 73.48 86.09
11 biencoder-distilbert-base-uncased-mmarcoFR 25.18 29.83 24.64 45.77 73.16 85.13
12 biencoder-msmarco-MiniLM-L12-cos-v5-mmarcoFR 26.22 30.99 25.69 47.29 73.09 84.95
13 biencoder-roberta-base-mmarcoFR 25.94 30.72 25.43 46.98 73.07 84.76
14 biencoder-distiluse-base-multilingual-cased-v1-mmarcoFR 24.57 29.08 24.04 44.51 72.54 85.13
15 biencoder-multi-qa-MiniLM-L6-cos-v1-mmarcoFR 24.72 29.58 24.25 46.05 72.19 84.6
16 biencoder-MiniLM-L12-H384-uncased-mmarcoFR 25.43 30.1 24.88 46.13 72.16 83.84
17 biencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR 24.74 29.41 24.23 45.4 71.52 84.42
18 biencoder-electra-base-discriminator-mmarcoFR 24.77 29.37 24.21 45.2 70.84 83.25
19 biencoder-bert-medium-mmarcoFR 23.86 28.56 23.39 44.47 70.57 83.58
20 biencoder-msmarco-MiniLM-L6-cos-v5-mmarcoFR 24.39 28.96 23.91 44.58 70.36 82.88
21 biencoder-distilroberta-base-mmarcoFR 23.94 28.44 23.46 43.77 70.08 82.86
22 biencoder-camemberta-base-mmarcoFR 24.78 29.24 24.23 44.58 69.59 82.18
23 biencoder-electra-base-french-europeana-cased-discriminator-mmarcoFR 23.38 27.97 22.91 43.5 68.96 81.61
24 biencoder-bert-small-mmarcoFR 22.4 26.84 21.95 41.96 68.88 82.14
25 biencoder-mMiniLM-L6-v2-mmarcoFR-v2-mmarcoFR 22.87 27.26 22.37 42.3 68.78 81.39
26 biencoder-MiniLM-L6-H384-uncased-mmarcoFR 22.86 27.34 22.41 42.62 68.4 81.54
27 biencoder-deberta-v3-small-mmarcoFR 22.44 26.84 21.97 41.84 68.17 80.9
28 biencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR 22.29 26.57 21.8 41.25 66.78 79.83
29 biencoder-bert-mini-mmarcoFR 20.06 24.09 19.66 37.78 64.27 77.39
30 biencoder-electra-small-discriminator-mmarcoFR 20.32 24.36 19.9 38.16 63.98 77.23
31 biencoder-deberta-v3-xsmall-mmarcoFR 17.7 21.29 17.31 33.59 58.76 73.45
32 biencoder-bert-tiny-mmarcoFR 14.94 18.22 14.59 29.46 51.94 66.3
33 biencoder-t5-small-mmarcoFR 12.44 15.1 12.14 24.28 47.82 63.37
34 biencoder-bert-small-mmarcoFR 0.22 0.28 0.21 0.5 1.25 2.34

Training

Background

We used the output/-/camembert-L2 model and fine-tuned it on a 500K sentence pairs dataset in French. We used a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset. Formally, we compute the cos similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss with a temperature of 0.05 by comparing with true pairs.

Hyperparameters

We trained the model on a single Tesla V100 GPU with 32GBs of memory during 40 epochs (i.e., 9.8k steps) using a batch size of 2048. We used the AdamW optimizer with an initial learning rate of 2e-05, weight decay of 0.01, learning rate warmup over the first 976 steps, and linear decay of the learning rate. The sequence length was limited to 128 tokens.

Data

We used the French version of the mMARCO dataset to fine-tune our model. mMARCO is a multi-lingual machine-translated version of the MS MARCO dataset, a large-scale IR dataset comprising:

  • a corpus of 8.8M passages;
  • a training set of ~533k queries (with at least one relevant passage);
  • a development set of ~101k queries;
  • a smaller dev set of 6,980 queries (which is actually used for evaluation in most published works). Link: https://ir-datasets.com/mmarco.html#mmarco/v2/fr/

Citation 

@online{louis2023,
   author    = 'Antoine Louis',
   title     = 'biencoder-camembert-L2-mmarcoFR: A Biencoder Model Trained on French mMARCO',
   publisher = 'Hugging Face',
   month     = 'may',
   year      = '2023',
   url       = 'https://huggingface.co/antoinelouis/biencoder-camembert-L2-mmarcoFR',
}