antoinelouis commited on
Commit
cd2e324
1 Parent(s): c50fdf1

Upload folder using huggingface_hub

Browse files
1_Pooling/config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": false
9
+ }
README.md ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: sentence-similarity
3
+ language: fr
4
+ license: apache-2.0
5
+ datasets:
6
+ - unicamp-dl/mmarco
7
+ metrics:
8
+ - recall
9
+
10
+ tags:
11
+ - sentence-transformers
12
+ - feature-extraction
13
+ - sentence-similarity
14
+ - transformers
15
+
16
+ ---
17
+
18
+ # biencoder-camembert-L10-mmarcoFR
19
+
20
+ This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. The model was trained on the **French** portion of the [mMARCO](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset.
21
+
22
+ ## Usage
23
+ ***
24
+
25
+ #### Sentence-Transformers
26
+
27
+ Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
28
+
29
+ ```
30
+ pip install -U sentence-transformers
31
+ ```
32
+
33
+ Then you can use the model like this:
34
+
35
+ ```python
36
+ from sentence_transformers import SentenceTransformer
37
+ sentences = ["This is an example sentence", "Each sentence is converted"]
38
+
39
+ model = SentenceTransformer('antoinelouis/biencoder-camembert-L10-mmarcoFR')
40
+ embeddings = model.encode(sentences)
41
+ print(embeddings)
42
+ ```
43
+
44
+
45
+
46
+ #### 🤗 Transformers
47
+
48
+ Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
49
+
50
+ ```python
51
+ from transformers import AutoTokenizer, AutoModel
52
+ import torch
53
+
54
+
55
+ #Mean Pooling - Take attention mask into account for correct averaging
56
+ def mean_pooling(model_output, attention_mask):
57
+ token_embeddings = model_output[0] #First element of model_output contains all token embeddings
58
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
59
+ return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
60
+
61
+
62
+ # Sentences we want sentence embeddings for
63
+ sentences = ['This is an example sentence', 'Each sentence is converted']
64
+
65
+ # Load model from HuggingFace Hub
66
+ tokenizer = AutoTokenizer.from_pretrained('antoinelouis/biencoder-camembert-L10-mmarcoFR')
67
+ model = AutoModel.from_pretrained('antoinelouis/biencoder-camembert-L10-mmarcoFR')
68
+
69
+ # Tokenize sentences
70
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
71
+
72
+ # Compute token embeddings
73
+ with torch.no_grad():
74
+ model_output = model(**encoded_input)
75
+
76
+ # Perform pooling. In this case, mean pooling.
77
+ sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
78
+
79
+ print("Sentence embeddings:")
80
+ print(sentence_embeddings)
81
+ ```
82
+
83
+
84
+
85
+ ## Evaluation
86
+ ***
87
+
88
+
89
+
90
+ <!--- Describe how your model was evaluated -->
91
+
92
+ Below, we compared its results with other biencoder models fine-tuned on the same dataset:
93
+ | | model | MRR@10 | NDCG@10 | MAP@10 | Recall@10 | Recall@100 (↑) | Recall@500 |
94
+ |---:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------:|----------:|---------:|------------:|-------------:|-------------:|
95
+ | 0 | [biencoder-camembert-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-camembert-base-mmarcoFR) | 28.53 | 33.72 | 27.93 | 51.46 | 77.82 | 89.13 |
96
+ | 1 | [biencoder-all-mpnet-base-v2-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-all-mpnet-base-v2-mmarcoFR) | 28.04 | 33.28 | 27.5 | 51.07 | 77.68 | 88.67 |
97
+ | 2 | [biencoder-multi-qa-mpnet-base-cos-v1-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-multi-qa-mpnet-base-cos-v1-mmarcoFR) | 27.6 | 32.92 | 27.09 | 50.97 | 77.41 | 87.79 |
98
+ | 3 | [biencoder-sentence-camembert-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-sentence-camembert-base-mmarcoFR) | 27.63 | 32.7 | 27.01 | 50.1 | 76.85 | 88.73 |
99
+ | 4 | [biencoder-distilcamembert-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-distilcamembert-base-mmarcoFR) | 26.8 | 31.87 | 26.23 | 49.2 | 76.44 | 87.87 |
100
+ | 5 | [biencoder-mpnet-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mpnet-base-mmarcoFR) | 27.2 | 32.22 | 26.63 | 49.41 | 75.71 | 86.88 |
101
+ | 6 | [biencoder-multi-qa-distilbert-cos-v1-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-multi-qa-distilbert-cos-v1-mmarcoFR) | 26.36 | 31.26 | 25.82 | 47.93 | 75.42 | 86.78 |
102
+ | 7 | [biencoder-bert-base-uncased-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-base-uncased-mmarcoFR) | 26.3 | 31.14 | 25.74 | 47.67 | 74.57 | 86.33 |
103
+ | 8 | [biencoder-msmarco-distilbert-cos-v5-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-msmarco-distilbert-cos-v5-mmarcoFR) | 25.75 | 30.63 | 25.24 | 47.22 | 73.96 | 85.64 |
104
+ | 9 | [biencoder-all-distilroberta-v1-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-all-distilroberta-v1-mmarcoFR) | 26.17 | 30.91 | 25.67 | 47.06 | 73.5 | 85.69 |
105
+ | 10 | [biencoder-all-MiniLM-L6-v2-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-all-MiniLM-L6-v2-mmarcoFR) | 25.49 | 30.39 | 24.99 | 47.1 | 73.48 | 86.09 |
106
+ | 11 | [biencoder-distilbert-base-uncased-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-distilbert-base-uncased-mmarcoFR) | 25.18 | 29.83 | 24.64 | 45.77 | 73.16 | 85.13 |
107
+ | 12 | [biencoder-msmarco-MiniLM-L12-cos-v5-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-msmarco-MiniLM-L12-cos-v5-mmarcoFR) | 26.22 | 30.99 | 25.69 | 47.29 | 73.09 | 84.95 |
108
+ | 13 | [biencoder-roberta-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-roberta-base-mmarcoFR) | 25.94 | 30.72 | 25.43 | 46.98 | 73.07 | 84.76 |
109
+ | 14 | [biencoder-distiluse-base-multilingual-cased-v1-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-distiluse-base-multilingual-cased-v1-mmarcoFR) | 24.57 | 29.08 | 24.04 | 44.51 | 72.54 | 85.13 |
110
+ | 15 | [biencoder-multi-qa-MiniLM-L6-cos-v1-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-multi-qa-MiniLM-L6-cos-v1-mmarcoFR) | 24.72 | 29.58 | 24.25 | 46.05 | 72.19 | 84.6 |
111
+ | 16 | [biencoder-MiniLM-L12-H384-uncased-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-MiniLM-L12-H384-uncased-mmarcoFR) | 25.43 | 30.1 | 24.88 | 46.13 | 72.16 | 83.84 |
112
+ | 17 | [biencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLMv2-L12-H384-distilled-from-XLMR-Large-mmarcoFR) | 24.74 | 29.41 | 24.23 | 45.4 | 71.52 | 84.42 |
113
+ | 18 | [biencoder-electra-base-discriminator-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-electra-base-discriminator-mmarcoFR) | 24.77 | 29.37 | 24.21 | 45.2 | 70.84 | 83.25 |
114
+ | 19 | [biencoder-bert-medium-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-medium-mmarcoFR) | 23.86 | 28.56 | 23.39 | 44.47 | 70.57 | 83.58 |
115
+ | 20 | [biencoder-msmarco-MiniLM-L6-cos-v5-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-msmarco-MiniLM-L6-cos-v5-mmarcoFR) | 24.39 | 28.96 | 23.91 | 44.58 | 70.36 | 82.88 |
116
+ | 21 | [biencoder-distilroberta-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-distilroberta-base-mmarcoFR) | 23.94 | 28.44 | 23.46 | 43.77 | 70.08 | 82.86 |
117
+ | 22 | [biencoder-camemberta-base-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-camemberta-base-mmarcoFR) | 24.78 | 29.24 | 24.23 | 44.58 | 69.59 | 82.18 |
118
+ | 23 | [biencoder-electra-base-french-europeana-cased-discriminator-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-electra-base-french-europeana-cased-discriminator-mmarcoFR) | 23.38 | 27.97 | 22.91 | 43.5 | 68.96 | 81.61 |
119
+ | 24 | [biencoder-bert-small-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-small-mmarcoFR) | 22.4 | 26.84 | 21.95 | 41.96 | 68.88 | 82.14 |
120
+ | 25 | [biencoder-mMiniLM-L6-v2-mmarcoFR-v2-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLM-L6-v2-mmarcoFR-v2-mmarcoFR) | 22.87 | 27.26 | 22.37 | 42.3 | 68.78 | 81.39 |
121
+ | 26 | [biencoder-MiniLM-L6-H384-uncased-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-MiniLM-L6-H384-uncased-mmarcoFR) | 22.86 | 27.34 | 22.41 | 42.62 | 68.4 | 81.54 |
122
+ | 27 | [biencoder-deberta-v3-small-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-deberta-v3-small-mmarcoFR) | 22.44 | 26.84 | 21.97 | 41.84 | 68.17 | 80.9 |
123
+ | 28 | [biencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-mMiniLMv2-L6-H384-distilled-from-XLMR-Large-mmarcoFR) | 22.29 | 26.57 | 21.8 | 41.25 | 66.78 | 79.83 |
124
+ | 29 | [biencoder-bert-mini-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-mini-mmarcoFR) | 20.06 | 24.09 | 19.66 | 37.78 | 64.27 | 77.39 |
125
+ | 30 | [biencoder-electra-small-discriminator-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-electra-small-discriminator-mmarcoFR) | 20.32 | 24.36 | 19.9 | 38.16 | 63.98 | 77.23 |
126
+ | 31 | [biencoder-deberta-v3-xsmall-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-deberta-v3-xsmall-mmarcoFR) | 17.7 | 21.29 | 17.31 | 33.59 | 58.76 | 73.45 |
127
+ | 32 | [biencoder-bert-tiny-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-tiny-mmarcoFR) | 14.94 | 18.22 | 14.59 | 29.46 | 51.94 | 66.3 |
128
+ | 33 | [biencoder-t5-small-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-t5-small-mmarcoFR) | 12.44 | 15.1 | 12.14 | 24.28 | 47.82 | 63.37 |
129
+ | 34 | [biencoder-bert-small-mmarcoFR](https://huggingface.co/antoinelouis/biencoder-bert-small-mmarcoFR) | 0.22 | 0.28 | 0.21 | 0.5 | 1.25 | 2.34 |
130
+
131
+
132
+
133
+ ## Training
134
+ ***
135
+
136
+ #### Background
137
+
138
+ We used the [output/-/camembert-L10](https://huggingface.co/output/-/camembert-L10) model and fine-tuned it on a 500K sentence pairs dataset in French. We used a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset. Formally, we compute the cos similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss with a temperature of 0.05 by comparing with true pairs.
139
+
140
+ #### Hyperparameters
141
+
142
+ We trained the model on a single Tesla V100 GPU with 32GBs of memory during 40 epochs (i.e., 52.0k steps) using a batch size of 384. We used the AdamW optimizer with an initial learning rate of 2e-05, weight decay of 0.01, learning rate warmup over the first 5200 steps, and linear decay of the learning rate. The sequence length was limited to 128 tokens.
143
+
144
+ #### Data
145
+
146
+ We used the French version of the [mMARCO](https://huggingface.co/datasets/unicamp-dl/mmarco) dataset to fine-tune our model. mMARCO is a multi-lingual machine-translated version of the MS MARCO dataset, a large-scale IR dataset comprising:
147
+ - a corpus of 8.8M passages;
148
+ - a training set of ~533k queries (with at least one relevant passage);
149
+ - a development set of ~101k queries;
150
+ - a smaller dev set of 6,980 queries (which is actually used for evaluation in most published works).
151
+ Link: [https://ir-datasets.com/mmarco.html#mmarco/v2/fr/](https://ir-datasets.com/mmarco.html#mmarco/v2/fr/)
152
+
153
+
154
+
155
+ ## Citation
156
+
157
+ ```bibtex
158
+ @online{louis2023,
159
+ author = 'Antoine Louis',
160
+ title = 'biencoder-camembert-L10-mmarcoFR: A Biencoder Model Trained on French mMARCO',
161
+ publisher = 'Hugging Face',
162
+ month = 'may',
163
+ year = '2023',
164
+ url = 'https://huggingface.co/antoinelouis/biencoder-camembert-L10-mmarcoFR',
165
+ }
166
+ ```
added_tokens.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</s>": 6,
3
+ "</s>NOTUSED": 2,
4
+ "<mask>": 32004,
5
+ "<pad>": 1,
6
+ "<s>": 5,
7
+ "<s>NOTUSED": 0,
8
+ "<unk>": 4
9
+ }
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "output/-/camembert-L10",
3
+ "architectures": [
4
+ "CamembertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": 5,
8
+ "classifier_dropout": null,
9
+ "eos_token_id": 6,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-05,
16
+ "max_position_embeddings": 514,
17
+ "model_type": "camembert",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 10,
20
+ "output_past": true,
21
+ "pad_token_id": 1,
22
+ "position_embedding_type": "absolute",
23
+ "torch_dtype": "float32",
24
+ "transformers_version": "4.35.0.dev0",
25
+ "type_vocab_size": 1,
26
+ "use_cache": true,
27
+ "vocab_size": 32005
28
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "2.2.2",
4
+ "transformers": "4.35.0.dev0",
5
+ "pytorch": "2.1.0+cu121"
6
+ }
7
+ }
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f871bc3833eab7a1e7fcea3adbafb70a56ab5134151109c766249d967bcdaa2
3
+ size 385841190
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 128,
3
+ "do_lower_case": false
4
+ }
sentencepiece.bpe.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:988bc5a00281c6d210a5d34bd143d0363741a432fefe741bf71e61b1869d4314
3
+ size 810912
special_tokens_map.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<s>NOTUSED",
4
+ "<pad>",
5
+ "</s>NOTUSED",
6
+ "<unk>",
7
+ "<s>",
8
+ "</s>",
9
+ "<mask>"
10
+ ],
11
+ "bos_token": "<s>",
12
+ "cls_token": "<s>",
13
+ "eos_token": "</s>",
14
+ "mask_token": "<mask>",
15
+ "pad_token": "<pad>",
16
+ "sep_token": "</s>",
17
+ "unk_token": "<unk>"
18
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>NOTUSED",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>NOTUSED",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "4": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "5": {
36
+ "content": "<s>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "6": {
44
+ "content": "</s>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ },
51
+ "32004": {
52
+ "content": "<mask>",
53
+ "lstrip": true,
54
+ "normalized": false,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": true
58
+ }
59
+ },
60
+ "additional_special_tokens": [
61
+ "<s>NOTUSED",
62
+ "<pad>",
63
+ "</s>NOTUSED",
64
+ "<unk>",
65
+ "<s>",
66
+ "</s>",
67
+ "<mask>"
68
+ ],
69
+ "bos_token": "<s>",
70
+ "clean_up_tokenization_spaces": true,
71
+ "cls_token": "<s>",
72
+ "eos_token": "</s>",
73
+ "mask_token": "<mask>",
74
+ "model_max_length": 128,
75
+ "pad_token": "<pad>",
76
+ "sep_token": "</s>",
77
+ "tokenizer_class": "CamembertTokenizer",
78
+ "unk_token": "<unk>"
79
+ }