cservan commited on
Commit
02d2ae0
1 Parent(s): 5a814f2

Add README

Browse files
Files changed (1) hide show
  1. README.md +142 -1
README.md CHANGED
@@ -1,3 +1,144 @@
1
  ---
2
- license: mit
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language: fr
3
+ license: apache-2.0
4
+ datasets:
5
+ - wikipedia
6
  ---
7
+
8
+ # mALBERT Base Cased 64k
9
+
10
+ Pretrained multilingual language model using a masked language modeling (MLM) objective. It was introduced in
11
+ [this paper](https://arxiv.org/abs/1909.11942) and first released in
12
+ [this repository](https://github.com/google-research/albert).
13
+ This model, unlike other ALBERT models, is cased: it does make a difference between french and French.
14
+
15
+ ## Model description
16
+
17
+ mALBERT is a transformers model pretrained on 16Go of French Wikipedia in a self-supervised fashion. This means it
18
+ was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
19
+ publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
20
+ was pretrained with two objectives:
21
+
22
+ - Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
23
+ the entire masked sentence through the model and has to predict the masked words. This is different from traditional
24
+ recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
25
+ GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
26
+ sentence.
27
+ - Sentence Ordering Prediction (SOP): mALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text.
28
+
29
+ This way, the model learns an inner representation of the languages that can then be used to extract features
30
+ useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
31
+ classifier using the features produced by the mALBERT model as inputs.
32
+
33
+ mALBERT is particular in that it shares its layers across its Transformer. Therefore, all layers have the same weights. Using repeating layers results in a small memory footprint, however, the computational cost remains similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same number of (repeating) layers.
34
+
35
+ This is the second version of the base model.
36
+
37
+ This model has the following configuration:
38
+
39
+ - 12 repeating layers
40
+ - 128 embedding dimension
41
+ - 768 hidden dimension
42
+ - 12 attention heads
43
+ - 11M parameters
44
+ - 64k of vocabulary size
45
+
46
+ ## Intended uses & limitations
47
+
48
+ You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
49
+ be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=malbert-base-cased-64k) to look for
50
+ fine-tuned versions on a task that interests you.
51
+
52
+ Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
53
+ to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
54
+ generation you should look at model like GPT2.
55
+
56
+ ### How to use
57
+
58
+ Here is how to use this model to get the features of a given text in PyTorch:
59
+
60
+ ```python
61
+ from transformers import AlbertTokenizer, AlbertModel
62
+ tokenizer = AlbertTokenizer.from_pretrained('cservan/malbert-base-cased-64k')
63
+ model = AlbertModel.from_pretrained("cservan/malbert-base-cased-64k")
64
+ text = "Remplacez-moi par le texte en français que vous souhaitez."
65
+ encoded_input = tokenizer(text, return_tensors='pt')
66
+ output = model(**encoded_input)
67
+ ```
68
+
69
+ and in TensorFlow:
70
+
71
+ ```python
72
+ from transformers import AlbertTokenizer, TFAlbertModel
73
+ tokenizer = AlbertTokenizer.from_pretrained('cservan/malbert-base-cased-64k')
74
+ model = TFAlbertModel.from_pretrained("cservan/malbert-base-cased-64k")
75
+ text = "Remplacez-moi par le texte en français que vous souhaitez."
76
+ encoded_input = tokenizer(text, return_tensors='tf')
77
+ output = model(encoded_input)
78
+ ```
79
+
80
+
81
+ ## Training data
82
+
83
+ The mALBERT model was pretrained on 4go of [French Wikipedia](https://fr.wikipedia.org/wiki/French_Wikipedia) (excluding lists, tables and
84
+ headers).
85
+
86
+ ## Training procedure
87
+
88
+ ### Preprocessing
89
+
90
+ The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 64,000. The inputs of the model are
91
+ then of the form:
92
+
93
+ ```
94
+ [CLS] Sentence A [SEP] Sentence B [SEP]
95
+ ```
96
+
97
+ ### Training
98
+
99
+ The mALBERT procedure follows the BERT setup.
100
+
101
+ The details of the masking procedure for each sentence are the following:
102
+ - 15% of the tokens are masked.
103
+ - In 80% of the cases, the masked tokens are replaced by `[MASK]`.
104
+ - In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
105
+ - In the 10% remaining cases, the masked tokens are left as is.
106
+
107
+ ## Evaluation results
108
+
109
+ When fine-tuned on downstream tasks, the ALBERT models achieve the following results:
110
+
111
+ Slot-filling:
112
+
113
+ |Models ⧹ Tasks | MMNLU | MultiATIS++ | CoNLL2003 | MultiCoNER | SNIPS | MEDIA |
114
+ |---------------|--------------|--------------|--------------|--------------|--------------|--------------|
115
+ |EnALBERT | N/A | N/A | 89.67 (0.34) | 42.36 (0.22) | 95.95 (0.13) | N/A |
116
+ |FrALBERT | N/A | N/A | N/A | N/A | N/A | 81.76 (0.59)
117
+ |mALBERT-128k | 65.81 (0.11) | 89.14 (0.15) | 88.27 (0.24) | 46.01 (0.18) | 91.60 (0.31) | 83.15 (0.38) |
118
+ |mALBERT-64k | 65.29 (0.14) | 88.88 (0.14) | 86.44 (0.37) | 44.70 (0.27) | 90.84 (0.47) | 82.30 (0.19) |
119
+ |mALBERT-32k | 64.83 (0.22) | 88.60 (0.27) | 84.96 (0.41) | 44.13 (0.39) | 89.89 (0.68) | 82.04 (0.28) |
120
+
121
+ Classification task:
122
+
123
+ |Models ⧹ Tasks | MMNLU | MultiATIS++ | SNIPS | SST2 |
124
+ |---------------|--------------|--------------|--------------|--------------|
125
+ |mALBERT-128k | 72.35 (0.09) | 90.58 (0.98) | 96.84 (0.49) | 34.66 (1.46) |
126
+ |mALBERT-64k | 71.26 (0.11) | 90.97 (0.70) | 96.53 (0.44) | 34.64 (1.02) |
127
+ |mALBERT-32k | 70.76 (0.11) | 90.55 (0.98) | 96.49 (0.45) | 34.18 (1.64) |
128
+
129
+ ### BibTeX entry and citation info
130
+
131
+ ```bibtex
132
+ @inproceedings{servan2024mALBERT,
133
+ author = {Christophe Servan and
134
+ Sahar Ghannay and
135
+ Sophie Rosset},
136
+ booktitle = {the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
137
+ title = {{mALBERT: Is a Compact Multilingual BERT Model Still Worth It?}},
138
+ year = {2024},
139
+ address = {Torino, Italy},
140
+ month = may,
141
+ }
142
+ ```
143
+
144
+ Link to the paper: [PDF](https://hal.science/hal-04520797)