sgugger Marissa commited on
Commit
dafb8ab
1 Parent(s): d5f927b

Add model card (#1)

Browse files

- Add model card (ce151e2111d25abf840d9e3c16570f53c50613ff)
- Update README.md (8170ea6e8b18f660ee3ea915006619e8ab838f4f)
- Update README.md (b8f0f9ed6e98177246f7c5bb014af410a767a53d)


Co-authored-by: Marissa Gerchick <Marissa@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +215 -0
README.md ADDED
@@ -0,0 +1,215 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ - en
5
+ - es
6
+ - fr
7
+ - de
8
+ - zh
9
+ - ru
10
+ - pt
11
+ - it
12
+ - ar
13
+ - ja
14
+ - id
15
+ - tr
16
+ - nl
17
+ - pl
18
+ - fa
19
+ - vi
20
+ - sv
21
+ - ko
22
+ - he
23
+ - ro
24
+ - no
25
+ - hi
26
+ - uk
27
+ - cs
28
+ - fi
29
+ - hu
30
+ - th
31
+ - da
32
+ - ca
33
+ - el
34
+ - bg
35
+ - sr
36
+ - ms
37
+ - bn
38
+ - hr
39
+ - sl
40
+ - az
41
+ - sk
42
+ - eo
43
+ - ta
44
+ - sh
45
+ - lt
46
+ - et
47
+ - ml
48
+ - la
49
+ - bs
50
+ - sq
51
+ - arz
52
+ - af
53
+ - ka
54
+ - mr
55
+ - eu
56
+ - tl
57
+ - ang
58
+ - gl
59
+ - nn
60
+ - ur
61
+ - kk
62
+ - be
63
+ - hy
64
+ - te
65
+ - lv
66
+ - mk
67
+ - als
68
+ - is
69
+ - wuu
70
+ - my
71
+ - sco
72
+ - mn
73
+ - ceb
74
+ - ast
75
+ - cy
76
+ - kn
77
+ - br
78
+ - an
79
+ - gu
80
+ - bar
81
+ - uz
82
+ - lb
83
+ - ne
84
+ - si
85
+ - war
86
+ - jv
87
+ - ga
88
+ - oc
89
+ - ku
90
+ - sw
91
+ - nds
92
+ - ckb
93
+ - ia
94
+ - yi
95
+ - fy
96
+ - scn
97
+ - gan
98
+ - tt
99
+ - am
100
+ license: cc-by-nc-4.0
101
+ ---
102
+
103
+ # xlm-mlm-100-1280
104
+
105
+ # Table of Contents
106
+
107
+ 1. [Model Details](#model-details)
108
+ 2. [Uses](#uses)
109
+ 3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
110
+ 4. [Training](#training)
111
+ 5. [Evaluation](#evaluation)
112
+ 6. [Environmental Impact](#environmental-impact)
113
+ 7. [Technical Specifications](#technical-specifications)
114
+ 8. [Citation](#citation)
115
+ 9. [Model Card Authors](#model-card-authors)
116
+ 10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
117
+
118
+
119
+ # Model Details
120
+
121
+ xlm-mlm-100-1280 is the XLM model, which was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau, trained on Wikipedia text in 100 languages. The model is a transformer pretrained using a masked language modeling (MLM) objective.
122
+
123
+ ## Model Description
124
+
125
+ - **Developed by:** See [associated paper](https://arxiv.org/abs/1901.07291) and [GitHub Repo](https://github.com/facebookresearch/XLM)
126
+ - **Model type:** Language model
127
+ - **Language(s) (NLP):** 100 languages, see [GitHub Repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages) for full list.
128
+ - **License:** CC-BY-NC-4.0
129
+ - **Related Models:** [xlm-mlm-17-1280](https://huggingface.co/xlm-mlm-17-1280)
130
+ - **Resources for more information:**
131
+ - [Associated paper](https://arxiv.org/abs/1901.07291)
132
+ - [GitHub Repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages)
133
+ - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
134
+
135
+ # Uses
136
+
137
+ ## Direct Use
138
+
139
+ The model is a language model. The model can be used for masked language modeling.
140
+
141
+ ## Downstream Use
142
+
143
+ To learn more about this task and potential downstream uses, see the Hugging Face [fill mask docs](https://huggingface.co/tasks/fill-mask) and the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs. Also see the [associated paper](https://arxiv.org/abs/1901.07291).
144
+
145
+ ## Out-of-Scope Use
146
+
147
+ The model should not be used to intentionally create hostile or alienating environments for people.
148
+
149
+ # Bias, Risks, and Limitations
150
+
151
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
152
+
153
+ ## Recommendations
154
+
155
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
156
+
157
+ # Training
158
+
159
+ This model is the XLM model trained on Wikipedia text in 100 languages. The preprocessing included tokenization with byte-pair-encoding. See the [GitHub repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages) and the [associated paper](https://arxiv.org/pdf/1911.02116.pdf) for further details on the training data and training procedure.
160
+
161
+ [Conneau et al. (2020)](https://arxiv.org/pdf/1911.02116.pdf) report that this model has 16 layers, 1280 hidden states, 16 attention heads, and the dimension of the feed-forward layer is 1520. The vocabulary size is 200k and the total number of parameters is 570M (see Table 7).
162
+
163
+ # Evaluation
164
+
165
+ ## Testing Data, Factors & Metrics
166
+
167
+ The model developers evaluated the model on the XNLI cross-lingual classification task (see the [XNLI data card](https://huggingface.co/datasets/xnli) for more details on XNLI) using the metric of test accuracy. See the [GitHub Repo](https://arxiv.org/pdf/1911.02116.pdf) for further details on the testing data, factors and metrics.
168
+
169
+ ## Results
170
+
171
+ For xlm-mlm-100-1280, the test accuracy on the XNLI cross-lingual classification task in English (en), Spanish (es), German (de), Arabic (ar), Chinese (zh) and Urdu (ur) are:
172
+
173
+ |Language| en | es | de | ar | zh | ur |
174
+ |:------:|:--:|:---:|:--:|:--:|:--:|:--:|
175
+ | |83.7|76.6 |73.6|67.4|71.7|62.9|
176
+
177
+ See the [GitHub repo](https://github.com/facebookresearch/XLM#ii-cross-lingual-language-model-pretraining-xlm) for further details.
178
+
179
+ # Environmental Impact
180
+
181
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
182
+
183
+ - **Hardware Type:** More information needed
184
+ - **Hours used:** More information needed
185
+ - **Cloud Provider:** More information needed
186
+ - **Compute Region:** More information needed
187
+ - **Carbon Emitted:** More information needed
188
+
189
+ # Technical Specifications
190
+
191
+ [Conneau et al. (2020)](https://arxiv.org/pdf/1911.02116.pdf) report that this model has 16 layers, 1280 hidden states, 16 attention heads, and the dimension of the feed-forward layer is 1520. The vocabulary size is 200k and the total number of parameters is 570M (see Table 7).
192
+
193
+ # Citation
194
+
195
+ **BibTeX:**
196
+
197
+ ```bibtex
198
+ @article{lample2019cross,
199
+ title={Cross-lingual language model pretraining},
200
+ author={Lample, Guillaume and Conneau, Alexis},
201
+ journal={arXiv preprint arXiv:1901.07291},
202
+ year={2019}
203
+ }
204
+ ```
205
+
206
+ **APA:**
207
+ - Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
208
+
209
+ # Model Card Authors
210
+
211
+ This model card was written by the team at Hugging Face.
212
+
213
+ # How to Get Started with the Model
214
+
215
+ More information needed. See the [ipython notebook](https://github.com/facebookresearch/XLM/blob/main/generate-embeddings.ipynb) in the associated [GitHub repo](https://github.com/facebookresearch/XLM#the-17-and-100-languages) for examples.