jarodrigues commited on
Commit
13d2dee
1 Parent(s): f7cbda3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +232 -0
README.md CHANGED
@@ -1,3 +1,235 @@
1
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: mit
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - pt
4
+ tags:
5
+ - albertina-pt*
6
+ - albertina-100m-portuguese-ptpt
7
+ - albertina-100m-portuguese-ptbr
8
+ - albertina-900m-portuguese-ptpt
9
+ - albertina-900m-portuguese-ptbr
10
+ - albertina-1b5-portuguese-ptpt
11
+ - albertina-1b5-portuguese-ptbr
12
+ - fill-mask
13
+ - bert
14
+ - deberta
15
+ - portuguese
16
+ - encoder
17
+ - foundation model
18
  license: mit
19
+ datasets:
20
+ - PORTULAN/glue-ptpt
21
+ widget:
22
+ - text: >-
23
+ A culinária portuguesa é rica em sabores e [MASK], tornando-se um dos
24
+ maiores tesouros do país.
25
  ---
26
+ ---
27
+ <img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png">
28
+ <p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for Albertina 1.5B PTBR.
29
+ You may be interested in some of the other models in the <a href="https://huggingface.co/PORTULAN">Albertina (encoders) and Gervásio (decoders) families</a>.
30
+ </p>
31
+
32
+ ---
33
+
34
+ # Albertina 1.5B PTBR
35
+
36
+
37
+ **Albertina 1.5B PTBR** is a foundation, large language model for the **American variant of Portuguese**.
38
+
39
+ It is an **encoder** of the BERT family, based on the neural architecture Transformer and
40
+ developed over the DeBERTa model, with most competitive performance for this language.
41
+ It has different versions that were trained for different variants of Portuguese (PT),
42
+ namely the European variant, spoken in Portugal (**PTPT**) and the American variant, spoken in Brazil (**PTBR**),
43
+ and it is openly distributed free of charge under an open license.
44
+
45
+ | Albertina's Family of Models |
46
+ |----------------------------------------------------------------------------------------------------------|
47
+ | [**Albertina 1.5B PTPT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) |
48
+ | [**Albertina 1.5B PTBR**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder) |
49
+ | [**Albertina 1.5B PTPT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256)|
50
+ | [**Albertina 1.5B PTBR 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder-256)|
51
+ | [**Albertina 900M PTPT**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptpt-encoder) |
52
+ | [**Albertina 900M PTBR**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptbr-encoder) |
53
+ | [**Albertina 100M PTPT**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptpt-encoder) |
54
+ | [**Albertina 100M PTBR**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptbr-encoder) |
55
+
56
+ **Albertina 1.5B PTBR** is a version for the **American variant of Portuguese**,
57
+ and to the best of our knowledge, this is an encoder specifically for this language and variant
58
+ that, at the time of its initial distribution, with its 1.5 billion parameters and performance scores sets a new state of the art for it,
59
+ and is made publicly available and distributed for reuse.
60
+
61
+ It is an **encoder** of the BERT family, based on the neural architecture Transformer and
62
+ developed over the DeBERTa model, with most competitive performance for this language.
63
+ It is distributed free of charge and under a most permissible license.
64
+
65
+
66
+ **Albertina 1.5B PTBR** is developed by a joint team from the University of Lisbon and the University of Porto, Portugal.
67
+ For a fully detailed description, check the respective [publication](https://arxiv.org/abs/?):
68
+
69
+ ``` latex
70
+ @misc{albertina-pt-fostering,
71
+ title={Fostering the Ecosystem of Open Neural Encoders for Portuguese
72
+ with Albertina PT-* family},
73
+ author={Rodrigo Santos and João Rodrigues and Luís Gomes and João Silva
74
+ and António Branco and Henrique Lopes Cardoso
75
+ and Tomás Freitas Osório and Bernardo Leite},
76
+ year={2024},
77
+ eprint={?},
78
+ archivePrefix={arXiv},
79
+ primaryClass={cs.CL}
80
+ }
81
+ ```
82
+
83
+ Please use the above cannonical reference when using or citing this model.
84
+
85
+ <br>
86
+
87
+
88
+ # Model Description
89
+
90
+ **This model card is for Albertina 1.5B PTBR**, with 1.5 billion parameters, 48 layers and a hidden size of 1536.
91
+
92
+ Albertina 1.5B PTBR is distributed under an [MIT license](https://huggingface.co/PORTULAN/albertina-ptpt/blob/main/LICENSE).
93
+
94
+ DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBERTa/blob/master/LICENSE).
95
+
96
+
97
+ <br>
98
+
99
+ # Training Data
100
+
101
+ [**Albertina 1.5B PTBR**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder) was trained over a 36 billion token data set that resulted from gathering some openly available corpora of American Portuguese from the following sources:
102
+
103
+ - [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX): the CulturaX is a multilingual corpus, freely available for research and AI development, created by combining and extensively cleaning two other large datasets, mC4 and OSCAR. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal.
104
+
105
+ ## Preprocessing
106
+
107
+ We filtered the PTBR corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline.
108
+ We skipped the default filtering of stopwords since it would disrupt the syntactic structure, and also the filtering
109
+ for language identification given the corpus was pre-selected as Portuguese.
110
+
111
+
112
+ ## Training
113
+
114
+ As codebase, we resorted to the [DeBERTa V2 xxlarge](https://huggingface.co/microsoft/deberta-v2-xxlarge), for English.
115
+
116
+ To train **Albertina 1.5B PTBR**, the data set was tokenized with the original DeBERTa tokenizer with a 128-token sequence
117
+ truncation and dynamic padding for 250k steps,
118
+ a 256-token sequence-truncation for 80k steps
119
+ ([**Albertina 1.5B PTBR 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder-256)) and finally a 512-token sequence-truncation for 60k steps.
120
+ These steps correspond to the equivalent setup of 48 hours on a2-megagpu-16gb Google Cloud A2 node for the 128-token input sequences, 24 hours of computation for the 256-token
121
+ input sequences and 24 hours of computation for the 512-token input sequences.
122
+ We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
123
+
124
+ <br>
125
+
126
+ # Performance
127
+
128
+
129
+ We resorted to [extraGLUE](https://huggingface.co/datasets/PORTULAN/extraglue), a **PTBR version of the GLUE and SUPERGLUE** benchmark.
130
+ We automatically translated the tasks from GLUE and SUPERGLUE using [DeepL Translate](https://www.deepl.com/), which specifically
131
+ provides translation from English to PTPT or PTBR as possible options.
132
+
133
+ | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
134
+ |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
135
+ | [**Albertina 1.5B PTPT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) | **0.8809** | 0.4742 | 0.8457 | **0.9034** | **0.8433** | **0.7840** | **0.7688** | **0.8602** |
136
+ | [**Albertina 1.5B PTPT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256) | 0.8809 | 0.5493 | 0.8752 | 0.8795 | 0.8400 | 0.5832 | 0.6791 | 0.8496 |
137
+ | [**Albertina 900M PTPT**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptpt-encoder) | 0.8339 | 0.4225 | **0.9171**| 0.8801 | 0.7033 | 0.6018 | 0.6728 | 0.8224 |
138
+ | [**Albertina 100M PTPT**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptpt-encoder) | 0.6919 | 0.4742 | 0.8047 | 0.8590 | n.a. | 0.4529 | 0.6481 | 0.7578 |
139
+ ||||||||||
140
+ | **DeBERTa 1.5B (English)** | 0.8147 | 0.4554 | 0.8696 | 0.8557 | 0.5167 | 0.4901 | 0.6687 | 0.8347 |
141
+ | **DeBERTa 100M (English)** | 0.6029 | **0.5634** | 0.7802 | 0.8320 | n.a. | 0.4698 | 0.6368 | 0.6829 |
142
+
143
+
144
+
145
+
146
+ **para modelo PTBR**
147
+
148
+ | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
149
+ |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
150
+ | [**Albertina 1.5B PTBR**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder) | **0.8676** | 0.4742 | 0.8622 | **0.9007** | 0.7767 | 0.6372 | **0.7667** | **0.8654** |
151
+ | [**Albertina 1.5B PTBR 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder-256) | 0.8123 | 0.4225 | 0.8638 | 0.8968 | **0.8533** | **0.6884** | 0.6799 | 0.8509 |
152
+ | [**Albertina 900M PTBR**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptbr-encoder) | 0.7545 | 0.4601 | **0.9071**| 0.8910 | 0.7767 | 0.5799 | 0.6731 | 0.8385 |
153
+ | **BERTimbau (335M)** | 0.6446 | **0.5634** | 0.8873 | 0.8842 | 0.6933 | 0.5438 | 0.6787 | 0.7783 |
154
+ | [**Albertina 100M PTBR**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptbr-encoder) | 0.6582 | **0.5634** | 0.8149 | 0.8489 | n.a. | 0.4771 | 0.6469 | 0.7537 |
155
+ ||||||||||
156
+ | **DeBERTa 1.5B (English)** | 0.7112 | **0.5634** | 0.8545 | 0.0123 | 0.5700 | 0.4307 | 0.3639 | 0.6217 |
157
+ | **DeBERTa 100M (English)** | 0.5716 | 0.5587 | 0.8060 | 0.8266 | n.a. | 0.4739 | 0.6391 | 0.6838 |
158
+
159
+
160
+ <br>
161
+
162
+ # How to use
163
+
164
+ You can use this model directly with a pipeline for masked language modeling:
165
+
166
+ ```python
167
+ >>> from transformers import pipeline
168
+ >>> unmasker = pipeline('fill-mask', model='PORTULAN/albertina-1b5-portuguese-ptbr-encoder')
169
+ >>> unmasker("A culinária portuguesa é rica em sabores e [MASK], tornando-se um dos maiores tesouros do país.")
170
+
171
+ [{'score': 0.8332648277282715, 'token': 14690, 'token_str': ' costumes', 'sequence': 'A culinária portuguesa é rica em sabores e costumes, tornando-se um dos maiores tesouros do país.'},
172
+ {'score': 0.07860890030860901, 'token': 29829, 'token_str': ' cores', 'sequence': 'A culinária portuguesa é rica em sabores e cores, tornando-se um dos maiores tesouros do país.'},
173
+ {'score': 0.03278181701898575, 'token': 35277, 'token_str': ' arte', 'sequence': 'A culinária portuguesa é rica em sabores e arte, tornando-se um dos maiores tesouros do país.'},
174
+ {'score': 0.009515956044197083, 'token': 9240, 'token_str': ' cor', 'sequence': 'A culinária portuguesa é rica em sabores e cor, tornando-se um dos maiores tesouros do país.'},
175
+ {'score': 0.009381960146129131, 'token': 33455, 'token_str': ' nuances', 'sequence': 'A culinária portuguesa é rica em sabores e nuances, tornando-se um dos maiores tesouros do país.'}]
176
+
177
+
178
+
179
+ ```
180
+
181
+ The model can be used by fine-tuning it for a specific task:
182
+
183
+ ```python
184
+ >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
185
+ >>> from datasets import load_dataset
186
+
187
+ >>> model = AutoModelForSequenceClassification.from_pretrained("PORTULAN/albertina-1b5-portuguese-ptbr-encoder", num_labels=2)
188
+ >>> tokenizer = AutoTokenizer.from_pretrained("PORTULAN/albertina-1b5-portuguese-ptbr-encoder")
189
+ >>> dataset = load_dataset("PORTULAN/glue-ptbr", "rte")
190
+
191
+ >>> def tokenize_function(examples):
192
+ ... return tokenizer(examples["sentence1"], examples["sentence2"], padding="max_length", truncation=True)
193
+
194
+ >>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
195
+
196
+ >>> training_args = TrainingArguments(output_dir="albertina-ptbr-rte", evaluation_strategy="epoch")
197
+ >>> trainer = Trainer(
198
+ ... model=model,
199
+ ... args=training_args,
200
+ ... train_dataset=tokenized_datasets["train"],
201
+ ... eval_dataset=tokenized_datasets["validation"],
202
+ ... )
203
+
204
+ >>> trainer.train()
205
+
206
+ ```
207
+
208
+ <br>
209
+
210
+ # Citation
211
+
212
+ When using or citing this model, kindly cite the following [publication](https://arxiv.org/abs/?):
213
+
214
+ ``` latex
215
+ @misc{albertina-pt-fostering,
216
+ title={Fostering the Ecosystem of Open Neural Encoders for Portuguese
217
+ with Albertina PT-* family},
218
+ author={Rodrigo Santos and João Rodrigues and Luís Gomes and João Silva
219
+ and António Branco and Henrique Lopes Cardoso
220
+ and Tomás Freitas Osório and Bernardo Leite},
221
+ year={2024},
222
+ eprint={?},
223
+ archivePrefix={arXiv},
224
+ primaryClass={cs.CL}
225
+ }
226
+ ```
227
+
228
+ <br>
229
+
230
+ # Acknowledgments
231
+
232
+ The research reported here was partially supported by: PORTULAN CLARIN—Research Infrastructure for the Science and Technology of Language,
233
+ funded by Lisboa 2020, Alentejo 2020 and FCT—Fundação para a Ciência e Tecnologia under the
234
+ grant PINFRA/22117/2016; research project ALBERTINA - Foundation Encoder Model for Portuguese and AI, funded by FCT—Fundação para a Ciência e Tecnologia under the
235
+ grant CPCA-IAC/AV/478394/2022; innovation project ACCELERAT.AI - Multilingual Intelligent Contact Centers, funded by IAPMEI, I.P. - Agência para a Competitividade e Inovação under the grant C625734525-00462629, of Plano de Recuperação e Resiliência, call RE-C05-i01.01 – Agendas/Alianças Mobilizadoras para a Reindustrialização; and LIACC - Laboratory for AI and Computer Science, funded by FCT—Fundação para a Ciência e Tecnologia under the grant FCT/UID/CEC/0027/2020.