nicholasKluge commited on
Commit
50279a8
1 Parent(s): b8b1eb6

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +397 -173
README.md CHANGED
@@ -1,173 +1,397 @@
1
- ---
2
- language:
3
- - pt
4
- license: apache-2.0
5
- library_name: transformers
6
- tags:
7
- - text-generation-inference
8
- datasets:
9
- - TucanoBR/GigaVerbo
10
- metrics:
11
- - perplexity
12
- pipeline_tag: text-generation
13
- widget:
14
- - text: "A floresta da Amazônia é conhecida por sua"
15
- example_title: Exemplo
16
- - text: "Uma das coisas que Portugal, Angola, Brasil e Moçambique tem em comum é o"
17
- example_title: Exemplo
18
- - text: "O Carnaval do Rio de Janeiro é"
19
- example_title: Exemplo
20
- inference:
21
- parameters:
22
- repetition_penalty: 1.2
23
- temperature: 0.2
24
- top_k: 20
25
- top_p: 0.2
26
- max_new_tokens: 150
27
- co2_eq_emissions:
28
- emissions: 89730
29
- source: CodeCarbon
30
- training_type: pre-training
31
- geographical_location: Germany
32
- hardware_used: NVIDIA A100-SXM4-80GB
33
- ---
34
- # Tucano-160m
35
-
36
- <img src="./logo.png" alt="An illustration of a Tucano bird showing vibrant colors like yellow, orange, blue, green, and black." height="200">
37
-
38
- ## Model Summary
39
-
40
- **[Tucano](https://huggingface.co/TucanoBR)** is a series of decoder-transformers based on the Llama 2 architecture, pretrained natively in Portuguese. All Tucano models were trained on **[GigaVerbo](https://huggingface.co/datasets/TucanoBR/GigaVerbo)**, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens.
41
-
42
- Read our preprint [here](https://arxiv.org/abs/xxxx.xxxxx).
43
-
44
- ## Details
45
-
46
- - **Architecture:** a Transformer-based model pre-trained via causal language modeling
47
- - **Size:** 162,417,408 parameters
48
- - **Context length:** 2048 tokens
49
- - **Dataset:** [TucanoBR/GigaVerbo](https://huggingface.co/datasets/TucanoBR/GigaVerbo)
50
- - **Language:** Portuguese
51
- - **Number of steps:** 320,000
52
- - **GPU:** 8 NVIDIA A100-SXM4-80GB
53
- - **Training time**: ~ 44 hours
54
- - **Emissions:** 89.73 KgCO2 (Germany)
55
- - **Total energy consumption:** 235.54 kWh
56
-
57
- This repository has the [source code](https://github.com/Nkluge-correa/Tucano) used to train this model. The main libraries used are:
58
-
59
- - [PyTorch](https://github.com/pytorch/pytorch)
60
- - [Transformers](https://github.com/huggingface/transformers)
61
- - [Datasets](https://github.com/huggingface/datasets)
62
- - [Tokenizers](https://github.com/huggingface/tokenizers)
63
- - [Sentencepiece](https://github.com/google/sentencepiece)
64
- - [Accelerate](https://github.com/huggingface/accelerate)
65
- - [FlashAttention](https://github.com/Dao-AILab/flash-attention)
66
- - [Liger Kernel](https://github.com/linkedin/Liger-Kernel)
67
- - [Codecarbon](https://github.com/mlco2/codecarbon)
68
-
69
- ## Intended Uses
70
-
71
- The primary intended use of the Tucano models is to serve as foundations for research and development involving native Portuguese language modeling. Checkpoints saved during training are designed to provide a controlled setting for performing comparative experiments, specifically regarding the effects of active pretraining on the performance of currently available benchmarks. You may also fine-tune and adapt Tucano models for deployment if your use follows the Apache 2.0 license. If you decide to use the Tucano models as a basis for your fine-tuned model, please conduct your own risk and bias assessment.
72
-
73
- ## Out-of-scope Use
74
-
75
- - Tucano models are **not intended for deployment**. They are not an out-of-the-box product and should not be used for human-facing interactions.
76
-
77
- - Tucano models are for **the Portuguese language only** and are unsuitable for text generation tasks in other languages.
78
-
79
- - Tucano models have **not been fine-tuned** for downstream tasks.
80
-
81
- ## Basic usage
82
-
83
- Using the `pipeline`:
84
-
85
- ```python
86
- from transformers import pipeline
87
-
88
- generator = pipeline("text-generation", model="TucanoBR/Tucano-160m")
89
-
90
- completions = generator("A floresta da Amazônia é conhecida por sua", num_return_sequences=2, max_new_tokens=100)
91
-
92
- for comp in completions:
93
- print(f"🤖 {comp['generated_text']}")
94
- ```
95
-
96
- Using the `AutoTokenizer` and `AutoModelForCausalLM`:
97
-
98
- ```python
99
- from transformers import AutoTokenizer, AutoModelForCausalLM
100
- import torch
101
-
102
- tokenizer = AutoTokenizer.from_pretrained("TucanoBR/Tucano-160m", revision='main')
103
- model = AutoModelForCausalLM.from_pretrained("TucanoBR/Tucano-160m", revision='main')
104
-
105
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
106
-
107
- model.eval()
108
- model.to(device)
109
-
110
- inputs = tokenizer("A floresta da Amazônia é conhecida por sua", return_tensors="pt").to(device)
111
-
112
- completions = model.generate(**inputs, num_return_sequences=2, max_new_tokens=100)
113
-
114
- for i, completion in enumerate(completions):
115
- print(f'🤖 {tokenizer.decode(completion)}')
116
- ```
117
-
118
- ## Limitations
119
-
120
- Like almost all other language models trained on large text datasets scraped from the web, the Tucano models show behavior that does not make them an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. Tucano models are all subject to the following:
121
-
122
- - **Hallucinations:** Tucano models can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.
123
-
124
- - **Biases and Toxicity:** Tucano models inherit the social and historical stereotypes from the data used to train them. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.
125
-
126
- - **Unreliable Code:** Tucano models may produce incorrect code snippets and statements. These code generations should not be treated as suggestions or accurate solutions.
127
-
128
- - **Language Limitations:** Tucano models are primarily designed to interact with Portuguese. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.
129
-
130
- - **Repetition and Verbosity:** Tucano models may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.
131
-
132
- Hence, even though our models are released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications. We also have humans moderating the outputs of these models in applications where they will interact with an audience, guaranteeing users are always aware they are interacting with a language model.
133
-
134
- ## Evaluations
135
-
136
- The table below compares our models against several Portuguese and multilingual language models on the evaluation harness used in our study. More information on it can be found [here](https://github.com/Nkluge-correa/Tucano/tree/main/evaluations/README.md). To learn more about our evaluation harness selection, [read our preprint](https://arxiv.org/abs/xxxx.xxxxx).
137
-
138
- | | Average | Calame-PT | Lambada-PT | Assin2 RTE | Assin2 STS | ARC-PT | HellaSwag-PT |
139
- |-------------------------|---------|-----------|------------|------------|------------|--------|--------------|
140
- | **Tucano-1b1** | 41.94 | 58.24 | 34.7 | 60.82 | 24.63 | 30.43 | 42.84 |
141
- | Llama-3.2-1B | 40.34 | 51.83 | 41.02 | 50.77 | 19.48 | 33.5 | 45.44 |
142
- | Bloom-1b1 | 36.95 | 52.94 | 30.22 | 54.32 | 14.64 | 29.83 | 39.74 |
143
- | Bloom-1b7 | 36.65 | 55.64 | 31.98 | 53.6 | 4.81 | 30.34 | 43.52 |
144
- | **Tucano-630m** | 36.29 | 56.55 | 33.13 | 57.79 | 1.99 | 28.89 | 39.41 |
145
- | Xglm-564m | 35.24 | 50.58 | 27.42 | 49.9 | 23.35 | 25.56 | 34.64 |
146
- | TTL-460m | 33.62 | 49.42 | 23.29 | 53.61 | 13 | 29.4 | 33 |
147
- | **Tucano-1b1-Instruct** | 33.19 | 56.74 | 34.66 | 33.42 | 0.87 | 30.6 | 42.83 |
148
- | **Tucano-160m** | 30.85 | 52.31 | 28.16 | 33.51 | 11.02 | 27.01 | 33.07 |
149
- | Bloom-560m | 29.85 | 49.95 | 25.44 | 33.33 | 8.48 | 24.74 | 37.15 |
150
- | TTL-160m | 29.56 | 46.72 | 20.98 | 53.97 | 0.24 | 26.15 | 29.29 |
151
- | GPorTuguese | 25.45 | 40.61 | 22.98 | 33.59 | 3.44 | 22.48 | 29.62 |
152
- | GlórIA-1b3 | 24.42 | 52.79 | 27.71 | 0 | 2.32 | 26.67 | 37.04 |
153
- | mGPT-1b3 | 21.3 | 47.14 | 29.92 | 0 | 0.58 | 23.81 | 26.37 |
154
- | Lola-v1 | 18.93 | 11.19 | 26.40 | 0 | 0 | 30.42 | 45.61 |
155
-
156
- ## Cite as 🤗
157
-
158
- ```latex
159
- @misc{correa24tucano,
160
- title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
161
- author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
162
- journal={arXiv preprint arXiv:xxxx.xxxxx},
163
- year={2024}
164
- }
165
- ```
166
-
167
- ## Aknowlegments
168
-
169
- We gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing \& Analytics Lab.
170
-
171
- ## License
172
-
173
- Tucano is licensed under the Apache License, Version 2.0. For more details, see the [LICENSE](LICENSE) file.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - pt
4
+ license: apache-2.0
5
+ library_name: transformers
6
+ tags:
7
+ - text-generation-inference
8
+ datasets:
9
+ - TucanoBR/GigaVerbo
10
+ metrics:
11
+ - perplexity
12
+ pipeline_tag: text-generation
13
+ widget:
14
+ - text: "A floresta da Amazônia é conhecida por sua"
15
+ example_title: Exemplo
16
+ - text: "Uma das coisas que Portugal, Angola, Brasil e Moçambique tem em comum é o"
17
+ example_title: Exemplo
18
+ - text: "O Carnaval do Rio de Janeiro é"
19
+ example_title: Exemplo
20
+ inference:
21
+ parameters:
22
+ repetition_penalty: 1.2
23
+ temperature: 0.2
24
+ top_k: 20
25
+ top_p: 0.2
26
+ max_new_tokens: 150
27
+ co2_eq_emissions:
28
+ emissions: 89730
29
+ source: CodeCarbon
30
+ training_type: pre-training
31
+ geographical_location: Germany
32
+ hardware_used: NVIDIA A100-SXM4-80GB
33
+ model-index:
34
+ - name: Tucano-160m
35
+ results:
36
+ - task:
37
+ type: text-generation
38
+ name: Text Generation
39
+ dataset:
40
+ name: CALAME-PT
41
+ type: NOVA-vision-language/calame-pt
42
+ split: all
43
+ args:
44
+ num_few_shot: 0
45
+ metrics:
46
+ - type: acc
47
+ value: 52.31
48
+ name: accuracy
49
+ source:
50
+ url: https://huggingface.co/datasets/NOVA-vision-language/calame-pt
51
+ name: Context-Aware LAnguage Modeling Evaluation for Portuguese
52
+ - task:
53
+ type: text-generation
54
+ name: Text Generation
55
+ dataset:
56
+ name: LAMBADA-PT
57
+ type: nicholasKluge/lambada-pt-br
58
+ split: train
59
+ args:
60
+ num_few_shot: 0
61
+ metrics:
62
+ - type: acc
63
+ value: 28.16
64
+ name: accuracy
65
+ source:
66
+ url: https://huggingface.co/datasets/nicholasKluge/lambada-pt-br
67
+ name: LAMBADA-PT-BR
68
+ - task:
69
+ type: text-generation
70
+ name: Text Generation
71
+ dataset:
72
+ name: ENEM Challenge (No Images)
73
+ type: eduagarcia/enem_challenge
74
+ split: train
75
+ args:
76
+ num_few_shot: 3
77
+ metrics:
78
+ - type: acc
79
+ value: 19.03
80
+ name: accuracy
81
+ source:
82
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
83
+ name: Open Portuguese LLM Leaderboard
84
+ - task:
85
+ type: text-generation
86
+ name: Text Generation
87
+ dataset:
88
+ name: BLUEX (No Images)
89
+ type: eduagarcia-temp/BLUEX_without_images
90
+ split: train
91
+ args:
92
+ num_few_shot: 3
93
+ metrics:
94
+ - type: acc
95
+ value: 22.11
96
+ name: accuracy
97
+ source:
98
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
99
+ name: Open Portuguese LLM Leaderboard
100
+ - task:
101
+ type: text-generation
102
+ name: Text Generation
103
+ dataset:
104
+ name: OAB Exams
105
+ type: eduagarcia/oab_exams
106
+ split: train
107
+ args:
108
+ num_few_shot: 3
109
+ metrics:
110
+ - type: acc
111
+ value: 25.1
112
+ name: accuracy
113
+ source:
114
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
115
+ name: Open Portuguese LLM Leaderboard
116
+ - task:
117
+ type: text-generation
118
+ name: Text Generation
119
+ dataset:
120
+ name: Assin2 RTE
121
+ type: assin2
122
+ split: test
123
+ args:
124
+ num_few_shot: 15
125
+ metrics:
126
+ - type: f1_macro
127
+ value: 33.51
128
+ name: f1-macro
129
+ source:
130
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
131
+ name: Open Portuguese LLM Leaderboard
132
+ - task:
133
+ type: text-generation
134
+ name: Text Generation
135
+ dataset:
136
+ name: Assin2 STS
137
+ type: eduagarcia/portuguese_benchmark
138
+ split: test
139
+ args:
140
+ num_few_shot: 10
141
+ metrics:
142
+ - type: pearson
143
+ value: 11.02
144
+ name: pearson
145
+ source:
146
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
147
+ name: Open Portuguese LLM Leaderboard
148
+ - task:
149
+ type: text-generation
150
+ name: Text Generation
151
+ dataset:
152
+ name: FaQuAD NLI
153
+ type: ruanchaves/faquad-nli
154
+ split: test
155
+ args:
156
+ num_few_shot: 15
157
+ metrics:
158
+ - type: f1_macro
159
+ value: 43.97
160
+ name: f1-macro
161
+ source:
162
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
163
+ name: Open Portuguese LLM Leaderboard
164
+ - task:
165
+ type: text-generation
166
+ name: Text Generation
167
+ dataset:
168
+ name: HateBR Binary
169
+ type: ruanchaves/hatebr
170
+ split: test
171
+ args:
172
+ num_few_shot: 25
173
+ metrics:
174
+ - type: f1_macro
175
+ value: 36.56
176
+ name: f1-macro
177
+ source:
178
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
179
+ name: Open Portuguese LLM Leaderboard
180
+ - task:
181
+ type: text-generation
182
+ name: Text Generation
183
+ dataset:
184
+ name: PT Hate Speech Binary
185
+ type: hate_speech_portuguese
186
+ split: test
187
+ args:
188
+ num_few_shot: 25
189
+ metrics:
190
+ - type: f1_macro
191
+ value: 22.99
192
+ name: f1-macro
193
+ source:
194
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
195
+ name: Open Portuguese LLM Leaderboard
196
+ - task:
197
+ type: text-generation
198
+ name: Text Generation
199
+ dataset:
200
+ name: tweetSentBR
201
+ type: eduagarcia-temp/tweetsentbr
202
+ split: test
203
+ args:
204
+ num_few_shot: 25
205
+ metrics:
206
+ - type: f1_macro
207
+ value: 16.86
208
+ name: f1-macro
209
+ source:
210
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
211
+ name: Open Portuguese LLM Leaderboard
212
+ - task:
213
+ type: text-generation
214
+ name: Text Generation
215
+ dataset:
216
+ name: ARC-Challenge (PT)
217
+ type: arc_pt
218
+ args:
219
+ num_few_shot: 25
220
+ metrics:
221
+ - type: acc_norm
222
+ value: 27.01
223
+ name: normalized accuracy
224
+ source:
225
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
226
+ name: Evaluation Framework for Multilingual Large Language Models
227
+ - task:
228
+ type: text-generation
229
+ name: Text Generation
230
+ dataset:
231
+ name: HellaSwag (PT)
232
+ type: hellaswag_pt
233
+ args:
234
+ num_few_shot: 10
235
+ metrics:
236
+ - type: acc_norm
237
+ value: 33.07
238
+ name: normalized accuracy
239
+ source:
240
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
241
+ name: Evaluation Framework for Multilingual Large Language Models
242
+ - task:
243
+ type: text-generation
244
+ name: Text Generation
245
+ dataset:
246
+ name: TruthfulQA (PT)
247
+ type: truthfulqa_pt
248
+ args:
249
+ num_few_shot: 0
250
+ metrics:
251
+ - type: mc2
252
+ value: 41.53
253
+ name: bleurt
254
+ source:
255
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
256
+ name: Evaluation Framework for Multilingual Large Language Models
257
+ ---
258
+ # Tucano-160m
259
+
260
+ <img src="./logo.png" alt="An illustration of a Tucano bird showing vibrant colors like yellow, orange, blue, green, and black." height="200">
261
+
262
+ ## Model Summary
263
+
264
+ **[Tucano](https://huggingface.co/TucanoBR)** is a series of decoder-transformers based on the Llama 2 architecture, pretrained natively in Portuguese. All Tucano models were trained on **[GigaVerbo](https://huggingface.co/datasets/TucanoBR/GigaVerbo)**, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens.
265
+
266
+ Read our preprint [here](https://arxiv.org/abs/xxxx.xxxxx).
267
+
268
+ ## Details
269
+
270
+ - **Architecture:** a Transformer-based model pre-trained via causal language modeling
271
+ - **Size:** 162,417,408 parameters
272
+ - **Context length:** 2048 tokens
273
+ - **Dataset:** [TucanoBR/GigaVerbo](https://huggingface.co/datasets/TucanoBR/GigaVerbo)
274
+ - **Language:** Portuguese
275
+ - **Number of steps:** 320,000
276
+ - **GPU:** 8 NVIDIA A100-SXM4-80GB
277
+ - **Training time**: ~ 44 hours
278
+ - **Emissions:** 89.73 KgCO2 (Germany)
279
+ - **Total energy consumption:** 235.54 kWh
280
+
281
+ This repository has the [source code](https://github.com/Nkluge-correa/Tucano) used to train this model. The main libraries used are:
282
+
283
+ - [PyTorch](https://github.com/pytorch/pytorch)
284
+ - [Transformers](https://github.com/huggingface/transformers)
285
+ - [Datasets](https://github.com/huggingface/datasets)
286
+ - [Tokenizers](https://github.com/huggingface/tokenizers)
287
+ - [Sentencepiece](https://github.com/google/sentencepiece)
288
+ - [Accelerate](https://github.com/huggingface/accelerate)
289
+ - [FlashAttention](https://github.com/Dao-AILab/flash-attention)
290
+ - [Liger Kernel](https://github.com/linkedin/Liger-Kernel)
291
+ - [Codecarbon](https://github.com/mlco2/codecarbon)
292
+
293
+ ## Intended Uses
294
+
295
+ The primary intended use of the Tucano models is to serve as foundations for research and development involving native Portuguese language modeling. Checkpoints saved during training are designed to provide a controlled setting for performing comparative experiments, specifically regarding the effects of active pretraining on the performance of currently available benchmarks. You may also fine-tune and adapt Tucano models for deployment if your use follows the Apache 2.0 license. If you decide to use the Tucano models as a basis for your fine-tuned model, please conduct your own risk and bias assessment.
296
+
297
+ ## Out-of-scope Use
298
+
299
+ - Tucano models are **not intended for deployment**. They are not an out-of-the-box product and should not be used for human-facing interactions.
300
+
301
+ - Tucano models are for **the Portuguese language only** and are unsuitable for text generation tasks in other languages.
302
+
303
+ - Tucano models have **not been fine-tuned** for downstream tasks.
304
+
305
+ ## Basic usage
306
+
307
+ Using the `pipeline`:
308
+
309
+ ```python
310
+ from transformers import pipeline
311
+
312
+ generator = pipeline("text-generation", model="TucanoBR/Tucano-160m")
313
+
314
+ completions = generator("A floresta da Amazônia é conhecida por sua", num_return_sequences=2, max_new_tokens=100)
315
+
316
+ for comp in completions:
317
+ print(f"🤖 {comp['generated_text']}")
318
+ ```
319
+
320
+ Using the `AutoTokenizer` and `AutoModelForCausalLM`:
321
+
322
+ ```python
323
+ from transformers import AutoTokenizer, AutoModelForCausalLM
324
+ import torch
325
+
326
+ tokenizer = AutoTokenizer.from_pretrained("TucanoBR/Tucano-160m", revision='main')
327
+ model = AutoModelForCausalLM.from_pretrained("TucanoBR/Tucano-160m", revision='main')
328
+
329
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
330
+
331
+ model.eval()
332
+ model.to(device)
333
+
334
+ inputs = tokenizer("A floresta da Amazônia é conhecida por sua", return_tensors="pt").to(device)
335
+
336
+ completions = model.generate(**inputs, num_return_sequences=2, max_new_tokens=100)
337
+
338
+ for i, completion in enumerate(completions):
339
+ print(f'🤖 {tokenizer.decode(completion)}')
340
+ ```
341
+
342
+ ## Limitations
343
+
344
+ Like almost all other language models trained on large text datasets scraped from the web, the Tucano models show behavior that does not make them an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. Tucano models are all subject to the following:
345
+
346
+ - **Hallucinations:** Tucano models can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.
347
+
348
+ - **Biases and Toxicity:** Tucano models inherit the social and historical stereotypes from the data used to train them. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.
349
+
350
+ - **Unreliable Code:** Tucano models may produce incorrect code snippets and statements. These code generations should not be treated as suggestions or accurate solutions.
351
+
352
+ - **Language Limitations:** Tucano models are primarily designed to interact with Portuguese. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.
353
+
354
+ - **Repetition and Verbosity:** Tucano models may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.
355
+
356
+ Hence, even though our models are released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications. We also have humans moderating the outputs of these models in applications where they will interact with an audience, guaranteeing users are always aware they are interacting with a language model.
357
+
358
+ ## Evaluations
359
+
360
+ The table below compares our models against several Portuguese and multilingual language models on the evaluation harness used in our study. More information on it can be found [here](https://github.com/Nkluge-correa/Tucano/tree/main/evaluations/README.md). To learn more about our evaluation harness selection, [read our preprint](https://arxiv.org/abs/xxxx.xxxxx).
361
+
362
+ | | Average | Calame-PT | Lambada-PT | Assin2 RTE | Assin2 STS | ARC-PT | HellaSwag-PT |
363
+ |-------------------------|---------|-----------|------------|------------|------------|--------|--------------|
364
+ | **Tucano-1b1** | 41.94 | 58.24 | 34.7 | 60.82 | 24.63 | 30.43 | 42.84 |
365
+ | Llama-3.2-1B | 40.34 | 51.83 | 41.02 | 50.77 | 19.48 | 33.5 | 45.44 |
366
+ | Bloom-1b1 | 36.95 | 52.94 | 30.22 | 54.32 | 14.64 | 29.83 | 39.74 |
367
+ | Bloom-1b7 | 36.65 | 55.64 | 31.98 | 53.6 | 4.81 | 30.34 | 43.52 |
368
+ | **Tucano-630m** | 36.29 | 56.55 | 33.13 | 57.79 | 1.99 | 28.89 | 39.41 |
369
+ | Xglm-564m | 35.24 | 50.58 | 27.42 | 49.9 | 23.35 | 25.56 | 34.64 |
370
+ | TTL-460m | 33.62 | 49.42 | 23.29 | 53.61 | 13 | 29.4 | 33 |
371
+ | **Tucano-1b1-Instruct** | 33.19 | 56.74 | 34.66 | 33.42 | 0.87 | 30.6 | 42.83 |
372
+ | **Tucano-160m** | 30.85 | 52.31 | 28.16 | 33.51 | 11.02 | 27.01 | 33.07 |
373
+ | Bloom-560m | 29.85 | 49.95 | 25.44 | 33.33 | 8.48 | 24.74 | 37.15 |
374
+ | TTL-160m | 29.56 | 46.72 | 20.98 | 53.97 | 0.24 | 26.15 | 29.29 |
375
+ | GPorTuguese | 25.45 | 40.61 | 22.98 | 33.59 | 3.44 | 22.48 | 29.62 |
376
+ | GlórIA-1b3 | 24.42 | 52.79 | 27.71 | 0 | 2.32 | 26.67 | 37.04 |
377
+ | mGPT-1b3 | 21.3 | 47.14 | 29.92 | 0 | 0.58 | 23.81 | 26.37 |
378
+ | Lola-v1 | 18.93 | 11.19 | 26.40 | 0 | 0 | 30.42 | 45.61 |
379
+
380
+ ## Cite as 🤗
381
+
382
+ ```latex
383
+ @misc{correa24tucano,
384
+ title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
385
+ author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
386
+ journal={arXiv preprint arXiv:xxxx.xxxxx},
387
+ year={2024}
388
+ }
389
+ ```
390
+
391
+ ## Aknowlegments
392
+
393
+ We gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing \& Analytics Lab.
394
+
395
+ ## License
396
+
397
+ Tucano is licensed under the Apache License, Version 2.0. For more details, see the [LICENSE](LICENSE) file.