RichardErkhov commited on
Commit
92d8c77
1 Parent(s): 246de20

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +328 -0
README.md ADDED
@@ -0,0 +1,328 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ sabia-7b - GGUF
11
+ - Model creator: https://huggingface.co/maritaca-ai/
12
+ - Original model: https://huggingface.co/maritaca-ai/sabia-7b/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [sabia-7b.Q2_K.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q2_K.gguf) | Q2_K | 2.36GB |
18
+ | [sabia-7b.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.IQ3_XS.gguf) | IQ3_XS | 2.6GB |
19
+ | [sabia-7b.IQ3_S.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.IQ3_S.gguf) | IQ3_S | 2.75GB |
20
+ | [sabia-7b.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q3_K_S.gguf) | Q3_K_S | 2.75GB |
21
+ | [sabia-7b.IQ3_M.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.IQ3_M.gguf) | IQ3_M | 2.9GB |
22
+ | [sabia-7b.Q3_K.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q3_K.gguf) | Q3_K | 3.07GB |
23
+ | [sabia-7b.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q3_K_M.gguf) | Q3_K_M | 3.07GB |
24
+ | [sabia-7b.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q3_K_L.gguf) | Q3_K_L | 3.35GB |
25
+ | [sabia-7b.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.IQ4_XS.gguf) | IQ4_XS | 3.4GB |
26
+ | [sabia-7b.Q4_0.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q4_0.gguf) | Q4_0 | 3.56GB |
27
+ | [sabia-7b.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.IQ4_NL.gguf) | IQ4_NL | 3.58GB |
28
+ | [sabia-7b.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q4_K_S.gguf) | Q4_K_S | 3.59GB |
29
+ | [sabia-7b.Q4_K.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q4_K.gguf) | Q4_K | 3.8GB |
30
+ | [sabia-7b.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q4_K_M.gguf) | Q4_K_M | 3.8GB |
31
+ | [sabia-7b.Q4_1.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q4_1.gguf) | Q4_1 | 3.95GB |
32
+ | [sabia-7b.Q5_0.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q5_0.gguf) | Q5_0 | 4.33GB |
33
+ | [sabia-7b.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q5_K_S.gguf) | Q5_K_S | 4.33GB |
34
+ | [sabia-7b.Q5_K.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q5_K.gguf) | Q5_K | 4.45GB |
35
+ | [sabia-7b.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q5_K_M.gguf) | Q5_K_M | 4.45GB |
36
+ | [sabia-7b.Q5_1.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q5_1.gguf) | Q5_1 | 4.72GB |
37
+ | [sabia-7b.Q6_K.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q6_K.gguf) | Q6_K | 5.15GB |
38
+ | [sabia-7b.Q8_0.gguf](https://huggingface.co/RichardErkhov/maritaca-ai_-_sabia-7b-gguf/blob/main/sabia-7b.Q8_0.gguf) | Q8_0 | 4.88GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ language:
46
+ - pt
47
+ model-index:
48
+ - name: sabia-7b
49
+ results:
50
+ - task:
51
+ type: text-generation
52
+ name: Text Generation
53
+ dataset:
54
+ name: ENEM Challenge (No Images)
55
+ type: eduagarcia/enem_challenge
56
+ split: train
57
+ args:
58
+ num_few_shot: 3
59
+ metrics:
60
+ - type: acc
61
+ value: 55.07
62
+ name: accuracy
63
+ source:
64
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
65
+ name: Open Portuguese LLM Leaderboard
66
+ - task:
67
+ type: text-generation
68
+ name: Text Generation
69
+ dataset:
70
+ name: BLUEX (No Images)
71
+ type: eduagarcia-temp/BLUEX_without_images
72
+ split: train
73
+ args:
74
+ num_few_shot: 3
75
+ metrics:
76
+ - type: acc
77
+ value: 47.71
78
+ name: accuracy
79
+ source:
80
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
81
+ name: Open Portuguese LLM Leaderboard
82
+ - task:
83
+ type: text-generation
84
+ name: Text Generation
85
+ dataset:
86
+ name: OAB Exams
87
+ type: eduagarcia/oab_exams
88
+ split: train
89
+ args:
90
+ num_few_shot: 3
91
+ metrics:
92
+ - type: acc
93
+ value: 41.41
94
+ name: accuracy
95
+ source:
96
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
97
+ name: Open Portuguese LLM Leaderboard
98
+ - task:
99
+ type: text-generation
100
+ name: Text Generation
101
+ dataset:
102
+ name: Assin2 RTE
103
+ type: assin2
104
+ split: test
105
+ args:
106
+ num_few_shot: 15
107
+ metrics:
108
+ - type: f1_macro
109
+ value: 46.68
110
+ name: f1-macro
111
+ source:
112
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
113
+ name: Open Portuguese LLM Leaderboard
114
+ - task:
115
+ type: text-generation
116
+ name: Text Generation
117
+ dataset:
118
+ name: Assin2 STS
119
+ type: eduagarcia/portuguese_benchmark
120
+ split: test
121
+ args:
122
+ num_few_shot: 15
123
+ metrics:
124
+ - type: pearson
125
+ value: 1.89
126
+ name: pearson
127
+ source:
128
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
129
+ name: Open Portuguese LLM Leaderboard
130
+ - task:
131
+ type: text-generation
132
+ name: Text Generation
133
+ dataset:
134
+ name: FaQuAD NLI
135
+ type: ruanchaves/faquad-nli
136
+ split: test
137
+ args:
138
+ num_few_shot: 15
139
+ metrics:
140
+ - type: f1_macro
141
+ value: 58.34
142
+ name: f1-macro
143
+ source:
144
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
145
+ name: Open Portuguese LLM Leaderboard
146
+ - task:
147
+ type: text-generation
148
+ name: Text Generation
149
+ dataset:
150
+ name: HateBR Binary
151
+ type: ruanchaves/hatebr
152
+ split: test
153
+ args:
154
+ num_few_shot: 25
155
+ metrics:
156
+ - type: f1_macro
157
+ value: 61.93
158
+ name: f1-macro
159
+ source:
160
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
161
+ name: Open Portuguese LLM Leaderboard
162
+ - task:
163
+ type: text-generation
164
+ name: Text Generation
165
+ dataset:
166
+ name: PT Hate Speech Binary
167
+ type: hate_speech_portuguese
168
+ split: test
169
+ args:
170
+ num_few_shot: 25
171
+ metrics:
172
+ - type: f1_macro
173
+ value: 64.13
174
+ name: f1-macro
175
+ source:
176
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
177
+ name: Open Portuguese LLM Leaderboard
178
+ - task:
179
+ type: text-generation
180
+ name: Text Generation
181
+ dataset:
182
+ name: tweetSentBR
183
+ type: eduagarcia-temp/tweetsentbr
184
+ split: test
185
+ args:
186
+ num_few_shot: 25
187
+ metrics:
188
+ - type: f1_macro
189
+ value: 46.64
190
+ name: f1-macro
191
+ source:
192
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=maritaca-ai/sabia-7b
193
+ name: Open Portuguese LLM Leaderboard
194
+ ---
195
+
196
+ Sabiá-7B is Portuguese language model developed by [Maritaca AI](https://www.maritaca.ai/).
197
+
198
+ **Input:** The model accepts only text input.
199
+
200
+ **Output:** The Model generates text only.
201
+
202
+ **Model Architecture:** Sabiá-7B is an auto-regressive language model that uses the same architecture of LLaMA-1-7B.
203
+
204
+ **Tokenizer:** It uses the same tokenizer as LLaMA-1-7B.
205
+
206
+ **Maximum sequence length:** 2048 tokens.
207
+
208
+ **Pretraining data:** The model was pretrained on 7 billion tokens from the Portuguese subset of ClueWeb22, starting with the weights of LLaMA-1-7B and further trained for an additional 10 billion tokens, approximately 1.4 epochs of the training dataset.
209
+
210
+ **Data Freshness:** The pretraining data has a cutoff of mid-2022.
211
+
212
+ **License:** The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.
213
+
214
+ **Paper:** For more details, please refer to our paper: [Sabiá: Portuguese Large Language Models](https://arxiv.org/pdf/2304.07880.pdf)
215
+
216
+
217
+ ## Few-shot Example
218
+
219
+ Given that Sabiá-7B was trained solely on a language modeling objective without fine-tuning for instruction following, it is recommended for few-shot tasks rather than zero-shot tasks, like in the example below.
220
+
221
+ ```python
222
+ import torch
223
+ from transformers import LlamaTokenizer, LlamaForCausalLM
224
+
225
+ tokenizer = LlamaTokenizer.from_pretrained("maritaca-ai/sabia-7b")
226
+ model = LlamaForCausalLM.from_pretrained(
227
+ "maritaca-ai/sabia-7b",
228
+ device_map="auto", # Automatically loads the model in the GPU, if there is one. Requires pip install acelerate
229
+ low_cpu_mem_usage=True,
230
+ torch_dtype=torch.bfloat16 # If your GPU does not support bfloat16, change to torch.float16
231
+ )
232
+
233
+ prompt = """Classifique a resenha de filme como "positiva" ou "negativa".
234
+
235
+ Resenha: Gostei muito do filme, é o melhor do ano!
236
+ Classe: positiva
237
+
238
+ Resenha: O filme deixa muito a desejar.
239
+ Classe: negativa
240
+
241
+ Resenha: Apesar de longo, valeu o ingresso.
242
+ Classe:"""
243
+
244
+ input_ids = tokenizer(prompt, return_tensors="pt")
245
+
246
+ output = model.generate(
247
+ input_ids["input_ids"].to("cuda"),
248
+ max_length=1024,
249
+ eos_token_id=tokenizer.encode("\n")) # Stop generation when a "\n" token is dectected
250
+
251
+ # The output contains the input tokens, so we have to skip them.
252
+ output = output[0][len(input_ids["input_ids"][0]):]
253
+
254
+ print(tokenizer.decode(output, skip_special_tokens=True))
255
+ ```
256
+
257
+ If your GPU does not have enough RAM, try using int8 precision.
258
+ However, expect some degradation in the model output quality when compared to fp16 or bf16.
259
+ ```python
260
+ model = LlamaForCausalLM.from_pretrained(
261
+ "maritaca-ai/sabia-7b",
262
+ device_map="auto",
263
+ low_cpu_mem_usage=True,
264
+ load_in_8bit=True, # Requires pip install bitsandbytes
265
+ )
266
+ ```
267
+
268
+ ## Results in Portuguese
269
+
270
+ Below we show the results on the Poeta benchmark, which consists of 14 Portuguese datasets.
271
+
272
+ For more information on the Normalized Preferred Metric (NPM), please refer to our paper.
273
+
274
+ |Model | NPM |
275
+ |--|--|
276
+ |LLaMA-1-7B| 33.0|
277
+ |LLaMA-2-7B| 43.7|
278
+ |Sabiá-7B| 48.5|
279
+
280
+ ## Results in English
281
+
282
+ Below we show the average results on 6 English datasets: PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OpenBookQA.
283
+
284
+ |Model | NPM |
285
+ |--|--|
286
+ |LLaMA-1-7B| 50.1|
287
+ |Sabiá-7B| 49.0|
288
+
289
+
290
+ ## Citation
291
+
292
+ Please use the following bibtex to cite our paper:
293
+ ```
294
+ @InProceedings{10.1007/978-3-031-45392-2_15,
295
+ author="Pires, Ramon
296
+ and Abonizio, Hugo
297
+ and Almeida, Thales Sales
298
+ and Nogueira, Rodrigo",
299
+ editor="Naldi, Murilo C.
300
+ and Bianchi, Reinaldo A. C.",
301
+ title="Sabi{\'a}: Portuguese Large Language Models",
302
+ booktitle="Intelligent Systems",
303
+ year="2023",
304
+ publisher="Springer Nature Switzerland",
305
+ address="Cham",
306
+ pages="226--240",
307
+ isbn="978-3-031-45392-2"
308
+ }
309
+ ```
310
+
311
+ # [Open Portuguese LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
312
+ Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/maritaca-ai/sabia-7b)
313
+
314
+ | Metric | Value |
315
+ |--------------------------|---------|
316
+ |Average |**47.09**|
317
+ |ENEM Challenge (No Images)| 55.07|
318
+ |BLUEX (No Images) | 47.71|
319
+ |OAB Exams | 41.41|
320
+ |Assin2 RTE | 46.68|
321
+ |Assin2 STS | 1.89|
322
+ |FaQuAD NLI | 58.34|
323
+ |HateBR Binary | 61.93|
324
+ |PT Hate Speech Binary | 64.13|
325
+ |tweetSentBR | 46.64|
326
+
327
+
328
+