LoneStriker commited on
Commit
8d27dc0
β€’
1 Parent(s): ea1a976

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -1,35 +1,5 @@
1
- *.7z filter=lfs diff=lfs merge=lfs -text
2
- *.arrow filter=lfs diff=lfs merge=lfs -text
3
- *.bin filter=lfs diff=lfs merge=lfs -text
4
- *.bz2 filter=lfs diff=lfs merge=lfs -text
5
- *.ckpt filter=lfs diff=lfs merge=lfs -text
6
- *.ftz filter=lfs diff=lfs merge=lfs -text
7
- *.gz filter=lfs diff=lfs merge=lfs -text
8
- *.h5 filter=lfs diff=lfs merge=lfs -text
9
- *.joblib filter=lfs diff=lfs merge=lfs -text
10
- *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
- *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
- *.model filter=lfs diff=lfs merge=lfs -text
13
- *.msgpack filter=lfs diff=lfs merge=lfs -text
14
- *.npy filter=lfs diff=lfs merge=lfs -text
15
- *.npz filter=lfs diff=lfs merge=lfs -text
16
- *.onnx filter=lfs diff=lfs merge=lfs -text
17
- *.ot filter=lfs diff=lfs merge=lfs -text
18
- *.parquet filter=lfs diff=lfs merge=lfs -text
19
- *.pb filter=lfs diff=lfs merge=lfs -text
20
- *.pickle filter=lfs diff=lfs merge=lfs -text
21
- *.pkl filter=lfs diff=lfs merge=lfs -text
22
- *.pt filter=lfs diff=lfs merge=lfs -text
23
- *.pth filter=lfs diff=lfs merge=lfs -text
24
- *.rar filter=lfs diff=lfs merge=lfs -text
25
- *.safetensors filter=lfs diff=lfs merge=lfs -text
26
- saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
- *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
- *.tflite filter=lfs diff=lfs merge=lfs -text
30
- *.tgz filter=lfs diff=lfs merge=lfs -text
31
- *.wasm filter=lfs diff=lfs merge=lfs -text
32
- *.xz filter=lfs diff=lfs merge=lfs -text
33
- *.zip filter=lfs diff=lfs merge=lfs -text
34
- *.zst filter=lfs diff=lfs merge=lfs -text
35
- *tfevents* filter=lfs diff=lfs merge=lfs -text
 
1
+ Einstein-v4-7B-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
2
+ Einstein-v4-7B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
3
+ Einstein-v4-7B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
4
+ Einstein-v4-7B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
5
+ Einstein-v4-7B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Einstein-v4-7B-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d7954f284d36017031fe27fcf31f0de2cdf50a7ed020e1239441186bca722e7
3
+ size 3822035328
Einstein-v4-7B-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb68812fd71d179d178b03f58d4e2b8544e160a19dbbe1e3ee7b01e8270224ba
3
+ size 4368451008
Einstein-v4-7B-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3fa6a0047c0496569ccedea8dfc8eb87d69dfdef1b022f7ebd3362d4b436068
3
+ size 5131422144
Einstein-v4-7B-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61f99baea2ca2e74a6d8e7e1400d9713363072aec05b85ec68f6b0d12a346ef1
3
+ size 5942078976
Einstein-v4-7B-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c21f514d02de372561f3d28a563e819bac020e323c31a29395800adf82f6b3ff
3
+ size 7695875456
README.md ADDED
@@ -0,0 +1,355 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ tags:
4
+ - axolotl
5
+ - generated_from_trainer
6
+ - Mistral
7
+ - instruct
8
+ - finetune
9
+ - chatml
10
+ - gpt4
11
+ - synthetic data
12
+ - science
13
+ - physics
14
+ - chemistry
15
+ - biology
16
+ - math
17
+ base_model: mistralai/Mistral-7B-v0.1
18
+ datasets:
19
+ - allenai/ai2_arc
20
+ - camel-ai/physics
21
+ - camel-ai/chemistry
22
+ - camel-ai/biology
23
+ - camel-ai/math
24
+ - metaeval/reclor
25
+ - openbookqa
26
+ - mandyyyyii/scibench
27
+ - derek-thomas/ScienceQA
28
+ - TIGER-Lab/ScienceEval
29
+ - jondurbin/airoboros-3.2
30
+ - LDJnr/Capybara
31
+ - Cot-Alpaca-GPT4-From-OpenHermes-2.5
32
+ - STEM-AI-mtl/Electrical-engineering
33
+ - knowrohit07/saraswati-stem
34
+ - sablo/oasst2_curated
35
+ - glaiveai/glaive-code-assistant
36
+ - lmsys/lmsys-chat-1m
37
+ - TIGER-Lab/MathInstruct
38
+ - bigbio/med_qa
39
+ - meta-math/MetaMathQA-40K
40
+ - openbookqa
41
+ - piqa
42
+ - metaeval/reclor
43
+ - derek-thomas/ScienceQA
44
+ - scibench
45
+ - sciq
46
+ - Open-Orca/SlimOrca
47
+ - migtissera/Synthia-v1.3
48
+ - TIGER-Lab/ScienceEval
49
+ model-index:
50
+ - name: Einstein-v4-7B
51
+ results:
52
+ - task:
53
+ type: text-generation
54
+ name: Text Generation
55
+ dataset:
56
+ name: AI2 Reasoning Challenge (25-Shot)
57
+ type: ai2_arc
58
+ config: ARC-Challenge
59
+ split: test
60
+ args:
61
+ num_few_shot: 25
62
+ metrics:
63
+ - type: acc_norm
64
+ value: 64.68
65
+ name: normalized accuracy
66
+ source:
67
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
68
+ name: Open LLM Leaderboard
69
+ - task:
70
+ type: text-generation
71
+ name: Text Generation
72
+ dataset:
73
+ name: HellaSwag (10-Shot)
74
+ type: hellaswag
75
+ split: validation
76
+ args:
77
+ num_few_shot: 10
78
+ metrics:
79
+ - type: acc_norm
80
+ value: 83.75
81
+ name: normalized accuracy
82
+ source:
83
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
84
+ name: Open LLM Leaderboard
85
+ - task:
86
+ type: text-generation
87
+ name: Text Generation
88
+ dataset:
89
+ name: MMLU (5-Shot)
90
+ type: cais/mmlu
91
+ config: all
92
+ split: test
93
+ args:
94
+ num_few_shot: 5
95
+ metrics:
96
+ - type: acc
97
+ value: 62.31
98
+ name: accuracy
99
+ source:
100
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
101
+ name: Open LLM Leaderboard
102
+ - task:
103
+ type: text-generation
104
+ name: Text Generation
105
+ dataset:
106
+ name: TruthfulQA (0-shot)
107
+ type: truthful_qa
108
+ config: multiple_choice
109
+ split: validation
110
+ args:
111
+ num_few_shot: 0
112
+ metrics:
113
+ - type: mc2
114
+ value: 55.15
115
+ source:
116
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
117
+ name: Open LLM Leaderboard
118
+ - task:
119
+ type: text-generation
120
+ name: Text Generation
121
+ dataset:
122
+ name: Winogrande (5-shot)
123
+ type: winogrande
124
+ config: winogrande_xl
125
+ split: validation
126
+ args:
127
+ num_few_shot: 5
128
+ metrics:
129
+ - type: acc
130
+ value: 76.24
131
+ name: accuracy
132
+ source:
133
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
134
+ name: Open LLM Leaderboard
135
+ - task:
136
+ type: text-generation
137
+ name: Text Generation
138
+ dataset:
139
+ name: GSM8k (5-shot)
140
+ type: gsm8k
141
+ config: main
142
+ split: test
143
+ args:
144
+ num_few_shot: 5
145
+ metrics:
146
+ - type: acc
147
+ value: 57.62
148
+ name: accuracy
149
+ source:
150
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v4-7B
151
+ name: Open LLM Leaderboard
152
+ ---
153
+
154
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6468ce47e134d050a58aa89c/U0zyXVGj-O8a7KP3BvPue.png)
155
+ # πŸ”¬ Einstein-v4-7B
156
+
157
+ This model is a full fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on diverse datasets.
158
+
159
+ This model is finetuned using `7xRTX3090` + `1xRTXA6000` using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl).
160
+
161
+ This model's training was sponsored by [sablo.ai](https://sablo.ai).
162
+
163
+ <details><summary>See axolotl config</summary>
164
+
165
+ axolotl version: `0.4.0`
166
+ ```yaml
167
+ base_model: mistralai/Mistral-7B-v0.1
168
+ model_type: MistralForCausalLM
169
+ tokenizer_type: LlamaTokenizer
170
+ is_mistral_derived_model: true
171
+
172
+ load_in_8bit: false
173
+ load_in_4bit: false
174
+ strict: false
175
+
176
+ chat_template: chatml
177
+ datasets:
178
+ - path: data/merged_all.json
179
+ ds_type: json
180
+ type: alpaca
181
+ conversation: chatml
182
+
183
+ - path: data/capybara_sharegpt.json
184
+ ds_type: json
185
+ type: sharegpt
186
+ conversation: chatml
187
+
188
+ - path: data/synthia-v1.3_sharegpt_12500.json
189
+ ds_type: json
190
+ type: sharegpt
191
+ conversation: chatml
192
+
193
+ - path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
194
+ ds_type: json
195
+ type: sharegpt
196
+ conversation: chatml
197
+
198
+ - path: data/slimorca_dedup_filtered_95k_sharegpt.json
199
+ ds_type: json
200
+ type: sharegpt
201
+ conversation: chatml
202
+
203
+ - path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
204
+ ds_type: json
205
+ type: sharegpt
206
+ conversation: chatml
207
+
208
+ dataset_prepared_path: last_run_prepared
209
+ val_set_size: 0.005
210
+ output_dir: ./Einstein-v4-model
211
+
212
+ sequence_len: 8192
213
+ sample_packing: true
214
+ pad_to_sequence_len: true
215
+ eval_sample_packing: false
216
+
217
+ wandb_project: Einstein
218
+ wandb_entity:
219
+ wandb_watch:
220
+ wandb_name:
221
+ wandb_log_model:
222
+ hub_model_id: Weyaxi/Einstein-v4-7B
223
+
224
+ save_safetensors: true
225
+
226
+ gradient_accumulation_steps: 4
227
+ micro_batch_size: 1
228
+ num_epochs: 1.5
229
+ optimizer: adamw_bnb_8bit
230
+ lr_scheduler: cosine
231
+ learning_rate: 0.000005
232
+
233
+ train_on_inputs: false
234
+ group_by_length: false
235
+ bf16: true
236
+ fp16: false
237
+ tf32: false
238
+
239
+ gradient_checkpointing: true
240
+ early_stopping_patience:
241
+ resume_from_checkpoint:
242
+ local_rank:
243
+ logging_steps: 1
244
+ xformers_attention:
245
+ flash_attention: true
246
+
247
+ warmup_steps: 10
248
+ evals_per_epoch: 2 # changed
249
+ eval_table_size:
250
+ eval_table_max_new_tokens: 128
251
+ saves_per_epoch: 4
252
+ debug:
253
+
254
+ deepspeed: zero3_bf16.json
255
+ weight_decay: 0.0
256
+ fsdp:
257
+ fsdp_config:
258
+ special_tokens:
259
+ bos_token: "<s>"
260
+ eos_token: "<|im_end|>"
261
+ unk_token: "<unk>"
262
+ tokens:
263
+ - "<|im_start|>"
264
+
265
+ resume_from_checkpoint: Einstein-v4-model/checkpoint-521
266
+
267
+ ```
268
+
269
+ </details><br>
270
+
271
+ # πŸ’¬ Prompt Template
272
+
273
+ You can use this prompt template while using the model:
274
+
275
+ ### ChatML
276
+
277
+ ```
278
+ <|im_start|>system
279
+ {system}<|im_end|>
280
+ <|im_start|>user
281
+ {user}<|im_end|>
282
+ <|im_start|>assistant
283
+ {asistant}<|im_end|>
284
+ ```
285
+
286
+ This prompt template is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating), which means you can format messages using the
287
+ `tokenizer.apply_chat_template()` method:
288
+
289
+ ```python
290
+ messages = [
291
+ {"role": "system", "content": "You are helpful AI asistant."},
292
+ {"role": "user", "content": "Hello!"}
293
+ ]
294
+ gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
295
+ model.generate(**gen_input)
296
+ ```
297
+
298
+ # πŸ”„ Quantizationed versions
299
+
300
+ Quantizationed versions of this model is available.
301
+
302
+ ## Exl2 [@bartowski](https://hf.co/bartowski):
303
+
304
+ - https://huggingface.co/bartowski/Einstein-v4-7B-exl2
305
+
306
+ You can switch up branches in the repo to use the one you want
307
+
308
+ | Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description |
309
+ | ----- | ---- | ------- | ------ | ------ | ------ | ------------ |
310
+ | [8_0](https://huggingface.co/bartowski/Einstein-v4-7B-exl2/tree/8_0) | 8.0 | 8.0 | 8.4 GB | 9.8 GB | 11.8 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
311
+ | [6_5](https://huggingface.co/bartowski/Einstein-v4-7B-exl2/tree/6_5) | 6.5 | 8.0 | 7.2 GB | 8.6 GB | 10.6 GB | Very similar to 8.0, good tradeoff of size vs performance, **recommended**. |
312
+ | [5_0](https://huggingface.co/bartowski/Einstein-v4-7B-exl2/tree/5_0) | 5.0 | 6.0 | 6.0 GB | 7.4 GB | 9.4 GB | Slightly lower quality vs 6.5, but usable on 8GB cards. |
313
+ | [4_25](https://huggingface.co/bartowski/Einstein-v4-7B-exl2/tree/4_25) | 4.25 | 6.0 | 5.3 GB | 6.7 GB | 8.7 GB | GPTQ equivalent bits per weight, slightly higher quality. |
314
+ | [3_5](https://huggingface.co/bartowski/Einstein-v4-7B-exl2/tree/3_5) | 3.5 | 6.0 | 4.7 GB | 6.1 GB | 8.1 GB | Lower quality, only use if you have to. |
315
+
316
+
317
+ # 🎯 [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
318
+ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Weyaxi__Einstein-v4-7B)
319
+
320
+ | Metric |Value|
321
+ |---------------------------------|----:|
322
+ |Avg. |66.62|
323
+ |AI2 Reasoning Challenge (25-Shot)|64.68|
324
+ |HellaSwag (10-Shot) |83.75|
325
+ |MMLU (5-Shot) |62.31|
326
+ |TruthfulQA (0-shot) |55.15|
327
+ |Winogrande (5-shot) |76.24|
328
+ |GSM8k (5-shot) |57.62|
329
+
330
+ # πŸ€– Additional information about training
331
+
332
+ This model is full fine-tuned for 1.5 epoch.
333
+
334
+ Total number of steps was 1562.
335
+
336
+ <details><summary>Loss graph</summary>
337
+
338
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6468ce47e134d050a58aa89c/UO0NJz9VN5NncIXi82Nk2.png)
339
+ </details><br>
340
+
341
+ # 🀝 Acknowledgments
342
+
343
+ Thanks to [sablo.ai](https://sablo.ai) for sponsoring this model.
344
+
345
+ Thanks to all the dataset authors mentioned in the datasets section.
346
+
347
+ Thanks to [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) for making the repository I used to make this model.
348
+
349
+ Thanks to all open source AI community.
350
+
351
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
352
+
353
+ If you would like to support me:
354
+
355
+ [β˜• Buy Me a Coffee](https://www.buymeacoffee.com/weyaxi)