bartowski commited on
Commit
97c2bcb
1 Parent(s): 7b46879

measurement.json

Browse files
Files changed (2) hide show
  1. README.md +124 -0
  2. measurement.json +0 -0
README.md ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: other
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ - instruct
9
+ - finetune
10
+ - chatml
11
+ - gpt4
12
+ - synthetic data
13
+ - science
14
+ - physics
15
+ - chemistry
16
+ - biology
17
+ - math
18
+ - llama
19
+ - llama3
20
+ base_model: meta-llama/Meta-Llama-3-8B
21
+ datasets:
22
+ - allenai/ai2_arc
23
+ - camel-ai/physics
24
+ - camel-ai/chemistry
25
+ - camel-ai/biology
26
+ - camel-ai/math
27
+ - metaeval/reclor
28
+ - openbookqa
29
+ - mandyyyyii/scibench
30
+ - derek-thomas/ScienceQA
31
+ - TIGER-Lab/ScienceEval
32
+ - jondurbin/airoboros-3.2
33
+ - LDJnr/Capybara
34
+ - Cot-Alpaca-GPT4-From-OpenHermes-2.5
35
+ - STEM-AI-mtl/Electrical-engineering
36
+ - knowrohit07/saraswati-stem
37
+ - sablo/oasst2_curated
38
+ - lmsys/lmsys-chat-1m
39
+ - TIGER-Lab/MathInstruct
40
+ - bigbio/med_qa
41
+ - meta-math/MetaMathQA-40K
42
+ - openbookqa
43
+ - piqa
44
+ - metaeval/reclor
45
+ - derek-thomas/ScienceQA
46
+ - scibench
47
+ - sciq
48
+ - Open-Orca/SlimOrca
49
+ - migtissera/Synthia-v1.3
50
+ - TIGER-Lab/ScienceEval
51
+ - allenai/WildChat
52
+ - microsoft/orca-math-word-problems-200k
53
+ - openchat/openchat_sharegpt4_dataset
54
+ - teknium/GPTeacher-General-Instruct
55
+ - m-a-p/CodeFeedback-Filtered-Instruction
56
+ - totally-not-an-llm/EverythingLM-data-V3
57
+ - HuggingFaceH4/no_robots
58
+ - OpenAssistant/oasst_top1_2023-08-25
59
+ - WizardLM/WizardLM_evol_instruct_70k
60
+ quantized_by: bartowski
61
+ pipeline_tag: text-generation
62
+ ---
63
+
64
+ ## Exllama v2 Quantizations of Einstein-v6.1-Llama3-8B
65
+
66
+ Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.0.19">turboderp's ExLlamaV2 v0.0.19</a> for quantization.
67
+
68
+ <b>The "main" branch only contains the measurement.json, download one of the other branches for the model (see below)</b>
69
+
70
+ Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.
71
+
72
+ Original model: https://huggingface.co/Weyaxi/Einstein-v6.1-Llama3-8B
73
+
74
+ ## Prompt format
75
+
76
+ ```
77
+ <|im_start|>system
78
+ {system_prompt}<|im_end|>
79
+ <|im_start|>user
80
+ {prompt}<|im_end|>
81
+ <|im_start|>assistant
82
+
83
+ ```
84
+
85
+ ## Available sizes
86
+
87
+
88
+ | Branch | Bits | lm_head bits | VRAM (4k) | VRAM (8K) | VRAM (16k) | VRAM (32k) | Description |
89
+ | ----- | ---- | ------- | ------ | ------ | ------ | ------ | ------------ |
90
+ | [8_0](https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2/tree/8_0) | 8.0 | 8.0 | 10.1 GB | 10.5 GB | 11.5 GB | 13.6 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
91
+ | [6_5](https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2/tree/6_5) | 6.5 | 8.0 | 8.9 GB | 9.3 GB | 10.3 GB | 12.4 GB | Very similar to 8.0, good tradeoff of size vs performance, **recommended**. |
92
+ | [5_0](https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2/tree/5_0) | 5.0 | 6.0 | 7.7 GB | 8.1 GB | 9.1 GB | 11.2 GB | Slightly lower quality vs 6.5, but usable on 8GB cards. |
93
+ | [4_25](https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2/tree/4_25) | 4.25 | 6.0 | 7.0 GB | 7.4 GB | 8.4 GB | 10.5 GB | GPTQ equivalent bits per weight, slightly higher quality. |
94
+ | [3_5](https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2/tree/3_5) | 3.5 | 6.0 | 6.4 GB | 6.8 GB | 7.8 GB | 9.9 GB | Lower quality, only use if you have to. |
95
+
96
+ ## Download instructions
97
+
98
+ With git:
99
+
100
+ ```shell
101
+ git clone --single-branch --branch 6_5 https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2 Einstein-v6.1-Llama3-8B-exl2-6_5
102
+ ```
103
+
104
+ With huggingface hub (credit to TheBloke for instructions):
105
+
106
+ ```shell
107
+ pip3 install huggingface-hub
108
+ ```
109
+
110
+ To download a specific branch, use the `--revision` parameter. For example, to download the 6.5 bpw branch:
111
+
112
+ Linux:
113
+
114
+ ```shell
115
+ huggingface-cli download bartowski/Einstein-v6.1-Llama3-8B-exl2 --revision 6_5 --local-dir Einstein-v6.1-Llama3-8B-exl2-6_5 --local-dir-use-symlinks False
116
+ ```
117
+
118
+ Windows (which apparently doesn't like _ in folders sometimes?):
119
+
120
+ ```shell
121
+ huggingface-cli download bartowski/Einstein-v6.1-Llama3-8B-exl2 --revision 6_5 --local-dir Einstein-v6.1-Llama3-8B-exl2-6.5 --local-dir-use-symlinks False
122
+ ```
123
+
124
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
measurement.json ADDED
The diff for this file is too large to render. See raw diff