Transformers
English
llama
text-generation-inference
TheBloke commited on
Commit
fb5470d
1 Parent(s): e129425

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +32 -0
README.md CHANGED
@@ -102,9 +102,41 @@ Refer to the Provided Files table below to see what files use which methods, and
102
  | [samantha-1.1-70b.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_0.bin) | q5_0 | 5 | 47.46 GB| 49.96 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
103
  | [samantha-1.1-70b.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 48.75 GB| 51.25 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
104
  | [samantha-1.1-70b.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 47.46 GB| 49.96 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
 
 
 
105
 
106
  **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
107
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
  ## How to run in `llama.cpp`
109
 
110
  I use the following command line; adjust for your tastes and needs:
 
102
  | [samantha-1.1-70b.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_0.bin) | q5_0 | 5 | 47.46 GB| 49.96 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
103
  | [samantha-1.1-70b.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 48.75 GB| 51.25 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
104
  | [samantha-1.1-70b.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/Samantha-1.1-70B-GGML/blob/main/samantha-1.1-70b.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 47.46 GB| 49.96 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
105
+ | samantha-1.1-70b.ggmlv3.q5_1.bin | q5_1 | 5 | 51.76 GB | 54.26 GB | Original quant method, 5-bit. Higher accuracy, slower inference than q5_0. |
106
+ | samantha-1.1-70b.ggmlv3.q6_K.bin | q6_K | 6 | 56.59 GB | 59.09 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
107
+ | samantha-1.1-70b.ggmlv3.q8_0.bin | q8_0 | 8 | 73.23 GB | 75.73 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
108
 
109
  **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
110
 
111
+ ### q5_1, q6_K and q8_0 files require expansion from archive
112
+
113
+ **Note:** HF does not support uploading files larger than 50GB. Therefore I have uploaded the q6_K and q8_0 files as multi-part ZIP files. They are not compressed, they are just for storing a .bin file in two parts.
114
+
115
+ <details>
116
+ <summary>Click for instructions regarding q5_1, q6_K and q8_0 files</summary>
117
+
118
+ ### q5_1
119
+ Please download:
120
+ * `samantha-1.1-70b.ggmlv3.q5_1.zip`
121
+ * `samantha-1.1-70b.ggmlv3.q5_1.z01`
122
+
123
+ ### q6_K
124
+ Please download:
125
+ * `samantha-1.1-70b.ggmlv3.q6_K.zip`
126
+ * `samantha-1.1-70b.ggmlv3.q6_K.z01`
127
+
128
+ ### q8_0
129
+ Please download:
130
+ * `samantha-1.1-70b.ggmlv3.q8_0.zip`
131
+ * `samantha-1.1-70b.ggmlv3.q8_0.z01`
132
+
133
+ Then extract the .zip archive. This will will expand both parts automatically. On Linux I found I had to use `7zip` - the basic `unzip` tool did not work. Example:
134
+ ```
135
+ sudo apt update -y && sudo apt install 7zip
136
+ 7zz x samantha-1.1-70b.ggmlv3.q6_K.zip
137
+ ```
138
+ </details>
139
+
140
  ## How to run in `llama.cpp`
141
 
142
  I use the following command line; adjust for your tastes and needs: