Transformers
English
llama
text-generation-inference
TheBloke commited on
Commit
58029c4
1 Parent(s): cd40570

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +66 -35
README.md CHANGED
@@ -1,10 +1,10 @@
1
  ---
2
  datasets:
3
- - garage-bAInd/OpenPlatypus
4
  inference: false
5
  language:
6
  - en
7
- license: other
8
  model_creator: garage-bAInd
9
  model_link: https://huggingface.co/garage-bAInd/Platypus2-13B
10
  model_name: Platypus2
@@ -13,17 +13,20 @@ quantized_by: TheBloke
13
  ---
14
 
15
  <!-- header start -->
16
- <div style="width: 100%;">
17
- <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
 
18
  </div>
19
  <div style="display: flex; justify-content: space-between; width: 100%;">
20
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
21
- <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
22
  </div>
23
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
24
- <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
25
  </div>
26
  </div>
 
 
27
  <!-- header end -->
28
 
29
  # Platypus2 - GGML
@@ -34,6 +37,13 @@ quantized_by: TheBloke
34
 
35
  This repo contains GGML format model files for [garage-bAInd's Platypus2](https://huggingface.co/garage-bAInd/Platypus2-13B).
36
 
 
 
 
 
 
 
 
37
  GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
38
  * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most popular web UI. Supports NVidia CUDA GPU acceleration.
39
  * [KoboldCpp](https://github.com/LostRuins/koboldcpp), a powerful GGML web UI with GPU acceleration on all platforms (CUDA and OpenCL). Especially good for story telling.
@@ -45,7 +55,8 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
45
  ## Repositories available
46
 
47
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/Platypus2-13B-GPTQ)
48
- * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/Platypus2-13B-GGML)
 
49
  * [garage-bAInd's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/garage-bAInd/Platypus2-13B)
50
 
51
  ## Prompt template: Alpaca
@@ -57,14 +68,19 @@ Below is an instruction that describes a task. Write a response that appropriate
57
  {prompt}
58
 
59
  ### Response:
 
60
  ```
61
 
62
  <!-- compatibility_ggml start -->
63
  ## Compatibility
64
 
65
- These quantised GGML files are compatible with llama.cpp as of June 6th, commit `2d43387`.
66
 
67
- They should also be compatible with all UIs, libraries and utilities which use GGML.
 
 
 
 
68
 
69
  ## Explanation of the new k-quant methods
70
  <details>
@@ -87,17 +103,17 @@ Refer to the Provided Files table below to see what files use which methods, and
87
  | Name | Quant method | Bits | Size | Max RAM required | Use case |
88
  | ---- | ---- | ---- | ---- | ---- | ----- |
89
  | [platypus2-13b.ggmlv3.q2_K.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q2_K.bin) | q2_K | 2 | 5.51 GB| 8.01 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
90
- | [platypus2-13b.ggmlv3.q3_K_L.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_L.bin) | q3_K_L | 3 | 6.93 GB| 9.43 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
91
- | [platypus2-13b.ggmlv3.q3_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_M.bin) | q3_K_M | 3 | 6.31 GB| 8.81 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
92
  | [platypus2-13b.ggmlv3.q3_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_S.bin) | q3_K_S | 3 | 5.66 GB| 8.16 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
 
 
93
  | [platypus2-13b.ggmlv3.q4_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_0.bin) | q4_0 | 4 | 7.37 GB| 9.87 GB | Original quant method, 4-bit. |
94
- | [platypus2-13b.ggmlv3.q4_1.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_1.bin) | q4_1 | 4 | 8.17 GB| 10.67 GB | Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
95
- | [platypus2-13b.ggmlv3.q4_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_K_M.bin) | q4_K_M | 4 | 7.87 GB| 10.37 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
96
  | [platypus2-13b.ggmlv3.q4_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_K_S.bin) | q4_K_S | 4 | 7.37 GB| 9.87 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
 
 
97
  | [platypus2-13b.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_0.bin) | q5_0 | 5 | 8.97 GB| 11.47 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
98
- | [platypus2-13b.ggmlv3.q5_1.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_1.bin) | q5_1 | 5 | 9.78 GB| 12.28 GB | Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
99
- | [platypus2-13b.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 9.23 GB| 11.73 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
100
  | [platypus2-13b.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 8.97 GB| 11.47 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
 
 
101
  | [platypus2-13b.ggmlv3.q6_K.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q6_K.bin) | q6_K | 6 | 10.68 GB| 13.18 GB | New k-quant method. Uses GGML_TYPE_Q8_K for all tensors - 6-bit quantization |
102
  | [platypus2-13b.ggmlv3.q8_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q8_0.bin) | q8_0 | 8 | 13.79 GB| 16.29 GB | Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
103
 
@@ -105,10 +121,12 @@ Refer to the Provided Files table below to see what files use which methods, and
105
 
106
  ## How to run in `llama.cpp`
107
 
108
- I use the following command line; adjust for your tastes and needs:
 
 
109
 
110
  ```
111
- ./main -t 10 -ngl 32 -m platypus2-13b.ggmlv3.q4_K_M.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
112
  ```
113
  Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
114
 
@@ -122,9 +140,10 @@ For other parameters and how to use them, please refer to [the llama.cpp documen
122
 
123
  ## How to run in `text-generation-webui`
124
 
125
- Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
126
 
127
  <!-- footer start -->
 
128
  ## Discord
129
 
130
  For further support, and discussions on these models and AI in general, join us at:
@@ -144,13 +163,15 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
144
  * Patreon: https://patreon.com/TheBlokeAI
145
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
146
 
147
- **Special thanks to**: Luke from CarbonQuill, Aemon Algiz.
148
 
149
- **Patreon special mentions**: Willem Michiel, Ajan Kanaga, Cory Kujawski, Alps Aficionado, Nikolai Manek, Jonathan Leane, Stanislav Ovsiannikov, Michael Levine, Luke Pendergrass, Sid, K, Gabriel Tamborski, Clay Pascal, Kalila, William Sang, Will Dee, Pieter, Nathan LeClaire, ya boyyy, David Flickinger, vamX, Derek Yates, Fen Risland, Jeffrey Morgan, webtim, Daniel P. Andersen, Chadd, Edmond Seymore, Pyrater, Olusegun Samson, Lone Striker, biorpg, alfie_i, Mano Prime, Chris Smitley, Dave, zynix, Trenton Dambrowitz, Johann-Peter Hartmann, Magnesian, Spencer Kim, John Detwiler, Iucharbius, Gabriel Puliatti, LangChain4j, Luke @flexchar, Vadim, Rishabh Srivastava, Preetika Verma, Ai Maven, Femi Adebogun, WelcomeToTheClub, Leonard Tan, Imad Khwaja, Steven Wood, Stefan Sabev, Sebastain Graf, usrbinkat, Dan Guido, Sam, Eugene Pentland, Mandus, transmissions 11, Slarti, Karl Bernard, Spiking Neurons AB, Artur Olbinski, Joseph William Delisle, ReadyPlayerEmma, Olakabola, Asp the Wyvern, Space Cruiser, Matthew Berman, Randy H, subjectnull, danny, John Villwock, Illia Dulskyi, Rainer Wilmers, theTransient, Pierre Kircher, Alexandros Triantafyllidis, Viktor Bowallius, terasurfer, Deep Realms, SuperWojo, senxiiz, Oscar Rangel, Alex, Stephen Murray, Talal Aujan, Raven Klaugh, Sean Connelly, Raymond Fosdick, Fred von Graf, chris gileta, Junyu Yang, Elle
150
 
151
 
152
  Thank you to all my generous patrons and donaters!
153
 
 
 
154
  <!-- footer end -->
155
 
156
  # Original model card: garage-bAInd's Platypus2
@@ -166,11 +187,11 @@ Platypus-13B is an instruction fine-tuned model based on the LLaMA2-13B transfor
166
 
167
  | Metric | Value |
168
  |-----------------------|-------|
169
- | MMLU (5-shot) | - |
170
- | ARC (25-shot) | - |
171
- | HellaSwag (10-shot) | - |
172
- | TruthfulQA (0-shot) | - |
173
- | Avg. | - |
174
 
175
  We use state-of-the-art [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, using the same version as the HuggingFace LLM Leaderboard. Please see below for detailed instructions on reproducing benchmark results.
176
 
@@ -192,7 +213,9 @@ We use state-of-the-art [Language Model Evaluation Harness](https://github.com/E
192
 
193
  ### Training Dataset
194
 
195
- STEM and logic based dataset [`garage-bAInd/OpenPlatypus`](https://huggingface.co/datasets/garage-bAInd/OpenPlatypus) [COMING SOON!]
 
 
196
 
197
  ### Training Procedure
198
 
@@ -211,7 +234,7 @@ cd lm-evaluation-harness
211
  # install
212
  pip install -e .
213
  ```
214
- Each task was evaluated on a single A100 80GB GPU.
215
 
216
  ARC:
217
  ```
@@ -239,21 +262,29 @@ Llama 2 and fine-tuned variants are a new technology that carries risks with use
239
  Please see the Responsible Use Guide available at https://ai.meta.com/llama/responsible-use-guide/
240
 
241
  ### Citations
242
-
 
 
 
 
 
 
 
243
  ```bibtex
244
  @misc{touvron2023llama,
245
  title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
246
- author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton Ferrer and Moya Chen and Guillem Cucurull and David Esiobu and Jude Fernandes and Jeremy Fu and Wenyin Fu and Brian Fuller and Cynthia Gao and Vedanuj Goswami and Naman Goyal and Anthony Hartshorn and Saghar Hosseini and Rui Hou and Hakan Inan and Marcin Kardas and Viktor Kerkez and Madian Khabsa and Isabel Kloumann and Artem Korenev and Punit Singh Koura and Marie-Anne Lachaux and Thibaut Lavril and Jenya Lee and Diana Liskovich and Yinghai Lu and Yuning Mao and Xavier Martinet and Todor Mihaylov and Pushkar Mishra and Igor Molybog and Yixin Nie and Andrew Poulton and Jeremy Reizenstein and Rashi Rungta and Kalyan Saladi and Alan Schelten and Ruan Silva and Eric Michael Smith and Ranjan Subramanian and Xiaoqing Ellen Tan and Binh Tang and Ross Taylor and Adina Williams and Jian Xiang Kuan and Puxin Xu and Zheng Yan and Iliyan Zarov and Yuchen Zhang and Angela Fan and Melanie Kambadur and Sharan Narang and Aurelien Rodriguez and Robert Stojnic and Sergey Edunov and Thomas Scialom},
247
- year={2023},
248
  eprint={2307.09288},
249
  archivePrefix={arXiv},
250
  }
251
  ```
252
  ```bibtex
253
- @article{hu2021lora,
254
- title={LoRA: Low-Rank Adaptation of Large Language Models},
255
- author={Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Chen, Weizhu},
256
- journal={CoRR},
257
- year={2021}
 
 
258
  }
259
  ```
 
1
  ---
2
  datasets:
3
+ - garage-bAInd/Open-Platypus
4
  inference: false
5
  language:
6
  - en
7
+ license: llama2
8
  model_creator: garage-bAInd
9
  model_link: https://huggingface.co/garage-bAInd/Platypus2-13B
10
  model_name: Platypus2
 
13
  ---
14
 
15
  <!-- header start -->
16
+ <!-- 200823 -->
17
+ <div style="width: auto; margin-left: auto; margin-right: auto">
18
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
19
  </div>
20
  <div style="display: flex; justify-content: space-between; width: 100%;">
21
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
22
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
23
  </div>
24
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
25
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
26
  </div>
27
  </div>
28
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
29
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
30
  <!-- header end -->
31
 
32
  # Platypus2 - GGML
 
37
 
38
  This repo contains GGML format model files for [garage-bAInd's Platypus2](https://huggingface.co/garage-bAInd/Platypus2-13B).
39
 
40
+ ### Important note regarding GGML files.
41
+
42
+ The GGML format has now been superseded by GGUF. As of August 21st 2023, [llama.cpp](https://github.com/ggerganov/llama.cpp) no longer supports GGML models. Third party clients and libraries are expected to still support it for a time, but many may also drop support.
43
+
44
+ Please use the GGUF models instead.
45
+ ### About GGML
46
+
47
  GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
48
  * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most popular web UI. Supports NVidia CUDA GPU acceleration.
49
  * [KoboldCpp](https://github.com/LostRuins/koboldcpp), a powerful GGML web UI with GPU acceleration on all platforms (CUDA and OpenCL). Especially good for story telling.
 
55
  ## Repositories available
56
 
57
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/Platypus2-13B-GPTQ)
58
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/Platypus2-13B-GGUF)
59
+ * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference (deprecated)](https://huggingface.co/TheBloke/Platypus2-13B-GGML)
60
  * [garage-bAInd's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/garage-bAInd/Platypus2-13B)
61
 
62
  ## Prompt template: Alpaca
 
68
  {prompt}
69
 
70
  ### Response:
71
+
72
  ```
73
 
74
  <!-- compatibility_ggml start -->
75
  ## Compatibility
76
 
77
+ These quantised GGML files are compatible with llama.cpp between June 6th (commit `2d43387`) and August 21st 2023.
78
 
79
+ For support with latest llama.cpp, please use GGUF files instead.
80
+
81
+ The final llama.cpp commit with support for GGML was: [dadbed99e65252d79f81101a392d0d6497b86caa](https://github.com/ggerganov/llama.cpp/commit/dadbed99e65252d79f81101a392d0d6497b86caa)
82
+
83
+ As of August 23rd 2023 they are still compatible with all UIs, libraries and utilities which use GGML. This may change in the future.
84
 
85
  ## Explanation of the new k-quant methods
86
  <details>
 
103
  | Name | Quant method | Bits | Size | Max RAM required | Use case |
104
  | ---- | ---- | ---- | ---- | ---- | ----- |
105
  | [platypus2-13b.ggmlv3.q2_K.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q2_K.bin) | q2_K | 2 | 5.51 GB| 8.01 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
 
 
106
  | [platypus2-13b.ggmlv3.q3_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_S.bin) | q3_K_S | 3 | 5.66 GB| 8.16 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
107
+ | [platypus2-13b.ggmlv3.q3_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_M.bin) | q3_K_M | 3 | 6.31 GB| 8.81 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
108
+ | [platypus2-13b.ggmlv3.q3_K_L.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q3_K_L.bin) | q3_K_L | 3 | 6.93 GB| 9.43 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
109
  | [platypus2-13b.ggmlv3.q4_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_0.bin) | q4_0 | 4 | 7.37 GB| 9.87 GB | Original quant method, 4-bit. |
 
 
110
  | [platypus2-13b.ggmlv3.q4_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_K_S.bin) | q4_K_S | 4 | 7.37 GB| 9.87 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
111
+ | [platypus2-13b.ggmlv3.q4_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_K_M.bin) | q4_K_M | 4 | 7.87 GB| 10.37 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
112
+ | [platypus2-13b.ggmlv3.q4_1.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q4_1.bin) | q4_1 | 4 | 8.17 GB| 10.67 GB | Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
113
  | [platypus2-13b.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_0.bin) | q5_0 | 5 | 8.97 GB| 11.47 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
 
 
114
  | [platypus2-13b.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 8.97 GB| 11.47 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
115
+ | [platypus2-13b.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 9.23 GB| 11.73 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
116
+ | [platypus2-13b.ggmlv3.q5_1.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q5_1.bin) | q5_1 | 5 | 9.78 GB| 12.28 GB | Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
117
  | [platypus2-13b.ggmlv3.q6_K.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q6_K.bin) | q6_K | 6 | 10.68 GB| 13.18 GB | New k-quant method. Uses GGML_TYPE_Q8_K for all tensors - 6-bit quantization |
118
  | [platypus2-13b.ggmlv3.q8_0.bin](https://huggingface.co/TheBloke/Platypus2-13B-GGML/blob/main/platypus2-13b.ggmlv3.q8_0.bin) | q8_0 | 8 | 13.79 GB| 16.29 GB | Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
119
 
 
121
 
122
  ## How to run in `llama.cpp`
123
 
124
+ Make sure you are using `llama.cpp` from commit [dadbed99e65252d79f81101a392d0d6497b86caa](https://github.com/ggerganov/llama.cpp/commit/dadbed99e65252d79f81101a392d0d6497b86caa) or earlier.
125
+
126
+ For compatibility with latest llama.cpp, please use GGUF files instead.
127
 
128
  ```
129
+ ./main -t 10 -ngl 32 -m platypus2-13b.ggmlv3.q4_K_M.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a story about llamas\n\n### Response:"
130
  ```
131
  Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
132
 
 
140
 
141
  ## How to run in `text-generation-webui`
142
 
143
+ Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
144
 
145
  <!-- footer start -->
146
+ <!-- 200823 -->
147
  ## Discord
148
 
149
  For further support, and discussions on these models and AI in general, join us at:
 
163
  * Patreon: https://patreon.com/TheBlokeAI
164
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
165
 
166
+ **Special thanks to**: Aemon Algiz.
167
 
168
+ **Patreon special mentions**: Russ Johnson, J, alfie_i, Alex, NimbleBox.ai, Chadd, Mandus, Nikolai Manek, Ken Nordquist, ya boyyy, Illia Dulskyi, Viktor Bowallius, vamX, Iucharbius, zynix, Magnesian, Clay Pascal, Pierre Kircher, Enrico Ros, Tony Hughes, Elle, Andrey, knownsqashed, Deep Realms, Jerry Meng, Lone Striker, Derek Yates, Pyrater, Mesiah Bishop, James Bentley, Femi Adebogun, Brandon Frisco, SuperWojo, Alps Aficionado, Michael Dempsey, Vitor Caleffi, Will Dee, Edmond Seymore, usrbinkat, LangChain4j, Kacper Wikieł, Luke Pendergrass, John Detwiler, theTransient, Nathan LeClaire, Tiffany J. Kim, biorpg, Eugene Pentland, Stanislav Ovsiannikov, Fred von Graf, terasurfer, Kalila, Dan Guido, Nitin Borwankar, 阿明, Ai Maven, John Villwock, Gabriel Puliatti, Stephen Murray, Asp the Wyvern, danny, Chris Smitley, ReadyPlayerEmma, S_X, Daniel P. Andersen, Olakabola, Jeffrey Morgan, Imad Khwaja, Caitlyn Gatomon, webtim, Alicia Loh, Trenton Dambrowitz, Swaroop Kallakuri, Erik Bjäreholt, Leonard Tan, Spiking Neurons AB, Luke @flexchar, Ajan Kanaga, Thomas Belote, Deo Leter, RoA, Willem Michiel, transmissions 11, subjectnull, Matthew Berman, Joseph William Delisle, David Ziegler, Michael Davis, Johann-Peter Hartmann, Talal Aujan, senxiiz, Artur Olbinski, Rainer Wilmers, Spencer Kim, Fen Risland, Cap'n Zoog, Rishabh Srivastava, Michael Levine, Geoffrey Montalvo, Sean Connelly, Alexandros Triantafyllidis, Pieter, Gabriel Tamborski, Sam, Subspace Studios, Junyu Yang, Pedro Madruga, Vadim, Cory Kujawski, K, Raven Klaugh, Randy H, Mano Prime, Sebastain Graf, Space Cruiser
169
 
170
 
171
  Thank you to all my generous patrons and donaters!
172
 
173
+ And thank you again to a16z for their generous grant.
174
+
175
  <!-- footer end -->
176
 
177
  # Original model card: garage-bAInd's Platypus2
 
187
 
188
  | Metric | Value |
189
  |-----------------------|-------|
190
+ | MMLU (5-shot) | 56.70 |
191
+ | ARC (25-shot) | 61.26 |
192
+ | HellaSwag (10-shot) | 82.56 |
193
+ | TruthfulQA (0-shot) | 44.86 |
194
+ | Avg. | 61.35 |
195
 
196
  We use state-of-the-art [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, using the same version as the HuggingFace LLM Leaderboard. Please see below for detailed instructions on reproducing benchmark results.
197
 
 
213
 
214
  ### Training Dataset
215
 
216
+ `garage-bAInd/Platypus2-13B` trained using STEM and logic based dataset [`garage-bAInd/Open-Platypus`](https://huggingface.co/datasets/garage-bAInd/Open-Platypus).
217
+
218
+ Please see our [paper](https://arxiv.org/abs/2308.07317) and [project webpage](https://platypus-llm.github.io) for additional information.
219
 
220
  ### Training Procedure
221
 
 
234
  # install
235
  pip install -e .
236
  ```
237
+ Each task was evaluated on 1 A100 80GB GPU.
238
 
239
  ARC:
240
  ```
 
262
  Please see the Responsible Use Guide available at https://ai.meta.com/llama/responsible-use-guide/
263
 
264
  ### Citations
265
+ ```bibtex
266
+ @article{platypus2023,
267
+ title={Platypus: Quick, Cheap, and Powerful Refinement of LLMs},
268
+ author={Ariel N. Lee and Cole J. Hunter and Nataniel Ruiz},
269
+ booktitle={arXiv preprint arxiv:2308.07317},
270
+ year={2023}
271
+ }
272
+ ```
273
  ```bibtex
274
  @misc{touvron2023llama,
275
  title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
276
+ author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov year={2023},
 
277
  eprint={2307.09288},
278
  archivePrefix={arXiv},
279
  }
280
  ```
281
  ```bibtex
282
+ @inproceedings{
283
+ hu2022lora,
284
+ title={Lo{RA}: Low-Rank Adaptation of Large Language Models},
285
+ author={Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
286
+ booktitle={International Conference on Learning Representations},
287
+ year={2022},
288
+ url={https://openreview.net/forum?id=nZeVKeeFYf9}
289
  }
290
  ```