Llamacpp quants

.gitattributes

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+airoboros-34b-3.2-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,46 @@
+---
+license: other
+license_name: yi-license
+license_link: https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE
+base_model: 01-ai/yi-34b-200k
+datasets:
+- jondurbin/airoboros-3.2
+- bluemoon-fandom-1-1-rp-cleaned
+- boolq
+- jondurbin/gutenberg-dpo-v0.1
+- LDJnr/Capybara
+- jondurbin/cinematika-v0.1
+- glaiveai/glaive-function-calling-v2
+- grimulkan/LimaRP-augmented
+- piqa
+- Vezora/Tested-22k-Python-Alpaca
+- mattpscott/airoboros-summarization
+- unalignment/toxic-dpo-v0.2
+quantized_by: bartowski
+pipeline_tag: text-generation
+---
+
+## Llamacpp Quantizations of airoboros-34b-3.2
+
+Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2334">b2334</a> for quantization.
+
+Original model: https://huggingface.co/jondurbin/airoboros-34b-3.2
+
+Download a file (not the whole branch) from below:
+
+| Filename | Quant type | File Size | Description |
+| -------- | ---------- | --------- | ----------- |
+| [airoboros-34b-3.2-Q8_0.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q8_0.gguf) | Q8_0 | 36.54GB | Extremely high quality, generally unneeded but max available quant. |
+| [airoboros-34b-3.2-Q6_K.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q6_K.gguf) | Q6_K | 28.21GB | Very high quality, near perfect, *recommended*. |
+| [airoboros-34b-3.2-Q5_K_M.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q5_K_M.gguf) | Q5_K_M | 24.32GB | High quality, very usable. |
+| [airoboros-34b-3.2-Q5_K_S.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q5_K_S.gguf) | Q5_K_S | 23.70GB | High quality, very usable. |
+| [airoboros-34b-3.2-Q5_0.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q5_0.gguf) | Q5_0 | 23.70GB | High quality, older format, generally not recommended. |
+| [airoboros-34b-3.2-Q4_K_M.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q4_K_M.gguf) | Q4_K_M | 20.65GB | Good quality, similar to 4.25 bpw. |
+| [airoboros-34b-3.2-Q4_K_S.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q4_K_S.gguf) | Q4_K_S | 19.59GB | Slightly lower quality with small space savings. |
+| [airoboros-34b-3.2-Q4_0.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q4_0.gguf) | Q4_0 | 19.46GB | Decent quality, older format, generally not recommended. |
+| [airoboros-34b-3.2-Q3_K_L.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q3_K_L.gguf) | Q3_K_L | 18.13GB | Lower quality but usable, good for low RAM availability. |
+| [airoboros-34b-3.2-Q3_K_M.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q3_K_M.gguf) | Q3_K_M | 16.65GB | Even lower quality. |
+| [airoboros-34b-3.2-Q3_K_S.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q3_K_S.gguf) | Q3_K_S | 14.96GB | Low quality, not recommended. |
+| [airoboros-34b-3.2-Q2_K.gguf](https://huggingface.co/bartowski/airoboros-34b-3.2-GGUF/blob/main/airoboros-34b-3.2-Q2_K.gguf) | Q2_K | 12.82GB | Extremely low quality, *not* recommended.
+
+Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

airoboros-34b-3.2-Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ce1cb6736f0255c717918c7f949d54052125263184c264eaee3a85b9524c4a9
+size 12825234080

airoboros-34b-3.2-Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75b92b3a4242ba629cc039cdc3d2ad375a912a9ba7b0b005f8c1c8bb6e74e876
+size 18139445920

airoboros-34b-3.2-Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f45a70a0c2b197172e70e6a2b618ed721b3939cc363d4e52a3e0db85abce8a7
+size 16654924448

airoboros-34b-3.2-Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b49fa20dcf223fc681e06068dc870af68384615f99c8f74e0fea1be9835353a
+size 14960294560

airoboros-34b-3.2-Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b366321ddf3ab5611a7d599bd12ca5916e311509d86bd1e19cd9f01e12254b4
+size 19466529440

airoboros-34b-3.2-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65fdf95748add41a934e1b13e0d8d268cefaef7166c6634fd6f6c9ed04fb24c2
+size 20658711200

airoboros-34b-3.2-Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c14b79a68173cb9d8060eb68eb876c6bdf3e278eeac81775208b887fed8462f0
+size 19598650016

airoboros-34b-3.2-Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85efbe041e37783ebcb7d2637406be1106df44ca32781ae3872d5966a92e1dc5
+size 23707691680

airoboros-34b-3.2-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b2b4db971ccbc0b520549bfa73bb92f2790615676f6b54b78777a0c70efa940
+size 24321845920

airoboros-34b-3.2-Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:72c2823fd254ebab6108b9694f51bc0b5ff5f1154ee4dc9776b4322fef9c6246
+size 23707691680

airoboros-34b-3.2-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b731ff22859a1b3cd79cb88c84af853e59f0ec804a6df2368e6abd51e6b7c9ac
+size 28213926560

airoboros-34b-3.2-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8866146ef05675a2322af42957a79ec32795bf6d548e45e2ad068d9a1f397499
+size 36542282400