Hugging Face's logo

bartowski
/
merlinite-7b-GGUF

Text Generation
GGUF
English
merlinite
mistral
ibm
lab
labrador
labradorite
Inference Endpoints
Model card Files Community
bartowski commited on
Commit
3d5d961
·
verified ·
1 Parent(s): e1f31f4

Llamacpp quants

Browse files
Files changed (15) hide show
  1. .gitattributes +13 -0
  2. README.md +40 -0
  3. merlinite-7b-Q2_K.gguf +3 -0
  4. merlinite-7b-Q3_K_L.gguf +3 -0
  5. merlinite-7b-Q3_K_M.gguf +3 -0
  6. merlinite-7b-Q3_K_S.gguf +3 -0
  7. merlinite-7b-Q4_0.gguf +3 -0
  8. merlinite-7b-Q4_K_M.gguf +3 -0
  9. merlinite-7b-Q4_K_S.gguf +3 -0
  10. merlinite-7b-Q5_0.gguf +3 -0
  11. merlinite-7b-Q5_K_M.gguf +3 -0
  12. merlinite-7b-Q5_K_S.gguf +3 -0
  13. merlinite-7b-Q6_K.gguf +3 -0
  14. merlinite-7b-Q8_0.gguf +3 -0
  15. merlinite-7b-fp16.gguf +3 -0
.gitattributes CHANGED
@@ -33,3 +33,16 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ merlinite-7b-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
37
+ merlinite-7b-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
38
+ merlinite-7b-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
39
+ merlinite-7b-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
40
+ merlinite-7b-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
41
+ merlinite-7b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
42
+ merlinite-7b-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
43
+ merlinite-7b-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
44
+ merlinite-7b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
45
+ merlinite-7b-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
46
+ merlinite-7b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
47
+ merlinite-7b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
48
+ merlinite-7b-fp16.gguf filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: text-generation
3
+ tags:
4
+ - merlinite
5
+ - mistral
6
+ - ibm
7
+ - lab
8
+ - labrador
9
+ - labradorite
10
+ license: apache-2.0
11
+ language:
12
+ - en
13
+ base_model: mistralai/Mistral-7B-v0.1
14
+ quantized_by: bartowski
15
+ ---
16
+
17
+ ## Llamacpp Quantizations of merlinite-7b
18
+
19
+ Using <a href="https://github.com/ggerganov/llama.cpp/commit/fa974646e1a2024fc7dc9e6f27cf1f2f5d4a3763">llama.cpp commit fa97464</a> for quantization.
20
+
21
+ Original model: https://huggingface.co/ibm/merlinite-7b
22
+
23
+ Download a file (not the whole branch) from below:
24
+
25
+ | Filename | Quant type | File Size | Description |
26
+ | -------- | ---------- | --------- | ----------- |
27
+ | [merlinite-7b-Q8_0.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q8_0.gguf) | Q8_0 | 7.69GB | Extremely high quality, generally unneeded but max available quant. |
28
+ | [merlinite-7b-Q6_K.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q6_K.gguf) | Q6_K | 5.94GB | Very high quality, near perfect, *recommended*. |
29
+ | [merlinite-7b-Q5_K_M.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q5_K_M.gguf) | Q5_K_M | 5.13GB | High quality, very usable. |
30
+ | [merlinite-7b-Q5_K_S.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q5_K_S.gguf) | Q5_K_S | 4.99GB | High quality, very usable. |
31
+ | [merlinite-7b-Q5_0.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q5_0.gguf) | Q5_0 | 4.99GB | High quality, older format, generally not recommended. |
32
+ | [merlinite-7b-Q4_K_M.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q4_K_M.gguf) | Q4_K_M | 4.36GB | Good quality, similar to 4.25 bpw. |
33
+ | [merlinite-7b-Q4_K_S.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q4_K_S.gguf) | Q4_K_S | 4.14GB | Slightly lower quality with small space savings. |
34
+ | [merlinite-7b-Q4_0.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q4_0.gguf) | Q4_0 | 4.10GB | Decent quality, older format, generally not recommended. |
35
+ | [merlinite-7b-Q3_K_L.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q3_K_L.gguf) | Q3_K_L | 3.82GB | Lower quality but usable, good for low RAM availability. |
36
+ | [merlinite-7b-Q3_K_M.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q3_K_M.gguf) | Q3_K_M | 3.51GB | Even lower quality. |
37
+ | [merlinite-7b-Q3_K_S.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q3_K_S.gguf) | Q3_K_S | 3.16GB | Low quality, not recommended. |
38
+ | [merlinite-7b-Q2_K.gguf](https://huggingface.co/bartowski/merlinite-7b-GGUF/blob/main/merlinite-7b-Q2_K.gguf) | Q2_K | 2.71GB | Extremely low quality, *not* recommended.
39
+
40
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
merlinite-7b-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:86be139410efadba3df35ebc4229fd45c2e89b19752747115a1a1e6f929d6dd0
3
+ size 2719279808
merlinite-7b-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:73e4ca0d82947bdf52fc8f42e5fc252aaee9e12c2dec1490378b4767a35e05ea
3
+ size 3822065600
merlinite-7b-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6df53c92a6940a893ee2ed14a7aae0777269fd8adca9060e04bbfd0e2327504f
3
+ size 3519027136
merlinite-7b-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6286b928ed15585db3d375702613efe039c8d308520bdf93d323f10f43ccefa8
3
+ size 3164608448
merlinite-7b-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b7248e6def189b6a35c493683fb41eef510e8154fb98078bef517a885fcbb94
3
+ size 4108961984
merlinite-7b-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b910d3cb28e4b31600447bef0b2572876d92d3aecdcb489552a90372228af463
3
+ size 4368484544
merlinite-7b-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:00dc1b9d1feff8c99a45d3a1bd05abb332c2d8fb54bfd3e7a2776ded58ae4d9a
3
+ size 4140419264
merlinite-7b-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62eca69094a0d646d338ccba6c6f210f09cf4e008cf4074dcd985451c5944d13
3
+ size 4997765312
merlinite-7b-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee7bc7a579910761bc9586120de9e6197e490381677a2f349b86e97289120303
3
+ size 5131458752
merlinite-7b-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5f5f89b7043767be3fbfe44a8e58fed0cb2490d169245454a2c554b1dbfab7c4
3
+ size 4997765312
merlinite-7b-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42955fc45554c6180e4b4d5c83ebda70accd6233be2a85e93af0ef40393337a5
3
+ size 5942118848
merlinite-7b-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:563a32e57fe1e951ae8e184d9ec07c402a33576b2bb313d64d2cdf2a27cc33f7
3
+ size 7695927232
merlinite-7b-fp16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fffd73c670a3a6a0b48b25a39243d3b28d986a10d5ad15803f8a2887b9957bf4
3
+ size 14484862880