Hugging Face's logo

bartowski
/
Mistral-7B-v0.2-hf-GGUF

Text Generation
GGUF
Inference Endpoints
Model card Files Community
bartowski commited on
Commit
871d03b
•
1 Parent(s): c4dd562

Llamacpp quants

Browse files
Files changed (18) hide show
  1. .gitattributes +16 -0
  2. Mistral-7B-v0.2-hf-IQ3_M.gguf +3 -0
  3. Mistral-7B-v0.2-hf-IQ3_S.gguf +3 -0
  4. Mistral-7B-v0.2-hf-IQ4_NL.gguf +3 -0
  5. Mistral-7B-v0.2-hf-IQ4_XS.gguf +3 -0
  6. Mistral-7B-v0.2-hf-Q2_K.gguf +3 -0
  7. Mistral-7B-v0.2-hf-Q3_K_L.gguf +3 -0
  8. Mistral-7B-v0.2-hf-Q3_K_M.gguf +3 -0
  9. Mistral-7B-v0.2-hf-Q3_K_S.gguf +3 -0
  10. Mistral-7B-v0.2-hf-Q4_0.gguf +3 -0
  11. Mistral-7B-v0.2-hf-Q4_K_M.gguf +3 -0
  12. Mistral-7B-v0.2-hf-Q4_K_S.gguf +3 -0
  13. Mistral-7B-v0.2-hf-Q5_0.gguf +3 -0
  14. Mistral-7B-v0.2-hf-Q5_K_M.gguf +3 -0
  15. Mistral-7B-v0.2-hf-Q5_K_S.gguf +3 -0
  16. Mistral-7B-v0.2-hf-Q6_K.gguf +3 -0
  17. Mistral-7B-v0.2-hf-Q8_0.gguf +3 -0
  18. README.md +35 -0
.gitattributes CHANGED
@@ -33,3 +33,19 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ Mistral-7B-v0.2-hf-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ Mistral-7B-v0.2-hf-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
38
+ Mistral-7B-v0.2-hf-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
39
+ Mistral-7B-v0.2-hf-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
40
+ Mistral-7B-v0.2-hf-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
41
+ Mistral-7B-v0.2-hf-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
42
+ Mistral-7B-v0.2-hf-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
43
+ Mistral-7B-v0.2-hf-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ Mistral-7B-v0.2-hf-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
45
+ Mistral-7B-v0.2-hf-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
46
+ Mistral-7B-v0.2-hf-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
47
+ Mistral-7B-v0.2-hf-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
48
+ Mistral-7B-v0.2-hf-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
49
+ Mistral-7B-v0.2-hf-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
50
+ Mistral-7B-v0.2-hf-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
51
+ Mistral-7B-v0.2-hf-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-7B-v0.2-hf-IQ3_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68dd2e0e65ee8c833e720b0577d3f0b27bd16400da5baeb55eef80f2741af886
3
+ size 3284891360
Mistral-7B-v0.2-hf-IQ3_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:69e8f5dffa524a667e316c9eb0095a1328551cd04ede29585ab656a593ce5471
3
+ size 3182393056
Mistral-7B-v0.2-hf-IQ4_NL.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8361bf68313be31122eb424afbf99effbdb59dc837e2917da752044b846e4b5e
3
+ size 4155053792
Mistral-7B-v0.2-hf-IQ4_XS.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:565faa3a1c56bc504a20aa23a90f7e7ccdcab7a259dcffedf2e303c88e9ab694
3
+ size 3944388320
Mistral-7B-v0.2-hf-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88e5202e44e71f35cca458e9884412cbab0e252756d7a7f4d9159dc67f012111
3
+ size 2719241952
Mistral-7B-v0.2-hf-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38cc887a8408f2a7c057c2b2232b37c5aebb1cb320c402c04a552a425e1380f4
3
+ size 3822024416
Mistral-7B-v0.2-hf-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:da462e9938706ea42873612fa5ff6a5cec6cac1730e261c69f2d15cfa4825372
3
+ size 3518985952
Mistral-7B-v0.2-hf-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2305a5c2434d0c09b48248957223efd7f2540b5c44ddf3c3bc97819a837db787
3
+ size 3164567264
Mistral-7B-v0.2-hf-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d301361cfde142a612a2fb28378db3d87ea1ebff4f7340388901580137ba47c5
3
+ size 4108916448
Mistral-7B-v0.2-hf-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f691ab1e80fd75aa5be44909af3c2a8b7d4146fa05e6e886dd0aab68f90af1d
3
+ size 4368439008
Mistral-7B-v0.2-hf-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33b29d534d56b10a46312ef56464638c36e49afb3b01d15bc23c64e1fc996e3e
3
+ size 4140373728
Mistral-7B-v0.2-hf-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:706537dc70a697be59ae9290dc92a751a26fe98c0ceece25ed588529a8846917
3
+ size 4997715680
Mistral-7B-v0.2-hf-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:275326e371b4c00c37206e1fb1e2abfe25b73bb1ee5bb937f8dbecbfe35ebcc1
3
+ size 5131409120
Mistral-7B-v0.2-hf-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e693fde6f1b26ba8ab392c11eec908914f08d6a86635a80f92e0ea801d8c942f
3
+ size 4997715680
Mistral-7B-v0.2-hf-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d01ba42c3937207fb7028398fb4aeedd7f27ca981aef1c3a0dbab7936d619a7
3
+ size 5942064864
Mistral-7B-v0.2-hf-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:53c948baa5fddb211fe616b7885f791549dbb0dd458f0b99de563b10133c087b
3
+ size 7695857376
README.md ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ pipeline_tag: text-generation
4
+ inference: true
5
+ quantized_by: bartowski
6
+ ---
7
+
8
+ ## Llamacpp Quantizations of Mistral-7B-v0.2-hf
9
+
10
+ Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2440">b2440</a> for quantization.
11
+
12
+ Original model: https://huggingface.co/alpindale/Mistral-7B-v0.2-hf/
13
+
14
+ Download a file (not the whole branch) from below:
15
+
16
+ | Filename | Quant type | File Size | Description |
17
+ | -------- | ---------- | --------- | ----------- |
18
+ | [Mistral-7B-v0.2-hf-Q8_0.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q8_0.gguf) | Q8_0 | 7.69GB | Extremely high quality, generally unneeded but max available quant. |
19
+ | [Mistral-7B-v0.2-hf-Q6_K.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q6_K.gguf) | Q6_K | 5.94GB | Very high quality, near perfect, *recommended*. |
20
+ | [Mistral-7B-v0.2-hf-Q5_K_M.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q5_K_M.gguf) | Q5_K_M | 5.13GB | High quality, very usable. |
21
+ | [Mistral-7B-v0.2-hf-Q5_K_S.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q5_K_S.gguf) | Q5_K_S | 4.99GB | High quality, very usable. |
22
+ | [Mistral-7B-v0.2-hf-Q5_0.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q5_0.gguf) | Q5_0 | 4.99GB | High quality, older format, generally not recommended. |
23
+ | [Mistral-7B-v0.2-hf-Q4_K_M.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q4_K_M.gguf) | Q4_K_M | 4.36GB | Good quality, similar to 4.25 bpw. |
24
+ | [Mistral-7B-v0.2-hf-Q4_K_S.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q4_K_S.gguf) | Q4_K_S | 4.14GB | Slightly lower quality with small space savings. |
25
+ | [Mistral-7B-v0.2-hf-IQ4_NL.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-IQ4_NL.gguf) | IQ4_NL | 4.15GB | Good quality, similar to Q4_K_S, new method of quanting, |
26
+ | [Mistral-7B-v0.2-hf-IQ4_XS.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-IQ4_XS.gguf) | IQ4_XS | 3.94GB | Decent quality, new method with similar performance to Q4. |
27
+ | [Mistral-7B-v0.2-hf-Q4_0.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q4_0.gguf) | Q4_0 | 4.10GB | Decent quality, older format, generally not recommended. |
28
+ | [Mistral-7B-v0.2-hf-IQ3_M.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-IQ3_M.gguf) | IQ3_M | 3.28GB | Medium-low quality, new method with decent performance. |
29
+ | [Mistral-7B-v0.2-hf-IQ3_S.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-IQ3_S.gguf) | IQ3_S | 3.18GB | Lower quality, new method with decent performance, recommended over Q3 quants. |
30
+ | [Mistral-7B-v0.2-hf-Q3_K_L.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q3_K_L.gguf) | Q3_K_L | 3.82GB | Lower quality but usable, good for low RAM availability. |
31
+ | [Mistral-7B-v0.2-hf-Q3_K_M.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q3_K_M.gguf) | Q3_K_M | 3.51GB | Even lower quality. |
32
+ | [Mistral-7B-v0.2-hf-Q3_K_S.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q3_K_S.gguf) | Q3_K_S | 3.16GB | Low quality, not recommended. |
33
+ | [Mistral-7B-v0.2-hf-Q2_K.gguf](https://huggingface.co/bartowski/Mistral-7B-v0.2-hf-GGUF/blob/main/Mistral-7B-v0.2-hf-Q2_K.gguf) | Q2_K | 2.71GB | Extremely low quality, *not* recommended.
34
+
35
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski