Hugging Face's logo

bartowski
/
Rhea-72b-v0.5-GGUF

Text Generation
Transformers
GGUF
English
Inference Endpoints
Model card Files Community
2
bartowski commited on
Commit
f88a025
•
1 Parent(s): aa5844b

Llamacpp quants

Browse files
Files changed (31) hide show
  1. .gitattributes +25 -0
  2. README.md +37 -0
  3. Rhea-72b-v0.5-IQ3_M.gguf +3 -0
  4. Rhea-72b-v0.5-IQ3_S.gguf +3 -0
  5. Rhea-72b-v0.5-IQ4_NL.gguf +3 -0
  6. Rhea-72b-v0.5-IQ4_XS.gguf +3 -0
  7. Rhea-72b-v0.5-Q2_K.gguf +3 -0
  8. Rhea-72b-v0.5-Q3_K_L.gguf +3 -0
  9. Rhea-72b-v0.5-Q3_K_M.gguf +3 -0
  10. Rhea-72b-v0.5-Q3_K_S.gguf +3 -0
  11. Rhea-72b-v0.5-Q4_0.gguf +3 -0
  12. Rhea-72b-v0.5-Q4_K_M.gguf +3 -0
  13. Rhea-72b-v0.5-Q4_K_S.gguf +3 -0
  14. Rhea-72b-v0.5-Q5_0.gguf +3 -0
  15. Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_a +3 -0
  16. Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_b +3 -0
  17. Rhea-72b-v0.5-Q5_K_M.gguf/combine.sh +2 -0
  18. Rhea-72b-v0.5-Q5_K_S.gguf +3 -0
  19. Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_a +3 -0
  20. Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_b +3 -0
  21. Rhea-72b-v0.5-Q6_K.gguf/combine.sh +2 -0
  22. Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_a +3 -0
  23. Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_b +3 -0
  24. Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_c +3 -0
  25. Rhea-72b-v0.5-Q8_0.gguf/combine.sh +2 -0
  26. Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_a +3 -0
  27. Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_b +3 -0
  28. Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_c +3 -0
  29. Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_d +3 -0
  30. Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_e +3 -0
  31. Rhea-72b-v0.5-fp16.gguf/combine.sh +2 -0
.gitattributes CHANGED
@@ -33,3 +33,28 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ Rhea-72b-v0.5-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ Rhea-72b-v0.5-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
38
+ Rhea-72b-v0.5-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
39
+ Rhea-72b-v0.5-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
40
+ Rhea-72b-v0.5-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
41
+ Rhea-72b-v0.5-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
42
+ Rhea-72b-v0.5-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
43
+ Rhea-72b-v0.5-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ Rhea-72b-v0.5-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
45
+ Rhea-72b-v0.5-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
46
+ Rhea-72b-v0.5-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
47
+ Rhea-72b-v0.5-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
48
+ Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_a filter=lfs diff=lfs merge=lfs -text
49
+ Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_b filter=lfs diff=lfs merge=lfs -text
50
+ Rhea-72b-v0.5-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
51
+ Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_a filter=lfs diff=lfs merge=lfs -text
52
+ Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_b filter=lfs diff=lfs merge=lfs -text
53
+ Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_a filter=lfs diff=lfs merge=lfs -text
54
+ Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_b filter=lfs diff=lfs merge=lfs -text
55
+ Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_c filter=lfs diff=lfs merge=lfs -text
56
+ Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_a filter=lfs diff=lfs merge=lfs -text
57
+ Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_b filter=lfs diff=lfs merge=lfs -text
58
+ Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_c filter=lfs diff=lfs merge=lfs -text
59
+ Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_d filter=lfs diff=lfs merge=lfs -text
60
+ Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_e filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ language:
5
+ - en
6
+ quantized_by: bartowski
7
+ pipeline_tag: text-generation
8
+ ---
9
+
10
+ ## Llamacpp Quantizations of Rhea-72b-v0.5
11
+
12
+ Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2536">b2536</a> for quantization.
13
+
14
+ Original model: https://huggingface.co/davidkim205/Rhea-72b-v0.5
15
+
16
+ Download a file (not the whole branch) from below:
17
+
18
+ | Filename | Quant type | File Size | Description |
19
+ | -------- | ---------- | --------- | ----------- |
20
+ | [Rhea-72b-v0.5-Q8_0.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/tree/main/Rhea-72b-v0.5-Q8_0.gguf) | Q8_0 | 76.82GB | Extremely high quality, generally unneeded but max available quant. |
21
+ | [Rhea-72b-v0.5-Q6_K.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/tree/main/Rhea-72b-v0.5-Q6_K.gguf) | Q6_K | 59.31GB | Very high quality, near perfect, *recommended*. |
22
+ | [Rhea-72b-v0.5-Q5_K_M.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/tree/main/Rhea-72b-v0.5-Q5_K_M.gguf) | Q5_K_M | 51.30GB | High quality, very usable. |
23
+ | [Rhea-72b-v0.5-Q5_K_S.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q5_K_S.gguf) | Q5_K_S | 49.88GB | High quality, very usable. |
24
+ | [Rhea-72b-v0.5-Q5_0.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q5_0.gguf) | Q5_0 | 49.88GB | High quality, older format, generally not recommended. |
25
+ | [Rhea-72b-v0.5-Q4_K_M.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q4_K_M.gguf) | Q4_K_M | 43.77GB | Good quality, uses about 4.83 bits per weight. |
26
+ | [Rhea-72b-v0.5-Q4_K_S.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q4_K_S.gguf) | Q4_K_S | 41.28GB | Slightly lower quality with small space savings. |
27
+ | [Rhea-72b-v0.5-IQ4_NL.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-IQ4_NL.gguf) | IQ4_NL | 41.25GB | Decent quality, similar to Q4_K_S, new method of quanting, |
28
+ | [Rhea-72b-v0.5-IQ4_XS.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-IQ4_XS.gguf) | IQ4_XS | 39.09GB | Decent quality, new method with similar performance to Q4. |
29
+ | [Rhea-72b-v0.5-Q4_0.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q4_0.gguf) | Q4_0 | 41.00GB | Decent quality, older format, generally not recommended. |
30
+ | [Rhea-72b-v0.5-Q3_K_L.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q3_K_L.gguf) | Q3_K_L | 38.48GB | Lower quality but usable, good for low RAM availability. |
31
+ | [Rhea-72b-v0.5-Q3_K_M.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q3_K_M.gguf) | Q3_K_M | 35.27GB | Even lower quality. |
32
+ | [Rhea-72b-v0.5-IQ3_M.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-IQ3_M.gguf) | IQ3_M | 33.26GB | Medium-low quality, new method with decent performance. |
33
+ | [Rhea-72b-v0.5-IQ3_S.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-IQ3_S.gguf) | IQ3_S | 31.56GB | Lower quality, new method with decent performance, recommended over Q3 quants. |
34
+ | [Rhea-72b-v0.5-Q3_K_S.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q3_K_S.gguf) | Q3_K_S | 31.56GB | Low quality, not recommended. |
35
+ | [Rhea-72b-v0.5-Q2_K.gguf](https://huggingface.co/bartowski/Rhea-72b-v0.5-GGUF/blob/main/Rhea-72b-v0.5-Q2_K.gguf) | Q2_K | 27.08GB | Extremely low quality, *not* recommended.
36
+
37
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
Rhea-72b-v0.5-IQ3_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57cf44adff40af2de1eaace2d14da117ec1d4b0338fb64e2727d278cdaea02db
3
+ size 33262220288
Rhea-72b-v0.5-IQ3_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:346713bb3614c12b6c0e8c62cf1393a3e9455e32d47dfd8bfe17b4dbf635246f
3
+ size 31568770048
Rhea-72b-v0.5-IQ4_NL.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3cb33b88f0b208beec87c413d236d7adb36e323a02f3d52e140e52f9b1b300ab
3
+ size 41255285760
Rhea-72b-v0.5-IQ4_XS.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:304b4f3f143edae182449ae4e5e199f052c25d1024d52fe4123a9f6a17faadc0
3
+ size 39098233856
Rhea-72b-v0.5-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf305bf932fb91d06692bdc8075e86f9fe9321d762c1fffb079afa5201b89eb4
3
+ size 27080176640
Rhea-72b-v0.5-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f530e5081c7d60aae8aac971638c6aa293b9536b94de01176abdf39866092e3
3
+ size 38489371648
Rhea-72b-v0.5-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e450f6c467eda160634e28ff7217e82b9070b57f3955480a3613e5981bc61b95
3
+ size 35276534784
Rhea-72b-v0.5-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:504d8aa21edeca9483dc7fe67c03dc209446a854222c3d0c93621fee64a72573
3
+ size 31568770048
Rhea-72b-v0.5-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1c8e275fd82ca7f03666826efdb48e2eea611a6f2074848f42b0970ddaa6f4df
3
+ size 41003627520
Rhea-72b-v0.5-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4debd3ee13a6216695718082c6b7b6d055daa545acfa4f26da147a2d7c7f6201
3
+ size 43771868160
Rhea-72b-v0.5-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:87f48d8cc69ffde5bfd48f1518c2a90985c5522b049b75c8d5eeb6f2728ae1c1
3
+ size 41288840192
Rhea-72b-v0.5-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13cee8ca0caff619e2d0d7a0cd11c53f5e8164f662e9296ce2c5a25c59a9588d
3
+ size 49883493376
Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b18699020c5a7dd4c6ee049d77abbe0c81669e79cf4235d8be78140397181135
3
+ size 25654778369
Rhea-72b-v0.5-Q5_K_M.gguf/Rhea-72b-v0.5-Q5_K_M_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bf5264535cb74bc5fde5ba74be8c6a7018a00efe89cb8b087c0a508320b2911
3
+ size 25654778367
Rhea-72b-v0.5-Q5_K_M.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat Rhea-72b-v0.5-Q5_K_M_part_* > "Rhea-72b-v0.5-Q5_K_M.gguf"
Rhea-72b-v0.5-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:44b8b57f5bfa22ed330ea377b74eb554804287554204bd2cc13380ce3c856085
3
+ size 49883493376
Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:809ea709598be44284ccf30e956f28d2c5639b4b05ff7f514745cbc10b954301
3
+ size 29659175425
Rhea-72b-v0.5-Q6_K.gguf/Rhea-72b-v0.5-Q6_K_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b532c3610ac8e0fc12494df46237befde20dca81c6b743bbed1ac0a102ae1400
3
+ size 29659175423
Rhea-72b-v0.5-Q6_K.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat Rhea-72b-v0.5-Q6_K_part_* > "Rhea-72b-v0.5-Q6_K.gguf"
Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:52f23af811f0d8678d463c949bd8bcec55f6683a0e6c2fd63eb8151a2738a950
3
+ size 25608261974
Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae7857d8aece6503e66238a593be7102173dcf748aa7f85f3c21a46b9bed06af
3
+ size 25608261974
Rhea-72b-v0.5-Q8_0.gguf/Rhea-72b-v0.5-Q8_0_part_c ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51e04b1a3d9653a478bbb4f7ad7be8d98542abaecabb2bcac7d7746a2bdc563d
3
+ size 25608261972
Rhea-72b-v0.5-Q8_0.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat Rhea-72b-v0.5-Q8_0_part_* > "Rhea-72b-v0.5-Q8_0.gguf"
Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30cc2b108ac71c842a1560bc9769b93c99278fa74b16124c9bd3003c715fc338
3
+ size 28918326266
Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7fb379049e77b73ff3b1ed0370f7295b7de835f3ffee6b9a1b238894e4c7c9a6
3
+ size 28918326266
Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_c ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6d59674286b32c10c264ee6cc8a689445612f07b979c6e6e553fe08853e16916
3
+ size 28918326266
Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_d ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:55a3ad593c726e56c24d466d4b11206643cf96d74f56525f499dbabd13d6a01b
3
+ size 28918326266
Rhea-72b-v0.5-fp16.gguf/Rhea-72b-v0.5-fp16_part_e ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b06d22f9763fa25f6b308b14a41238e933157d64b0e074e5d2043c20f5e49c5
3
+ size 28918326264
Rhea-72b-v0.5-fp16.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat Rhea-72b-v0.5-fp16_part_* > "Rhea-72b-v0.5-fp16.gguf"