Commit
•
e980b0f
1
Parent(s):
9141f72
Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
inference: false
|
3 |
+
tags:
|
4 |
+
- gguf
|
5 |
+
- quantized
|
6 |
+
- roleplay
|
7 |
+
- multimodal
|
8 |
+
- vision
|
9 |
+
- llava
|
10 |
+
- sillytavern
|
11 |
+
- merge
|
12 |
+
- mistral
|
13 |
+
- conversational
|
14 |
+
base_model:
|
15 |
+
- grimjim/kukulemon-7B
|
16 |
+
- Inv/Konstanta-V4-Alpha-7B
|
17 |
+
---
|
18 |
+
# #Roleplay #Multimodal #Vision
|
19 |
+
|
20 |
+
This repository hosts GGUF-IQ-Imatrix quants for [Nitral-AI/KukulStanta-7B](https://huggingface.co/Nitral-AI/KukulStanta-7B).
|
21 |
+
|
22 |
+
"It's Kukul time!"
|
23 |
+
|
24 |
+
**Recommended starting [SillyTavern presets here](https://huggingface.co/Lewdiculous/Eris_PrimeV4-Vision-32k-7B-GGUF-IQ-Imatrix/tree/main/sillytavern-presets-lewdicu-3.0.2-mistral-0.2).**
|
25 |
+
|
26 |
+
This is a **#multimodal** model that also has **#vision** capabilities. <br> Read the full card information if you also want to use that functionality.
|
27 |
+
|
28 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/65d4cf2693a0a3744a27536c/G_Gfnn3JP_CMYQQFMG591.png)
|
29 |
+
|
30 |
+
**What does "Imatrix" mean?**
|
31 |
+
|
32 |
+
<details><summary>
|
33 |
+
⇲ Click here to expand/hide more information about this topic.
|
34 |
+
</summary>
|
35 |
+
|
36 |
+
It stands for **Importance Matrix**, a technique used to improve the quality of quantized models.
|
37 |
+
The **Imatrix** is calculated based on calibration data, and it helps determine the importance of different model activations during the quantization process.
|
38 |
+
The idea is to preserve the most important information during quantization, which can help reduce the loss of model performance, especially when the calibration data is diverse.
|
39 |
+
[[1]](https://github.com/ggerganov/llama.cpp/discussions/5006) [[2]](https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-8395384)
|
40 |
+
|
41 |
+
For imatrix data generation, kalomaze's `groups_merged.txt` with added roleplay chats was used, you can find it [here](https://huggingface.co/Lewdiculous/Datura_7B-GGUF-Imatrix/blob/main/imatrix-with-rp-format-data.txt). This was just to add a bit more diversity to the data.
|
42 |
+
|
43 |
+
</details><br>
|
44 |
+
|
45 |
+
# Vision/multimodal capabilities:
|
46 |
+
|
47 |
+
<details><summary>
|
48 |
+
⇲ Click here to expand/hide how this would work in practice in a roleplay chat.
|
49 |
+
</summary>
|
50 |
+
|
51 |
+
![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/65d4cf2693a0a3744a27536c/Kkfi_CizIk0ZXMRF8N5jo.jpeg)
|
52 |
+
|
53 |
+
</details><br>
|
54 |
+
|
55 |
+
<details><summary>
|
56 |
+
⇲ Click here to expand/hide what your SillyTavern Image Captions extension settings should look like.
|
57 |
+
</summary>
|
58 |
+
|
59 |
+
![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/65d4cf2693a0a3744a27536c/UpXOnVrzvsMRYeqMaSOaa.jpeg)
|
60 |
+
|
61 |
+
</details><br>
|
62 |
+
|
63 |
+
**If you want to use vision functionality:**
|
64 |
+
|
65 |
+
* Make sure you are using the latest version of [KoboldCpp](https://github.com/LostRuins/koboldcpp).
|
66 |
+
|
67 |
+
To use the multimodal capabilities of this model, such as **vision**, you also need to load the specified **mmproj** file, you can get it [here](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/blob/main/mmproj-model-f16.gguf) or as uploaded in the repository.
|
68 |
+
|
69 |
+
* You can load the **mmproj** by using the corresponding section in the interface:
|
70 |
+
|
71 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/65d4cf2693a0a3744a27536c/UX6Ubss2EPNAT3SKGMLe0.png)
|
72 |
+
|
73 |
+
* For CLI users, you can load the **mmproj file** by adding the respective flag to your usual command:
|
74 |
+
|
75 |
+
```
|
76 |
+
--mmproj your-mmproj-file.gguf
|
77 |
+
```
|
78 |
+
|
79 |
+
# Quantization information:
|
80 |
+
|
81 |
+
|
82 |
+
<details><summary>
|
83 |
+
⇲ Click here to expand/hide more information about this topic.
|
84 |
+
</summary>
|
85 |
+
|
86 |
+
```python
|
87 |
+
quantization_options = [
|
88 |
+
"Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
|
89 |
+
"Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
|
90 |
+
]
|
91 |
+
```
|
92 |
+
|
93 |
+
**Steps performed:**
|
94 |
+
|
95 |
+
```
|
96 |
+
Base⇢ GGUF(F16)⇢ Imatrix-Data(F16)⇢ GGUF(Imatrix-Quants)
|
97 |
+
```
|
98 |
+
*Using the latest llama.cpp at the time.*
|
99 |
+
|
100 |
+
</details><br>
|