|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- en |
|
--- |
|
|
|
# WinterGoddess-1.4x-70B-L2 IQ2-GGUF |
|
|
|
## Description |
|
IQ2-GGUF quants of [Sao10K/WinterGoddess-1.4x-70B-L2](https://huggingface.co/Sao10K/WinterGoddess-1.4x-70B-L2) |
|
|
|
Unlike regular GGUF quants this uses important matrix similar to Quip# to keep the quant from degrading too much even at 2bpw allowing you to run larger models on less powerful machines. |
|
|
|
***NOTE:*** Currently you will need experimental branches of Koboldcpp or Ooba for this to work. |
|
- Nexesenex have compiled Windows binaries [HERE](https://github.com/Nexesenex/kobold.cpp/releases/tag/v1.55.1_b1842) |
|
- [llamacpp_0.2.29 branch](https://github.com/oobabooga/text-generation-webui/tree/llamacpp_0.2.29) of Ooba also works |
|
|
|
|
|
[More info about IQ2](https://github.com/ggerganov/llama.cpp/pull/4897) |
|
|
|
|
|
|
|
# Models |
|
|
|
Models: [IQ2-XS](https://huggingface.co/Kooten/WinterGoddess-1.4x-70B-L2-IQ2-GGUF/blob/main/WinterGoddess-1.4x-70B-L2-IQ2_XS.gguf), [IQ2-XXS](https://huggingface.co/Kooten/WinterGoddess-1.4x-70B-L2-IQ2-GGUF/blob/main/WinterGoddess-1.4x-70B-L2-IQ2_XXS.gguf) |
|
|
|
Regular GGUF Quants: [Here](https://huggingface.co/TheBloke/WinterGoddess-1.4x-70B-L2-GGUF) |
|
|
|
## Prompt Format |
|
|
|
### Alpaca: |
|
``` |
|
### Instruction: |
|
<Prompt> |
|
|
|
### Response: |
|
|
|
``` |
|
|
|
OR |
|
|
|
``` |
|
### Instruction: |
|
<Prompt> |
|
|
|
### Input: |
|
<Insert Context Here> |
|
|
|
### Response: |
|
``` |
|
|
|
## Contact |
|
Kooten on discord |
