Exl2 version of Undi95/SolarMaid-v0.1.1
branch
3bh8 : 3bpw h8
3.5bh8 : 3.5bpw h8
3.8bh8 : 3.8bpw h8
4bh8 : 4bpw h8
6bh8 : 6bpw h8
8bh8 : 8bpw h8
Using ThePile 0007.parquet as dataset
Quantization settings : python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp -cf SolarMaid-v0.1.1-8bpw-h8-exl2 -c 0007.parquet -l 8192 -b 8 -hb 8 -ml 8192python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp2 -cf SolarMaid-v0.1.1-6bpw-h8-exl2 -c 0007.parquet -l 8192 -b 6 -hb 8 -m SolarMaid-v0.1.1-temp/measurement.json -ml 8192python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp3 -cf SolarMaid-v0.1.1-4bpw-h8-exl2 -c 0007.parquet -l 8192 -b 4 -hb 8 -m SolarMaid-v0.1.1-temp/measurement.json -ml 8192python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp4 -cf SolarMaid-v0.1.1-3bpw-h8-exl2 -c 0007.parquet -l 8192 -b 3.8 -hb 8 -m SolarMaid-v0.1.1-temp/measurement.json -ml 8192python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp5 -cf SolarMaid-v0.1.1-2.4bpw-h8-exl2 -c 0007.parquet -l 8192 -b 3.5 -hb 8 -m SolarMaid-v0.1.1-temp/measurement.json -ml 8192python convert.py -i models/Undi95_SolarMaid-v0.1.1 -o SolarMaid-v0.1.1-temp5 -cf SolarMaid-v0.1.1-2.4bpw-h8-exl2 -c 0007.parquet -l 8192 -b 3 -hb 8 -m SolarMaid-v0.1.1-temp/measurement.json -ml 8192
below this line is original readme
SolarMaid is a model made primarely for ERP (Erotic Roleplay) without being too dumb, it use 40 out of 48 layers of "SOLAR Instruct 10.7B", and 32 out of 32 layers of "Noromaid 7B v0.1.1".
Despite being a little unstable, benchmark stay high, and can output some hot sh*t.
I'm working on making it more stable.
Description
This repo contains fp16 files of SolarMaid v0.1.1.
Models used
- upstage/SOLAR-10.7B-Instruct-v1.0
- NeverSleep/Noromaid-7b-v0.1.1
Prompt template: Alpaca
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{prompt}
### Response:
Prompt template: Custom (SOLAR)
### User:
{prompt}
### Assistant:
If you want to support me, you can here.
- Downloads last month
- 3