File size: 3,891 Bytes
170f51c d9c596d 170f51c b2995bf 170f51c d9c596d 170f51c d9c596d 170f51c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 |
---
license: other
license_name: yi-34b
license_link: https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE
tags:
- merge
- roleplay
- exl2
- not-for-all-audiences
---
# Merged-RP-Stew-V2-34B - EXL2 3.5bpw
This is a 3.5bpw EXL2 quant of [ParasiticRogue/Merged-RP-Stew-V2-34B](https://huggingface.co/ParasiticRogue/Merged-RP-Stew-V2-34B)
Details about the model and the merge info can be found at the above mode page.
## EXL2 Version
These quants were made with exllamav2 version 0.0.17. Quants made on this version of EXL2 may not working on older versions of the exllamav2 library.
If you have problems loading these models, please update Text Generation WebUI to the latest version.
## Perplexity Scoring
Below are the perplexity scores for the EXL2 models. A lower score is better.
| Quant Level | Perplexity Score |
|-------------|------------------|
| 8.0 | 5.2805 |
| 7.0 | 5.2841 |
| 6.0 | 5.2789 |
| 5.0 | 5.2930 |
| 4.5 | 5.3110 |
| 4.0 | 5.3663 |
| 3.5 | 5.4860 |
## EQ Bench
Here are the EQ Bench scores for the EXL2 quants using Alpaca, ChatML and Chat-Vicuna prompt templates. A higher score is better.
| Quant Size | ChatML | Alpaca | Chat-Vicuna |
|------------|--------|--------|-------------|
| 8.0 | 73.82 | 75.02 | 73.53 |
| 7.0 | 74.22 | 74.96 | 72.71 |
| 6.0 | 74.29 | 75.11 | 74.16 |
| 5.0 | 72.74 | 74.11 | 73.28 |
| 4.5 | 73.62 | 73.18 | 73.32 |
| 4.0 | 73.99 | 71.85 | 74.74 |
| 3.5 | 72.07 | 73.73 | 73.56 |
### Chat-Vicuna Template
This is the Chat-Vicuna template yaml that was used in EQ bench. It was tested in Text Generation Web UI and seemed to produce accurate results.
```yaml
user: "USER:"
bot: "ASSISTANT:"
turn_template: "<|user|> <|user-message|><|im_end|>\n<|bot|> <|bot-message|><|im_end|></s>\n"
context: "<|system-message|><|im_end|>\n\n"
system_message: "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
```
### Perplexity Script
This was the script used for perplexity testing.
```bash
#!/bin/bash
# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2
# Set the model name
MODEL_NAME="Merged-RP-Stew-V2-34B"
BIT_PRECISIONS=(8.0 7.0 6.0 5.0 4.5 4.0 3.5 3.0 2.65 2.4)
for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
MODEL_DIR="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"
if [ -d "$MODEL_DIR" ]; then
output=$(python test_inference.py -m "$MODEL_DIR" -gs 21,24 -ed data/wikitext/wikitext-2-v1.parquet)
score=$(echo "$output" | grep -oP 'Evaluation perplexity: \K[\d.]+')
echo "BPW: $BIT_PRECISION, Score: $score"
fi
done
```
## Quant Details
This is the script used for quantization. These quants were calibrated against Bluemoon-Light's Chat Vicuna training data.
```bash
#!/bin/bash
# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2
# Set the model name and bit size
MODEL_NAME="Merged-RP-Stew-V2-34B"
BIT_PRECISION=3.5
# Define variables
MODEL_DIR="models/$MODEL_NAME"
OUTPUT_DIR="exl2_$MODEL_NAME"
MEASUREMENT_FILE="measurements/$MODEL_NAME.json"
CONVERTED_FOLDER="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"
CALIBRATION_DATASET="data/Bluemoon-Light/chat-vicuna.parquet"
# Create directories
rm -r "$OUTPUT_DIR"
mkdir "$OUTPUT_DIR"
mkdir "$CONVERTED_FOLDER"
# Run conversion commands
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -om $MEASUREMENT_FILE -c $CALIBRATION_DATASET
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -m $MEASUREMENT_FILE -b $BIT_PRECISION -c $CALIBRATION_DATASET -cf $CONVERTED_FOLDER
```
|