GamaTech/baize-v2-13b-GPTQ

Baize v2 13b GPTQ

GPTQ 4bit quantized version of Baize V2 13b by Project Baize.
Please check the original repo for more information about this model!

This model was quantized using GPTQ-for-LLaMa with the settings:

python llama.py ./input/baize-v2-13b c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors ./output/baize-v2-13b-4bit-128g.safetensors

---

Available Sizes:

• Baize V2 7b GPTQ
• Baize V2 13b GPTQ

---

Usage with Oobabooga's Text Generation WebUI:

1. In the Model tab, enter "GamaTech/baize-v2-13b-GPTQ" and click Download.
2. Once the download is complete, click the Refresh button next to the Model drop-down in the top left.
3. Uncheck the Autoload the model checkbox. This allows you to select the correct settings before attempting to load the model.
4. Select the model from the Model drop-down list.
5. On the right, set the following settings:
   • Wbits = 4
   • Groupsize = 128
   • Model_Type = LLaMA
6. Click Save settings for this model.
7. Click Load Model.

---

license: cc-by-nc-4.0