TheBloke/Puma-3b-GPTQ

Chat & support: TheBloke's Discord server

Want to contribute? TheBloke's Patreon page

TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)

---

Puma 3B - GPTQ

• Model creator: Bohan Du
• Original model: Puma 3B

Description

This repo contains GPTQ model files for Bohan Du's Puma 3B.

Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.

Repositories available

• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference
• Bohan Du's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: Human-Response

### HUMAN:
{prompt}

### RESPONSE:

Provided files and GPTQ parameters

Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.

Each separate quant is in a different branch. See below for instructions on fetching from different branches.

All GPTQ files are made with AutoGPTQ.

Explanation of GPTQ parameters

• Bits: The bit size of the quantised model.
• GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
• Act Order: True or False. Also known as desc_act. True results in better quantisation accuracy. Some GPTQ clients have issues with models that use Act Order plus Group Size.
• Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.
• GPTQ dataset: The dataset used for quantisation. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Note that the GPTQ dataset is not the same as the dataset used to train the model - please refer to the original model repo for details of the training dataset(s).
• Sequence Length: The length of the dataset sequences used for quantisation. Ideally this is the same as the model sequence length. For some very long sequence models (16+K), a lower sequence length may have to be used. Note that a lower sequence length does not limit the sequence length of the quantised model. It only impacts the quantisation accuracy on longer inference sequences.
• ExLlama Compatibility: Whether this file can be loaded with ExLlama, which currently only supports Llama models in 4-bit.

Branch	Bits	GS	Act Order	Damp %	GPTQ Dataset	Seq Len	Size	ExLlama	Desc
main	4	128	No	0.1	wikitext	2048	2.09 GB	Yes	Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options.
gptq-4bit-32g-actorder_True	4	32	Yes	0.1	wikitext	2048	2.28 GB	Yes	4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed.
gptq-4bit-64g-actorder_True	4	64	Yes	0.1	wikitext	2048	2.15 GB	Yes	4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed.
gptq-4bit-128g-actorder_True	4	128	Yes	0.1	wikitext	2048	2.09 GB	Yes	4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed.
gptq-8bit--1g-actorder_True	8	None	Yes	0.1	wikitext	2048	3.64 GB	No	8-bit, with Act Order. No group size, to lower VRAM requirements and to improve AutoGPTQ speed.
gptq-8bit-128g-actorder_True	8	128	Yes	0.1	wikitext	2048	3.71 GB	No	8-bit, with group size 128g for higher inference quality and with Act Order for even higher accuracy. Poor AutoGPTQ CUDA speed.

How to download from branches

• In text-generation-webui, you can add :branch to the end of the download name, eg TheBloke/Puma-3b-GPTQ:gptq-4bit-32g-actorder_True
• With Git, you can clone a branch with:

git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/Puma-3b-GPTQ

• In Python Transformers code, the branch is the revision parameter; see below.

How to easily download and use this model in text-generation-webui.

Please make sure you're using the latest version of text-generation-webui.

It is strongly recommended to use the text-generation-webui one-click-installers unless you know how to make a manual install.

1. Click the Model tab.
2. Under Download custom model or LoRA, enter TheBloke/Puma-3b-GPTQ.

• To download from a specific branch, enter for example TheBloke/Puma-3b-GPTQ:gptq-4bit-32g-actorder_True
• see Provided Files above for the list of branches for each option.

3. Click Download.
4. The model will start downloading. Once it's finished it will say "Done"
5. In the top left, click the refresh icon next to Model.
6. In the Model dropdown, choose the model you just downloaded: Puma-3b-GPTQ
7. The model will automatically load, and is now ready for use!
8. If you want any custom settings, set them and then click Save settings for this model followed by Reload the Model in the top right.

• Note that you do not need to set GPTQ parameters any more. These are set automatically from the file quantize_config.json.

9. Once you're ready, click the Text Generation tab and enter a prompt to get started!

How to use this GPTQ model from Python code

First make sure you have AutoGPTQ 0.3.1 or later installed:

pip3 install auto-gptq

If you have problems installing AutoGPTQ, please build from source instead:

pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip3 install .

Then try the following example code:

from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name_or_path = "TheBloke/Puma-3b-GPTQ"

use_triton = False

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
        use_safetensors=True,
        trust_remote_code=False,
        device="cuda:0",
        use_triton=use_triton,
        quantize_config=None)

"""
# To download from a specific branch, use the revision parameter, as in this example:
# Note that `revision` requires AutoGPTQ 0.3.1 or later!

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
        revision="gptq-4bit-32g-actorder_True",
        use_safetensors=True,
        trust_remote_code=False,
        device="cuda:0",
        quantize_config=None)
"""

prompt = "Tell me about AI"
prompt_template=f'''### HUMAN:
{prompt}

### RESPONSE:
'''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))

# Inference can also be done using transformers' pipeline

# Prevent printing spurious transformers error when using pipeline with AutoGPTQ
logging.set_verbosity(logging.CRITICAL)

print("*** Pipeline:")
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.15
)

print(pipe(prompt_template)[0]['generated_text'])

Compatibility

The files provided will work with AutoGPTQ (CUDA and Triton modes), GPTQ-for-LLaMa (only CUDA has been tested), and Occ4m's GPTQ-for-LLaMa fork.

ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.

Discord

For further support, and discussions on these models and AI in general, join us at:

TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

• Patreon: https://patreon.com/TheBlokeAI
• Ko-Fi: https://ko-fi.com/TheBlokeAI

Special thanks to: Aemon Algiz.

Patreon special mentions: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter

Thank you to all my generous patrons and donaters!

And thank you again to a16z for their generous grant.

Original model card: Bohan Du's Puma 3B

This is OpenLLaMA 3B V2 finetuned on ShareGPT Hyperfiltered for 1 epochs.

Prompt template:

### HUMAN:
{prompt}

### RESPONSE:
<leave a newline for the model to answer>

q4_1 GGML quant available here.

Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!

Benchmarks:

arc_challenge	acc 0.3660409556313993 acc_stderr 0.014077223108470142 acc_norm 0.4121160409556314 acc_norm_stderr 0.014383915302225403
hellaswag	acc 0.5401314479187412 acc_stderr 0.004973683026202171 acc_norm 0.7177853017327226 acc_norm_stderr 0.004491574539441884
hendrycksTest-abstract_algebra	acc 0.21 acc_stderr 0.04093601807403326 acc_norm 0.21 acc_norm_stderr 0.04093601807403326
hendrycksTest-anatomy	acc 0.25925925925925924 acc_stderr 0.03785714465066653 acc_norm 0.25925925925925924 acc_norm_stderr 0.03785714465066653
hendrycksTest-astronomy	acc 0.28289473684210525 acc_stderr 0.03665349695640767 acc_norm 0.28289473684210525 acc_norm_stderr 0.03665349695640767
hendrycksTest-business_ethics	acc 0.34 acc_stderr 0.04760952285695236 acc_norm 0.34 acc_norm_stderr 0.04760952285695236
hendrycksTest-clinical_knowledge	acc 0.2528301886792453 acc_stderr 0.02674989977124123 acc_norm 0.2528301886792453 acc_norm_stderr 0.02674989977124123
hendrycksTest-college_biology	acc 0.24305555555555555 acc_stderr 0.03586879280080339 acc_norm 0.24305555555555555 acc_norm_stderr 0.03586879280080339
hendrycksTest-college_chemistry	acc 0.19 acc_stderr 0.03942772444036623 acc_norm 0.19 acc_norm_stderr 0.03942772444036623
hendrycksTest-college_computer_science	acc 0.21 acc_stderr 0.040936018074033256 acc_norm 0.21 acc_norm_stderr 0.040936018074033256
hendrycksTest-college_mathematics	acc 0.36 acc_stderr 0.04824181513244218 acc_norm 0.36 acc_norm_stderr 0.04824181513244218
hendrycksTest-college_medicine	acc 0.20809248554913296 acc_stderr 0.030952890217749877 acc_norm 0.20809248554913296 acc_norm_stderr 0.030952890217749877
hendrycksTest-college_physics	acc 0.2647058823529412 acc_stderr 0.04389869956808778 acc_norm 0.2647058823529412 acc_norm_stderr 0.04389869956808778
hendrycksTest-computer_security	acc 0.33 acc_stderr 0.047258156262526045 acc_norm 0.33 acc_norm_stderr 0.047258156262526045
hendrycksTest-conceptual_physics	acc 0.33617021276595743 acc_stderr 0.030881618520676942 acc_norm 0.33617021276595743 acc_norm_stderr 0.030881618520676942
hendrycksTest-econometrics	acc 0.24561403508771928 acc_stderr 0.04049339297748144 acc_norm 0.24561403508771928 acc_norm_stderr 0.04049339297748144
hendrycksTest-electrical_engineering	acc 0.2689655172413793 acc_stderr 0.03695183311650232 acc_norm 0.2689655172413793 acc_norm_stderr 0.03695183311650232
hendrycksTest-elementary_mathematics	acc 0.2724867724867725 acc_stderr 0.022930973071633366 acc_norm 0.2724867724867725 acc_norm_stderr 0.022930973071633366
hendrycksTest-formal_logic	acc 0.21428571428571427 acc_stderr 0.03670066451047181 acc_norm 0.21428571428571427 acc_norm_stderr 0.03670066451047181
hendrycksTest-global_facts	acc 0.32 acc_stderr 0.046882617226215034 acc_norm 0.32 acc_norm_stderr 0.046882617226215034
hendrycksTest-high_school_biology	acc 0.24193548387096775 acc_stderr 0.024362599693031093 acc_norm 0.24193548387096775 acc_norm_stderr 0.024362599693031093
hendrycksTest-high_school_chemistry	acc 0.26108374384236455 acc_stderr 0.030903796952114485 acc_norm 0.26108374384236455 acc_norm_stderr 0.030903796952114485
hendrycksTest-high_school_computer_science	acc 0.27 acc_stderr 0.04461960433384739 acc_norm 0.27 acc_norm_stderr 0.04461960433384739
hendrycksTest-high_school_european_history	acc 0.30303030303030304 acc_stderr 0.035886248000917075 acc_norm 0.30303030303030304 acc_norm_stderr 0.035886248000917075
hendrycksTest-high_school_geography	acc 0.24242424242424243 acc_stderr 0.030532892233932026 acc_norm 0.24242424242424243 acc_norm_stderr 0.030532892233932026
hendrycksTest-high_school_government_and_politics	acc 0.24352331606217617 acc_stderr 0.030975436386845426 acc_norm 0.24352331606217617 acc_norm_stderr 0.030975436386845426
hendrycksTest-high_school_macroeconomics	acc 0.2564102564102564 acc_stderr 0.022139081103971545 acc_norm 0.2564102564102564 acc_norm_stderr 0.022139081103971545
hendrycksTest-high_school_mathematics	acc 0.2518518518518518 acc_stderr 0.026466117538959905 acc_norm 0.2518518518518518 acc_norm_stderr 0.026466117538959905
hendrycksTest-high_school_microeconomics	acc 0.24789915966386555 acc_stderr 0.028047967224176892 acc_norm 0.24789915966386555 acc_norm_stderr 0.028047967224176892
hendrycksTest-high_school_physics	acc 0.31125827814569534 acc_stderr 0.037804458505267334 acc_norm 0.31125827814569534 acc_norm_stderr 0.037804458505267334
hendrycksTest-high_school_psychology	acc 0.26422018348623855 acc_stderr 0.01890416417151019 acc_norm 0.26422018348623855 acc_norm_stderr 0.01890416417151019
hendrycksTest-high_school_statistics	acc 0.2175925925925926 acc_stderr 0.028139689444859672 acc_norm 0.2175925925925926 acc_norm_stderr 0.028139689444859672
hendrycksTest-high_school_us_history	acc 0.24019607843137256 acc_stderr 0.02998373305591362 acc_norm 0.24019607843137256 acc_norm_stderr 0.02998373305591362
hendrycksTest-high_school_world_history	acc 0.28270042194092826 acc_stderr 0.029312814153955924 acc_norm 0.28270042194092826 acc_norm_stderr 0.029312814153955924
hendrycksTest-human_aging	acc 0.3901345291479821 acc_stderr 0.03273766725459156 acc_norm 0.3901345291479821 acc_norm_stderr 0.03273766725459156
hendrycksTest-human_sexuality	acc 0.1984732824427481 acc_stderr 0.03498149385462473 acc_norm 0.1984732824427481 acc_norm_stderr 0.03498149385462473
hendrycksTest-international_law	acc 0.30578512396694213 acc_stderr 0.04205953933884124 acc_norm 0.30578512396694213 acc_norm_stderr 0.04205953933884124
hendrycksTest-jurisprudence	acc 0.24074074074074073 acc_stderr 0.0413311944024384 acc_norm 0.24074074074074073 acc_norm_stderr 0.0413311944024384
hendrycksTest-logical_fallacies	acc 0.22085889570552147 acc_stderr 0.03259177392742177 acc_norm 0.22085889570552147 acc_norm_stderr 0.03259177392742177
hendrycksTest-machine_learning	acc 0.24107142857142858 acc_stderr 0.04059867246952685 acc_norm 0.24107142857142858 acc_norm_stderr 0.04059867246952685
hendrycksTest-management	acc 0.2912621359223301 acc_stderr 0.04498676320572922 acc_norm 0.2912621359223301 acc_norm_stderr 0.04498676320572922
hendrycksTest-marketing	acc 0.28205128205128205 acc_stderr 0.029480360549541194 acc_norm 0.28205128205128205 acc_norm_stderr 0.029480360549541194
hendrycksTest-medical_genetics	acc 0.31 acc_stderr 0.04648231987117316 acc_norm 0.31 acc_norm_stderr 0.04648231987117316
hendrycksTest-miscellaneous	acc 0.30140485312899107 acc_stderr 0.016409091097268787 acc_norm 0.30140485312899107 acc_norm_stderr 0.016409091097268787
hendrycksTest-moral_disputes	acc 0.30346820809248554 acc_stderr 0.024752411960917202 acc_norm 0.30346820809248554 acc_norm_stderr 0.024752411960917202
hendrycksTest-moral_scenarios	acc 0.24692737430167597 acc_stderr 0.014422292204808835 acc_norm 0.24692737430167597 acc_norm_stderr 0.014422292204808835
hendrycksTest-nutrition	acc 0.28431372549019607 acc_stderr 0.025829163272757475 acc_norm 0.28431372549019607 acc_norm_stderr 0.025829163272757475
hendrycksTest-philosophy	acc 0.29260450160771706 acc_stderr 0.02583989833487798 acc_norm 0.29260450160771706 acc_norm_stderr 0.02583989833487798
hendrycksTest-prehistory	acc 0.2993827160493827 acc_stderr 0.02548311560119547 acc_norm 0.2993827160493827 acc_norm_stderr 0.02548311560119547
hendrycksTest-professional_accounting	acc 0.2624113475177305 acc_stderr 0.02624492034984301 acc_norm 0.2624113475177305 acc_norm_stderr 0.02624492034984301
hendrycksTest-professional_law	acc 0.24641460234680573 acc_stderr 0.011005971399927242 acc_norm 0.24641460234680573 acc_norm_stderr 0.011005971399927242
hendrycksTest-professional_medicine	acc 0.21691176470588236 acc_stderr 0.02503584522771125 acc_norm 0.21691176470588236 acc_norm_stderr 0.02503584522771125
hendrycksTest-professional_psychology	acc 0.27941176470588236 acc_stderr 0.018152871051538812 acc_norm 0.27941176470588236 acc_norm_stderr 0.018152871051538812
hendrycksTest-public_relations	acc 0.2636363636363636 acc_stderr 0.04220224692971987 acc_norm 0.2636363636363636 acc_norm_stderr 0.04220224692971987
hendrycksTest-security_studies	acc 0.37551020408163266 acc_stderr 0.031001209039894836 acc_norm 0.37551020408163266 acc_norm_stderr 0.031001209039894836
hendrycksTest-sociology	acc 0.26865671641791045 acc_stderr 0.03134328358208954 acc_norm 0.26865671641791045 acc_norm_stderr 0.03134328358208954
hendrycksTest-us_foreign_policy	acc 0.35 acc_stderr 0.0479372485441102 acc_norm 0.35 acc_norm_stderr 0.0479372485441102
hendrycksTest-virology	acc 0.3253012048192771 acc_stderr 0.03647168523683227 acc_norm 0.3253012048192771 acc_norm_stderr 0.03647168523683227
hendrycksTest-world_religions	acc 0.30409356725146197 acc_stderr 0.035282112582452334 acc_norm 0.30409356725146197 acc_norm_stderr 0.035282112582452334
truthfulqa_mc	mc1 0.2460220318237454 mc1_stderr 0.015077219200662597 mc2 0.3821583918021301 mc2_stderr 0.013892826466923648