Quantized version of meta-llama/LlamaGuard-7b

Model Description

The model meta-llama/LlamaGuard-7b was quantized to 4bit, group_size 128, and act-order=True with auto-gptq integration in transformers (https://huggingface.co/blog/gptq-integration).

Evaluation

To evaluate the qunatized model and compare it with the full precision model, I performed binary classification on the "toxicity" label from the ~5k samples test set of lmsys/toxic-chat.

๐Ÿ“Š Full Precision Model:

Average Precision Score: 0.3625

๐Ÿ“Š 4-bit Quantized Model:

Average Precision Score: 0.3450

Downloads last month
19
Safetensors
Model size
1.13B params
Tensor type
I32
ยท
FP16
ยท
Inference Examples
Inference API (serverless) has been turned off for this model.

Model tree for SebastianSchramm/LlamaGuard-7b-GPTQ-4bit-128g-actorder_True

Quantized
(1)
this model