Disclaimer:

The model is reproduced based on the paper VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models github and arXiv

The model itself is sourced from a community release.

It is intended only for experimental purposes.

Users are responsible for any consequences arising from the use of this model.

Note:

The PPL test results are for reference only and were collected using GPTQ testing script.

{
    "ctx_2048": {
        "wikitext2": 10.379325866699219
    },
    "ctx_4096": {
        "wikitext2": 10.027759552001953
    },
    "ctx_8192": {
        "wikitext2": 10.061389923095703
    }
}
Downloads last month
211
Safetensors
Model size
1.75B params
Tensor type
I32
ยท
BF16
ยท
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no library tag.

Model tree for VPTQ-community/Meta-Llama-3.1-8B-Instruct-v12-k65536-4096-woft

Quantized
(354)
this model

Spaces using VPTQ-community/Meta-Llama-3.1-8B-Instruct-v12-k65536-4096-woft 2

Collection including VPTQ-community/Meta-Llama-3.1-8B-Instruct-v12-k65536-4096-woft