|
--- |
|
library_name: transformers |
|
tags: |
|
- phi-3 |
|
- phi-3-mini |
|
- phi-3-mini-4k-instruct |
|
- conversational |
|
- text-generation-inference |
|
pipeline_tag: text-generation |
|
language: |
|
- en |
|
--- |
|
|
|
Official quantization of [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using [PV-Tuning](https://arxiv.org/abs/2405.14852) on top of [AQLM](https://arxiv.org/abs/2401.06118). |
|
|
|
For this quantization, we used 1 codebook of 16 bits for groups of 8 weights. |
|
|
|
Results (0-shot `acc`): |
|
|
|
Results: |
|
| Model | Quantization | ArcC| ArcE| Hellaswag | PiQA | Winogrande | Model size, Gb | |
|
|------|------|-------|------|------|------|------|------| |
|
| microsoft/Phi-3-mini-4k-instruct| None | 0.5529 | 0.8325 | 0.6055 | 0.8020 | 0.7364 | 7.6 | |
|
| | 1x16 | 0.5051 | 0.7950 | 0.5532 | 0.7949 | 73.01 | 1.4 | |
|
|
|
You can also find Phi-3-medium models compressed with AQLM+PV: [2-bit](https://huggingface.co/ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf) and [1-bit](https://huggingface.co/ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-1Bit-1x16-hf) |
|
|
|
The 1x16g16 (1-bit) models are on the way, as soon as we update the inference lib with their respective kernels. |
|
|
|
To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the [official GitHub repo](https://github.com/Vahe1994/AQLM). |
|
The original code for PV-Tuning can be found in the [AQLM@pv-tuning](https://github.com/Vahe1994/AQLM/tree/pv-tuning) branch. |
|
|
