---
language:
- en
- fr
- de
- es
- it
- pt
- zh
- ja
- ru
- ko
license: other
license_name: mrl
inference: false
license_link: https://mistral.ai/licenses/MRL-0.1.md
base_model:
- anthracite-org/magnum-v4-123b
---

# Magnum-v4-123b HQQ

This repo contains magnum-v4-123b quantized to 4-bit precision using [HQQ](https://github.com/mobiusml/hqq/).

HQQ provides a similar level of precision to AWQ at 4-bit, but with no need for calibration.

This quant was generated using 8xA40s within only 10 minutes.

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig


model_path = "anthracite-org/magnum-v4-123b"
quant_config = HqqConfig(nbits=4, group_size=128, axis=1)

model = AutoModelForCausalLM.from_pretrained(model_path,
                                             torch_dtype=torch.float16,
                                             cache_dir='.',
                                             device_map="cuda:0",
                                             quantization_config=quant_config,
                                             low_cpu_mem_usage=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

output_path = "magnum-v4-123b-hqq-4bit"
model.save_pretrained(output_path)
tokenizer.save_pretrained(output_path)
```

## Inference

You can perform inference directly with transformers, or using [aphrodite](https://github.com/PygmalionAI/aphrodite-engine):

```sh
pip install aphrodite-engine

aphrodite run alpindale/magnum-v4-123b-hqq-4bit -tp 2
```