ConstDemi/vikhr-nemo-12b-instruct-r-awq

Vikhr-Nemo-12B-Instruct-R — AWQ (W4A16)

4-bit AWQ quantization of Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24, a Russian-language grounded-RAG model. Produced with llm-compressor in the compressed-tensors format for vLLM.

Main motivation: on pre-Ampere GPUs (sm75, e.g. Quadro RTX 6000 / RTX 20xx) GPTQ checkpoints fall back to slow non-Marlin kernels in vLLM, while this checkpoint runs via MarlinLinearKernel for CompressedTensorsWNA16, giving a large speedup in prefill and batched decode.

Quantization details

• Method: AWQ, scheme W4A16, lm_head kept unquantized
• Tool: llm-compressor (AWQModifier, one-shot)
• Calibration: 256 samples x 2048 tokens from GrandMaster-PRO-MAX, rendered through the model chat template

Usage

Ampere and newer (sm80+): dtype is picked from config automatically

vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq

Turing (sm75, e.g. Quadro RTX 6000): no native bfloat16

vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq \
    --dtype float16

Verified on vLLM 0.25.1, sm75 GPU, --dtype float16. The model follows the two-step grounded generation protocol of the original Vikhr-R (document selection as JSON, then the answer); see the original model card for the prompt format.

Limitations

• No perplexity or benchmark-based quality evaluation was performed; the checkpoint was validated on an internal Russian grounded-RAG task (answer quality, source citation, and the JSON document-selection step).
• Calibrated on general instruction data, not domain-specific text.
• bf16 scale tensors: use --dtype float16 on GPUs without native bf16 support.