cgg507/Behemoth-R1-123B-v2-QTIP-2bit

Behemoth-R1-123B-v2 QTIP 2-bit

A 2-bit QTIP quantization of TheDrummer/Behemoth-R1-123B-v2, which is a fine-tune of Mistral-Large-Instruct-2411 (88 layers, hidden 12288, GQA 96/8). It is about 32 GB on disk and runs a 123B model on two 32 GB AMD Radeon AI PRO R9700 cards.

The quantizer and the inference runtime are in c000507/Behemoth-qtip: a C++/HIP reimplementation of the QTIP pipeline for ROCm (gfx1201), plus behemoth_server, an OpenAI-compatible server with QTIP trellis-decode kernels and two-GPU tensor parallelism.

Format

q/o, gate/up/down projections	QTIP 2-bit HYB code: bitshift trellis L=16, K=2, V=2, quantlut_sym with a 2^9-entry LUT, 16x16 tiles
k/v projections	FP8 e4m3 with per-channel scale, kept from the source FP8 model (kv_format: fp8)
Embeddings, LM head, norms	bf16
Layout	QTIP-hfized llama layout (trellis, SU, SV, tlut, rcp, tp_rank per linear); quip_params in config.json
Incoherence processing	random sign vectors + Hadamard transforms on both sides
Rounding	block LDLQ (16-blocks) with tail-biting Viterbi, sigma_reg 1e-2, scale override 0.9

The weights are not loadable with stock transformers. Use behemoth_server from the Behemoth-qtip repo. The 2-bit tensors follow the reference QTIP (Cornell-RelaxML/qtip) hfized layout, but the mixed FP8 k/v projections are specific to Behemoth-qtip (see its docs/MIXED_FP8_RECONSTRUCTION.md).

Calibration

• 4.19M tokens: 512 sequences of 8192 tokens.
• 75% chat and roleplay conversations with chain-of-thought, in the model's chat template with the reasoning inlined. 25% general text: OpenWebText and SmolTalk.
• Hessians were collected layer by layer from the original FP8 model (FP8 weights used exactly, per-channel scale applied in fp32), one Hessian group at a time: q, o, gate+up, down.

Serving

Build runtime/ from the Behemoth-qtip repository (ROCm, see its README), then:

runtime/build-rocm/behemoth_server --model /path/to/Behemoth-R1-123B-v2-QTIP-2bit \
  --devices 0,1 --transport zerocopy --code-layout lane --memory-fraction 0.9 \
  --token-budget 2048 --prefill-chunk 2048 --mixed-prefill-tokens 512 --prefill-interval 24 \
  --host 0.0.0.0 --port 1337 --threads 4 --profile false

Single-request decode measures about 44 ms/token at short context and about 52 ms/token at 16k on 2x R9700.

Chat template

Mistral [INST] format, the same as the base model (chat_template.jinja):

<s>[SYSTEM_PROMPT] ...[/SYSTEM_PROMPT][INST] ...[/INST] assistant reply</s>

License

This is a derivative of Mistral-Large-Instruct-2411 and is distributed under the Mistral AI Research License: research and non-commercial use only. See the base models for their terms.