Harsha001/flint-ROCmFPX

Flint — ROCmFPX GGUF

RecursiveSI/flint, Recursive's 39.8B mixture-of-experts model for coding agents (about 2.5B active per token), quantized to the ROCmFPX formats, a llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

Start with Q6_0_ROCMFPX if it fits. The 4-bit files lose clearly more quality on this model than usual (see below).

These files need HarshaVardhan3002/ROCmFPX at commit b130d4995 or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. These files have not been tested on AMD hardware yet.

Not an official release of Recursive. All credit for the model goes to its authors.

Files and measured quality

Measured against the BF16 model (perplexity 19.1433) on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the next-token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16.

File	Size	PPL	KLD	Same top
flint-Q8_0_ROCMFPX.gguf	41.1 GB	19.1675	0.0747	91.02%	8-bit, closest to BF16
flint-Q6_0_ROCMFPX.gguf	32.9 GB	19.7449	0.0917	89.63%	6-bit, almost as close as 8-bit
flint-Q4_0_ROCMFP4_STRIX.gguf	21.3 GB	24.295	0.5360	71.89%	4-bit recipe tuned for Strix Halo (gfx1151); clear quality loss
flint-Q4_0_ROCMFP4_FAST.gguf	21.2 GB	24.743	0.5428	71.69%	speed-first 4-bit (one scale per block); clear quality loss

For comparison, a stock llama.cpp Q4_K_M of the same model (23 GB) measured PPL 20.02, KLD 0.250, same top 82.8%. If you need a 4-bit file and do not need the ROCmFPX types, a stock Q4_K_M is the better choice for Flint.

Not published: Q3_0_ROCMFPX (22.1 GB, KLD 0.5523) is larger than both 4-bit files and further from BF16; Q2_0_ROCMFPX does not work on this model.

Why these files need a newer build

Flint's weights are about 20 times smaller than those of most models. ROCmFPX stores one small scale per block of 32 weights, and that scale has a fixed smallest step, so on Flint many blocks lost most of their precision: a first 8-bit file had KLD 3.9 (same top 33%) (ROCmFPX#118).

These files were made with an opt-in fix in llama-quantize (LLAMA_ROCMFPX_TENSOR_SCALE=1): each weight tensor (each expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing, and the inverse factor is stored as a small extra .scale tensor that llama.cpp already applies to the matmul output. The factor is a power of two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject.

How these were made

1. convert_hf_to_gguf.py --outtype bf16.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (calibration_datav3), 512-token chunks.
3. LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix, one run per type, from HarshaVardhan3002/ROCmFPX b130d4995.
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels. Each quant was expanded back to F16 (llama-quantize --allow-requantize ... F16), which keeps its stored weights and scales, and scored on the same GPU with stock F16 kernels. The fork's own CUDA kernels for these types are not correct on NVIDIA, so they were not used. AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m flint-Q6_0_ROCMFPX.gguf -ngl 999 -fa on --jinja --temp 1.0 --top-p 0.95 --top-k 20

Sampling settings are the ones Recursive used for its evaluations.

License

Apache-2.0, same as the base model.