antirez

/

deepseek-v4.1-flash-gguf

Like 35

Propose DeepSeek-V4.1-Flash-Q2-AProjQ4-requant.gguf as an experimental attention-weight variant of DeepSeek-V4.1-Flash-Q2.gguf, for testing with the V4.1 AProjQ4 Metal implementation in GiorgioOppo/ds4.

Quantization scope and provenance

• Requantize Q8_0 to Q4_K for attn_q_a.weight, attn_q_b.weight, attn_kv.weight, attn_output_a.weight and attn_output_b.weight in all 40 layers: 200 tensors.
• Preserve IQ2_XXS routed gate/up experts, Q2_K routed down experts, Q8_0 shared experts and output head, and the existing Engram, indexer, compressor and other tensors.
• Preserve the model architecture and attention algorithm; this does not change KV-cache precision.
• This artifact is derived from the mixed-quantization Q2 GGUF, whose attention weights are Q8_0, not directly from the original safetensors. The new attention quantization is uncalibrated. The source model's expert imatrix does not establish attention calibration.

The APFS copy-on-write converter retains the original tensor offsets and logical file size: 365,713,686,528 bytes (340.60 GiB). The attention payload decreases from 5,381,160,960 to 2,848,849,920 bytes, but the file is not repacked: this is not a smaller download. Untouched tensor payloads are preserved by the clone operation.

The source checkpoint revision recorded by the source GGUF is df42c109f1defefcbfcedbe7d905718a12266e40. The source GGUF header-only SHA256 is 194e59dbe1ec9814e434e381ea31039a2123f74c65b41b5c0eafc79346dc174c, also recorded in the requantized artifact. This is a provenance fingerprint, not a full-file checksum.

Reproduction and runtime

Converter and implementation reference: commit a6c27dd, including requantizer and quantization documentation.

From that checkout on macOS/APFS, with the documented quantizer dependencies:

python3 gguf-tools/deepseek41_requantize.py \
  --source-gguf gguf/DeepSeek-V4.1-Flash-Q2.gguf \
  --out gguf/DeepSeek-V4.1-Flash-Q2-AProjQ4-requant.gguf

The Metal prefill implementation uses FP16 matrix tiles with FP32 accumulation and preserves BF16 activation boundaries. This is an implementation detail of the runner; the projection weights in the GGUF are Q4_K. Compatibility with unmodified upstream runners is not claimed.

Validation and remaining work

The local artifact's GGUF header matches the converter plan byte for byte, including tensor types, offsets and source-header provenance. The converter's existing --check audit passed for all 200 changed Q4 attention payloads. It has also run with Metal SSD streaming on an Apple M1 Max with 32 GiB memory, context 4096 and prefill chunk 128.

Runtime optimization comparisons on this same Q4 artifact produced matching prefill logits and generated text. Those comparisons do not measure quantization quality against the original Q8-attention model, and no end-to-end speedup over that model is claimed here. An earlier quantization-quality smoke comparison covered only five cases and 129 target tokens, with mixed metrics; it is insufficient to establish general quality parity. Quantization quality and target-hardware performance need independent evaluation before recommending this artifact as a replacement for the source model with calibrated expert weights.