Developer note

SMQ alone is not going to produce a good model; it must be paired with the accompanying QAT.

The QAT stage is now a closed-form algebraic solver β€” no optimizer, no gradients, no backpropagation. On a 27B block it drives the held-out block-output relative error from ~12.7% (plain SMQ) down to ~3.4% after real Q2_K re-quantization, in a few minutes on a single 12 GiB GPU, with every output row guarded so it can never end up worse than plain SMQ.

SMQ: Quadratic Q2_K Quantization and Algebraic Quantization-Aware Adaptation

SMQ is a GGUF quantization pipeline built around a closed-form quadratic factorization of the native llama.cpp Q2_K representation. Its companion training path, SMQ-QAT, adapts decoder-block weights against the actual packed-and-decoded SMQ quantizer rather than a proxy quantizer.

The project has two related but distinct parts:

  1. SMQ quantizer (smq_gguf_convert_live.py) converts eligible GGUF tensors to Q2_K using a deterministic, blockwise quadratic fit.
  2. SMQ-QAT (smq_qat.py, driven by train_smq_qat.py or its Python train() function) performs sequential, blockwise, algebraic quantization-aware adaptation. It solves each projection's Q2_K scales/offsets in closed form from output-error Gram matrices, refines the 2-bit codes by output-aware coordinate descent, and guards every output row on both fitting and held-out tokens.

The converter can be used by itself. QAT is an optional adaptation stage before conversion.

Terminology: β€œsource weights” below means the values decoded from the input GGUF. If the input is Q8_0, for example, the QAT teacher is the dequantized Q8_0 modelβ€”not the original pre-quantization FP16/BF16 checkpoint unless that checkpoint is what was supplied as input.


Contents


Overview

Conventional round-to-nearest (RTN) quantization chooses the representable code nearest to each value under a selected scale and offset grid. SMQ starts from an RTN-derived code assignment, then uses a closed-form least-squares projection to fit the scale and offset parameters to those codes. The fitted parameters are snapped back onto the parameter lattice used by Q2_K, and codes are recalculated against that final grid.

The important distinction is that SMQ does not introduce a new GGML tensor type. It writes the standard Q2_K 84-byte block layout. The quantization algorithm used to choose the codes and parameters is different; the packed result remains compatible with the existing Q2_K representation.

SMQ-QAT adds a second idea: rather than asking only how well the fixed source weights reconstruct, it re-solves the Q2_K parameters to minimize the block's output error on calibration activations. Because Q2_K decodes each 16-weight sub-block as w = s*q - o with fixed 2-bit codes q, a linear layer's output is linear in (s, o) for fixed codes. That makes the output-error fit a least-squares problem solvable in closed form from Gram matrices β€” no iterative optimizer and no backward pass are required.

Pipeline at a glance

Input GGUF
   β”‚
   β”œβ”€β”€ Optional importance matrix
   β”‚
   β”œβ”€β”€ [Optional] SMQ-QAT (algebraic, no optimizer)
   β”‚       β”œβ”€β”€ Build corpus token sequences; hold out a tail split
   β”‚       β”œβ”€β”€ Process decoder blocks sequentially
   β”‚       β”œβ”€β”€ Fake-quantize projections with real SMQ Q2_K
   β”‚       β”œβ”€β”€ Solve (s, o) in closed form from output-error Gram matrices
   β”‚       β”œβ”€β”€ Snap to the Q2_K lattice in the output metric
   β”‚       β”œβ”€β”€ Refine 2-bit codes by guarded coordinate descent
   β”‚       β”œβ”€β”€ Guard every row on fit + held-out tokens; revert block if worse
   β”‚       └── Export an adapted, mostly-F16 staging GGUF
   β”‚
   └── SMQ converter
           β”œβ”€β”€ Find every eligible tensor (unless filtered)
           β”œβ”€β”€ Factorize 256-weight blocks
           β”œβ”€β”€ Pack native Q2_K data
           └── Write a complete GGUF

Q2_K representation

The converter targets GGML's native Q2_K format. Each quantization super-block contains 256 weights in 84 bytes:

Component Size Meaning
scales 16 bytes One packed scale/minimum pair per 16-value sub-block; each is a 4-bit value
qs 64 bytes 256 two-bit codes
d 2 bytes FP16 super-block scale
dmin 2 bytes FP16 super-block minimum/offset scale
Total 84 bytes 256 weights

The nominal storage cost is therefore:

[ 84 / 256 = 0.328125\text{ bytes/weight} = 2.625\text{ bits/weight}. ]

This is the storage cost of the quantized blocks, not a guarantee that an entire model file averages exactly 2.625 bits per parameter. GGUF metadata, non-eligible tensors, and any tensors deliberately passed through at their original type affect the whole-file average.

Each 256-value super-block is divided into sixteen 16-value sub-blocks. For sub-block index (j), the decoded value is:

[ \hat{x}i = d,ls_j,q_i - d{min},lm_j, ]

where:

  • (q_i\in{0,1,2,3}) is the two-bit code;
  • (ls_j\in{0,\ldots,15}) is the four-bit scale multiplier;
  • (lm_j\in{0,\ldots,15}) is the four-bit minimum multiplier;
  • (d) and (d_{min}) are FP16 super-block parameters.

Writing (s_j = d,ls_j) and (o_j = d_{min},lm_j), the sub-block decode is simply (w = s_j q - o_j). This identity is what makes the algebraic QAT fit possible: for fixed codes, the layer output is linear in the per-sub-block scales and offsets.

The converter explicitly packs the scale/minimum nibbles, code planes, and FP16 parameters into the GGML Q2_K byte layout. Its self-test checks that packing and its independent decoder reproduce the expected values exactly.


SMQ quadratic factorization

SMQ processes a tensor as a stream of independent 256-weight Q2_K super-blocks. Within each super-block, the parameter fit is performed over its sixteen 16-value sub-blocks. The implementation is closed-form; it does not run iterative coordinate descent or sweep candidate objective values.

1. Construct an RTN starting grid

For each 16-value sub-block (x), the converter computes a nonnegative minimum offset and a three-level step:

[ \ell_{min}=\max(-\min(x),0), \qquad s=\frac{\max(\max(x)+\ell_{min},0)}{3}. ]

The maximum step and maximum offset across the sixteen sub-blocks define the super-block parameters:

[ d=\operatorname{FP16}!\left(\frac{\max_j s_j}{15}\right), \qquad d_{min}=\operatorname{FP16}!\left(\frac{\max_j \ell_{min,j}}{15}\right). ]

The local four-bit multipliers are rounded and clipped to [0, 15]:

[ ls_j=\operatorname{clip}!\left(\operatorname{round}(s_j/d),0,15\right), \qquad lm_j=\operatorname{clip}!\left(\operatorname{round}(\ell_{min,j}/d_{min}),0,15\right). ]

This produces the initial RTN grid and initial two-bit codes:

[ q_0=\operatorname{clip}!\left(\operatorname{round} \left(\frac{x+d_{min}lm}{dls}\right),0,3\right). ]

The implementation uses safe denominators for zero-scale cases and stores d and dmin on the FP16 lattice required by Q2_K.

2. Fit scale and offset for the fixed codes

With the initial codes fixed, SMQ fits each 16-value sub-block to the model (x_i \approx s q_i-o) by minimizing the quadratic objective

[ E(s,o)=\sum_i w_i(x_i-sq_i+o)^2. ]

For uniform weights, (w_i=1). When an importance matrix is available, its values supply the weights. Define the weighted moments:

[ S_w=\sum_i w_i,\quad S_x=\sum_i w_ix_i,\quad S_q=\sum_i w_iq_i,\quad S_{qq}=\sum_i w_iq_i^2,\quad S_{xq}=\sum_i w_ix_iq_i. ]

The 2-by-2 normal equations have determinant (\Delta=S_wS_{qq}-S_q^2). For a non-degenerate sub-block, the closed-form solution is

[ s=\frac{S_wS_{xq}-S_qS_x}{\Delta}, \qquad o=\frac{S_qS_{xq}-S_{qq}S_x}{\Delta}. ]

The Q2_K minimum term cannot represent a negative offset. If the fitted (o) is negative, SMQ clamps it to zero and refits the scale on that boundary ((s=S_{xq}/S_{qq},\ o=0)). If the system is degenerate, the implementation falls back to the RTN step and minimum for that sub-block.

3. Snap the fitted parameters to the Q2_K lattice

The continuous fitted values are not written directly. SMQ converts the maximum fitted scale and offset across the sixteen sub-blocks into FP16 super-block parameters:

[ d_2=\operatorname{FP16}!\left(\frac{\max_j s_j}{15}\right), \qquad d_{min,2}=\operatorname{FP16}!\left(\frac{\max_j o_j}{15}\right). ]

It then recomputes the four-bit multipliers and re-derives the two-bit codes against that final grid. The final codes are selected for the parameters that can actually be stored, not for the unsnapped floating-point fit.

4. Apply the per-super-block RTN guard

By default, the converter compares the candidate quadratic reconstruction with the initial RTN reconstruction for each 256-weight super-block, comparing importance-weighted squared error over the entire super-block and keeping the lower-error candidate. This guard makes the packed choice no worse than the RTN candidate under the same per-super-block objective. It is not a guarantee about downstream model quality, perplexity, or benchmark accuracy.

The --no-rtn-metrics option disables the RTN comparison and guard as well as its associated metrics. In that mode the decomposition is emitted without the RTN fallback safeguard; it is not merely a β€œhide the RTN statistics” option.

5. Pack the native block

The selected codes and quantization parameters are packed into 84 bytes per 256 weights using the GGML Q2_K ordering. The output is a normal GGUF with general.file_type marked as mostly-Q2_K and the GGML quantization version set by the converter.


Importance matrices

An importance matrix changes the objective used by the local quadratic fit. For each sub-block, SMQ minimizes (\sum_i w_i(x_i-sq_i+o)^2) instead of the unweighted sum of squared errors. The same weights are used in the per-super-block RTN guard and in the converter's weighted reconstruction statistics.

The supplied reader supports two forms:

  • Legacy .dat imatrix: per-tensor entries containing per-element importance values.
  • GGUF imatrix: general.type = imatrix, with per-tensor <tensor>.in_sum2 vectors and optional <tensor>.counts values. The input-channel sums are divided by their token count when a positive count is present.

The reader supports either a per-element weight array or a per-input-channel vector. A channel vector is tiled across the output rows as required by the Q2_K block layout. If a tensor is missing from the matrix, that tensor is quantized with uniform weights. The converter reports matrix hits and misses.

An imatrix is optional. Without one, the quadratic fit and the guard use uniform weights. In QAT, the imatrix is also passed to the fake-quant refresh so the final re-quantization matches what the converter will do with the same matrix.


Tensor eligibility and coverage

The converter's default is all eligible tensors, not a hand-picked set of named projections. A tensor is eligible when:

  1. It has at least two GGML dimensions.
  2. No dimension after the first is equal to one (degenerate elementwise tensors are excluded).
  3. ne[0] is divisible by 256.
  4. Its element count is divisible by 256.

The first dimension, ne[0], is the contiguous row width used to align Q2_K super-blocks. The test is based on GGML dimensions, not on the apparent orientation of a NumPy matrix.

When no tensor filters or maximum-tensor limit are supplied, every eligible tensor is selected. This can include eligible embedding and output matrices; the converter does not exclude them merely because of their names. Tensors that fail eligibility are copied through in their original GGML type and byte representation. Typical pass-through tensors include one-dimensional norms and biases, which must remain floating point for the corresponding GGML multiply/add operations.

Selection controls:

  • --tensor NAME: select an exact tensor name or a substring; repeatable.
  • --tensor-pattern REGEX: select tensor names matching a regular expression.
  • --max-tensors N: stop after the first N selected eligible tensors; 0 means no limit.
  • --list-tensors: list eligible tensors and report tensors that will be skipped for shape/eligibility reasons.

If selection filters are supplied, eligible tensors not selected are copied unchanged.


Installing dependencies

Converter

The converter requires NumPy and gguf-py from the llama.cpp checkout used by the runtime:

pip install numpy
pip install -e /path/to/llama.cpp/gguf-py

The converter can use PyTorch CPU reductions as an optional accelerator. If PyTorch is not importable, the NumPy reduction paths are used.

QAT

The QAT module requires PyTorch, NumPy, GGUF-Py, Transformers, and Safetensors, plus the tokenizer package appropriate for the model tokenizer:

pip install torch numpy transformers safetensors
pip install -e /path/to/llama.cpp/gguf-py

# BPE tokenizers (for example, Qwen/Mistral families)
pip install tokenizers

# SentencePiece tokenizers (for example, Llama)
pip install sentencepiece

CUDA is used when available and selected. The algebraic fit runs its Gram statistics and coordinate descent on the training device and uses BF16 autocast only for the block forwards that collect activations and evaluate block output. There is no optimizer state and no backward pass, so VRAM use is dominated by the weights, the decoded fake-quant cache, and the per-group Gram matrices.


Using the converter

Assuming the file is named smq_gguf_convert_live.py:

Quantize every eligible tensor

python smq_gguf_convert_live.py model-f16.gguf \
  --output-gguf model-smq-q2k.gguf

Quantize using an importance matrix

python smq_gguf_convert_live.py model-f16.gguf \
  --imatrix model.imatrix.gguf \
  --output-gguf model-smq-q2k.gguf \
  --out reports/model-smq

This writes a complete GGUF and benchmark reports at reports/model-smq.csv and reports/model-smq.json.

Inspect tensor eligibility first

python smq_gguf_convert_live.py model.gguf --list-tensors

Quantize selected tensors only

# One tensor or substring
python smq_gguf_convert_live.py model.gguf \
  --tensor blk.12.ffn_up.weight \
  --output-gguf selected-q2k.gguf

# Regular-expression selection
python smq_gguf_convert_live.py model.gguf \
  --tensor-pattern 'blk\\.(0|1|2)\\.' \
  --output-gguf early-blocks-q2k.gguf

Unselected tensors are copied unchanged.

Save per-tensor reconstructions

python smq_gguf_convert_live.py model.gguf \
  --output-gguf model-smq-q2k.gguf \
  --save-recon reconstructions/

The converter writes per-tensor float32 .npy reconstructions for analysis. See the metric caveat under Implementation notes and limitations when interpreting the current guarded-output reporting path.

Useful converter flags

Flag Default Purpose
--output-gguf, --out-gguf none Write a complete Q2_K GGUF
--list-tensors off List eligible and skipped tensors
--tensor none Restrict selection by exact name or substring; repeatable
--tensor-pattern none Restrict selection by regular expression
--imatrix none Apply per-element or per-channel importance weights
--max-tensors 0 Maximum selected eligible tensors; zero means all
--chunk-blocks 32768 Number of 256-weight blocks processed per chunk
--progress-interval 5 seconds Heartbeat interval for long stages
--save-recon none Save float32 reconstructions as .npy files
--out smq_gguf_bench CSV/JSON report prefix
--quiet off Reduce progress output
--no-rtn-metrics off Disable RTN comparison and guard; emits unguarded decomposition
--no-metrics off Skip residual metric accumulation
--self-test off Run packing/factorization self-tests

Run python smq_gguf_convert_live.py --help for the CLI help generated by the installed file.


SMQ-QAT: the algebraic solver

SMQ-QAT makes each decoder block learn to operate with SMQ's actual Q2_K representation, but it does so algebraically. There is no AdamW, no learning rate, no gradient, and no optimizer state. Each projection's Q2_K scales and offsets are solved in closed form from output-error normal equations, snapped to the storable lattice in the output metric, and the 2-bit codes are refined by output-aware coordinate descent. Every candidate is guarded per output row on both the fitting tokens and a held-out split.

What gets fake-quantized and trained

For each decoder block, the trainer identifies eligible projection matrices that the converter will emit as Q2_K and attaches a fake-quant parametrization to each corresponding linear weight. The parametrization computes the actual SMQ Q2_K quantize/decode round trip; it does not substitute a different low-bit quantizer or an approximate codebook.

The forward weight is

[ W_{eff}=W+\big(Q(W)-W\big)_{detach}, ]

where Q(W) is the decoded weight reconstructed from the packed Q2_K payload. During the algebraic fit the parametrization is deferred (_defer=True) so forwards use the master weight directly while Gram statistics are collected; the fitted decoded lattice is installed at the end of the block.

The quantizer itself is a NumPy engine running on CPU. The full-precision tensor is transferred to CPU for quantization, and the decoded tensor is returned to the training device or kept in a pinned CPU cache when offloading is enabled.

What is not adapted

The algebraic fit adapts decoder-block projection matrices only. One-dimensional block auxiliary parameters (normalization weights, biases) are not co-adapted; they pass through unchanged. The trainer does not train the model-level token embedding matrix or the output/LM-head matrix.

This is separate from converter coverage:

  • The final converter can still quantize token_embd.weight and output.weight if they meet the generic eligibility rule.
  • The QAT trainer uses the embedding matrix to create the initial corpus activations, but does not optimize it.
  • The output head is not part of the block-output target and is not adapted.

Therefore β€œall eligible matrices are quantized” and β€œall eligible matrices receive QAT adaptation” are not the same claim.

Block targets and the held-out split

For each block, the trainer computes target outputs using that block's source weights and the current incoming activation buffer. The target is the block output captured before the final model norm, computed in FP32 without autocast so the distillation signal is exact.

A tail of the corpus is held out for guarding and evaluation: n_eval = min(8, max(1, n_seq // 4)) sequences. The remaining sequences are the fitting split. A block needs at least two sequences (one to fit, one to hold out); with only one sequence the block is left at plain SMQ and a message is logged. The held-out split is what makes the per-row and per-block guards meaningful rather than self-referential.

The source GGUF may itself contain quantized weights (for example, Q8_0). In that case the block teacher is the source GGUF decoded into the partial HF modelβ€”not an unseen original FP16 checkpoint.


The output-error objective and Gram matrices

For a projection with weight rows (w_r\in\mathbb{R}^d), student input (X_q\in\mathbb{R}^{N\times d}), and target output (Y\in\mathbb{R}^{N\times m}), the per-row output error is

[ e_r(w_r)=\sum_{t=1}^{N}\big(X_{q,t},w_r - Y_{t,r}\big)^2 = w_r^\top C,w_r - 2,w_r^\top g_r + yy_r, ]

where the Gram matrices are accumulated by streaming over tokens (the activations themselves are never stored for the fit):

[ C = X_q^\top X_q\in\mathbb{R}^{d\times d},\qquad G = X_q^\top Y\in\mathbb{R}^{d\times m},\qquad yy_r=\sum_t Y_{t,r}^2. ]

The solve cost therefore does not depend on the token count. Held-out error is evaluated explicitly on the held-out activations (X_v, Y_v) via _row_err_val.

Input groups and execution order

Projections that consume the very same input tensor (for example q/k/v, or gate/up) share one Gram matrix and are solved together as a group. Groups are discovered by registering forward pre-hooks and ordering modules by execution. A later group sees the already-quantized outputs of the earlier ones: its student input (X_q) is captured with the earlier groups' fitted weights installed, while its target (Y=X_{fp}W_{fp}^\top) comes from the pure full-precision block. This is what lets downstream projections compensate for upstream quantization error.


Closed-form scale/offset solve

For fixed codes (q), each row's weights are (w_{r,j,a}=s_{r,j}q_{r,j,a}-o_{r,j}), linear in the (2J) unknowns (\theta_r=(s_{r,1..J},o_{r,1..J})) where (J=d/16). Substituting into (e_r) gives a quadratic form (\theta_r^\top A_r\theta_r-2b_r^\top\theta_r+\text{const}), where (A_r) is built from (C) contracted with the codes and (b_r) from (G). _ls_fit_gram solves the resulting (2J\times 2J) normal equations per row:

  • The system is Jacobi-scaled to a unit diagonal first; the proximal ridge is relative to that unit diagonal and pulls undetermined directions toward the current scales (so dead input channels never zero a weight).
  • Rows are processed in chunks sized from free GPU memory; a chunk that still runs out of memory is retried with half the rows.
  • The intermediate contraction tensor is built in slices of sub-blocks and written directly into (A_r), so even a 17,408-wide projection (ffn_down, (J=1088), (2J=2176)) fits without materializing a full-size temporary.
  • Normal equations run at float32_matmul_precision="highest"; Gram statistics and coordinate descent may use TF32.

The continuous solution ((s_{ls},o_{ls})) is then snapped to the storable Q2_K lattice (next section).


Metric-coupled lattice snapping

A naive snap rounds each sub-block scale independently in weight space. That throws away the output metric and lets a single low-energy, noisy sub-block dictate the super-block scale, crushing the 4-bit resolution of the important sub-blocks. The solver instead snaps in the Gram metric using each sub-block's (2\times2) block ((A_{ss},A_{so},A_{oo})):

  1. Schur-complement scale weights. The effective curvature for snapping (s_j) (with (o_j) free to adjust) is (w_s = A_{ss}-A_{so}^2/A_{oo}). A small grid of super-block scale multipliers around (\max(s)/15) is searched under these weights.
  2. Conditional offset compensation. When (s_j) is snapped to (s_{\text{snap},j}), the rounding residual (\delta s_j=s_{\text{snap},j}-s_{ls,j}) shifts the sub-block along (q_j). The optimal offset shifts by (-(A_{so}/A_{oo}),\delta s) to cancel that mean shift, so the target offset is adjusted to (O_{adj}=O_{ls}-(A_{so}/A_{oo})(S_{\text{snap}}-S_{ls})) before snapping ((d_{min},lm)) under (w_o=A_{oo}).
  3. Converter-canonical super-scales. The snapped super-block scale is normalized to (d=\operatorname{FP16}(\max_j s_{\text{snap},j}/15)) so that (\max_j ls_j=15) (and likewise for (d_{min})). This is exactly the grid the converter's factorize_q2k reconstructs from min/max, which is what makes the fitted lattice survive the final re-quantization losslessly (see Per-row guards and exact repacking).

Prior annealing (why high-dimensional projections need it)

The empirical Gram matrix (C=X_q^\top X_q) is estimated from a finite calibration set. For wide, low-rank activations β€” especially the MLP projections gate/up/down, whose input dimension can exceed the token count β€” (C) has a large nullspace and many tiny eigenvalues. Solving with a tiny ridge lets the fit chase sample noise in those directions: training error collapses while held-out error explodes, and the held-out guard then rejects every row.

The fix is an empirical-Bayes / Tikhonov prior toward the true teacher weights, applied to the Gram matrices themselves:

[ C_\gamma = C + \gamma,\operatorname{diag}(C),\qquad G_\gamma = G + \gamma,\operatorname{diag}(C),W_{fp}^\top. ]

This is equivalent to augmenting the calibration tokens with synthetic orthogonal tokens of energy (\gamma C_{ii}) whose teacher output is exact. In high-energy directions the real data dominates; in the nullspace the fit stays anchored to (W_{fp}), weighted by channel importance.

The prior strength (\gamma) is annealed from strong to weak across the rounds:

  • (d\le 8192): (\gamma\in(2.0,\ 0.75,\ 0.15,\ 0.0)) over ls_iters rounds.
  • (d>8192) (e.g. ffn_down): (\gamma\in(3.0,\ 1.0)) over a reduced round count.

Crucially, every candidate is judged on the true, unregularized training error (_row_err_gram with the original (C,G)) and the held-out error (_row_err_val). The prior only shapes the proposal; the per-row guard decides acceptance. So each row automatically lands at its own bias–variance optimum: rows that benefit from drifting away from (W_{fp}) (e.g. an attention output projection compensating upstream error) take the weak-prior steps, while rows that would overfit take the strong-prior steps. The prior is added to and removed from (C) and (G) in place, so it costs no extra VRAM.


Output-aware code coordinate descent

After each scale solve, the 2-bit codes are refined by _cd_codes, which minimizes the same output-error quadratic directly over (q\in{0,1,2,3}):

  • For a row with weights (w) and fixed scales/offsets, moving weight (i) to lattice value (c) changes the error by (2(c-w_i)h_i + C_{ii}(c-w_i)^2) where (h=Cw-g). The minimizing lattice point is nearest to (t_i=w_i-h_i/C_{ii}).
  • Weights are visited 16 at a time (one sub-block). Inside a sub-block the gradient (h) is updated exactly column by column; the rest of the row receives one GEMM per sub-block. This keeps the update output-aware rather than weight-space nearest.
  • cd_damp shrinks every step (denominator (C_{ii}(1+\text{damp}))) as a trust region; cd_ridge adds a proximal term toward the starting weights, lifting small eigenvalues the way GPTQ's percdamp does.
  • Each sweep is guarded per row on fit + held-out error; a sweep that improves no row stops early. The first round uses cd_sweeps sweeps, later rounds use one.

When cd_sweeps=0, the solver falls back to weight-space nearest-code re-derivation (_rederive_codes), still guarded per row.

For very wide projections ((d>8192)) the number of scale rounds is capped at two (eff_ls_iters = min(ls_iters, 2)) because each round solves a (2176\times2176) Cholesky system per row; two rounds plus the final alternate-ridge phase capture essentially all of the achievable reduction at a fraction of the cost.


Per-row guards and exact repacking

The best candidate is tracked per output row, including its codes and all Q2_K lattice fields ((q,s,o,d,d_{min},ls,lm)):

  • A row only takes a new candidate if its output error improves on both the fitting tokens and the held-out tokens (e_c <= e_best and v_c <= v_best).
  • Rejected rows restart the next round from their own best state, so a rejected candidate can never poison a later solve.
  • After the normal rounds, an alternate-ridge phase (ls_ridge_alt, default 3e-2) runs one more scale solve with a stronger prior. Because it runs after the fit has reached its best state and is row-guarded, it can only improve on that state.
  • The winning rows are repacked with _q2k_pack_from_parts (no re-quantization) and verified bit-exactly against the independent test decoder; a mismatch raises rather than silently emitting a wrong payload.

Converter round-trip invariance

The staging GGUF stores F16 weights that the converter later re-quantizes with factorize_q2k. That function recovers a Q2_K lattice tensor bit-exactly when each active sub-block spans both boundary codes ((\min q=0,\ \max q=3)) and each super-block uses the canonical super-scales (d=\operatorname{FP16}(\max_j s_j/15)), (d_{min}=\operatorname{FP16}(\max_j o_j/15)). The metric-coupled snap produces canonical super-scales, and the code re-derivation enforces the boundary codes on the extreme continuous ratios. The trainer's final manager.refresh() therefore measures exactly what the F16 export followed by the converter will emit, and the block-level guard (below) is evaluated on that real re-quantized result.


Block-output residual absorption

A decoder block computes (\text{Out}=h+\text{Attn}(h)+\text{FFN}(h+\text{Attn}(h))). By the time the final projection group runs (typically ffn_down / mlp.down_proj, which writes linearly into the residual stream), the block already carries the accumulated quantization error of the attention sub-block and the activation shift from quantized gate/up.

Fitting that final projection against its isolated teacher output (X_{fp}W_{fp}^\top) would ignore this upstream error. Instead, when the last group is a single projection that adds directly into the block output, the solver sets its target to the true block-output residual:

[ Y^\star = \text{targets} - \big(\text{BlockOut}{stu} - X_q W{stu}^\top\big) = \text{targets} - \text{residual}_{stu}. ]

Because (\text{BlockOut}{stu}(W)=\text{residual}{stu}+X_qW^\top), minimizing (|X_qW^\top-Y^\star|^2) is identically minimizing the end-to-end block-output error (|\text{BlockOut}_{stu}(W)-\text{targets}|^2), row by row. The final projection thus absorbs the upstream quantization residual in closed form.

Whether a projection qualifies is verified at runtime, not assumed: a one-token probe adds +1.0 to the module's output and checks that the decoder-layer output changes by exactly +1.0 (identity residual, no post-projection norm). If the check fails, the standard teacher target is used. This keeps the absorption correct across architectures without hardcoding module names.


Block-level revert guard

After the algebraic fit, the trainer re-quantizes the fitted weights (manager.refresh()) and evaluates the held-out block output. If the fitted-and-re-quantized relative error is worse than the plain-SMQ baseline by more than 1e-6, the entire block is reverted to plain SMQ and re-evaluated. Combined with the per-row guards, this means a block can never end up worse than plain SMQ on the held-out split.


Sequential block adaptation

The QAT flow is:

  1. Read the GGUF architecture metadata and discover the decoder block structure from tensor names and metadata.
  2. Build corpus token sequences using the tokenizer embedded in the GGUF, or a supplied tokenizer path.
  3. For the first trained block, embed the corpus with the source embedding table. For later blocks, use the activation buffer produced by the previous block.
  4. Prepare a one-decoder-layer HF model containing the current block. Tensor-name/shape mappings and supported conversion inversions are checked before training.
  5. Compute source-weight block-output targets (FP32, pre-norm) and record the held-out tail split.
  6. Attach fake-quant modules to eligible linear projections.
  7. Run the algebraic fit: group discovery β†’ Gram statistics β†’ closed-form scale solve β†’ metric-coupled snap β†’ prior-annealed code coordinate descent β†’ per-row guards β†’ alternate-ridge phase β†’ residual absorption on the final projection.
  8. Re-quantize, evaluate the held-out block output, and revert the block if it regressed.
  9. Save adapted block tensors in safetensors under the QAT work directory (trained/blk{b}.safetensors).
  10. Refresh Q2_K caches and propagate the adapted, fake-quantized block output to the next decoder block (propagate="qat").
  11. Export a complete mostly-F16 staging GGUF. Trained 2-D projections are written as F16; tensors with no trained counterpart pass through unchanged.
  12. Run the SMQ converter on that staging GGUF to produce the final Q2_K model.

The trainer discovers block tensors from the GGUF and uses explicit architecture/name mappings for the HF bridge. The supplied mapping includes llama, qwen2, qwen3, mistral, qwen3next, and qwen35; unsupported architectures or unmapped tensor names fail validation rather than being silently guessed. NextN/MTP draft blocks are excluded from training because they are not part of the main forward path.

Resume

Each block's fitted weights are saved to trained/blk{b}.safetensors. On a rerun, an existing file is loaded and the block is skipped. To retrain a block with new settings, delete its trained/blk{b}.safetensors first.

Corpus construction

The trainer reads .txt, .text, .md, and .jsonl inputs. For JSONL it uses a text field, falling back to content; malformed JSON lines are treated as plain text. A directory is read recursively in sorted path order.

After tokenization, it selects num_seqs evenly spaced windows of length seq_len from the available token stream, then shuffles their order using the supplied seed. max_tokens=0 means use the full token stream; a positive value truncates the stream before windows are selected.

Activation buffers are stored as BF16, in RAM for smaller corpora or memory-mapped files when the sequence count exceeds the configured threshold (64 by default). Targets are stored as FP32.


QAT training example

The training CLI is train_smq_qat.py:

python train_smq_qat.py \
  --gguf model-source-Q8_0.gguf \
  --corpus calibration_data.txt \
  --output model-adapted-f16.gguf \
  --imatrix model.imatrix.gguf \
  --num-seqs 32 --seq-len 512 \
  --micro-batch 2 \
  --ls-iters 4 --cd-sweeps 2 \
  --device cuda:0

Then quantize the staging GGUF with the converter:

python smq_gguf_convert_live.py model-adapted-f16.gguf \
  --imatrix model.imatrix.gguf \
  --output-gguf model-smq-q2k.gguf \
  --out reports/model-smq-q2k

The Python API is equivalent:

from pathlib import Path
from smq_qat import train

train(
    gguf_path=Path("model-source-Q8_0.gguf"),
    corpus=Path("calibration_data.txt"),
    output=Path("model-adapted-f16.gguf"),
    num_seqs=32,
    seq_len=512,
    micro_batch=2,
    ls_iters=4,
    ls_ridge=1.0e-3,
    ls_ridge_alt=3.0e-2,
    cd_sweeps=2,
    cd_damp=0.01,
    cd_ridge=0.0,
    seed=1234,
    imatrix=Path("model.imatrix.gguf"),
    device="cuda:0",
    propagate="qat",
)

Algebraic fit parameters

Parameter / flag Default Meaning
ls_iters / --ls-iters 4 Scale-solve β†’ snap β†’ code-update rounds per projection (capped at 2 for d>8192)
ls_ridge / --ls-ridge 1e-3 Proximal ridge of the scale solve toward current scales, relative to a unit diagonal
ls_ridge_alt / --ls-ridge-alt 3e-2 Extra final scale-solve phase with a stronger prior; row-guarded, so it cannot worsen the fit (0 disables)
ls_row_chunk / --ls-row-chunk 0 Rows per batched scale solve (0 = auto from free VRAM)
cd_sweeps / --cd-sweeps 2 Output-aware coordinate-descent sweeps over the 2-bit codes after each scale solve (0 = weight-space re-derivation fallback)
cd_damp / --cd-damp 0.01 Relative step damping of the code update (trust region)
cd_ridge / --cd-ridge 0.0 Proximal ridge of the code update toward its starting weights (GPTQ-style percdamp)

Other useful parameters

Parameter / flag Default Meaning
gguf_path / --gguf required Source GGUF used to construct block teachers
corpus / --corpus required Text file, JSONL file, or directory of supported text files
output / --output required Complete adapted staging GGUF output path
blocks / --blocks LO-HI all Optional block range; HI exclusive
num_seqs / --num-seqs 32 Number of corpus windows
seq_len / --seq-len 512 Tokens per window
max_tokens / --max-tokens 0 Optional token-stream limit; zero means all
micro_batch / --micro-batch 4 Sequences per forward micro-batch while collecting Gram statistics
seed / --seed 1234 (CLI) Corpus-window shuffle seed
imatrix / --imatrix none Optional .dat or GGUF importance matrix
tokenizer / --tokenizer none Optional external tokenizer path; otherwise GGUF tokenizer metadata is used
device / --device cuda Training device, e.g. cuda:0 or cpu
propagate / --propagate qat Activation propagation mode (qat or full)
train_types / --train-types all Comma-separated tensor types to train
offload_q / --offload-q auto Decoded fake-quant cache placement (auto/on/off)
offload_threshold 64 Sequence count above which activations are memory-mapped
quant_workers / --quant-workers auto Tensors quantized concurrently during a refresh
work_dir / --work-dir <output>.qat_work Work directory (holds trained/ and temp buffers)
keep_block_dirs / --keep-block-dirs off Keep per-block HF partial directories
validate true Run structural/transform/HF-mapping preflight before training
verbose / --verbose off Log every trainable tensor and per-iteration row counts
dry_run / --dry-run off Print the plan and exit

Legacy optimizer arguments

The CLI and train() still accept the old optimizer arguments (--lr, --steps, --weight-decay, --weight-rmse-lambda, --cosine-lambda, --grad-accum, --requant-every, --requant-drift, --warmup-frac, --norm-lr-mult, --cache-quant-error, --fresh-penalty, --adaptive-step, etc.) so existing callers do not break, but they are ignored by the algebraic fit. There is no optimizer to configure.


Preflight and export utilities

The smq_qat.py command-line entry point is for validation and export; full training is driven by train_smq_qat.py.

Validate architecture and tensor mappings

python smq_qat.py model.gguf
python smq_qat.py model.gguf --blocks 0 8     # upper bound exclusive
python smq_qat.py model.gguf --no-deep        # skip the Transformers name/shape comparison

The preflight checks tensor mappings and dimensions, verifies GGUF-to-HF transforms against their source values, and can compare supplied parameters with the actual Transformers model structure.

Export already-trained blocks

python smq_qat.py source.gguf \
  --export-trained /path/to/output.qat_work/trained \
  --output adapted-f16.gguf

An optional --tmp-dir controls where the GGUF export spool file is written.


Reading the metrics

Converter metrics

The converter's per-tensor CSV/JSON reports include quantization statistics such as relative RMSE and its comparison with the RTN baseline. With an imatrix, these statistics are importance-weighted. They describe weight reconstruction, not model-level perplexity or downstream task performance.

QAT weight RMSE

The trainer logs absolute and relative RMSE (||W-Q(W)|| / ||W||) for each trained projection and an aggregate relative RMSE, measured after the final re-quantization β€” i.e. the trained master weights against the decoded Q2_K representation the converter will actually emit. A lower weight RMSE means the adapted weights are more accurately represented by Q2_K; it does not by itself establish better PPL or generalization.

QAT output RMSE and cosine

The block-output diagnostic compares the trained, re-quantized block's output with its source-weight target, captured before the final model norm. Unlike earlier optimizer-based versions, this diagnostic is evaluated on the held-out tail split, not on training sequences. The trainer logs:

  • BASELINE, plain SMQ before training
  • ALGEBRAIC, fitted + re-quantized
  • REVERTED, plain SMQ (only if the block-level guard fired)
  • FINAL, trained + re-quantized

Each line reports out_rmse, relative error, and cosine similarity on the held-out sequences. Per-projection lines additionally report fit-split and held-out relative error before/after, and the number of rows improved.

Use an independent corpus and evaluate the complete exported Q2_K model for final quality comparisons; block-output error is a necessary but not sufficient condition for end-to-end quality.


Reported experimental observations

The following are experimental observations from development, not universal guarantees or independently reproduced benchmark claims.

Block 0: algebraic fit on a 27B model

A single-block run on ThinkingCap-Qwen3.8-27B-abliterated-Q8_0.gguf (block 0, 32 sequences Γ— 512 tokens, imatrix supplied, ls_iters=4, cd_sweeps=2, ls_ridge_alt=3e-2) completed in ~266 s on one RTX 3060 (12 GiB) with every output row improved on every tensor:

Tensor Rows improved Train rel error Held-out rel error Fit time
linear_attn.in_proj_qkv.weight 10240 / 10240 8.06% β†’ 4.26% 7.94% β†’ 5.31% 25.1 s
linear_attn.in_proj_z.weight 6144 / 6144 10.76% β†’ 5.85% 10.48% β†’ 7.21% 15.4 s
linear_attn.in_proj_b.weight 48 / 48 7.96% β†’ 4.00% 7.81% β†’ 4.92% 2.0 s
linear_attn.in_proj_a.weight 48 / 48 8.82% β†’ 4.59% 8.46% β†’ 5.55% 2.0 s
linear_attn.out_proj.weight 5120 / 5120 16.73% β†’ 1.63% 16.72% β†’ 2.26% 18.6 s
mlp.gate_proj.weight 17408 / 17408 22.52% β†’ 13.13% 24.72% β†’ 18.82% 43.5 s
mlp.up_proj.weight 17408 / 17408 23.25% β†’ 13.48% 25.43% β†’ 19.24% 43.3 s
mlp.down_proj.weight 5120 / 5120 22.93% β†’ 12.10% 30.29% β†’ 26.28% 104.3 s

End-to-end held-out block output, after real Q2_K re-quantization:

Measurement Plain SMQ baseline After algebraic QAT
Block-output RMSE (out_rmse) 2.609e-02 7.016e-03
Relative block-output error 12.68% 3.41%
Cosine similarity 0.99351 0.99950
Aggregate relative weight RMSE (post-requant) 29.36% 1.81%

This is a ~3.7Γ— reduction in held-out block-output RMSE with no optimizer and no backward pass. The mlp.down_proj row uses residual absorption, so its before/after numbers reflect the full block-output residual rather than an isolated layer target. These are single-block diagnostics on a held-out calibration split, not a whole-model perplexity or benchmark claim.

Whole-model observations

During development, an approximately 8.6 GB Q2 model from a reported ~45 GB Q8_0 source was reported, along with perplexity around 8 and a higher ARC-Challenge score than the compared baseline. The exact evaluation corpus, ARC scoring protocol, baseline artifacts, and reproducibility procedure were not included in the supplied source files; treat these as reported experimental observations rather than a benchmark claim.


Memory and performance notes

  • No optimizer state. Per trainable parameter the fit needs roughly the student weight (4 B) + decoded cache (4 B) + codes (1 B) + decode/snap temporaries (~4 B) β‰ˆ 13 B/param, plus one (d\times d) Gram matrix and a (d\times m) cross-Gram per input group. There are no momentum/variance buffers and no activation graph for backprop.
  • Chunked, OOM-resilient solves. Scale-solve row chunks and coordinate-descent row chunks are sized from free VRAM and halved on OutOfMemoryError. The scale-solve contraction tensor is built in sub-block slices so wide projections do not materialize a full-size temporary.
  • Precision. Gram statistics and coordinate descent may use TF32 ("high"); the normal-equation Cholesky solves run at "highest". Because the codes are 2-bit integers, TF32 in the statistics does not degrade the lattice fit.
  • Wide projections. For d>8192 the scale-round count is capped at two and the prior schedule is shortened, keeping the (2176\times2176) per-row Cholesky cost bounded.
  • Fragmentation. On tight VRAM, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True reduces fragmentation across the many differently-sized allocations of a block.
  • Offloading. The decoded fake-quant cache can be kept in pinned CPU RAM (--offload-q on or auto) when the memory plan indicates it will not fit on the GPU.

Validation recommendations

For a fair evaluation of quantizer changes or QAT hyperparameters:

  1. Keep the source model fixed. Record its exact GGUF file, tensor types, and checksum.
  2. Keep imatrix provenance fixed. Record the imatrix file and whether each tensor hit or missed it.
  3. Use the same corpus windows and seed when comparing settings.
  4. Separate training and evaluation text. The block diagnostic uses a held-out calibration tail, but final quality should be measured on independent data with the complete exported model.
  5. Measure the actual emitted Q2_K file. Decode the final packed Q2_K tensors and compute reconstruction error against the corresponding adapted staging weights.
  6. Evaluate end-to-end. Record PPL, KLD, token-probability changes, and benchmark accuracy with identical prompts, tokenization, scoring rules, and evaluation subsets.
  7. Ablate the fit. Compare ls_ridge, ls_ridge_alt, cd_sweeps, cd_damp, and cd_ridge while holding corpus, seed, and source checkpoint constant.
  8. Keep a copy of the adapted staging GGUF. It makes it possible to distinguish QAT effects from the final converter effects.

For multiple-choice benchmarks such as ARC-Challenge, report both accuracy and the exact scoring method. Perplexity and multiple-choice accuracy measure different behavior and can move in different directions.


Implementation notes and limitations

QAT does not adapt every matrix that the converter quantizes

The converter's selection is generic and can include eligible embeddings and output matrices. The QAT module's trainable projection list is built from decoder-block tensors. The token embedding is used to construct the first activation buffer but is not optimized; the output/LM head is not part of the block-output objective. One-dimensional norms/biases are not co-adapted by the algebraic fit. Those matrices can still be quantized by the final converter if eligible.

Source precision defines the teacher

β€œFull-precision target” in the QAT code means the target produced by the source GGUF weights after the GGUF-to-HF conversion. It does not restore precision discarded before the input GGUF was created.

Block diagnostics are not a held-out model benchmark

The block-output diagnostic uses a held-out tail of the calibration corpus, which is a genuine validation split for the fit, but it is still a single-block, calibration-domain measurement. Evaluate the complete exported model on independent data before drawing conclusions about generalization.

Guarded payload versus reported candidate metrics

In the supplied converter snapshot, the per-super-block guard chooses between the quadratic candidate and RTN for the packed parameters. However, the current metric accumulator uses the quadratic candidate's residual moments, and the optional reconstruction buffer is built from the quadratic candidate residual. If any super-block falls back to RTN, these candidate metrics/reconstruction files may not exactly equal the mixed guarded payload that was packed. For an exact report on the emitted GGUF, decode the written Q2_K tensor and compare it directly with the source tensor.

Disabling the RTN guard

--no-rtn-metrics disables both the RTN comparison and its per-block safeguard. Use it only when an unguarded quadratic decomposition is intended.

Supported architectures

The QAT HF bridge uses an explicit architecture-to-Transformers model mapping plus GGUF metadata and tensor-name transforms. Unsupported architectures, unknown tensor suffixes, or shape mismatches fail preflight rather than being guessed. Converter-only GGUF quantization does not require the HF bridge. Residual absorption is verified per block by a runtime probe rather than assumed from architecture names.

File-size comparisons

The nominal Q2_K block payload is 2.625 bits per weight. Whole-file size depends on the number and size of eligible tensors, pass-through tensors, metadata, and any selected tensor filters. Always compare actual file sizes and list-tensor reports rather than assuming every model parameter is stored at exactly 2.625 bits.


Project status

SMQ and SMQ-QAT are experimental implementations. The converter includes packing/factorization self-tests and validates GGUF tensor shapes when writing. The QAT module includes preflight checks for GGUF/HF mappings and conversion transforms, per-row and per-block held-out guards, and bit-exact repack verification. Model-level PPL and benchmark claims should be reported with independent evaluation data and reproducible settings.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support