CompressedGemma/HPC-Quantize-V2

GUI screencap

Developer Note

This requires a CUDA GPU, expect a 50GB model to take around 3-4 hours to quantize on a 3060 12GB.

I prefer to llama-quantize to 8_0, then to generate a imatrix from that 8_0, and finally to quantize the 8_0 file to Q2K.

HPC-Quantize V2

Holographic Phase Contraction, Geometry-Preserving Weight Analogs, and Algebraic Coupled Bilinear Factorization for Ultra-Low-Bit (Q2_K) LLM Quantization

HPC-Quantize V2 is a standalone Python/NumPy ultra-low-bit quantization engine and GGUF re-quantizer (with a real-time telemetry desktop GUI in gui.py) built for 2-bit class (Q2_K, 2.625 bpw) compression of large language models.

Core thesis: At 2 bits per weight (4 levels $q \in {0,1,2,3}$ and 4-bit sub-block scales $Ls, Lm \in {0..15}$), quantization is not an independent rounding problem and should not rely on brute-force grid scans. It is a coupled bilinear factorization and global sequence-optimization problem best solved by extracting discrete lattice coordinates from geometry-preserving weight analogs, projecting them onto the true weights via closed-form orthogonalized sufficient statistics, and optimizing cross-block state trajectories with a 6-state sequential Sieve and 36-state Viterbi dynamic program.

HPC-Quantize V2 (hpc_quantize_v2/ + gui.py) has zero C or ctypes dependencies, reads and writes stock llama.cpp-compatible .gguf models and .gguf/.dat importance matrices (iMatrix), scales across all CPU cores with bit-for-bit determinism, and requires no custom inference kernels.

---

Key Highlights in V2

1. Geometry-Preserving Weight Analogs (hpc_quantize_v2/analog.py): Constructs a portfolio of surrogate weight blocks $\tilde{x}$ that preserve exact sub-block first and second importance-weighted moments ($\mu_j, \sigma_j^2$), rank order, unit sub-block direction ($\cos\theta = 1$), total block energy, and the $2\times 2$ D6/Vesica fold Gram matrix while regularizing heavy-tailed outliers toward the 4-level Q2_K lattice.
2. Coupled Orthogonal-Basis Bilinear Q2_K Factorization (hpc_quantize_v2/factored.py): Replaces independent 1-D scale/offset heuristics with an orthogonalized sub-block decomposition $\hat{x}{j,k} = s_j(q{j,k} - \bar{q}j) + (s_j\bar{q}j - t_j)$, coupled offset compensation $t_j^{\text{coupled}} = \max(0, (d \cdot Ls_j \cdot S{wq,j} - S{x,j})/S_{w,j})$, rank-1 algebraic 1-D code refinement ($S_{vv} - S_{lv}^2/S_{l2}$), and a single-pass vectorized 36-seed candidate lattice evaluator (81x faster than per-seed loops).
3. Algebraic Coupled Phase-4 Block Assembly (hpc_quantize_v2/pipeline.py): Replaces legacy brute-force $16\times 16$ sub-block scans, $9\times 9$ weight-level ULP loops, and 256-pass greedy code searches with closed-form sufficient-statistic projections (1,200–1,475 blocks/sec in Phase 4, achieving +2.64% lower objective error than brute-force Phase 4 in 1/9th the time).
4. Bit-Identical Multi-Core Execution (--workers 0): Distributes block-independent candidate and analog solves across all CPU cores via zero-copy fork shared memory while keeping cross-block Sieve, 36-state Viterbi, and row-aware DC carry strictly sequential — producing 100% bit-for-bit identical GGUF output regardless of worker count.
5. Real-Time Telemetry GUI (gui.py) & Direct GGUF-to-GGUF Pipeline (hpc_quantize_v2/gguf.py): Ingests F32, F16, BF16, Q8_0, Q4_0, and Q2_K source GGUFs plus llama.cpp iMatrix files, displays live intra-tensor stage/block progress, dual Tensor ETA and block-weighted Whole-Model ETA, live RMSE/RMSE/σ, Sieve temperature, and floor hits, and emits a stock llama.cpp Q2_K GGUF.

---

Why Standard Q2_K Quantization Breaks Down

A Q2_K superblock encodes 256 weights in 84 bytes (16 sub-blocks of 16 weights each):

$${\widehat{x}}_{16j+k}=d\cdot L{s}_j\cdot q_{16j+k}-d_{\min }\cdot L{m}_j,q\in \{0,1,2,3\},L{s}_j,L{m}_j\in \{0,\dots ,15\},d,d_{\min }\in {\mathbb{F}}_{16}\mathrm{.}\backslash hat\{x\}\_\{16j\; +\; k\}\; =\; d\; \backslash cdot\; Ls\_j\; \backslash cdot\; q\_\{16j\; +\; k\}\; -\; d\_\{\backslash min\}\; \backslash cdot\; Lm\_j,\; \backslash qquad\; q\; \backslash in\; \backslash \{0,1,2,3\backslash \},\; \backslash quad\; Ls\_j,\; Lm\_j\; \backslash in\; \backslash \{0,\backslash ldots,15\backslash \},\; \backslash quad\; d,\; d\_\{\backslash min\}\; \backslash in\; \backslash mathbb\{F\}\_\{16\}.$$

When factoring raw LLM weights directly into this bilinear form, three structural failure modes occur:

1. Intra-sub-block outlier collapse: A single outlier weight in a 16-weight sub-block stretches the span $x_{\max} - x_{\min}$, forcing 13–15 of the 16 weights into a single bin ($q=1$ or $q=2$) and reducing effective precision below 1 bit.
2. Inter-sub-block scale crushing: A single high-dispersion sub-block inflates the shared superblock scale $d$, forcing $Ls_j \in {1, 2}$ across the remaining 15 sub-blocks.
3. Scale-offset cross-term neglect: Because $q_{j,k} \in {0,1,2,3}$ is non-negative, its importance-weighted mean $\bar{q}j = \frac{\sum_k w{j,k} q_{j,k}}{\sum_k w_{j,k}}$ is strictly positive. Rounding the scale $s_j \to d \cdot Ls_j$ shifts the sub-block mean by $(d \cdot Ls_j - s_j)\bar{q}j$. Factoring $s_j \approx d \cdot Ls_j$ and $t_j \approx d{\min} \cdot Lm_j$ independently leaves this mean shift uncompensated.

---

Mathematical Architecture of HPC-Quantize V2

1. Geometry-Preserving Weight Analogs (analog.py)

To prevent outliers from collapsing the discrete code assignment $q \in {0,1,2,3}$ while respecting the true linear-layer geometry, V2 constructs a portfolio of geometry-preserving analogs $\tilde{x}$ of each eligible weight block before extracting candidate lattice coordinates:

A. Sub-block Iso-Moment Quantile-Companded Analog (subblock_isomoment_analog)

For each 16-weight sub-block $j$ with importance weights $w_{j,k}$:

$${\mu }_j=\frac{\sum _k{w}_{j,k}{x}_{j,k}}{\sum _k{w}_{j,k}},{\sigma }_j^2=\frac{\sum _k{w}_{j,k}(x_{j,k}-{\mu }_j{)}^2}{\sum _k{w}_{j,k}},z_{j,k}=\frac{x_{j,k}-{\mu }_j}{{\sigma }_j}\mathrm{.}\backslash mu\_j\; =\; \backslash frac\{\backslash sum\_k\; w\_\{j,k\}\; x\_\{j,k\}\}\{\backslash sum\_k\; w\_\{j,k\}\},\; \backslash qquad\; \backslash sigma\_j^2\; =\; \backslash frac\{\backslash sum\_k\; w\_\{j,k\}(x\_\{j,k\}\; -\; \backslash mu\_j)^2\}\{\backslash sum\_k\; w\_\{j,k\}\},\; \backslash qquad\; z\_\{j,k\}\; =\; \backslash frac\{x\_\{j,k\}\; -\; \backslash mu\_j\}\{\backslash sigma\_j\}.$$

Standardized coordinates are mapped through an importance-weighted hyperbolic transport blended with ideal 4-level quantiles $q^{\text{ideal}}_r \in [-1.35, +1.35]$:

$${\tilde{z}}_{j,k}=(1-\alpha )\tau \tanh \text{\hspace{0.17em}⁣}\left(\frac{(w_{j,k}\mathrm{/}{\bar{w}}_j{)}^{\beta }{z}_{j,k}}{\tau }\right)+\alpha q_{\text{rank}(z_{j,k})}^{\text{ideal}},\backslash tilde\{z\}\_\{j,k\}\; =\; (1\; -\; \backslash alpha)\backslash ,\backslash tau\; \backslash tanh\backslash !\backslash left(\backslash frac\{(w\_\{j,k\}/\backslash bar\{w\}\_j)^\backslash beta\; z\_\{j,k\}\}\{\backslash tau\}\backslash right)\; +\; \backslash alpha\backslash ,\; q^\{\backslash text\{ideal\}\}\_\{\backslash text\{rank\}(z\_\{j,k\})\},$$

and then orthogonally projected back onto the exact sub-block sphere:

$${\tilde{x}}_{j,k}={\mu }_j+{\sigma }_j\frac{{\tilde{z}}_{j,k}-{\bar{z}}_j}{\sqrt{\frac{1}{W_j}\sum _k{w}_{j,k}({\tilde{z}}_{j,k}-{\bar{z}}_j{)}^2}}\mathrm{.}\backslash tilde\{x\}\_\{j,k\}\; =\; \backslash mu\_j\; +\; \backslash sigma\_j\; \backslash frac\{\backslash tilde\{z\}\_\{j,k\}\; -\; \backslash bar\{z\}\_j\}\{\backslash sqrt\{\backslash frac\{1\}\{W\_j\}\backslash sum\_k\; w\_\{j,k\}(\backslash tilde\{z\}\_\{j,k\}\; -\; \backslash bar\{z\}\_j)^2\}\}.$$

This preserves the exact sub-block weighted mean $\mu_j$, weighted variance $\sigma_j^2$, and rank ordering while equalizing spacing across the 4 quantization levels.

B. D6 / Vesica Gram-Preserving Fold Analog (vesica_gram_analog)

Decomposes the 256-weight superblock into symmetric (vesica) and antisymmetric (wave) fold modes:

$$v_p=\frac{1}{2}(x_p+x_{p+128}),u_p=\frac{1}{2}(x_p-x_{p+128}),p\in \{0,\dots ,127\}\mathrm{.}v\_p\; =\; \backslash frac\{1\}\{2\}(x\_p\; +\; x\_\{p+128\}),\; \backslash qquad\; u\_p\; =\; \backslash frac\{1\}\{2\}(x\_p\; -\; x\_\{p+128\}),\; \backslash qquad\; p\; \backslash in\; \backslash \{0,\backslash ldots,127\backslash \}.$$

Let $G \in \mathbb{R}^{2\times 2}$ be the importance-weighted covariance (Gram) matrix of $(v, u)$. V2 whitens $(v, u)$ via $G^{-1/2}$, applies smooth hyperbolic saturation along the principal axes, re-orthogonalizes, and colors back via $G^{1/2}$ so that:

$${\mathrm{Gram}}_w(\tilde{v},\tilde{u})={\mathrm{Gram}}_w(v,u),{\mathbb{E}}_w[\tilde{v}]={\mathbb{E}}_w[v],{\mathbb{E}}_w[\tilde{u}]={\mathbb{E}}_w[u]\mathrm{.}\backslash operatorname\{Gram\}\_w(\backslash tilde\{v\},\; \backslash tilde\{u\})\; =\; \backslash operatorname\{Gram\}\_w(v,\; u),\; \backslash qquad\; \backslash mathbb\{E\}\_w[\backslash tilde\{v\}]\; =\; \backslash mathbb\{E\}\_w[v],\; \backslash qquad\; \backslash mathbb\{E\}\_w[\backslash tilde\{u\}]\; =\; \backslash mathbb\{E\}\_w[u].$$

C. Cross-Sub-block Scale-Lattice Analog (subscale_lattice_analog)

Tempers extreme sub-block standard deviations $\sigma_j \mapsto \sigma_j^\gamma \sigma_{\text{block}}^{1-\gamma}$ so all 16 sub-blocks fit cleanly into the shared 4-bit ${0..15}$ scale lattice while preserving every sub-block's unit direction $\hat{u}_j$ ($\cos\theta = 1$), mean $\mu_j$, and total superblock weighted energy.

---

2. Coupled Orthogonal-Basis Bilinear Q2_K Factorization (factored.py)

Given candidate discrete codes $q_{j,k} \in {0,1,2,3}$ (from raw starts and the analog portfolio), V2 projects the continuous and 4-bit scale parameters directly onto the true weights $x$:

1. Orthogonalized Sub-block Basis: Rewriting $\hat{x}{j,k} = s_j(q{j,k} - \bar{q}j) + (s_j\bar{q}j - t_j)$ diagonalizes the $2\times 2$ sub-block normal equations. The true sensitivity of sub-block $j$ to scale error $(s_j - d \cdot Ls_j)^2$ is the centered variance weight: $$ V{qq,j} = \sum{k=0}^{15} w_{j,k}(q_{j,k} - \bar{q}j)^2 = S{qq,j} - S_{wq,j}\bar{q}_j. $$
2. Monotone Rank-1 1-D Factorization (factor_1d): Factors $s_j \approx d \cdot Ls_j$ under weights $V_{qq,j}$ using multi-start Lloyd iterations followed by an exact vectorized rank-1 coordinate sweep maximizing $S_{lv}^2 / S_{l2}$ (since $\sum_j w_j(v_j - d^* L_j)^2 = S_{vv} - S_{lv}^2/S_{l2}$).
3. Coupled Offset Target: Once $(d, Ls_j)$ are chosen, the optimal offset for sub-block $j$ is shifted in closed form to absorb the scale rounding residual: $$ t_j^{\text{coupled}} = \max!\left(0,; \frac{d \cdot Ls_j \cdot S_{wq,j} - S_{x,j}}{S_{w,j}}\right) = \max!\left(0,; t_j + \bar{q}j(d \cdot Ls_j - s_j)\right), $$ which is then factored into $d{\min} \cdot Lm_j$ under weights $S_{w,j}$.

---

3. Batched Algebraic Phase-4 Assembly (pipeline.py)

Instead of brute-force $16\times 16$ sub-block scans and $9\times 9$ weight-level ULP searches, Phase 4 refines both the Viterbi-selected state and the analog-factored state across all blocks in vectorized batches using three exact algebraic identities:

1. Closed-Form Coupled Sub-scale Projection: From sufficient statistics $(V_{qq,j}, C_{qx,j}, S_{wq,j}, S_{x,j}, S_{w,j})$, the unconstrained scale minimizer is $s_j^* = \max(0, C_{qx,j}/V_{qq,j})$. Evaluating the local integer neighborhood $Ls_j \in \operatorname{round}(s_j^*/d) + {-3,\ldots,+3}$ paired with its closed-form coupled offset $Lm_j \in \operatorname{round}(t_j^*(Ls_j)/m) + {-1,0,+1}$ replaces $16\times 16 = 256$ brute-force pairs with $21$ algebraic states.
2. Vesica + DC Augmented $2\times 2$ Cramér Solve & $O(1)$ Scalar Quadratic ULP Polish: With $A_i = Ls_{i\gg 4} q_i$ and $B_i = Lm_{i\gg 4}$, the full extended objective (weighted SSE + Vesica fold pairs + DC carry) is an exact quadratic form in $(d, m)$: $$ J(d, m) = \text{const} + \tilde{S}{aa} d^2 - 2\tilde{S}{ab} d m + \tilde{S}{bb} m^2 - 2\tilde{S}{xa} d + 2\tilde{S}{xb} m, $$ where $(\tilde{S}{aa}, \tilde{S}{ab}, \tilde{S}{bb}, \tilde{S}{xa}, \tilde{S}{xb})$ include the Vesica pair sums $(A_p + A_{p+128}, B_p + B_{p+128}, x_p + x_{p+128})$ and the row DC sum $(\sum_i A_i, \sum_i B_i, \sum_i x_i + \text{carry})$. The continuous optimum $(d^*, m^*)$ is solved by $2\times 2$ Cramér's rule and the $\mathbb{F}_{16}$ ULP neighborhood is polished in $O(1)$ scalar arithmetic without touching the 256 weights.
3. Closed-Form 1-D Quadratic Vesica/DC Code Projection: Each weight's optimal discrete step $\Delta q_k^*$ under the combined weighted SSE + Vesica + DC objective is computed directly from the 1-D quadratic minimizer: $$ \Delta q_k^* = \operatorname{round}!\left(\frac{d_{\text{sub},k}\left(w_k e_k + \frac{\lambda_{\text{vw}}}{N}(e_k + e_{k\oplus 128}) + \frac{\lambda_{\text{dc}}}{N} D\right)}{\left(w_k + \frac{\lambda_{\text{vw}} + \lambda_{\text{dc}}}{N}\right) d_{\text{sub},k}^2}\right). $$

---

4. End-to-End HPC Pipeline

Source Weights + Optional iMatrix
        │
        ▼
Kurtosis-Gated Clamp & Batched GGML Seeds (make_qkx2_batch / make_qp_batch)
        │
        ▼
Vectorized Superblock WLS + Multi-Core Analog-Guided Factored Candidate Field
        │
        ▼
6-State Sequential Sieve (Bounded Neighbor Parity Back-Action + Log-Slack)
        │
        ▼
36-State Viterbi Global Sequence Optimization (Free-Energy Bins + Manhattan Prior)
        │
        ▼
Batched Algebraic Phase-4 Coupled Assembly + Quadratic ULP Polish + Closed-Form Code Projection
        │
        ▼
Row-Aware Sequential DC Residual Nudge & Monotone Local-Safety Floors
        │
        ▼
Standard Q2_K GGUF Output

---

Measured Benchmarks

Evaluated across 48 synthetic blocks (4 distributions) and a full row (47 superblocks = 12,032 weights) of blk.0.ffn_down.weight from Raon.gguf weighted by imatrix.gguf:

Benchmark Suite	Mean Objective Reduction	Median Reduction	Min Reduction	Worse Cases
Gaussian ($\mathcal{N}(0, 0.02^2)$)	+8.85%	+8.45%	+3.17%	0 / 12
Heavy-Tailed ($\text{Student-}t_3$)	+10.25%	+8.30%	+5.04%	0 / 12
Offset-Sparse (50% density + DC shift)	+14.07%	+14.83%	+4.16%	0 / 12
Sparse (10% density)	+35.40%	+32.78%	+17.01%	0 / 12
Real Raon.gguf (blk.0.ffn_down.weight + iMatrix)	+15.96%	—	—	0 / 47

Reproduce at any time with:

python3 -m hpc_quantize_v2 --bench --bench-per-kind 12

---

Quickstart

Requirements

Only Python 3.10+ and NumPy (tkinter is included with standard Python installations for the GUI). No C compiler or shared libraries are needed.

pip install numpy

1. Desktop GUI with Live Telemetry (gui.py)

Launch the graphical interface to browse a source .gguf, attach an .imatrix/.gguf importance file, inspect eligible tensors, tune analog/HPC parameters, set CPU worker count, copy the equivalent CLI command, and monitor live quantization with dual progress bars (Model Progress and Tensor Superblocks) and dual ETAs (Tensor ETA and Model ETA):

python3 gui.py

Or pre-populate paths on launch:

python3 gui.py --gguf Raon.gguf --imatrix imatrix.gguf --output Raon-Q2K.gguf

2. Command-Line Interface (hpc_quantize_v2)

Quantize a full GGUF model to a Q2_K GGUF with an iMatrix (using all CPU cores by default):

python3 -m hpc_quantize_v2 \
  --gguf Raon.gguf \
  --all-q2k \
  --imatrix-gguf imatrix.gguf \
  --output Raon-Q2K.gguf \
  --workers 0 \
  --verbose

Inspect tensors in a GGUF file:

python3 -m hpc_quantize_v2 --gguf Raon.gguf --list-tensors

Smoke-test a single tensor into a valid GGUF (copies remaining tensors verbatim):

python3 -m hpc_quantize_v2 \
  --gguf Raon.gguf \
  --tensor blk.0.ssm_beta.weight \
  --imatrix-gguf imatrix.gguf \
  --output Raon-smoke.gguf

Quantize a NumPy .npy weight array to .npz:

python3 -m hpc_quantize_v2 --input weights.npy --output weights_q2k.npz --verbose

3. Python API

import numpy as np
from hpc_quantize_v2 import (
    HPCConfig,
    Q2KQuantizer,
    read_imatrix,
)
from hpc_quantize_v2.gguf import write_quantized_gguf

# 1. Quantize an in-memory tensor (workers=0 auto-uses all CPU cores)
cfg = HPCConfig(
    n_cand_d=6,
    n_cand_m=6,
    use_weight_analogs=True,
    factor_coupled_q2k=True,
    factor_joint_subscales=True,
    use_spectral=True,
    workers=0,
)
quantizer = Q2KQuantizer(cfg)

weights = np.random.default_rng(0).normal(0, 0.02, size=(64, 256)).astype(np.float32)
result = quantizer.quantize(weights, row_width=256)
print("RMSE:", result.stats.rmse)
reconstructed = result.dequantize()

# 2. Convert a source GGUF + iMatrix into a quantized Q2_K GGUF
imat = read_imatrix("imatrix.gguf")
stats = write_quantized_gguf(
    "Raon.gguf",
    "Raon-Q2K.gguf",
    quantizer,
    imat=imat,
    quantize_all=True,
)
print(stats)

---

Package Structure

HPC-Quantize/
├── gui.py                         # Standalone Tkinter/TTK GUI with live telemetry dashboard
└── hpc_quantize_v2/               # Pure-Python V2 package (zero C dependencies)
    ├── __init__.py                # Public API exports
    ├── __main__.py                # CLI entry point (--gguf, --input, --demo, --bench, --workers)
    ├── config.py                  # Validated HPCConfig dataclass
    ├── fp16.py                    # Bit-exact GGML nearest_int & IEEE fp16 ULP primitives
    ├── q2k.py                     # Q2_K 84-byte block packing, unpacking & batched dequantization
    ├── analog.py                  # Geometry-preserving weight analogs (Iso-Moment, Vesica Gram, Subscale)
    ├── factored.py                # Coupled orthogonal-basis bilinear Q2_K block & vectorized candidate solver
    ├── seeds.py                   # Kurtosis-gated outlier clamping & batched GGML seed solvers
    ├── spectral.py                # Folded error pyramid, DC/Vesica penalty & Hadamard code descent
    ├── sieve.py                   # 6-state sequential Sieve with neighbor parity back-action
    ├── viterbi.py                 # Vectorized 36-state Viterbi sequence optimizer
    ├── pipeline.py                # End-to-end Q2KQuantizer with multi-core Phase 2b & algebraic Phase 4
    ├── gguf.py                    # GGUF reader, iMatrix loader & GGUF-to-GGUF streaming writer
    ├── reference.py               # Faithful C-reference solver port for A/B benchmarking

Running the Test Suite

python3 -m unittest discover -s hpc_quantize_v2/tests -v