Add PolarQuant quantization: Hadamard-rotated Lloyd-Max optimal weights + KV cache
Author: caiovicentino
Pull request: no

## 🚀 Feature request

### Motivation

PolarQuant is a quantization method that uses **Walsh-Hadamard rotation + Lloyd-Max optimal centroids** for both weight compression and KV cache compression. It achieves better PPL per bit than existing methods because:

1. **Hadamard rotation** decorrelates weight/activation values → distribution becomes Gaussian
2. **Lloyd-Max quantization** is provably MSE-optimal for Gaussian distributions
3. **No calibration data needed** — unlike GPTQ/AWQ, works on any model instantly

### Results

Tested on Qwen3.5-9B (WikiText-2 PPL, lower is better):

| Method | Bits | PPL | Δ vs FP16 |
|--------|------|-----|-----------|
| FP16 Baseline | 16 | 6.37 | — |
| **PolarQuant Q5 + INT4** | **~4** | **6.54** | **+0.17** |
| torchao INT4 (absmax) | 4 | 6.68 | +0.31 |
| BnB NF4 | 4 | ~6.7 | +0.33 |

PolarQuant beats torchao INT4 by **0.14 PPL** at the same effective bit-width — a significant gap.

### Proposal

Add `PolarQuantConfig` as a native quantization method in transformers, with two components:

#### 1. Weight Quantization (`PolarQuantConfig`)

```python
from transformers import AutoModelForCausalLM, PolarQuantConfig

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-31B-it",
    quantization_config=PolarQuantConfig(weight_bits=5, kv_bits=3),
    device_map="auto",
)
```

Pipeline: Load BF16 → Hadamard rotate → Lloyd-Max Q5 quantize → dequant → torchao INT4

#### 2. KV Cache Compression (`PolarQuantCache`)

```python
from transformers import PolarQuantCache

outputs = model.generate(
    input_ids,
    past_key_values=PolarQuantCache(num_layers=60, nbits=3),
    max_new_tokens=512,
)
```

This provides **5.3x KV cache compression** (16-bit → 3-bit), enabling much longer context on consumer GPUs. Currently, `QuantoQuantizedCache` is the only built-in quantized cache option.

### Why not use existing methods?

| Feature | PolarQuant | torchao | BnB | GPTQ | AWQ |
|---------|-----------|---------|-----|------|-----|
| Calibration-free | ✅ | ✅ | ✅ | ❌ | ❌ |
| KV cache compression | ✅ (Q2/Q3/Q4) | ❌ | ❌ | ❌ | ❌ |
| Hadamard rotation | ✅ | ❌ | ❌ | ❌ | ❌ |
| Optimal centroids | ✅ (Lloyd-Max) | ❌ | NF4 | ❌ | ❌ |
| Best PPL at 4-bit | ✅ | ❌ | ❌ | Close | Close |

### Implementation plan

1. `PolarQuantConfig` in `quantization_config.py`
2. `PolarQuantHfQuantizer` in `quantizers/quantizer_polarquant.py`
3. `PolarQuantCache` in `cache_utils.py` (extending existing `QuantizedCacheConfig`)
4. Tests + documentation

The implementation is self-contained (~500 lines) with scipy as the only additional dependency (for Lloyd-Max centroid computation, runs once at init).

### References

- **Paper**: [PolarQuant: Hadamard-Rotated Lloyd-Max Quantization for LLM Compression](https://arxiv.org/abs/2603.29078)
- **Code**: [github.com/caiovicentino/polarengine-vllm](https://github.com/caiovicentino/polarengine-vllm)
- **Models**: [HuggingFace Collection](https://huggingface.co/collections/caiovicentino1/polarquant-models-69cbc96292c5174df2088b08) (36 quantized models)

### Proven at scale

- Gemma 4 31B-it: 62.5 GB → 21.5 GB (fits RTX 4090), 24.9 tok/s
- Qwen3.5 9B: 17.9 GB → 6.5 GB, 43.1 tok/s, PPL 6.54
- Qwen3.5 27B: → 17.7 GB, PPL 5.37
- 36 models published on HuggingFace

Happy to implement if there's interest from maintainers!

Comments:
## Update: 49 models quantized, video models validated

Since opening this issue, we've significantly expanded PolarQuant validation:

### Scale
- **49 models** published on HuggingFace (was 36)
- Now includes **LLMs, MoE, video generation, image generation, and diffusion models**
- Models from Google, Qwen, NVIDIA, Tencent, Netflix, Lightricks, H Company

### New results

**Video models (first quantization method to compress video generators):**

| Model | Original | PQ5 | Reduction | cos_sim |
|-------|----------|-----|-----------|---------|
| HY-OmniWeaving (Tencent) | 79 GB | 28 GB | -65% | 0.9986 |
| LTX-2.3 (22B, Lightricks) | 46 GB | 15 GB | -68% | 0.9986 |
| VOID (Netflix) | 42 GB | 13 GB | -69% | 0.9986 |
| CogVideoX-5b-I2V | 21.6 GB | 8.7 GB | -60% | 0.9986 |

Video generation validated end-to-end: [demo video](https://huggingface.co/caiovicentino1/HY-OmniWeaving-PolarQuant-Q5)

**MoE models:**

| Model | Experts | Original | PQ5 Codes |
|-------|---------|----------|-----------|
| Holo3-35B-A3B | 256 experts, 8 active | 70 GB | 21.9 GB |
| GLM-4.7-Flash | 64 experts, 4 active | 61 GB | 19 GB |
| Gemma-4-26B-A4B | 128 experts | 25.2 GB | — |

**Benchmark (PQ5 beats BF16):**

| Benchmark | BF16 | PQ5+INT4 | Delta |
|-----------|------|----------|-------|
| HellaSwag | 64.5% | **67.0%** | **+2.5%** |
| Winogrande | 72.5% | **73.0%** | **+0.5%** |
| VRAM | 56.4 GB | 18.7 GB | -67% |

*(Qwopus3.5-27B, 0-shot, n=200)*

### Implementation readiness

The `PolarQuantConfig` + `PolarQuantCache` implementation is ready (~500 lines). We'd be happy to submit a PR if there's interest from maintainers.

Key advantage over existing quantizers: **no calibration data needed** + **KV cache compression** (currently only `QuantoQuantizedCache` exists in transformers).

cc @SunMarc @TheBloke @sayakpaul — would love feedback on whether this fits the transformers quantization roadmap.

All models: https://huggingface.co/collections/caiovicentino1/polarquant-models-69cbc96292c5174df2088b08

We use Hadamard rotation too but followed it with E8 lattice VQ instead of Lloyd-Max scalar quantization. One thing we found that might be relevant:

Phi-3-mini has head_dim=96 (not power of 2). Hadamard needs power-of-2 dimensions, so we zero-pad to 128 before the transform and slice back after. Works fine, no quality penalty from the padding. Might be worth handling in PolarQuant too if you hit non-standard models.

Also, if you test on Qwen models, heads up that they need the first/last 2 layers left uncompressed or quality collapses. We saw this on Qwen2.5-7B with both our E8 approach and basic Hadamard+quant. Seems to be a Qwen architecture thing, not quantizer-specific.

Our code for reference: https://github.com/jagmarques/nexusquant

cc @SunMarc 

Hey @caiovicentino, I'm not sure if this is worth adding for model weights unless it is supported by popular serving tools like vllm or there is enough interest from the community.  However, for KV Cache, I think it can be worth adding it but from this blogpost maybe it will be even better consider adding turboquant as it also relies on polarquant ? https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/

Thanks @SunMarc and @jagmarques — both comments are actionable. Let me address each.

## Re @SunMarc — KV cache first, weights conditional

Agreed this is the right framing. Let me clarify where PolarQuant stands on both axes:

**vLLM adoption (weights):**
- Our PR [vllm-project/vllm#37190](https://github.com/vllm-project/vllm/pull/37190) is open and under active review. Latest validated numbers: 37.4 tok/s on RTX PRO 6000 for a 35B MoE in INT4 with expert offloading (1.58x the all-in-GPU baseline of 23.6 tok/s)
- Several models already serve natively via standard GPTQ / CompressedTensors formats (zero plugin):
  - `Qwopus3.5-9B-v3-PolarQuant-v7-GPTQ` — 67.07% HumanEval (narrowly edges out 66.87% BF16 on this specific benchmark — at 9B scale and standard mode only; the 27B thinking-mode story is more complicated and the BF16 baseline wins there)
  - `Qwopus-MoE-35B-A3B-PolarQuant-Q5` — PPL 6.56 on full WikiText-2 (295K tokens), low-PPL regime
- The honest take: for serving, PolarQuant is primarily a **distribution format** that decodes into standard INT4/GPTQ. The runtime story is already covered by existing transformers quantizers. Which is why we agree weight quantization doesn't have a clear case here.

**TurboQuant connection — you're right to flag it:**
Google's TurboQuant uses random rotations + uniform quantization, which is conceptually adjacent to our Hadamard rotation + Lloyd-Max. The shared insight is that **rotation before quantization kills outliers cheaply**. They extend it with dimension-adaptive rotations which we haven't explored — potentially foldable into a unified "rotation-based cache quantization" path.

**Proposed scope for a KV cache PR:**
1. `PolarQuantizedCache` class implementing the `Cache` interface
2. 5-bit codes for K and V with per-head-group scales
3. Hadamard rotation on `head_dim` before quantization (shared utility)
4. Test suite: PPL correctness + memory reduction + latency benchmarks
5. Docs + example in `docs/source/en/kv_cache.md`
6. Benchmarks on Llama-3.1-8B and Qwen3-7B comparing with `QuantoQuantizedCache`

If that lands cleanly and shows clear wins vs Quanto, weight quantization can follow later based on ecosystem pull. Does this scope work for you? Happy to start the PR once we have your ack on direction.

## Re @jagmarques — Phi-3 padding + Qwen layer quirks

Both points are extremely useful, thank you.

**Phi-3-mini head_dim=96:**
Yes — we handle this by zero-padding to 128 before the transform and slicing back after. Confirmed no quality penalty in our testing either. I'll document this clearly in the `PolarQuantizedCache` implementation so non-power-of-2 head dims "just work".

**Qwen first/last 2 layers uncompressed:**
This aligns with what another tester ([Arien0](https://huggingface.co/caiovicentino1/Qwopus3.5-9B-v3-PolarQuant-Q5/discussions/3)) independently reported today on Qwen3.5 weight quantization. Their broader list of precision-sensitive layers:
- `linear_attn.*`
- `mtp.*` (MTP head)
- `lm_head`
- `embed_tokens`
- `norm.*`
- First/last 2 decoder layers (your observation)

Would you mind if I tag you on the KV cache PR when it's up? You've clearly tested more edge cases than I have — a cross-check against the E8 lattice VQ approach would be valuable.

Also curious: how did E8 lattice VQ compare to scalar Lloyd-Max at equivalent bits on Qwen2.5-7B? Happy to compare notes if you're open to it.


Yeah tag me on the PR, happy to cross-check. On E8 vs Lloyd-Max on Qwen: we only have 1.5B data so far (2 KV heads, needs boundary=1 or it blows up). Haven't run 7B yet. The Arien0 layer list is useful, we found the same first/last 2 layers thing independently on Qwen2.5-1.5B. Will share numbers when we have them on 7B.

@jagmarques thanks — yes, absolutely will tag you on the PR when it's up. But thinking about this more, reviewing a finished PR isn't the right shape of engagement here when you already have complementary infrastructure running. Let me propose something better than just tagging, on top of the PR review.

Independent confirmation of the first/last 2 layers thing across two testers is genuinely useful — you on Qwen2.5-1.5B, Arien0 on Qwen3.5-9B. Two independent observations on different Qwen sizes converging on the same layer carve-out is a strong signal. We'll bake that into the default config for the `PolarQuantizedCache` implementation.

## Technical question on your E8 setup

When you say "2 KV heads, needs boundary=1 or it blows up" — is that the lattice decoder boundary parameter (how far to search from the query point during codebook lookup)? And is the failure mode runtime divergence, quality collapse, or numerical overflow on the small-head case?

We use scalar Lloyd-Max in PolarQuant specifically to sidestep the codebook-search complexity of lattice VQ, but we haven't rigorously compared at the same bitrate.

Also: does your E8 setup also skip `lm_head`, `embed_tokens`, and the norm layers (per Arien0's broader list), or does lattice VQ handle those fine without the carve-out?

## Proposal — align before the PR lands, not just at review time

You'll still get tagged on the PR when it opens (as you asked), but there's real value in aligning earlier since you already have the parallel setup. Concrete ideas, pick whichever works:

1. **Shared scope doc / gist.** I can start one with the 6-point scope from my earlier reply to @SunMarc, and you can redline what's missing or wrong from the E8 / lattice VQ angle. This locks the PR boundary before code lands.

2. **Your E8 reference, if sharable.** A paper, repo, or even just a sketch of the head_dim padding + lattice decoder. Not to copy — to make the scalar vs lattice comparison clean when numbers land.

3. **Your 1.5B numbers are already load-bearing.** If E8 meaningfully beats scalar Lloyd-Max at 5-bit on the same head_dim, the cache implementation probably needs to support both. If they're roughly equivalent, scalar wins on throughput and we simplify. I'd rather see your 1.5B results now and align on harness than discover a meaningful gap after the PR is written.

4. **Harness convergence.** Are you using `lm_eval` for PPL + downstream, or something custom? Our weight-quant benchmarks have been WikiText-2 PPL + HumanEval, but KV cache eval is a different beast. Happy to converge on one protocol so our numbers are directly comparable when your 7B run completes.

We can keep the technical back-and-forth on this issue thread if you prefer visibility, or move to a gist / private channel if it gets noisy. Your call.

Our current runs have been on Qwen3.5 series (9B, 27B, MoE). We haven't tested Qwen2.5-1.5B or Qwen2.5-7B yet, so your 7B results would be a genuine apples-to-apples comparison against PolarQuant scalar Lloyd-Max if we coordinate on the harness first.

boundary=1 means the first and last decoder layers stay at full precision (fp16 KV, no quantization). On Qwen2.5-1.5B with 2 KV heads, compressing those layers produces PPL in the millions. With boundary=1, E8 3-bit quant-only drops to +0.22%. Pure quality collapse, not numerical overflow.

We don't touch lm_head, embed_tokens, or norms since our pipeline only compresses the KV cache, not weights. The broader layer list from Arien0 is weight-quant specific.

Harness: PPL on multi-topic passages plus needle-in-a-haystack for factual recall. Our repo has the pipeline code, the E8 implementation is in nexusquant/core/e8_lattice.py.

At 3-bit, E8 lattice and scalar quant are both near-lossless on Gemma-2-2b. The gap shows at 2-bit where E8 gives +0.15% vs uniform scalar at +5.33%. Whether Lloyd-Max closes that gap at 2-bit is the comparison we haven't run yet.

Happy to have a PR for `PolarQuantizedCache` @caiovicentino ! 

Saw @caiovicentino closed the first PR at #45364 after flagging the TurboQuant overlap. Left a longer reply there with our multi-needle numbers on Qwen2.5-7B and Gemma-2-2b: tldr is single-needle tests at position 0.25 happened to favor different quantizers on different models, but at 0.5 and 0.75 the story changes. On the 8+ KV head models (Mistral-7B, Llama-3-8B, Qwen3-8B) all quantizers we tested pass NIAH at every position. Will run whatever the next PR shape looks like on our harness.

> unless it is supported by popular serving tools like vllm or there is enough interest from the community 

Incredibly excited about @caiovicentino's models - they are incredibly tiny which unlocks very real & very practical use-cases. 
My modest hardware was able to cram STT/TSS models, as well as a VT model in the freed space... It's ridiculously amazing, and unlocked the multi-modal capabilities I was seeking without having to fork another $15,000 in GPUs. Scale that up to a datacenter-scale, and there are literal billions to be saved.

The lack of calibration requirement is another major improvement.
PolarQuant / HLWQ truly can't be ignored. 

Thank you for your work @caiovicentino. 
I am following your models closely, I will swap my Voxtral by your HLWQ version today.

Thank you both — @Quentin-M for the deployment story (the $15k GPU displacement is exactly what I hoped this work would unlock for people on modest hardware), and @jagmarques for being willing to run future work on your NIAH harness.

Quick context for the rest of the thread, since the naming changed mid-stream: the WEIGHT-quantization track has been renamed **HLWQ** (Hadamard-Lloyd Weight Quantization) because I discovered "PolarQuant" was already published as a KV-cache method by Han et al. at Google ([arXiv:2502.02617](https://arxiv.org/abs/2502.02617), Feb 2025). The two methods are technically distinct (HLWQ is weight quantization with deterministic Walsh-Hadamard rotation; Han et al. is KV cache with random polar rotation), but the name collision was real and worth fixing. I closed the KV-cache PR (#45364) and the longer technical reply to @jagmarques's multi-needle data lives there.

@Quentin-M — one calibration honesty note: the lighter HLWQ variants are calibration-free, but the strongest results (v7 on Qwopus3.5-9B-v3, 67.07% HumanEval — matches BF16 at 66.87%, the +0.20pp edge is within the benchmark noise floor on 164 samples) use GPTQ calibration with gs64+FOEM. "No calibration required" is true for the lighter configurations and a tradeoff for the strongest ones.

Thanks again to both of you.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Quick follow-up on rotation comparisons since you're using Hadamard too. Tested NQ pipeline with Iso (4D quaternion) and Planar (2D Givens) drop-ins from RotorQuant on Mistral-7B K3V2: Hadamard +0.27%, Iso +1.06%, Planar +1.29% wikitext-2 PPL. Hadamard wins by ~0.8pp at iso bit-budget, which I didn't expect going in. Curious if PolarQuant sees the same when you swap rotations.

Thanks for the rotation ablation — that's a useful empirical confirmation. Worth flagging that Hadamard rotation in this stack comes from Han et al. ([arXiv:2502.02617](https://arxiv.org/abs/2502.02617), Feb 2025), the original PolarQuant paper — this issue tracks an HF-native implementation, not a new technique on the KV cache side. The weight trilha (Hadamard-Lloyd Weight Quantization, HLWQ, [arXiv:2603.29078](https://arxiv.org/abs/2603.29078)) is separate and uses Hadamard for a different reason (decorrelating outliers before scalar quant).

Your finding (Hadamard +0.27% < Iso +1.06% < Planar +1.29% on Mistral-7B K3V2) lines up with theory: Hadamard's bounded coherence (μ²=1/d) gives the tightest worst-case quantization error guarantee. Iso has no such guarantee; Planar covers only 2D. i haven't run this ablation — would be a natural addition. RotorQuant link?

https://github.com/scrya-com/rotorquant

Han et al.'s PolarQuant (arXiv:2502.02617) uses random polar transformation per its abstract, not Hadamard.

Correcting my prior comment — I had the attribution backwards.

Han et al. PolarQuant ([arXiv:2502.02617](https://arxiv.org/abs/2502.02617)) uses **random preconditioning + polar transformation**, not Hadamard. Hadamard rotation for KV cache traces to **TurboQuant** ([Zandieh et al., arXiv:2504.19874](https://arxiv.org/abs/2504.19874)), the follow-up that builds on PolarQuant. Apologies for the confusion.

Re the numbers: your Mistral-7B result (Hadamard +0.27% < Iso +1.06% < Planar +1.29%) is the inverse of the RotorQuant README for Llama 3.1 8B (Iso 6.91 PPL beats Hadamard/Turbo 7.07 PPL). If both replicate, the rotation hierarchy is architecture-dependent — clean ablation paper on its own.

I'll update the issue body to (a) cite Han et al. and TurboQuant as prior art, (b) rename the weight proposal to HLWQ to disambiguate from the PolarQuant name owned by Han et al.

Could be configuration, not architecture. RotorQuant's iso/turbo/planar are on Q4_K_M weights through llama-perplexity on Llama 3.1 8B Instruct; ours are real FP16 weights through HF prefill+continue on Mistral-7B-v0.1. Iso-config (same weights, same harness, both models) is the cleaner test before calling it architecture-dependent.

Fair — confounded comparison (Q4_K_M + llama-perplexity vs FP16 + HF prefill). Can't separate harness/quantization/architecture from the numbers we have. Iso-config sweep is the cleaner test.

Ran the iso-config sweep. Mistral-7B-Inst-v0.3 wikitext (n=60 chunks, 1024-prefix, same Hadamard + per-token amax on both sides, only the post-rotation quantizer differs): K3V2 E8 +0.28%, K3V2 uniform +0.62%. K2V2 E8 +1.14%, K2V2 uniform +2.08%. Replicated on Llama-3.1-Inst, same direction with bigger gaps (K3 0.78pp, K2 2.92pp at the same protocol). Paper PDF at github.com/jagmarques/nexusquant has the numbers in context.


Following up on the iso-config thread. Ran the AQUA-KV protocol (arxiv:2501.19392 Tab. 2, wikitext-2, 6 sequences of 8192 tokens, 1024-prefix) on Llama-3.1-8B-base. NQ K3V2 pb=0 lands at 2.625 bpe with +0.487% PPL vs FP16, against AQUA-KV reported 3.06 bpe at +0.53%, no calibration step on our side. Paper PDF at github.com/jagmarques/nexusquant.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.