Soulfate24

/

AutoRound-ASHQ1-Remix_Double-Quantization_Suite

like 1

And One technical question: for the higher tiers
(Quality, possibly Compact), did you keep
allow_q3_or_lower disabled? I found that flag
is only beneficial on tight budgets like Nano/Mini,
but on larger budgets it can hurt perplexity since
the optimizer doesn't need Q3 to fit the target ratio.

Also wanted to expand on the allow_q3_or_lower point -
this affects even the low tiers, not just Quality/Compact.

The issue is the algorithm is just too greedy. Once that
flag is on, it will drop tensors all the way down to IQ2_XXS
if it decides they're low priority, when something like Q4_K
would have fit the budget just fine and preserved way more
signal. Anything below Q4 tends to hurt perplexity more than
it helps, even though it technically saves bytes.

The tiers still come out ahead of uniform IQ2 quants overall
since the important tensors stay protected, but there's
noticeable perplexity left on the table from being too
aggressive on the "unimportant" ones. A less greedy allocation
that stops around Q3/Q4 as a floor instead of IQ2 would
probably close that gap.

Hey wepiqx,

Thank you so much for this, coming from you it means a lot. ASHQ1 is literally the foundation everything here builds on; your priority-queue knapsack formulation and tied-group activation hashing are what make this whole suite possible, and they're credited as such in the README. The AutoRound pre-stage was indeed the piece that unlocked things your original couldn't do from raw BF16, but without your core there'd be nothing to integrate into.

On your technical question, great instincts, and I now have measurements to answer it precisely:

1. allow_q3_or_lower stayed disabled everywhere in my standard tier runs. It remains opt-in in the CLI, but nothing ships through it.
2. Your "too greedy" diagnosis turned out to be exactly right, and measurable. I built a knockout-probe harness (90_attribution-probe.py) to marginalize each decision, and forced IQ3_S on gate/up measured +0.022 KLD (~2×10⁻⁴ damage per MiB saved), an order of magnitude worse per byte than any winning promotion I found. So v2.0.0 went further than a Q3/Q4 stopping rule: MLP classes (ffn_gate/up/down) now carry a hard IQ4_XS declarative floor at plain budgets. Your suggested fix, but stopped one notch earlier because the data said so.
3. Scaling to other families also surfaced two real bugs in the classifier path worth knowing about, untied lm_heads expose an output.weight the exact-name matcher shelved as unknown, and on one hybrid the recurrent readout (ssm_out) needed a dedicated precision floor below which output entropy collapses. Both are fixed and documented (CHARTER.md, laws L0–L5).

On the stock-quants comparison: absolutely yes, and it's nearly free with the tooling I already have, same imatrix, uniform llama-quantize pass per type, then straight into the existing perplexity/KLD sweep. Size-matched against Nano→Fidelity it should show exactly where the selective allocation pays. I'll post the full table once the runs finish.

And yes, very interested in collaborating. Ping me when you're back at your machine; I'd love to get your eyes on the calibration ledger and see where you'd want to take it next.

.>python 03_perplexity_test.py
Binary : .\llama-cpp\llama-perplexity.exe
GPU : cuda · 7930/8188 MiB VRAM available
[Corpus] Using existing .\wiki.test.raw

[KL-Base] Generating reference logits over the full corpus (ngl=7, ~50 min)…
0.04.519.350 I cmn init: llama threadpool init, n_threads = 15
0.04.519.930 I 
0.04.519.968 I system_info: n_threads = 15 (n_threads_batch = 15) / 16 | CUDA : ARCHS = 750,800,860,890,900,1200,1210 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.04.521.028 I perplexity: saving all logits to .\kld-bf16.dat
0.04.521.033 I perplexity: tokenizing the input ..
0.04.864.087 I perplexity: tokenization took 343.048 ms
0.04.864.439 I perplexity: calculating perplexity over 580 chunks, n_ctx=512, batch_size=512, n_seq=1
0.09.981.668 I perplexity: 5.10 seconds per pass - ETA 49.25 minutes
[1]5.7284,…,[580]9.5298,
25.24.692.262 I Final estimate: PPL = 9.5298 +/- 0.06948

 ✓ Base compressed with Xpress8K
 ✓ Base ready: kld-bf16.dat

Available models:
 1. model-no-mtp-BF16.gguf (17091 MiB · ngl=7)
 2. model-ASHQ1-Fidelity-48pc.gguf (9243 MiB · ngl=22)
 3. _stock-q8_0.gguf (9086 MiB · ngl=22)
 4. _stock-q6_K.gguf (7018 MiB · ngl=32)
 5. model-ASHQ1-Quality-39pc.gguf (6675 MiB · ngl=32)
 6. _stock-q5_k_m.gguf (6168 MiB · ngl=32)
 7. model-ASHQ1-Compact-33pc.gguf (5650 MiB · ngl=32)
 8. _stock-q4_k_m.gguf (5368 MiB · ngl=32)
 9. model-ASHQ1-Mini-30pc.gguf (5225 MiB · ngl=32)
 10. _stock-q4_k_s.gguf (5104 MiB · ngl=32)
 11. model-ASHQ1-Nano-27pc.gguf (4625 MiB · ngl=32)
 12. _stock-q3_k_m.gguf (4409 MiB · ngl=32)
 13. _stock-q3_k_s.gguf (4062 MiB · ngl=32)
 a. ALL · q. quit
Select model(s) or file path [1/a/model.gguf/q]: a
 layout: 32 layers · 232 MiB/layer · non-layer 1826 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── model-ASHQ1-Fidelity-48pc.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.5381 · KLD=0.0084 · RMS Δp=2.46% · top-p=97.4% · 345.9 tok/s · 11.2 min
 layout: 32 layers · 220 MiB/layer · non-layer 2061 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q8_0.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.5448 · KLD=0.0075 · RMS Δp=2.45% · top-p=97.7% · 365.7 tok/s · 11.3 min
 layout: 32 layers · 170 MiB/layer · non-layer 1591 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q6_K.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.4530 · KLD=0.0145 · RMS Δp=3.23% · top-p=96.2% · 731.4 tok/s · 6.7 min
 layout: 32 layers · 159 MiB/layer · non-layer 1591 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── model-ASHQ1-Quality-39pc.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.2284 · KLD=0.0303 · RMS Δp=4.55% · top-p=94.1% · 775.8 tok/s · 6.3 min
 layout: 32 layers · 147 MiB/layer · non-layer 1463 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q5_k_m.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.0280 · KLD=0.0755 · RMS Δp=6.76% · top-p=90.5% · 1113.0 tok/s · 5.2 min
 layout: 32 layers · 127 MiB/layer · non-layer 1591 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── model-ASHQ1-Compact-33pc.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.6526 · KLD=0.0505 · RMS Δp=5.83% · top-p=91.7% · 1066.7 tok/s · 4.9 min
 layout: 32 layers · 126 MiB/layer · non-layer 1341 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q4_k_m.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.1993 · KLD=0.0896 · RMS Δp=7.55% · top-p=89.1% · 1219.0 tok/s · 4.8 min
 layout: 32 layers · 114 MiB/layer · non-layer 1591 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── model-ASHQ1-Mini-30pc.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.8564 · KLD=0.0649 · RMS Δp=6.67% · top-p=90.3% · 1219.0 tok/s · 4.9 min
 layout: 32 layers · 118 MiB/layer · non-layer 1341 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q4_k_s.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=9.2654 · KLD=0.0927 · RMS Δp=7.68% · top-p=88.9% · 1383.8 tok/s · 4.7 min
 layout: 32 layers · 111 MiB/layer · non-layer 1061 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── model-ASHQ1-Nano-27pc.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=10.0184 · KLD=0.0856 · RMS Δp=7.70% · top-p=88.1% · 1312.8 tok/s · 4.8 min
 layout: 32 layers · 100 MiB/layer · non-layer 1213 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q3_k_m.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=10.2358 · KLD=0.1543 · RMS Δp=10.30% · top-p=84.5% · 1219.0 tok/s · 4.7 min
 layout: 32 layers · 89 MiB/layer · non-layer 1213 MiB · KV 4×256
 probe: native auto-fit … ok → llama.cpp sizes the split itself

── _stock-q3_k_s.gguf ──
 auto-fit · ctx=512 · batch=512 · threads=15 · Flash-Attention · KL-divergence
 PPL=10.4404 · KLD=0.2372 · RMS Δp=12.58% · top-p=80.3% · 1280.0 tok/s · 5.5 min

========================================================================================
 EVALUATION SUMMARY (KLD/RMS = fidelity to BF16 reference; PPL = entropy indicator)
========================================================================================
 Model PPL +ΔPPL KLD RMS Δp top-p Speed
 -------------------------------------------------------------------------------------------
 model-ASHQ1-Fidelity-48pc.gguf 9.5381 +0.0000 0.0084 2.46% 97.4% 345.9 t/s
 _stock-q8_0.gguf 9.5448 +0.0067 0.0075 2.45% 97.7% 365.7 t/s
 _stock-q6_K.gguf 9.4530 -0.0851 0.0145 3.23% 96.2% 731.4 t/s
 model-ASHQ1-Quality-39pc.gguf 9.2284 -0.3098 0.0303 4.55% 94.1% 775.8 t/s
 _stock-q5_k_m.gguf 9.0280 -0.5102 0.0755 6.76% 90.5% 1113.0 t/s
 model-ASHQ1-Compact-33pc.gguf 9.6526 +0.1144 0.0505 5.83% 91.7% 1066.7 t/s
 _stock-q4_k_m.gguf 9.1993 -0.3388 0.0896 7.55% 89.1% 1219.0 t/s
 model-ASHQ1-Mini-30pc.gguf 9.8564 +0.3183 0.0649 6.67% 90.3% 1219.0 t/s
 _stock-q4_k_s.gguf 9.2654 -0.2727 0.0927 7.68% 88.9% 1383.8 t/s
 model-ASHQ1-Nano-27pc.gguf 10.0184 +0.4802 0.0856 7.70% 88.1% 1312.8 t/s
 _stock-q3_k_m.gguf 10.2358 +0.6977 0.1543 10.30% 84.5% 1219.0 t/s
 _stock-q3_k_s.gguf 10.4404 +0.9023 0.2372 12.58% 80.3% 1280.0 t/s

 Δ PPL relative to model-ASHQ1-Fidelity-48pc.gguf · corpus: wiki.test.raw
 Reading grid: KLD < 0.02 quasi-lossless · 0.02-0.10 solid · 0.10-0.20 usable · >0.20 degraded.
 PPL below the BF16 base = entropy collapse (over-confidence), not a quality gain.
 wiki.test.raw corpus: absolute PPL comparable to llama.cpp published benchmarks.

Update: your stock-quant request turned into the most valuable experiment of the release. Full size-matched sweep on Ornith-9B: ASHQ1 wins decisively at 27–33% (Nano vs Q3_K_M: KLD −45%, top-p +3.6; Mini vs Q4_K_S: −30%; Compact vs Q4_K_M: −44%), but the edge decays with ratio: Quality at 39% is only ~22% ahead of size-interpolated stock, and Fidelity ties/loses to straight Q8_0. That's a clean L-law now: allocation alpha ∝ distance-from-uniform-budget. It reshaped our defaults (batch = Mini/Compact/Nano; higher tiers opt-in) and raises a fun open question you might enjoy: even stock quants show entropy collapse on this distillate, worth investigating together whether it's the distillation (fine-tune) or the arch.