Repository: ggml-org/llama.cpp
Pull request: [ggml-quants] Add memsets and other fixes for IQ quants

While trying to stop my Qwen3.5 quants from getting a ton of "Oops: found point X not on grid ...", I (and claude) came across a potential big issue

Using gdb, it seems that `L` is often initialized to non-zero memory, and so when it's read, it has garbage data in it that's causing the quantizations to go awry when there's no candidates found during the search

With this change, with Qwen3.5, I no longer saw ANY "Oops: found point.." errors, and the PPL seems totally as expected

This affected `IQ2_XXS`, `IQ2_XS`, `IQ3_XXS`, and `IQ3_S`. It seems that it's most often caused when the imatrix data contains a full block of 0s (as is common in MoE models with a large number of experts) or when the block itself is all 0s (for a particularly sparse model)

Now, `memset` is called on `L` before it is used.

Additional changes:

~~`L` -> `Laux`: basically a no-op, we only use the returned scale so we can use `L` or `Laux` here, since `L` is immediately memset right after this change makes it more obvious that this is a buffer, but I don't care if we leave this line out~~

`eff_max <= 0` check: guards against dividing by 0 when `scale` is 0, we set the scales to 0 and exit early like when `max < GROUP_MAX_EPS` is triggered

PPL tested on **Qwen3-Code-Next**:

PPL `IQ2_XS` **before**:

```
Final estimate: PPL = 9.3646 +/- 0.16983
```

PPL `IQ2_XS` **after**:

```
Final estimate: PPL = 9.3703 +/- 0.17014
```

Difference: +0.0057

`IQ2_XXS` is now possible to make (with my imatrix at least) whereas before I got:

```
Oops: found point 16384 not on grid: 0 0 0 0 0 0 0 1
[   9/ 843]           blk.0.ffn_gate_exps.weight - [ 2048,   512,   512,     1], type =   bf16, converting to iq2_xs .. /llama.cpp/ggml/src/ggml-quants.c:3355: fatal error
libggml-base.so.0(+0x1848b)[0x70bdefd5648b]
libggml-base.so.0(ggml_print_backtrace+0x21f)[0x70bdefd568ef]
libggml-base.so.0(ggml_abort+0x152)[0x70bdefd56ac2]
libggml-base.so.0(+0x46b8c)[0x70bdefd84b8c]
libggml-base.so.0(quantize_iq2_xs+0x81)[0x70bdefda3d21]
```

For sanity, PPL `IQ2_XXS`:

```
Final estimate: PPL = 10.4755 +/- 0.19246
```

-----------
For IQ1_S and IQ1_M:

Similar to above, the assert happens when we have all-zero weights meaning the search loop failed to find a candidate. Instead now we treat is as a zero block, and we populate the scales and shifts to avoid uninitialized memory

This was the assert I was seeing:

```
blk.7.ffn_down_exps.weight - [ 1024,  4096,   512,     1], type =   bf16, converting to iq1_m .. /llama.cpp/ggml/src/ggml-quants.c:4534: GGML_ASSERT(besti1 >= 0 && besti2 >= 0 && best_k >= 0) failed                                                                                          
  /llama.cpp/ggml/src/ggml-quants.c:4534: GGML_ASSERT(besti1 >= 0 && besti2 >= 0 && best_k >= 0) failed
```

PPL `IQ1_S` **before**: 

```
Final estimate: PPL = 15.6349 +/- 0.31023
```

PPL `IQ1_S` **after**:

```
Final estimate: PPL = 15.6509 +/- 0.31119
```

Difference: +0.016

ALL PPL calculations done with:

```
./llama-perplexity -m /models/Qwen3-Coder-Next.gguf --flash-attn on -f /training_data/wiki.test.raw --chunks 100 --batch-size 4096 --ubatch-size 4096
```

I used `Qwen3-Coder-Next` because I had existing "before" quants to compare against, and IQ2_XXS previously failed with the "Oops" crash.

**Note**, I think the PPL change is from changes to the model conversion, I ran PPL against the full bf16 model

Conversion done with old release [b7936](https://github.com/ggml-org/llama.cpp/releases/tag/b7936):

```
Final estimate: PPL = 8.0452 +/- 0.15022
```

Conversion from master:

```
Final estimate: PPL = 8.0508 +/- 0.15029
```

Difference: +0.0056

(in hindsight, wikitext for a coding model wasn't the best dataset... willing to redo the PPLs against a different dataset if necessary)

**Disclaimer**:

I don't fully understand this area of code yet, so while the `memset` stuff is obvious and had immediate tangible improvements, the rest is outside my wheelhouse. I really would appreciate another set of experienced eyes looking at this, I provide my test results in an attempt to prove nothing has been broken by this, and have reviewed this to the best of my ability

If any additional tests should be run before people are willing to review I'll more than happily run them

Changed files:
filename: ggml/src/ggml-quants.c
patch: @@ -3104,6 +3104,11 @@ static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML
             }
             float scale = make_qp_quants(32, kMaxQ+1, xval, (uint8_t*)L, weight);
             float eff_max = scale*kMaxQ;
+            if (eff_max <= 0) {
+                scales[ib] = 0;
+                memset(L, 0, 32);
+                continue;
+            }
             float best = 0;
             for (int is = -6; is <= 6; ++is) {
                 float id = (2*kMaxQ-1+is*0.1f)/eff_max;
@@ -3273,9 +3278,9 @@ static void quantize_row_iq2_xs_impl(const float * GGML_RESTRICT x, void * GGML_
             }
             float max = xval[0];
             for (int i = 1; i < 16; ++i) max = MAX(max, xval[i]);
+            memset(L, 0, 16);
             if (max < GROUP_MAX_EPS) {
                 scales[ib] = 0;
-                memset(L, 0, 16);
                 continue;
             }
             float best = 0;
@@ -3714,9 +3719,9 @@ static void quantize_row_iq3_xxs_impl(int grid_size, const float * GGML_RESTRICT
             }
             float max = xval[0];
             for (int i = 1; i < 32; ++i) max = MAX(max, xval[i]);
+            memset(L, 0, 32);
             if (max < GROUP_MAX_EPS_IQ3_XXS) {
                 scales[ib] = 0;
-                memset(L, 0, 32);
                 continue;
             }
             float best = 0;
@@ -3922,6 +3927,7 @@ static void quantize_row_iq3_s_impl(int block_size, const float * GGML_RESTRICT
             }
             float max = xval[0];
             for (int i = 1; i < block_size; ++i) max = MAX(max, xval[i]);
+            memset(L, 0, block_size);
             if (!max) {
                 scales[ib] = 0;
                 continue;
@@ -4245,6 +4251,7 @@ static void quantize_row_iq1_s_impl(const float * GGML_RESTRICT x, void * GGML_R
             for (int i = 1; i < block_size; ++i) max = MAX(max, fabsf(xb[i]));
             if (max < GROUP_MAX_EPS_IQ1_S) {
                 scales[ib] = 0;
+                shifts[ib] = 1;
                 memset(L, 1, block_size);
                 continue;
             }
@@ -4285,7 +4292,12 @@ static void quantize_row_iq1_s_impl(const float * GGML_RESTRICT x, void * GGML_R
                     }
                 }
             }
-            GGML_ASSERT(besti1 >= 0 && besti2 >= 0 && best_shift != 0);
+            if (besti1 < 0 || besti2 < 0 || best_shift == 0) {
+                scales[ib] = 0;
+                shifts[ib] = 1;
+                memset(L, 1, block_size);
+                continue;
+            }
             for (int j =      0; j < besti1; ++j) L[idx[2*j]] = 0;
             for (int j = besti1; j < besti2; ++j) L[idx[2*j]] = 1;
             for (int j = besti2; j < block_size; ++j) L[idx[2*j]] = 2;
@@ -4429,6 +4441,7 @@ static void quantize_row_iq1_m_impl(const float * GGML_RESTRICT x, void * GGML_R
             for (int i = 1; i < block_size; ++i) max = MAX(max, fabsf(xb[i]));
             if (max < GROUP_MAX_EPS_IQ1_M) {
                 scales[ib] = 0;
+                shifts[ib] = 0;
                 memset(L, 1, block_size);
                 continue;
             }
@@ -4527,7 +4540,12 @@ static void quantize_row_iq1_m_impl(const float * GGML_RESTRICT x, void * GGML_R
                     }
                 }
             }
-            GGML_ASSERT(besti1 >= 0 && besti2 >= 0 && best_k >= 0);
+            if (besti1 < 0 || besti2 < 0 || best_k < 0) {
+                scales[ib] = 0;
+                shifts[ib] = 0;
+                memset(L, 1, block_size);
+                continue;
+            }
             for (int j =      0; j < besti1; ++j) L[idx[2*j]] = 0;
             for (int j = besti1; j < besti2; ++j) L[idx[2*j]] = 1;
             for (int j = besti2; j < block_size; ++j) L[idx[2*j]] = 2;
@@ -4874,6 +4892,7 @@ static void quantize_row_iq2_s_impl(const float * GGML_RESTRICT x, void * GGML_R
             }
             float max = xval[0];
             for (int i = 1; i < 16; ++i) max = MAX(max, xval[i]);
+            memset(L, 0, 16);
             if (max < GROUP_MAX_EPS_IQ2_S) {
                 scales[ib] = 0;
                 continue;