Repository: ggml-org/llama.cpp
Pull request: mmq.cu: tune mmq/rocblas switching for RDNA

Continuing from #18442 I applied similar benchmarking as #14949 and #18202 to try and minimize bad cases on RDNA while keeping the logic simple.

TL;DR
---

Over an average of all models on https://huggingface.co/Beinsezii/mmq_test over a variety of µbatch sizes I have
| mmq% | blas% | tuned% |
| --- | --- | --- |
| 95.9 | 85.7 | 98.8|

Where 100% is a theoretical maximum if it were to optimally choose mmq or rocblas in each case.

Current master is functionally equivalent to `mmq` for this and all other benchmarks.

Benchmarks
---

<details><summary><b>compare-llama-bench affected quants</b></summary>

| GPU         | Model                         |   Microbatch size | Test   |   t/s b7600 |   t/s beinsezii/rocm_mmq_tune |   Speedup |
|:------------|:------------------------------|------------------:|:-------|------------:|------------------------------:|----------:|
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw      |               256 | pp2048 |    11710.08 |                      14162.79 |      1.21 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw      |               512 | pp2048 |    13389.07 |                      13443.75 |      1.00 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw      |              1024 | pp2048 |    15232.29 |                      16217.56 |      1.06 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw  |               256 | pp2048 |    11419.26 |                      14492.09 |      1.27 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw  |               512 | pp2048 |    12980.69 |                      13579.75 |      1.05 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw  |              1024 | pp2048 |    14666.17 |                      16202.63 |      1.10 |
| RX 7900 XTX | llama 1B Q2_K_M               |               256 | pp2048 |    10308.52 |                      13559.73 |      1.32 |
| RX 7900 XTX | llama 1B Q2_K_M               |               512 | pp2048 |    11137.57 |                      12940.25 |      1.16 |
| RX 7900 XTX | llama 1B Q2_K_M               |              1024 | pp2048 |    12807.04 |                      15554.49 |      1.21 |
| RX 7900 XTX | llama 1B Q6_K                 |               256 | pp2048 |     8985.03 |                      14162.37 |      1.58 |
| RX 7900 XTX | llama 1B Q6_K                 |               512 | pp2048 |    10344.84 |                      13421.27 |      1.30 |
| RX 7900 XTX | llama 1B Q6_K                 |              1024 | pp2048 |    11870.41 |                      15930.02 |      1.34 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw     |               256 | pp2048 |     1324.46 |                       1455.10 |      1.10 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw     |               512 | pp2048 |     1391.35 |                       1503.48 |      1.08 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw     |              1024 | pp2048 |     1498.16 |                       1591.97 |      1.06 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw |               256 | pp2048 |     1282.21 |                       1465.35 |      1.14 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw |               512 | pp2048 |     1363.45 |                       1476.22 |      1.08 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw |              1024 | pp2048 |     1458.91 |                       1562.08 |      1.07 |
| RX 7900 XTX | qwen3 14B Q2_K_M              |               256 | pp2048 |     1060.96 |                       1421.48 |      1.34 |
| RX 7900 XTX | qwen3 14B Q2_K_M              |               512 | pp2048 |     1112.72 |                       1447.63 |      1.30 |
| RX 7900 XTX | qwen3 14B Q2_K_M              |              1024 | pp2048 |     1198.96 |                       1623.59 |      1.35 |
| RX 7900 XTX | qwen3 14B Q6_K                |               256 | pp2048 |      987.74 |                       1414.90 |      1.43 |
| RX 7900 XTX | qwen3 14B Q6_K                |               512 | pp2048 |     1036.21 |                       1530.71 |      1.48 |
| RX 7900 XTX | qwen3 14B Q6_K                |              1024 | pp2048 |     1114.71 |                       1577.11 |      1.41 |

</details>

<details><summary><b>compare-llama-bench full</b></summary>

| GPU         | Model                          |   Microbatch size | Test   |   t/s b7600 |   t/s beinsezii/rocm_mmq_tune |   Speedup |
|:------------|:-------------------------------|------------------:|:-------|------------:|------------------------------:|----------:|
| RX 7900 XTX | gpt-oss 20B MXFP4 MoE          |                64 | pp2048 |     1526.87 |                       1533.15 |      1.00 |
| RX 7900 XTX | gpt-oss 20B MXFP4 MoE          |               128 | pp2048 |     1736.72 |                       1744.08 |      1.00 |
| RX 7900 XTX | gpt-oss 20B MXFP4 MoE          |               256 | pp2048 |     2670.39 |                       2723.63 |      1.02 |
| RX 7900 XTX | gpt-oss 20B MXFP4 MoE          |               512 | pp2048 |     3844.00 |                       3857.77 |      1.00 |
| RX 7900 XTX | gpt-oss 20B MXFP4 MoE          |              1024 | pp2048 |     4767.12 |                       4776.70 |      1.00 |
| RX 7900 XTX | granitehybrid 1B Q4_K_M        |                64 | pp2048 |     1996.18 |                       2051.94 |      1.03 |
| RX 7900 XTX | granitehybrid 1B Q4_K_M        |               128 | pp2048 |     2363.13 |                       2458.62 |      1.04 |
| RX 7900 XTX | granitehybrid 1B Q4_K_M        |               256 | pp2048 |     3752.94 |                       3848.70 |      1.03 |
| RX 7900 XTX | granitehybrid 1B Q4_K_M        |               512 | pp2048 |     5144.10 |                       5239.24 |      1.02 |
| RX 7900 XTX | granitehybrid 1B Q4_K_M        |              1024 | pp2048 |     5951.86 |                       6029.21 |      1.01 |
| RX 7900 XTX | llama 1B IQ1_S - 1.5625 bpw    |                64 | pp2048 |     7027.68 |                       6666.87 |      0.95 |
| RX 7900 XTX | llama 1B IQ1_S - 1.5625 bpw    |               128 | pp2048 |     9170.36 |                       9228.60 |      1.01 |
| RX 7900 XTX | llama 1B IQ1_S - 1.5625 bpw    |               256 | pp2048 |    13441.39 |                      13776.57 |      1.02 |
| RX 7900 XTX | llama 1B IQ1_S - 1.5625 bpw    |               512 | pp2048 |    15024.81 |                      15560.55 |      1.04 |
| RX 7900 XTX | llama 1B IQ1_S - 1.5625 bpw    |              1024 | pp2048 |    17155.51 |                      17523.91 |      1.02 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw       |                64 | pp2048 |     6637.23 |                       6414.86 |      0.97 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw       |               128 | pp2048 |     8048.83 |                       8034.94 |      1.00 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw       |               256 | pp2048 |    11710.08 |                      14162.79 |      1.21 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw       |               512 | pp2048 |    13389.07 |                      13443.75 |      1.00 |
| RX 7900 XTX | llama 1B IQ2_S - 2.5 bpw       |              1024 | pp2048 |    15232.29 |                      16217.56 |      1.06 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw   |                64 | pp2048 |     6507.77 |                       6242.87 |      0.96 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw   |               128 | pp2048 |     7801.62 |                       7795.86 |      1.00 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw   |               256 | pp2048 |    11419.26 |                      14492.09 |      1.27 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw   |               512 | pp2048 |    12980.69 |                      13579.75 |      1.05 |
| RX 7900 XTX | llama 1B IQ2_XS - 2.3125 bpw   |              1024 | pp2048 |    14666.17 |                      16202.63 |      1.10 |
| RX 7900 XTX | llama 1B IQ2_XXS - 2.0625 bpw  |                64 | pp2048 |     6667.97 |                       6304.72 |      0.95 |
| RX 7900 XTX | llama 1B IQ2_XXS - 2.0625 bpw  |               128 | pp2048 |     9311.87 |                       9311.16 |      1.00 |
| RX 7900 XTX | llama 1B IQ2_XXS - 2.0625 bpw  |               256 | pp2048 |    13530.44 |                      13821.29 |      1.02 |
| RX 7900 XTX | llama 1B IQ2_XXS - 2.0625 bpw  |               512 | pp2048 |    14918.22 |                      15340.57 |      1.03 |
| RX 7900 XTX | llama 1B IQ2_XXS - 2.0625 bpw  |              1024 | pp2048 |    16953.24 |                      17495.25 |      1.03 |
| RX 7900 XTX | llama 1B IQ3_S - 3.4375 bpw    |                64 | pp2048 |     6758.54 |                       6809.36 |      1.01 |
| RX 7900 XTX | llama 1B IQ3_S - 3.4375 bpw    |               128 | pp2048 |     9395.78 |                       9383.74 |      1.00 |
| RX 7900 XTX | llama 1B IQ3_S - 3.4375 bpw    |               256 | pp2048 |    13616.08 |                      13611.88 |      1.00 |
| RX 7900 XTX | llama 1B IQ3_S - 3.4375 bpw    |               512 | pp2048 |    15014.23 |                      15088.34 |      1.00 |
| RX 7900 XTX | llama 1B IQ3_S - 3.4375 bpw    |              1024 | pp2048 |    16841.61 |                      17049.62 |      1.01 |
| RX 7900 XTX | llama 1B IQ3_XXS - 3.0625 bpw  |                64 | pp2048 |     6989.48 |                       6992.81 |      1.00 |
| RX 7900 XTX | llama 1B IQ3_XXS - 3.0625 bpw  |               128 | pp2048 |     9329.01 |                       9360.47 |      1.00 |
| RX 7900 XTX | llama 1B IQ3_XXS - 3.0625 bpw  |               256 | pp2048 |    13667.47 |                      14401.52 |      1.05 |
| RX 7900 XTX | llama 1B IQ3_XXS - 3.0625 bpw  |               512 | pp2048 |    15279.04 |                      15682.87 |      1.03 |
| RX 7900 XTX | llama 1B IQ3_XXS - 3.0625 bpw  |              1024 | pp2048 |    17511.66 |                      17846.33 |      1.02 |
| RX 7900 XTX | llama 1B IQ4_NL - 4.5 bpw      |                64 | pp2048 |     7739.35 |                       7772.91 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_NL - 4.5 bpw      |               128 | pp2048 |    10303.88 |                      10303.47 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_NL - 4.5 bpw      |               256 | pp2048 |    14758.83 |                      14790.00 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_NL - 4.5 bpw      |               512 | pp2048 |    16346.91 |                      16396.66 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_NL - 4.5 bpw      |              1024 | pp2048 |    18466.96 |                      18591.02 |      1.01 |
| RX 7900 XTX | llama 1B IQ4_XS - 4.25 bpw     |                64 | pp2048 |     7811.50 |                       7863.27 |      1.01 |
| RX 7900 XTX | llama 1B IQ4_XS - 4.25 bpw     |               128 | pp2048 |    10306.37 |                      10308.72 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_XS - 4.25 bpw     |               256 | pp2048 |    14751.24 |                      14769.14 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_XS - 4.25 bpw     |               512 | pp2048 |    16341.84 |                      16349.54 |      1.00 |
| RX 7900 XTX | llama 1B IQ4_XS - 4.25 bpw     |              1024 | pp2048 |    18228.46 |                      18714.99 |      1.03 |
| RX 7900 XTX | llama 1B Q2_K_M                |                64 | pp2048 |     5192.98 |                       4812.87 |      0.93 |
| RX 7900 XTX | llama 1B Q2_K_M                |               128 | pp2048 |     7344.04 |                       7361.62 |      1.00 |
| RX 7900 XTX | llama 1B Q2_K_M                |               256 | pp2048 |    10308.52 |                      13559.73 |      1.32 |
| RX 7900 XTX | llama 1B Q2_K_M                |               512 | pp2048 |    11137.57 |                      12940.25 |      1.16 |
| RX 7900 XTX | llama 1B Q2_K_M                |              1024 | pp2048 |    12807.04 |                      15554.49 |      1.21 |
| RX 7900 XTX | llama 1B Q3_K_M                |                64 | pp2048 |     6953.68 |                       6973.70 |      1.00 |
| RX 7900 XTX | llama 1B Q3_K_M                |               128 | pp2048 |     8895.70 |                       8907.37 |      1.00 |
| RX 7900 XTX | llama 1B Q3_K_M                |               256 | pp2048 |    12954.66 |                      13047.81 |      1.01 |
| RX 7900 XTX | llama 1B Q3_K_M                |               512 | pp2048 |    14688.59 |                      14692.35 |      1.00 |
| RX 7900 XTX | llama 1B Q3_K_M                |              1024 | pp2048 |    16542.45 |                      16624.59 |      1.00 |
| RX 7900 XTX | llama 1B Q4_0                  |                64 | pp2048 |     7562.55 |                       7610.92 |      1.01 |
| RX 7900 XTX | llama 1B Q4_0                  |               128 | pp2048 |    10106.39 |                      10096.22 |      1.00 |
| RX 7900 XTX | llama 1B Q4_0                  |               256 | pp2048 |    14566.00 |                      14613.30 |      1.00 |
| RX 7900 XTX | llama 1B Q4_0                  |               512 | pp2048 |    16083.62 |                      16129.93 |      1.00 |
| RX 7900 XTX | llama 1B Q4_0                  |              1024 | pp2048 |    18178.06 |                      18300.13 |      1.01 |
| RX 7900 XTX | llama 1B Q4_1                  |                64 | pp2048 |     7509.82 |                       6766.76 |      0.90 |
| RX 7900 XTX | llama 1B Q4_1                  |               128 | pp2048 |     8963.73 |                       9026.38 |      1.01 |
| RX 7900 XTX | llama 1B Q4_1                  |               256 | pp2048 |    13117.91 |                      13168.37 |      1.00 |
| RX 7900 XTX | llama 1B Q4_1                  |               512 | pp2048 |    14950.75 |                      15028.10 |      1.01 |
| RX 7900 XTX | llama 1B Q4_1                  |              1024 | pp2048 |    16740.56 |                      16966.23 |      1.01 |
| RX 7900 XTX | llama 1B Q4_K_M                |                64 | pp2048 |     6909.62 |                       6912.13 |      1.00 |
| RX 7900 XTX | llama 1B Q4_K_M                |               128 | pp2048 |     8315.25 |                       8360.44 |      1.01 |
| RX 7900 XTX | llama 1B Q4_K_M                |               256 | pp2048 |    12427.88 |                      13770.19 |      1.11 |
| RX 7900 XTX | llama 1B Q4_K_M                |               512 | pp2048 |    14203.03 |                      15579.25 |      1.10 |
| RX 7900 XTX | llama 1B Q4_K_M                |              1024 | pp2048 |    16391.75 |                      17443.34 |      1.06 |
| RX 7900 XTX | llama 1B Q5_0                  |                64 | pp2048 |     7118.02 |                       7161.32 |      1.01 |
| RX 7900 XTX | llama 1B Q5_0                  |               128 | pp2048 |     9771.66 |                       9755.90 |      1.00 |
| RX 7900 XTX | llama 1B Q5_0                  |               256 | pp2048 |    14112.98 |                      14134.09 |      1.00 |
| RX 7900 XTX | llama 1B Q5_0                  |               512 | pp2048 |    15561.64 |                      15640.29 |      1.01 |
| RX 7900 XTX | llama 1B Q5_0                  |              1024 | pp2048 |    17557.71 |                      17678.22 |      1.01 |
| RX 7900 XTX | llama 1B Q5_1                  |                64 | pp2048 |     6777.00 |                       6776.62 |      1.00 |
| RX 7900 XTX | llama 1B Q5_1                  |               128 | pp2048 |     8387.54 |                       8441.55 |      1.01 |
| RX 7900 XTX | llama 1B Q5_1                  |               256 | pp2048 |    12443.72 |                      12487.28 |      1.00 |
| RX 7900 XTX | llama 1B Q5_1                  |               512 | pp2048 |    14397.02 |                      14466.45 |      1.00 |
| RX 7900 XTX | llama 1B Q5_1                  |              1024 | pp2048 |    16301.91 |                      16438.11 |      1.01 |
| RX 7900 XTX | llama 1B Q5_K_M                |                64 | pp2048 |     6959.61 |                       6974.76 |      1.00 |
| RX 7900 XTX | llama 1B Q5_K_M                |               128 | pp2048 |     7993.31 |                       8064.42 |      1.01 |
| RX 7900 XTX | llama 1B Q5_K_M                |               256 | pp2048 |    11951.11 |                      13219.55 |      1.11 |
| RX 7900 XTX | llama 1B Q5_K_M                |               512 | pp2048 |    13692.80 |                      14974.54 |      1.09 |
| RX 7900 XTX | llama 1B Q5_K_M                |              1024 | pp2048 |    15581.72 |                      16945.35 |      1.09 |
| RX 7900 XTX | llama 1B Q6_K                  |                64 | pp2048 |     5576.08 |                       5604.39 |      1.01 |
| RX 7900 XTX | llama 1B Q6_K                  |               128 | pp2048 |     5925.98 |                       5955.94 |      1.01 |
| RX 7900 XTX | llama 1B Q6_K                  |               256 | pp2048 |     8985.03 |                      14162.37 |      1.58 |
| RX 7900 XTX | llama 1B Q6_K                  |               512 | pp2048 |    10344.84 |                      13421.27 |      1.30 |
| RX 7900 XTX | llama 1B Q6_K                  |              1024 | pp2048 |    11870.41 |                      15930.02 |      1.34 |
| RX 7900 XTX | llama 1B Q8_0                  |                64 | pp2048 |     7481.56 |                       7506.52 |      1.00 |
| RX 7900 XTX | llama 1B Q8_0                  |               128 | pp2048 |    10159.60 |                      10164.84 |      1.00 |
| RX 7900 XTX | llama 1B Q8_0                  |               256 | pp2048 |    14538.35 |                      14473.59 |      1.00 |
| RX 7900 XTX | llama 1B Q8_0                  |               512 | pp2048 |    16438.17 |                      16492.02 |      1.00 |
| RX 7900 XTX | llama 1B Q8_0                  |              1024 | pp2048 |    18724.04 |                      18843.78 |      1.01 |
| RX 7900 XTX | qwen3 14B IQ1_S - 1.5625 bpw   |                64 | pp2048 |     1140.61 |                       1141.17 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ1_S - 1.5625 bpw   |               128 | pp2048 |     1359.67 |                       1352.72 |      0.99 |
| RX 7900 XTX | qwen3 14B IQ1_S - 1.5625 bpw   |               256 | pp2048 |     1563.62 |                       1597.59 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ1_S - 1.5625 bpw   |               512 | pp2048 |     1655.34 |                       1685.87 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ1_S - 1.5625 bpw   |              1024 | pp2048 |     1753.76 |                       1786.72 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw      |                64 | pp2048 |     1030.98 |                       1029.35 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw      |               128 | pp2048 |     1135.86 |                       1132.83 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw      |               256 | pp2048 |     1324.46 |                       1455.10 |      1.10 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw      |               512 | pp2048 |     1391.35 |                       1503.48 |      1.08 |
| RX 7900 XTX | qwen3 14B IQ2_S - 2.5 bpw      |              1024 | pp2048 |     1498.16 |                       1591.97 |      1.06 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw  |                64 | pp2048 |     1001.96 |                       1000.14 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw  |               128 | pp2048 |     1099.44 |                       1094.66 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw  |               256 | pp2048 |     1282.21 |                       1465.35 |      1.14 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw  |               512 | pp2048 |     1363.45 |                       1476.22 |      1.08 |
| RX 7900 XTX | qwen3 14B IQ2_XS - 2.3125 bpw  |              1024 | pp2048 |     1458.91 |                       1562.08 |      1.07 |
| RX 7900 XTX | qwen3 14B IQ2_XXS - 2.0625 bpw |                64 | pp2048 |     1032.14 |                       1034.30 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_XXS - 2.0625 bpw |               128 | pp2048 |     1366.24 |                       1361.42 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ2_XXS - 2.0625 bpw |               256 | pp2048 |     1544.25 |                       1580.51 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ2_XXS - 2.0625 bpw |               512 | pp2048 |     1632.63 |                       1667.30 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ2_XXS - 2.0625 bpw |              1024 | pp2048 |     1748.30 |                       1779.36 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ3_S - 3.4375 bpw   |                64 | pp2048 |     1045.70 |                       1044.43 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ3_S - 3.4375 bpw   |               128 | pp2048 |     1377.53 |                       1374.77 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ3_S - 3.4375 bpw   |               256 | pp2048 |     1563.80 |                       1558.34 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ3_S - 3.4375 bpw   |               512 | pp2048 |     1640.04 |                       1648.42 |      1.01 |
| RX 7900 XTX | qwen3 14B IQ3_S - 3.4375 bpw   |              1024 | pp2048 |     1762.73 |                       1737.97 |      0.99 |
| RX 7900 XTX | qwen3 14B IQ3_XXS - 3.0625 bpw |                64 | pp2048 |     1096.26 |                       1093.64 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ3_XXS - 3.0625 bpw |               128 | pp2048 |     1394.21 |                       1390.97 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ3_XXS - 3.0625 bpw |               256 | pp2048 |     1600.20 |                       1649.71 |      1.03 |
| RX 7900 XTX | qwen3 14B IQ3_XXS - 3.0625 bpw |               512 | pp2048 |     1688.29 |                       1721.06 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ3_XXS - 3.0625 bpw |              1024 | pp2048 |     1801.51 |                       1794.65 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_NL - 4.5 bpw     |                64 | pp2048 |     1247.63 |                       1243.49 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_NL - 4.5 bpw     |               128 | pp2048 |     1523.49 |                       1522.58 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_NL - 4.5 bpw     |               256 | pp2048 |     1755.56 |                       1753.71 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_NL - 4.5 bpw     |               512 | pp2048 |     1874.83 |                       1872.05 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_NL - 4.5 bpw     |              1024 | pp2048 |     1961.00 |                       2002.52 |      1.02 |
| RX 7900 XTX | qwen3 14B IQ4_XS - 4.25 bpw    |                64 | pp2048 |     1262.09 |                       1263.23 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_XS - 4.25 bpw    |               128 | pp2048 |     1524.40 |                       1523.44 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_XS - 4.25 bpw    |               256 | pp2048 |     1751.32 |                       1752.22 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_XS - 4.25 bpw    |               512 | pp2048 |     1872.45 |                       1870.28 |      1.00 |
| RX 7900 XTX | qwen3 14B IQ4_XS - 4.25 bpw    |              1024 | pp2048 |     1984.84 |                       1963.28 |      0.99 |
| RX 7900 XTX | qwen3 14B Q2_K_M               |                64 | pp2048 |      718.31 |                        718.29 |      1.00 |
| RX 7900 XTX | qwen3 14B Q2_K_M               |               128 | pp2048 |      952.23 |                        951.22 |      1.00 |
| RX 7900 XTX | qwen3 14B Q2_K_M               |               256 | pp2048 |     1060.96 |                       1421.48 |      1.34 |
| RX 7900 XTX | qwen3 14B Q2_K_M               |               512 | pp2048 |     1112.72 |                       1447.63 |      1.30 |
| RX 7900 XTX | qwen3 14B Q2_K_M               |              1024 | pp2048 |     1198.96 |                       1623.59 |      1.35 |
| RX 7900 XTX | qwen3 14B Q3_K_M               |                64 | pp2048 |     1079.69 |                       1079.12 |      1.00 |
| RX 7900 XTX | qwen3 14B Q3_K_M               |               128 | pp2048 |     1296.56 |                       1295.70 |      1.00 |
| RX 7900 XTX | qwen3 14B Q3_K_M               |               256 | pp2048 |     1511.63 |                       1510.58 |      1.00 |
| RX 7900 XTX | qwen3 14B Q3_K_M               |               512 | pp2048 |     1594.04 |                       1583.39 |      0.99 |
| RX 7900 XTX | qwen3 14B Q3_K_M               |              1024 | pp2048 |     1711.89 |                       1711.06 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_0                 |                64 | pp2048 |     1201.50 |                       1200.02 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_0                 |               128 | pp2048 |     1496.31 |                       1495.05 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_0                 |               256 | pp2048 |     1710.85 |                       1707.70 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_0                 |               512 | pp2048 |     1814.45 |                       1813.08 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_0                 |              1024 | pp2048 |     1942.44 |                       1924.26 |      0.99 |
| RX 7900 XTX | qwen3 14B Q4_1                 |                64 | pp2048 |     1211.52 |                       1210.78 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_1                 |               128 | pp2048 |     1321.09 |                       1319.58 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_1                 |               256 | pp2048 |     1559.01 |                       1557.82 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_1                 |               512 | pp2048 |     1646.61 |                       1645.66 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_1                 |              1024 | pp2048 |     1759.53 |                       1758.46 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_K_M               |                64 | pp2048 |     1091.35 |                       1090.71 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_K_M               |               128 | pp2048 |     1223.08 |                       1222.06 |      1.00 |
| RX 7900 XTX | qwen3 14B Q4_K_M               |               256 | pp2048 |     1463.26 |                       1594.43 |      1.09 |
| RX 7900 XTX | qwen3 14B Q4_K_M               |               512 | pp2048 |     1551.95 |                       1693.88 |      1.09 |
| RX 7900 XTX | qwen3 14B Q4_K_M               |              1024 | pp2048 |     1663.48 |                       1776.79 |      1.07 |
| RX 7900 XTX | qwen3 14B Q5_0                 |                64 | pp2048 |     1125.17 |                       1124.47 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_0                 |               128 | pp2048 |     1431.73 |                       1430.83 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_0                 |               256 | pp2048 |     1637.76 |                       1638.62 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_0                 |               512 | pp2048 |     1736.37 |                       1737.31 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_0                 |              1024 | pp2048 |     1862.31 |                       1860.42 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_1                 |                64 | pp2048 |     1151.87 |                       1151.59 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_1                 |               128 | pp2048 |     1269.13 |                       1269.21 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_1                 |               256 | pp2048 |     1492.63 |                       1492.59 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_1                 |               512 | pp2048 |     1573.52 |                       1574.66 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_1                 |              1024 | pp2048 |     1691.33 |                       1692.26 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_K_M               |                64 | pp2048 |     1095.75 |                       1095.21 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_K_M               |               128 | pp2048 |     1178.49 |                       1177.27 |      1.00 |
| RX 7900 XTX | qwen3 14B Q5_K_M               |               256 | pp2048 |     1399.05 |                       1516.18 |      1.08 |
| RX 7900 XTX | qwen3 14B Q5_K_M               |               512 | pp2048 |     1478.38 |                       1608.14 |      1.09 |
| RX 7900 XTX | qwen3 14B Q5_K_M               |              1024 | pp2048 |     1594.73 |                       1684.33 |      1.06 |
| RX 7900 XTX | qwen3 14B Q6_K                 |                64 | pp2048 |      827.01 |                        826.16 |      1.00 |
| RX 7900 XTX | qwen3 14B Q6_K                 |               128 | pp2048 |      829.64 |                        829.41 |      1.00 |
| RX 7900 XTX | qwen3 14B Q6_K                 |               256 | pp2048 |      987.74 |                       1414.90 |      1.43 |
| RX 7900 XTX | qwen3 14B Q6_K                 |               512 | pp2048 |     1036.21 |                       1530.71 |      1.48 |
| RX 7900 XTX | qwen3 14B Q6_K                 |              1024 | pp2048 |     1114.71 |                       1577.11 |      1.41 |
| RX 7900 XTX | qwen3 14B Q8_0                 |                64 | pp2048 |     1192.57 |                       1196.07 |      1.00 |
| RX 7900 XTX | qwen3 14B Q8_0                 |               128 | pp2048 |     1490.71 |                       1495.53 |      1.00 |
| RX 7900 XTX | qwen3 14B Q8_0                 |               256 | pp2048 |     1748.60 |                       1753.73 |      1.00 |
| RX 7900 XTX | qwen3 14B Q8_0                 |               512 | pp2048 |     1854.19 |                       1859.18 |      1.00 |
| RX 7900 XTX | qwen3 14B Q8_0                 |              1024 | pp2048 |     1974.66 |                       1983.18 |      1.00 |
| RX 7900 XTX | qwen3moe 30B.A3B Q4_K_M        |                64 | pp2048 |     1095.47 |                       1100.05 |      1.00 |
| RX 7900 XTX | qwen3moe 30B.A3B Q4_K_M        |               128 | pp2048 |     1152.03 |                       1158.15 |      1.01 |
| RX 7900 XTX | qwen3moe 30B.A3B Q4_K_M        |               256 | pp2048 |     1801.08 |                       1823.55 |      1.01 |
| RX 7900 XTX | qwen3moe 30B.A3B Q4_K_M        |               512 | pp2048 |     2523.29 |                       2555.23 |      1.01 |
| RX 7900 XTX | qwen3moe 30B.A3B Q4_K_M        |              1024 | pp2048 |     3331.65 |                       3374.67 |      1.01 |

</details>

<details><summary><b>mmq/blas/tuned breakdown</b></summary>

| model_filename | model_n_params | n_ubatch | n_prompt | mmq_ts | blas_ts | tuned_ts | mmq&#x2f;blas | mmq% | blas% | tuned% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 14B/IQ1_S.gguf | 14768307200 | 64 | 2048 | 1140.61 | 557.03 | 1141.17 | 104.76 | 100.0 | 48.84 | 100.05 |
| 14B/IQ1_S.gguf | 14768307200 | 128 | 2048 | 1359.67 | 988.76 | 1352.72 | 37.51 | 100.0 | 72.72 | 99.49 |
| 14B/IQ1_S.gguf | 14768307200 | 256 | 2048 | 1563.62 | 1513.45 | 1597.59 | 3.31 | 100.0 | 96.79 | 102.17 |
| 14B/IQ1_S.gguf | 14768307200 | 512 | 2048 | 1655.34 | 1485.16 | 1685.87 | 11.46 | 100.0 | 89.72 | 101.84 |
| 14B/IQ1_S.gguf | 14768307200 | 1024 | 2048 | 1753.76 | 1566.91 | 1786.72 | 11.92 | 100.0 | 89.35 | 101.88 |
| 14B/IQ2_S.gguf | 14768307200 | 64 | 2048 | 1030.98 | 549.34 | 1029.35 | 87.68 | 100.0 | 53.28 | 99.84 |
| 14B/IQ2_S.gguf | 14768307200 | 128 | 2048 | 1135.86 | 977.31 | 1132.83 | 16.22 | 100.0 | 86.04 | 99.73 |
| 14B/IQ2_S.gguf | 14768307200 | 256 | 2048 | 1324.46 | 1486.75 | 1455.1 | -10.92 | 89.08 | 100.0 | 97.87 |
| 14B/IQ2_S.gguf | 14768307200 | 512 | 2048 | 1391.35 | 1477.13 | 1503.48 | -5.81 | 94.19 | 100.0 | 101.78 |
| 14B/IQ2_S.gguf | 14768307200 | 1024 | 2048 | 1498.16 | 1552.33 | 1591.97 | -3.49 | 96.51 | 100.0 | 102.55 |
| 14B/IQ2_XS.gguf | 14768307200 | 64 | 2048 | 1001.96 | 550.12 | 1000.14 | 82.13 | 100.0 | 54.9 | 99.82 |
| 14B/IQ2_XS.gguf | 14768307200 | 128 | 2048 | 1099.44 | 976.01 | 1094.66 | 12.65 | 100.0 | 88.77 | 99.56 |
| 14B/IQ2_XS.gguf | 14768307200 | 256 | 2048 | 1282.21 | 1489.22 | 1465.35 | -13.9 | 86.1 | 100.0 | 98.4 |
| 14B/IQ2_XS.gguf | 14768307200 | 512 | 2048 | 1363.45 | 1492.71 | 1476.22 | -8.66 | 91.34 | 100.0 | 98.9 |
| 14B/IQ2_XS.gguf | 14768307200 | 1024 | 2048 | 1458.91 | 1555.26 | 1562.08 | -6.2 | 93.8 | 100.0 | 100.44 |
| 14B/IQ2_XXS.gguf | 14768307200 | 64 | 2048 | 1032.14 | 551.64 | 1034.3 | 87.11 | 100.0 | 53.45 | 100.21 |
| 14B/IQ2_XXS.gguf | 14768307200 | 128 | 2048 | 1366.24 | 987.0 | 1361.42 | 38.42 | 100.0 | 72.24 | 99.65 |
| 14B/IQ2_XXS.gguf | 14768307200 | 256 | 2048 | 1544.25 | 1501.45 | 1580.51 | 2.85 | 100.0 | 97.23 | 102.35 |
| 14B/IQ2_XXS.gguf | 14768307200 | 512 | 2048 | 1632.63 | 1496.13 | 1667.3 | 9.12 | 100.0 | 91.64 | 102.12 |
| 14B/IQ2_XXS.gguf | 14768307200 | 1024 | 2048 | 1748.3 | 1566.41 | 1779.36 | 11.61 | 100.0 | 89.6 | 101.78 |
| 14B/IQ3_S.gguf | 14768307200 | 64 | 2048 | 1045.7 | 546.83 | 1044.43 | 91.23 | 100.0 | 52.29 | 99.88 |
| 14B/IQ3_S.gguf | 14768307200 | 128 | 2048 | 1377.53 | 981.26 | 1374.77 | 40.38 | 100.0 | 71.23 | 99.8 |
| 14B/IQ3_S.gguf | 14768307200 | 256 | 2048 | 1563.8 | 1474.86 | 1558.34 | 6.03 | 100.0 | 94.31 | 99.65 |
| 14B/IQ3_S.gguf | 14768307200 | 512 | 2048 | 1640.04 | 1500.24 | 1648.42 | 9.32 | 100.0 | 91.48 | 100.51 |
| 14B/IQ3_S.gguf | 14768307200 | 1024 | 2048 | 1762.73 | 1554.7 | 1737.97 | 13.38 | 100.0 | 88.2 | 98.59 |
| 14B/IQ3_XXS.gguf | 14768307200 | 64 | 2048 | 1096.26 | 544.75 | 1093.64 | 101.24 | 100.0 | 49.69 | 99.76 |
| 14B/IQ3_XXS.gguf | 14768307200 | 128 | 2048 | 1394.21 | 980.82 | 1390.97 | 42.15 | 100.0 | 70.35 | 99.77 |
| 14B/IQ3_XXS.gguf | 14768307200 | 256 | 2048 | 1600.2 | 1439.52 | 1649.71 | 11.16 | 100.0 | 89.96 | 103.09 |
| 14B/IQ3_XXS.gguf | 14768307200 | 512 | 2048 | 1688.29 | 1507.53 | 1721.06 | 11.99 | 100.0 | 89.29 | 101.94 |
| 14B/IQ3_XXS.gguf | 14768307200 | 1024 | 2048 | 1801.51 | 1550.74 | 1794.65 | 16.17 | 100.0 | 86.08 | 99.62 |
| 14B/IQ4_NL.gguf | 14768307200 | 64 | 2048 | 1247.63 | 542.82 | 1243.49 | 129.84 | 100.0 | 43.51 | 99.67 |
| 14B/IQ4_NL.gguf | 14768307200 | 128 | 2048 | 1523.49 | 984.15 | 1522.58 | 54.8 | 100.0 | 64.6 | 99.94 |
| 14B/IQ4_NL.gguf | 14768307200 | 256 | 2048 | 1755.56 | 1464.65 | 1753.71 | 19.86 | 100.0 | 83.43 | 99.89 |
| 14B/IQ4_NL.gguf | 14768307200 | 512 | 2048 | 1874.83 | 1505.38 | 1872.05 | 24.54 | 100.0 | 80.29 | 99.85 |
| 14B/IQ4_NL.gguf | 14768307200 | 1024 | 2048 | 1961.0 | 1558.94 | 2002.52 | 25.79 | 100.0 | 79.5 | 102.12 |
| 14B/IQ4_XS.gguf | 14768307200 | 64 | 2048 | 1262.09 | 540.68 | 1263.23 | 133.43 | 100.0 | 42.84 | 100.09 |
| 14B/IQ4_XS.gguf | 14768307200 | 128 | 2048 | 1524.4 | 985.87 | 1523.44 | 54.63 | 100.0 | 64.67 | 99.94 |
| 14B/IQ4_XS.gguf | 14768307200 | 256 | 2048 | 1751.32 | 1468.94 | 1752.22 | 19.22 | 100.0 | 83.88 | 100.05 |
| 14B/IQ4_XS.gguf | 14768307200 | 512 | 2048 | 1872.45 | 1495.29 | 1870.28 | 25.22 | 100.0 | 79.86 | 99.88 |
| 14B/IQ4_XS.gguf | 14768307200 | 1024 | 2048 | 1984.84 | 1560.43 | 1963.28 | 27.2 | 100.0 | 78.62 | 98.91 |
| 14B/Q2_K.gguf | 14768307200 | 64 | 2048 | 718.31 | 547.06 | 718.29 | 31.3 | 100.0 | 76.16 | 100.0 |
| 14B/Q2_K.gguf | 14768307200 | 128 | 2048 | 952.23 | 971.48 | 951.22 | -1.98 | 98.02 | 100.0 | 97.91 |
| 14B/Q2_K.gguf | 14768307200 | 256 | 2048 | 1060.96 | 1445.4 | 1421.48 | -26.6 | 73.4 | 100.0 | 98.35 |
| 14B/Q2_K.gguf | 14768307200 | 512 | 2048 | 1112.72 | 1483.19 | 1447.63 | -24.98 | 75.02 | 100.0 | 97.6 |
| 14B/Q2_K.gguf | 14768307200 | 1024 | 2048 | 1198.96 | 1553.28 | 1623.59 | -22.81 | 77.19 | 100.0 | 104.53 |
| 14B/Q3_K.gguf | 14768307200 | 64 | 2048 | 1079.69 | 573.68 | 1079.12 | 88.2 | 100.0 | 53.13 | 99.95 |
| 14B/Q3_K.gguf | 14768307200 | 128 | 2048 | 1296.56 | 1005.21 | 1295.7 | 28.98 | 100.0 | 77.53 | 99.93 |
| 14B/Q3_K.gguf | 14768307200 | 256 | 2048 | 1511.63 | 1386.17 | 1510.58 | 9.05 | 100.0 | 91.7 | 99.93 |
| 14B/Q3_K.gguf | 14768307200 | 512 | 2048 | 1594.04 | 1521.6 | 1583.39 | 4.76 | 100.0 | 95.46 | 99.33 |
| 14B/Q3_K.gguf | 14768307200 | 1024 | 2048 | 1711.89 | 1566.33 | 1711.06 | 9.29 | 100.0 | 91.5 | 99.95 |
| 14B/Q4_0.gguf | 14768307200 | 64 | 2048 | 1201.5 | 548.27 | 1200.02 | 119.15 | 100.0 | 45.63 | 99.88 |
| 14B/Q4_0.gguf | 14768307200 | 128 | 2048 | 1496.31 | 995.61 | 1495.05 | 50.29 | 100.0 | 66.54 | 99.92 |
| 14B/Q4_0.gguf | 14768307200 | 256 | 2048 | 1710.85 | 1475.79 | 1707.7 | 15.93 | 100.0 | 86.26 | 99.82 |
| 14B/Q4_0.gguf | 14768307200 | 512 | 2048 | 1814.45 | 1523.29 | 1813.08 | 19.11 | 100.0 | 83.95 | 99.92 |
| 14B/Q4_0.gguf | 14768307200 | 1024 | 2048 | 1942.44 | 1565.44 | 1924.26 | 24.08 | 100.0 | 80.59 | 99.06 |
| 14B/Q4_1.gguf | 14768307200 | 64 | 2048 | 1211.52 | 545.63 | 1210.78 | 122.04 | 100.0 | 45.04 | 99.94 |
| 14B/Q4_1.gguf | 14768307200 | 128 | 2048 | 1321.09 | 994.21 | 1319.58 | 32.88 | 100.0 | 75.26 | 99.89 |
| 14B/Q4_1.gguf | 14768307200 | 256 | 2048 | 1559.01 | 1466.78 | 1557.82 | 6.29 | 100.0 | 94.08 | 99.92 |
| 14B/Q4_1.gguf | 14768307200 | 512 | 2048 | 1646.61 | 1527.81 | 1645.66 | 7.78 | 100.0 | 92.79 | 99.94 |
| 14B/Q4_1.gguf | 14768307200 | 1024 | 2048 | 1759.53 | 1564.76 | 1758.46 | 12.45 | 100.0 | 88.93 | 99.94 |
| 14B/Q4_K.gguf | 14768307200 | 64 | 2048 | 1091.35 | 547.94 | 1090.71 | 99.17 | 100.0 | 50.21 | 99.94 |
| 14B/Q4_K.gguf | 14768307200 | 128 | 2048 | 1223.08 | 990.97 | 1222.06 | 23.42 | 100.0 | 81.02 | 99.92 |
| 14B/Q4_K.gguf | 14768307200 | 256 | 2048 | 1463.26 | 1443.11 | 1594.43 | 1.4 | 100.0 | 98.62 | 108.96 |
| 14B/Q4_K.gguf | 14768307200 | 512 | 2048 | 1551.95 | 1532.37 | 1693.88 | 1.28 | 100.0 | 98.74 | 109.15 |
| 14B/Q4_K.gguf | 14768307200 | 1024 | 2048 | 1663.48 | 1567.9 | 1776.79 | 6.1 | 100.0 | 94.25 | 106.81 |
| 14B/Q5_0.gguf | 14768307200 | 64 | 2048 | 1125.17 | 509.11 | 1124.47 | 121.01 | 100.0 | 45.25 | 99.94 |
| 14B/Q5_0.gguf | 14768307200 | 128 | 2048 | 1431.73 | 890.35 | 1430.83 | 60.81 | 100.0 | 62.19 | 99.94 |
| 14B/Q5_0.gguf | 14768307200 | 256 | 2048 | 1637.76 | 1255.97 | 1638.62 | 30.4 | 100.0 | 76.69 | 100.05 |
| 14B/Q5_0.gguf | 14768307200 | 512 | 2048 | 1736.37 | 1462.34 | 1737.31 | 18.74 | 100.0 | 84.22 | 100.05 |
| 14B/Q5_0.gguf | 14768307200 | 1024 | 2048 | 1862.31 | 1524.64 | 1860.42 | 22.15 | 100.0 | 81.87 | 99.9 |
| 14B/Q5_1.gguf | 14768307200 | 64 | 2048 | 1151.87 | 505.03 | 1151.59 | 128.08 | 100.0 | 43.84 | 99.98 |
| 14B/Q5_1.gguf | 14768307200 | 128 | 2048 | 1269.13 | 891.1 | 1269.21 | 42.42 | 100.0 | 70.21 | 100.01 |
| 14B/Q5_1.gguf | 14768307200 | 256 | 2048 | 1492.63 | 1260.43 | 1492.59 | 18.42 | 100.0 | 84.44 | 100.0 |
| 14B/Q5_1.gguf | 14768307200 | 512 | 2048 | 1573.52 | 1460.43 | 1574.66 | 7.74 | 100.0 | 92.81 | 100.07 |
| 14B/Q5_1.gguf | 14768307200 | 1024 | 2048 | 1691.33 | 1509.13 | 1692.26 | 12.07 | 100.0 | 89.23 | 100.06 |
| 14B/Q5_K.gguf | 14768307200 | 64 | 2048 | 1095.75 | 533.91 | 1095.21 | 105.23 | 100.0 | 48.73 | 99.95 |
| 14B/Q5_K.gguf | 14768307200 | 128 | 2048 | 1178.49 | 964.74 | 1177.27 | 22.16 | 100.0 | 81.86 | 99.9 |
| 14B/Q5_K.gguf | 14768307200 | 256 | 2048 | 1399.05 | 1376.83 | 1516.18 | 1.61 | 100.0 | 98.41 | 108.37 |
| 14B/Q5_K.gguf | 14768307200 | 512 | 2048 | 1478.38 | 1517.64 | 1608.14 | -2.59 | 97.41 | 100.0 | 105.96 |
| 14B/Q5_K.gguf | 14768307200 | 1024 | 2048 | 1594.73 | 1558.64 | 1684.33 | 2.32 | 100.0 | 97.74 | 105.62 |
| 14B/Q6_K.gguf | 14768307200 | 64 | 2048 | 827.01 | 526.74 | 826.16 | 57.01 | 100.0 | 63.69 | 99.9 |
| 14B/Q6_K.gguf | 14768307200 | 128 | 2048 | 829.64 | 972.01 | 829.41 | -14.65 | 85.35 | 100.0 | 85.33 |
| 14B/Q6_K.gguf | 14768307200 | 256 | 2048 | 987.74 | 1412.01 | 1414.9 | -30.05 | 69.95 | 100.0 | 100.2 |
| 14B/Q6_K.gguf | 14768307200 | 512 | 2048 | 1036.21 | 1527.21 | 1530.71 | -32.15 | 67.85 | 100.0 | 100.23 |
| 14B/Q6_K.gguf | 14768307200 | 1024 | 2048 | 1114.71 | 1556.72 | 1577.11 | -28.39 | 71.61 | 100.0 | 101.31 |
| 14B/Q8_0.gguf | 14768307200 | 64 | 2048 | 1192.57 | 503.44 | 1196.07 | 136.89 | 100.0 | 42.21 | 100.29 |
| 14B/Q8_0.gguf | 14768307200 | 128 | 2048 | 1490.71 | 933.45 | 1495.53 | 59.7 | 100.0 | 62.62 | 100.32 |
| 14B/Q8_0.gguf | 14768307200 | 256 | 2048 | 1748.6 | 1384.78 | 1753.73 | 26.27 | 100.0 | 79.19 | 100.29 |
| 14B/Q8_0.gguf | 14768307200 | 512 | 2048 | 1854.19 | 1485.42 | 1859.18 | 24.83 | 100.0 | 80.11 | 100.27 |
| 14B/Q8_0.gguf | 14768307200 | 1024 | 2048 | 1974.66 | 1558.26 | 1983.18 | 26.72 | 100.0 | 78.91 | 100.43 |
| MOE/128e.gguf | 30532122624 | 64 | 2048 | 1095.47 | 388.54 | 1100.05 | 181.95 | 100.0 | 35.47 | 100.42 |
| MOE/128e.gguf | 30532122624 | 128 | 2048 | 1152.03 | 608.09 | 1158.15 | 89.45 | 100.0 | 52.78 | 100.53 |
| MOE/128e.gguf | 30532122624 | 256 | 2048 | 1801.08 | 952.55 | 1823.55 | 89.08 | 100.0 | 52.89 | 101.25 |
| MOE/128e.gguf | 30532122624 | 512 | 2048 | 2523.29 | 1342.15 | 2555.23 | 88.0 | 100.0 | 53.19 | 101.27 |
| MOE/128e.gguf | 30532122624 | 1024 | 2048 | 3331.65 | 1724.47 | 3374.67 | 93.2 | 100.0 | 51.76 | 101.29 |
| MOE/32e.gguf | 20914757184 | 64 | 2048 | 1526.87 | 1262.36 | 1533.15 | 20.95 | 100.0 | 82.68 | 100.41 |
| MOE/32e.gguf | 20914757184 | 128 | 2048 | 1736.72 | 1913.88 | 1744.08 | -9.26 | 90.74 | 100.0 | 91.13 |
| MOE/32e.gguf | 20914757184 | 256 | 2048 | 2670.39 | 2701.05 | 2723.63 | -1.14 | 98.86 | 100.0 | 100.84 |
| MOE/32e.gguf | 20914757184 | 512 | 2048 | 3844.0 | 3490.39 | 3857.77 | 10.13 | 100.0 | 90.8 | 100.36 |
| MOE/32e.gguf | 20914757184 | 1024 | 2048 | 4767.12 | 4317.11 | 4776.7 | 10.42 | 100.0 | 90.56 | 100.2 |
| MOE/64e.gguf | 6939037248 | 64 | 2048 | 1996.18 | 660.28 | 2051.94 | 202.32 | 100.0 | 33.08 | 102.79 |
| MOE/64e.gguf | 6939037248 | 128 | 2048 | 2363.13 | 1104.21 | 2458.62 | 114.01 | 100.0 | 46.73 | 104.04 |
| MOE/64e.gguf | 6939037248 | 256 | 2048 | 3752.94 | 1663.58 | 3848.7 | 125.59 | 100.0 | 44.33 | 102.55 |
| MOE/64e.gguf | 6939037248 | 512 | 2048 | 5144.1 | 2498.93 | 5239.24 | 105.85 | 100.0 | 48.58 | 101.85 |
| MOE/64e.gguf | 6939037248 | 1024 | 2048 | 5951.86 | 3331.43 | 6029.21 | 78.66 | 100.0 | 55.97 | 101.3 |

</details>

Edge Cases
---

When excluding the 1B model which is noisy, there's exactly two outliers
| model_filename | model_n_params | n_ubatch | n_prompt | mmq_ts | blas_ts | tuned_ts | mmq&#x2f;blas | mmq% | blas% | tuned% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MOE/32e.gguf | 20914757184 | 128 | 2048 | 1736.72 | 1913.88 | 1744.08 | -9.26 | 90.74 | 100.0 | 91.13 |
| 14B/Q6_K.gguf | 14768307200 | 128 | 2048 | 829.64 | 972.01 | 829.41 | -14.65 | 85.35 | 100.0 | 85.33 |

Both of which are on bs=128, and both of which quickly flip < 128. If you wanted to fudge this case, you could probably do something like
```diff
diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu
index ccb9ebed5..b7c1b7dc2 100644
--- a/ggml/src/ggml-cuda/mmq.cu
+++ b/ggml/src/ggml-cuda/mmq.cu
@@ -344,6 +344,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
             // These quants are really bad on MMQ
             case GGML_TYPE_Q2_K:
             case GGML_TYPE_Q6_K:
+                return ne11 < 128;  // ==128 specifically is much better on rocblas
             // These quants are usually worse but not always
             case GGML_TYPE_IQ2_XS:
             case GGML_TYPE_IQ2_S:
```
to get avg tuned% > 99, but that might be considered splitting hairs.

Testing Setup
---

100% of my testing was on GFX1100, ROCm 7.1.1, compile flags
```
GGML_CUDA_FA_ALL_QUANTS=ON
GGML_HIP=ON
GGML_HIP_GRAPHS=ON
```
and forced mmq/cublas as appropriate for measuring.

I do not own any RDNA3.5 or RDNA4 hardware. I'm assuming RDNA3.5 will behave pretty much the same, but since RDNA4 has some implementation differences in MMQ, it may be worth for someone to re-measure in the future.

Methodology
---

The raw data for every combination of model / batch / backend can be viewed at [measurements.csv](https://github.com/user-attachments/files/24405010/measurements.csv) which was generated by the scripts on huggingface.
Different from the other PRs, I've made the baseline MMQ as it seems to better handle most cases.
In general, I put little weight on the 1B results as it's extremely noisy, even with hip graphs. For cases that were a wash between µbatch sizes like Q4_K and Q5_K, I simply preferred MMQ.

Changed files:
filename: ggml/src/ggml-cuda/mmq.cu
patch: @@ -333,6 +333,28 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
     }
 
     if (amd_wmma_available(cc)) {
+        // RDNA 4 is consistently worse on rocblas
+        // https://github.com/ggml-org/llama.cpp/pull/18537#issuecomment-3706422301
+        if (GGML_CUDA_CC_IS_RDNA3(cc)) {
+            // High expert counts almost always better on MMQ
+            // due to a large amount of graph splits
+            // https://github.com/ggml-org/llama.cpp/pull/18202
+            if (n_experts >= 64) {
+                return true;
+            }
+
+            switch (type) {
+                // These quants are really bad on MMQ
+                case GGML_TYPE_Q2_K:
+                case GGML_TYPE_Q6_K:
+                // These quants are usually worse but not always
+                case GGML_TYPE_IQ2_XS:
+                case GGML_TYPE_IQ2_S:
+                    return ne11 <= 128;
+                default:
+                    return true;
+            }
+        }
         return true;
     }