Repository: ggml-org/llama.cpp
Pull request: llama : bump max seq limit from 64 to 256

ref https://github.com/ggml-org/llama.cpp/commit/4f81b33e324b1669b282eb81104e4a1131be7dce#commitcomment-165464897

I'm not able to measure any significant impact on the overall performance from bumping this value, so I guess it's OK to do so. Probably we should refactor the implementation to support dynamically configure `LLAMA_MAX_SEQ` to remove any potential concerns about performance.

Here is a basic command to test the parallel text generation performance for various batch sizes:

```bash
llama-batched-bench -hf ggml-org/gemma-3-4b-it-GGUF:Q8_0 -c 150000 -npp 512 -ntg 64 -npl 1,2,4,8,16,32,64,80,96,112,128,144,160,186,192,208,224,240,256
```

Sample output on M2 Ultra:

```
main: n_kv_max = 196608, n_batch = 2048, n_ubatch = 512, flash_attn = -1, is_pp_shared = 0, n_gpu_layers = -1, n_threads = 16, n_threads_batch = 16

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|   512 |     64 |    1 |    576 |    0.197 |  2594.53 |    0.654 |    97.86 |    0.851 |   676.58 |
|   512 |     64 |    2 |   1152 |    0.387 |  2648.99 |    0.712 |   179.89 |    1.098 |  1049.06 |
|   512 |     64 |    4 |   2304 |    0.770 |  2659.82 |    0.951 |   269.05 |    1.721 |  1338.40 |
|   512 |     64 |    8 |   4608 |    1.542 |  2656.28 |    1.296 |   395.01 |    2.838 |  1623.59 |
|   512 |     64 |   16 |   9216 |    3.078 |  2661.10 |    3.221 |   317.89 |    6.300 |  1462.93 |
|   512 |     64 |   32 |  18432 |    6.175 |  2653.36 |    3.422 |   598.44 |    9.597 |  1920.60 |
|   512 |     64 |   64 |  36864 |   12.318 |  2660.25 |    4.063 |  1008.05 |   16.381 |  2250.42 |
|   512 |     64 |   80 |  46080 |   15.393 |  2661.00 |    5.032 |  1017.58 |   20.424 |  2256.14 |
|   512 |     64 |   96 |  55296 |   18.473 |  2660.69 |    5.210 |  1179.32 |   23.683 |  2334.83 |
|   512 |     64 |  112 |  64512 |   21.548 |  2661.17 |    6.157 |  1164.16 |   27.706 |  2328.48 |
|   512 |     64 |  128 |  73728 |   24.635 |  2660.23 |    6.365 |  1287.09 |   31.000 |  2378.31 |
|   512 |     64 |  144 |  82944 |   27.716 |  2660.13 |    7.384 |  1248.14 |   35.100 |  2363.09 |
|   512 |     64 |  160 |  92160 |   30.802 |  2659.55 |    7.688 |  1331.86 |   38.491 |  2394.35 |
|   512 |     64 |  186 | 107136 |   38.703 |  2460.56 |    8.963 |  1328.18 |   47.666 |  2247.64 |
|   512 |     64 |  192 | 110592 |   39.155 |  2510.66 |    9.029 |  1360.96 |   48.184 |  2295.22 |
|   512 |     64 |  208 | 119808 |   40.047 |  2659.27 |    9.854 |  1350.91 |   49.901 |  2400.91 |
|   512 |     64 |  224 | 129024 |   43.099 |  2661.06 |   10.151 |  1412.32 |   53.249 |  2423.02 |
|   512 |     64 |  240 | 138240 |   46.167 |  2661.63 |   11.240 |  1366.60 |   57.407 |  2408.08 |
|   512 |     64 |  256 | 147456 |   49.251 |  2661.33 |   11.502 |  1424.50 |   60.752 |  2427.17 |
```





Changed files:
filename: src/llama-cparams.h
patch: @@ -4,7 +4,7 @@
 
 #include <cstdint>
 
-#define LLAMA_MAX_SEQ 64
+#define LLAMA_MAX_SEQ 256
 
 struct llama_cparams {
     uint32_t n_ctx;           // context size used during inference