Repository: ggml-org/llama.cpp
Pull request: [CUDA] Reduce CPU-side stalls due to the CUDA command buffer being full

With pipeline parallelism, during prompt processing, the CPU-side CUDA command buffer gets full, stalling the CPU. Due to this, enough work doesn't get submitted to the GPU, resulting in bubbles in the GPU timeline. This PR fixes this by setting the CUDA environment variable CUDA_SCALE_LAUNCH_QUEUES to 4x to increase the command buffer size.

The NSight profile below shows the issue in more detail:

<img width="1958" height="983" alt="image" src="https://github.com/user-attachments/assets/3efdaaf3-dd58-464b-a9d1-3cd31d3f0030" />

After setting the environment variable, this is how the new profile looks:

<img width="1162" height="247" alt="image" src="https://github.com/user-attachments/assets/aa1923da-817b-4e7e-981c-bfcc3bf3f92b" />

The GPU0 is busy for the most part, but there are small bubbles on GPU 1. I think the reason for this is that for a constant batch size, batch n+1 takes more time than batch n due to causal attention. That's why GPU 0 on batch n+1 has more work to do than GPU 1 on batch n. This can be fixed by setting non-uniform tensor-split between GPUs.

Performance Gains
============

- Significant perf improvement of 25% in PP throughout for larger models with pipeline parallelism.
- Less but measurable perf improvement on single GPU for larger models.
- The smaller the GPU and the larger the model, the greater the performance benefit is expected with this environment variable.
- No change in performance for smaller models
- No change in decode phase throughput
- No change in VRAM usage
- ~120 MB higher system RAM usage per GPU. For two GPUs, sys RAM usage increases by 240 MB.

**Pipeline parallelism with 2x RTX Pro 6000 Blackwell GPUs.**

Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf

| ISL  | Env disabled | Env enabled | Speed-up |
-- | -- | -- | --
512 | 1682.79 | 1678.15 | 1.00
1024 | 1884.01 | 2064.24 | 1.10
2048 | 1948.14 | 2289.02 | 1.17
4096 | 1841.07 | 2266.42 | 1.23
8192 | 1563.33 | 1959.12 | 1.25

Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

ISL | Env disabled | Env enabled | Speed-up
-- | -- | -- | --
512 | 11467.79 | 11597.3 | 1.01
1024 | 14371.86 | 14381.97 | 1.00
2048 | 15551.36 | 15537.17 | 1.00
4096 | 14545.61 | 14522.35 | 1.00
8192 | 11896.39 | 11874.36 | 1.00

**Single GPU: RTX Pro 6000 Blackwell**

Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf
ISL | Env disabled | Env enabled | Speed-up
-- | -- | -- | --
512 | 1656.24 | 1686.89 | 1.02
1024 | 1567.71 | 1597.92 | 1.02
2048 | 1455.61 | 1484.2 | 1.02
4096 | 1235.89 | 1314.54 | 1.06
8192 | 953.8 | 976.03 | 1.02

Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
ISL | Env disabled | Env enabled | Speed-up
-- | -- | -- | --
512 | 12109.8 | 12031.88 | 0.99
1024 | 11426.8 | 11426.9 | 1.00
2048 | 10589.84 | 10594.17 | 1.00
4096 | 8902.05 | 8909.34 | 1.00
8192 | 6874.66 | 6872.47 | 1.00


Changed files:
filename: docs/build.md
patch: @@ -248,6 +248,14 @@ You may set the [cuda environmental variables](https://docs.nvidia.com/cuda/cuda
 CUDA_VISIBLE_DEVICES="-0" ./build/bin/llama-server --model /srv/models/llama.gguf
 ```
 
+#### CUDA_SCALE_LAUNCH_QUEUES
+
+The environment variable [`CUDA_SCALE_LAUNCH_QUEUES`](https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/environment-variables.html#cuda-scale-launch-queues) controls the size of CUDA's command buffer, which determines how many GPU operations can be queued before the CPU must wait for the GPU to catch up. A larger buffer reduces CPU-side stalls and allows more work to be queued on a GPU.
+
+**Default behavior:** llama.cpp automatically sets `CUDA_SCALE_LAUNCH_QUEUES=4x`, which increases the CUDA command buffer to 4 times its default size. This optimization is particularly beneficial for **Multi-GPU setups with pipeline parallelism**, where it significantly improves prompt processing throughput by allowing more operations to be enqueued across GPUs.
+
+See PR [#19042](https://github.com/ggml-org/llama.cpp/pull/19042) for performance benchmarks and technical details.
+
 ### Unified Memory
 
 The environment variable `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` can be used to enable unified memory in Linux. This allows swapping to system RAM instead of crashing when the GPU VRAM is exhausted. In Windows this setting is available in the NVIDIA control panel as `System Memory Fallback`.

filename: ggml/src/ggml-cuda/ggml-cuda.cu
patch: @@ -4876,6 +4876,16 @@ ggml_backend_reg_t ggml_backend_cuda_reg() {
         static std::mutex mutex;
         std::lock_guard<std::mutex> lock(mutex);
         if (!initialized) {
+            // Set CUDA_SCALE_LAUNCH_QUEUES before any CUDA API call to improve multi-GPU pipeline parallelism performance
+            // PR: https://github.com/ggml-org/llama.cpp/pull/19042
+            if (getenv("CUDA_SCALE_LAUNCH_QUEUES") == nullptr) {
+#ifdef _WIN32
+                _putenv_s("CUDA_SCALE_LAUNCH_QUEUES", "4x");
+#else
+                setenv("CUDA_SCALE_LAUNCH_QUEUES", "4x", 0); // don't overwrite if already set
+#endif // _WIN32
+            }
+
             ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context;
             const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;