Repository: ggml-org/llama.cpp
Pull request: CUDA: build archs as virtual for GGML_NATIVE=OFF

See https://github.com/ggml-org/ggml/issues/1154 .

CMake give you the option to build CUDA architectures as either real or virtual or both (default). My understanding is that if at runtime a real architecture is present it can be used directly, otherwise JIT compilation is used to create the binary code if a suiting virtual architecture is present. However, the CUDA architectures we define are the lowest possible ones for the features that we use and as a result the compiled real architectures basically never see any use. So we may as well skip them to speed up the compilation process and reduce binary size.

On my systems binary size and total compilation time of llama.cpp without CCache and with `GGML_NATIVE=OFF` change as follows:

| Setup  | Size of ggml-cuda.so [MiB] | CT Epyc 7742 (64C/128T) [s] | CT Xeon E5-2683 v4 (16C/32T) [s] |
|--------|----------------------------|-----------------------------|----------------------------------|
| master |                        114 |                         324 |                              562 |
| PR     |                         60 |                         222 |                              406 |

The difference seems to be particularly noticeable on CPUs with more cores since the compilation of the entire program spends a long time waiting for only 2 CUDA kernels.

Changed files:
filename: ggml/src/ggml-cuda/CMakeLists.txt
patch: @@ -12,12 +12,30 @@ if (CUDAToolkit_FOUND)
         # 61     == Pascal, __dp4a instruction (per-byte integer dot product)
         # 70     == V100, FP16 tensor cores
         # 75     == Turing, int8 tensor cores
+        # 80     == Ampere, asynchronous data loading, faster tensor core instructions
+        # 86     == RTX 3000, needs CUDA v11.1
+        # 89     == RTX 4000, needs CUDA v11.8
+        #
+        # XX-virtual == compile CUDA code as PTX, do JIT compilation to binary code on first run
+        # XX-real    == compile CUDA code as device code for this specific architecture
+        # no suffix  == compile as both PTX and device code
+        #
+        # The default behavior for a non-native is to build virtual architectures as needed to cover all features needed
+        #     for best performance and to also build real architectures for the most commonly used GPUs.
         if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24")
             set(CMAKE_CUDA_ARCHITECTURES "native")
         elseif(GGML_CUDA_F16 OR GGML_CUDA_DMMV_F16)
-            set(CMAKE_CUDA_ARCHITECTURES "60;61;70;75;80")
+            if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
+                set(CMAKE_CUDA_ARCHITECTURES "60-virtual;61-virtual;70-virtual;75-virtual;80-virtual;86-real;89-real")
+            else()
+                set(CMAKE_CUDA_ARCHITECTURES "60-virtual;61-virtual;70-virtual;75-virtual;80-virtual;86-real")
+            endif()
         else()
-            set(CMAKE_CUDA_ARCHITECTURES "50;61;70;75;80")
+            if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
+                set(CMAKE_CUDA_ARCHITECTURES "50-virtual;61-virtual;70-virtual;75-virtual;80-virtual;86-real;89-real")
+            else()
+                set(CMAKE_CUDA_ARCHITECTURES "50-virtual;61-virtual;70-virtual;75-virtual;80-virtual;86-real")
+            endif()
         endif()
     endif()
     message(STATUS "Using CUDA architectures: ${CMAKE_CUDA_ARCHITECTURES}")