Repository: ggml-org/llama.cpp
Pull request: vulkan: Enable VK_KHR_cooperative_matrix extension for Intel Xe2 GPUs

Enabling VK_KHR_cooperative_matrix on Intel Xe2 GPUs (currently Lunar Lake and Battlemage) have significant performance improvement, while we also see performance regressions with older GPUs like Arc A770.
This PR will enable VK_KHR_cooperative_matrix only for Xe2 GPUs until performance regression is resolved for older GPUs.

Reference: https://github.com/ggml-org/llama.cpp/discussions/13530

# llama-bench results
OS | Platform | Benchmark | b5589-xe2-enabled | b5583-master | Difference
-- | -- | -- | -- | -- | --
Windows 11 24H2 (gfx driver   32.0.101.6795) | U7-268V | pp512 | 416.53 | 148.60 | 280%
  |   | tg128 | 37.86 | 36.87 | 103%
  | i5-13400 + Arc B580 | pp512 | 1631.44 | 490.89 | 332%
  |   | tg128 | 126.69 | 129.95 | 97%
  | i9-12900K + Arc A770 | pp512 | 977.81 | 974.92 | 100%
  |   | tg128 | 96.83 | 96.42 | 100%
Ubuntu 24.04.2 (Mesa 24.2.8) | U7-268V | pp512 | 167.94 | 122.42 | 137%
  |   | tg128 | 13.67 | 13.75 | 99%
  | i5-13400 + Arc B580 | pp512 | 583.28 | 420.57 | 139%
  |   | tg128 | 41.80 | 41.82 | 100%
  | i9-12900K + Arc A770 | pp512 | 328.25 | 333.88 | 98%
  |   | tg128 | 39.00 | 39.14 | 100%

## Windows
### Lunar Lake Core Ultra 7 268V
#### Before
```
λ llama-bench.exe -m ..\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) Graphics (Intel Corporation) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        148.60 ± 3.52 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         36.87 ± 0.61 |

build: 7e00e60e (5583)
```
#### After
```
λ llama-bench.exe -m ..\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) Graphics (Intel Corporation) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |       416.53 ± 25.15 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         37.86 ± 0.19 |

build: a0dd7795 (5589)
```
### Battlemage Arc B580

#### Before
```
λ llama-bench.exe -m ..\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) B580 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |       490.89 ± 12.28 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |        129.95 ± 0.27 |

build: 7e00e60e (5583)
```
#### After
```
λ llama-bench.exe -m ..\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) B580 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |       1631.44 ± 4.42 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |        126.69 ± 0.50 |

build: a0dd7795 (5589)
```
### Alchemist Arc A770
#### Before
```
λ llama-bench.exe -m C:\Users\cpie-ace\Documents\Axell\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        974.92 ± 4.06 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         96.42 ± 0.40 |

build: 7e00e60e (5583)
```
#### After
```
λ llama-bench.exe -m C:\Users\cpie-ace\Documents\Axell\gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        977.81 ± 1.79 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         96.83 ± 0.30 |

build: a0dd7795 (5589)
```
## Linux
### Lunar Lake Core Ultra 7 268V

#### Before
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (LNL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 131072 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        122.42 ± 0.63 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         13.75 ± 0.01 |

build: 7e00e60e (5583)
```
#### After
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (LNL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 131072 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        167.94 ± 0.63 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         13.67 ± 0.05 |

build: a0dd7795 (5589)
```
### Battlemage Arc B580

#### Before
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 163840 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        420.52 ± 0.19 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         41.82 ± 0.31 |

build: 7e00e60e (5583)
```
#### After
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 163840 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        583.28 ± 0.69 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         41.80 ± 0.27 |

build: a0dd7795 (5589)
```
### Alchemist Arc A770
#### Before
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        333.88 ± 0.34 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         39.14 ± 0.10 |

build: 7e00e60e (5583)

```
#### After
```
$ ./llama-bench -m ~/Downloads/gemma-2-2b-it-Q4_K_M.gguf 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 65536 | int dot: 1 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           pp512 |        328.35 ± 5.95 |
| gemma2 2B Q4_K - Medium        |   1.59 GiB |     2.61 B | Vulkan     |  99 |           tg128 |         39.00 ± 0.24 |

build: a0dd7795 (5589)
```


Changed files:
filename: ggml/src/ggml-vulkan/ggml-vulkan.cpp
patch: @@ -196,6 +196,7 @@ enum vk_device_architecture {
     AMD_RDNA1,
     AMD_RDNA2,
     AMD_RDNA3,
+    INTEL_XE2,
 };
 
 static vk_device_architecture get_device_architecture(const vk::PhysicalDevice& device) {
@@ -246,6 +247,34 @@ static vk_device_architecture get_device_architecture(const vk::PhysicalDevice&
             }
             return vk_device_architecture::AMD_RDNA2;
         }
+    } else if (props.vendorID == VK_VENDOR_ID_INTEL) {
+        const std::vector<vk::ExtensionProperties> ext_props = device.enumerateDeviceExtensionProperties();
+
+        bool subgroup_size_control = false;
+
+        for (const auto& properties : ext_props) {
+            if (strcmp("VK_EXT_subgroup_size_control", properties.extensionName) == 0) {
+                subgroup_size_control = true;
+            }
+        }
+
+        if (!subgroup_size_control) {
+            return vk_device_architecture::OTHER;
+        }
+
+        vk::PhysicalDeviceProperties2 props2;
+        vk::PhysicalDeviceSubgroupSizeControlPropertiesEXT subgroup_size_control_props;
+
+        props2.pNext = &subgroup_size_control_props;
+        device.getProperties2(&props2);
+
+        if (subgroup_size_control_props.minSubgroupSize == 16) {
+            // Xe2 architecture uses SIMD16 while previous Xe and Gen architecture uses SIMD8.
+            // Minimum subgroup size matches the SIMD width so we distinguish architecture by checking this value.
+            // https://www.intel.com/content/www/us/en/content-details/824434/2024-intel-tech-tour-xe2-and-lunar-lake-s-gpu.html
+            // https://www.intel.com/content/www/us/en/docs/oneapi/optimization-guide-gpu/2025-0/intel-xe-gpu-architecture.html
+            return vk_device_architecture::INTEL_XE2;
+        }
     }
     return vk_device_architecture::OTHER;
 }
@@ -10263,8 +10292,9 @@ static bool ggml_vk_instance_portability_enumeration_ext_available(const std::ve
 static bool ggml_vk_khr_cooperative_matrix_support(const vk::PhysicalDeviceProperties& props, const vk::PhysicalDeviceDriverProperties& driver_props, vk_device_architecture arch) {
     switch (props.vendorID) {
     case VK_VENDOR_ID_INTEL:
-        // Intel drivers don't support coopmat properly yet
-        return false;
+        // Only allowing Xe2 GPU at the moment since Xe2 GPU can gain significant performance boost,
+        // while some older hardware (ex. Arc A770) has performance regressions
+        return arch == vk_device_architecture::INTEL_XE2;
     case VK_VENDOR_ID_AMD:
         if (driver_props.driverID == vk::DriverId::eAmdProprietary || driver_props.driverID == vk::DriverId::eAmdOpenSource) {
             // Workaround for AMD proprietary driver reporting support on all GPUs