Repository: vllm-project/vllm

Issue:
 Issue Building vllm Wheel: Unsupported GPU Architecture 'compute_89'
I'm trying to build the vllm package from source (pip install vllm), but I'm encountering an error related to CUDA and GPU architecture. I am using RTX 4090. Here are the crucial parts of the log:


```
Building wheels for collected packages: vllm
Building wheel for vllm (pyproject.toml) ... error
error: subprocess-exited-with-error

...

building 'vllm.cache_ops' extension
Emitting ninja build file ...

...

/usr/local/cuda/bin/nvcc  -I/tmp/pip-build-env-... (other flags and paths)
nvcc fatal   : Unsupported gpu architecture 'compute_89'
```

Here is my compute cluster:
---
6790722
Age:15 min.[Remaining](https://vast.ai/faq#Instances):1d 19h71.69.80.64:44536-44750North Carolina, US1x RTX 409081.8  TFLOPS
m:9663host:39244verified0.3/24.6 GB
3506.9 GB/s
ROME2D16
PCIE 4.0, 8x12.5 GB/s
AMD EPYC 7662 64-Core Processor
25.6/256 cpu
4/52 GB
nvme
4972 MB/s6.9/21.0 GB
516.4 Mbps514.0 MbpsMax CUDA: 12.2
 
GPU: 0% 35C , CPU: 4% Status: success, running pytorch/pytorch_2.0.1-cuda11.7-cudnn8-devel/jupyter
$0.263/hr 
Current State : running Next State : running
Docker Image: pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel

Image Runtype : jupyter_direc ssh_direc ssh_proxy

Environment :
{
  "JUPYTER_DIR": "/"
}

Docker Args:

Onstart: env | grep _ >> /etc/environment; echo 'starting up'


---

It seems that the nvcc compiler doesn't support the 'compute_89' GPU architecture. I'm using CUDA version 11.7 on a RTX 4090.

Has anyone encountered this issue before? Any suggestions on how to resolve it?


Discussion:
Upgrading your Cuda version to 11.8 can solve this problem
@ryanshrott As @fengenbao answered, compiling CUDA kernels for Lovelace GPUs (i.e., SM 89) requires CUDA 11.8 or later. ([link](https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/)). You can either upgrade your CUDA version or build vLLM for SM80/86 architectures with CUDA 11.7, since [SM89 is compatible with SM80 and SM86](https://docs.nvidia.com/cuda/ada-compatibility-guide/index.html#applications-built-using-cuda-toolkit-11-0-through-11-7).

For the later option, please insert `compute_capabilities = {80}` below the following line:
https://github.com/vllm-project/vllm/blob/c393af6cd70373ab88b22dbbe59e14bb80ea343d/setup.py#L52


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
 
 if CUDA_HOME is None:
     raise RuntimeError(
-        f"Cannot find CUDA_HOME. CUDA must be available in order to build the package.")
+        f"Cannot find CUDA_HOME. CUDA must be available to build the package.")
 
 
 def get_nvcc_cuda_version(cuda_dir: str) -> Version:
@@ -55,6 +55,14 @@ def get_nvcc_cuda_version(cuda_dir: str) -> Version:
 if 86 in compute_capabilities and nvcc_cuda_version < Version("11.1"):
     raise RuntimeError(
         "CUDA 11.1 or higher is required for GPUs with compute capability 8.6.")
+if 89 in compute_capabilities and nvcc_cuda_version < Version("11.8"):
+    # CUDA 11.8 is required to generate the code targeting compute capability 8.9.
+    # However, GPUs with compute capability 8.9 can also run the code generated by
+    # the previous versions of CUDA 11 and targeting compute capability 8.0.
+    # Therefore, if CUDA 11.8 is not available, we target compute capability 8.0
+    # instead of 8.9.
+    compute_capabilities.remove(89)
+    compute_capabilities.add(80)
 if 90 in compute_capabilities and nvcc_cuda_version < Version("11.8"):
     raise RuntimeError(
         "CUDA 11.8 or higher is required for GPUs with compute capability 9.0.")
@@ -65,6 +73,7 @@ def get_nvcc_cuda_version(cuda_dir: str) -> Version:
     if nvcc_cuda_version >= Version("11.1"):
         compute_capabilities.add(86)
     if nvcc_cuda_version >= Version("11.8"):
+        compute_capabilities.add(89)
         compute_capabilities.add(90)
 
 # Add target compute capabilities to NVCC flags.


Test patch: