Repository: vllm-project/vllm

Issue:
Error with docker container `vllm/vllm-openai:v0.3.0` when `--kv-cache-dtype=fp8_e5m2`
When I run the following command, it works as expected:

```
docker run --rm --gpus "device=8" vllm/vllm-openai:v0.3.0 --model=facebook/opt-125m
```

However, when I add `--kv-cache-dtype=fp8_e5m2`, it results in an error:

```
docker run --rm --gpus "device=8" vllm/vllm-openai:v0.3.0 --model=facebook/opt-125m --kv-cache-dtype=fp8_e5m2
```

Error log:
```txt
INFO 02-06 01:02:56 api_server.py:209] args: Namespace(host=None, port=8000, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, served_model_name=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, root_path=None, middleware=[], model='facebook/opt-125m', tokenizer=None, revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='fp8_e5m2', max_model_len=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, block_size=16, seed=0, swap_space=4, gpu_memory_utilization=0.9, max_num_batched_tokens=None, max_num_seqs=256, max_paddings=256, disable_log_stats=False, quantization=None, enforce_eager=False, max_context_len_to_capture=8192, disable_custom_all_reduce=False, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', max_cpu_loras=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
config.json: 100%|██████████| 651/651 [00:00<00:00, 1.39MB/s]
INFO 02-06 01:02:56 utils.py:188] CUDA_HOME is not found in the environment. Using /usr/local/cuda as CUDA_HOME.
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/vllm/entrypoints/openai/api_server.py", line 217, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/workspace/vllm/engine/async_llm_engine.py", line 617, in from_engine_args
    engine_configs = engine_args.create_engine_configs()
  File "/workspace/vllm/engine/arg_utils.py", line 279, in create_engine_configs
    cache_config = CacheConfig(self.block_size,
  File "/workspace/vllm/config.py", line 296, in __init__
    self._verify_cache_dtype()
  File "/workspace/vllm/config.py", line 312, in _verify_cache_dtype
    nvcc_cuda_version = get_nvcc_cuda_version()
  File "/workspace/vllm/utils.py", line 191, in get_nvcc_cuda_version
    nvcc_output = subprocess.check_output([cuda_home + "/bin/nvcc", "-V"],
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/cuda/bin/nvcc'
```


Discussion:
The error happened when checking cuda version in `get_nvcc_cuda_version` function in `vllm/utils.py`. The reason is that `CUDA_HOME` is not set on environment and we try to use `/usr/local/cuda/bin/nvcc` to verify cuda version. But there is not `/usr/local/cuda/bin/nvcc` in vllm/vllm-openai:v0.3.0.

Could you please help me to verify the valid path of `nvcc` in vllm/vllm-openai:v0.3.0? Thanks a lot.
@zhaoyang-star, it seems there is no `nvcc` present either:

```sh
$ docker run --rm -d --name vllm --gpus "device=8" vllm/vllm-openai:v0.3.0 --model=facebook/opt-125m
$ docker exec -it vllm /bin/bash -c "find / | grep nvcc"

/usr/local/lib/python3.10/dist-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/run_nvcc.cmake
```
the vllm docker do not have nvcc
> @zhaoyang-star, it seems there is no `nvcc` present either:
> 
> ```shell
> $ docker run --rm -d --name vllm --gpus "device=8" vllm/vllm-openai:v0.3.0 --model=facebook/opt-125m
> $ docker exec -it vllm /bin/bash -c "find / | grep nvcc"
> 
> /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/run_nvcc.cmake
> ```

You could find `*/bin/nvcc` by  `python -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"`.
@garycaokai @PenutChen 
You could comment Line 296 in [config.py](https://github.com/vllm-project/vllm/blob/main/vllm/config.py#L296). This will ignore the cuda version check.
> You could find `*/bin/nvcc` by `python -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"`.

@zhaoyang-star thank you for the reply. I've tried executing the following command:

```
docker exec -it vllm /usr/bin/python3 -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"
```

This returned `/usr/local/cuda`, but unfortunately, `nvcc` is still not found within that directory:

```
docker exec -it vllm /bin/bash -c 'find /usr/local/cuda | grep nvcc'
```

> You could comment Line 296 in [config.py](https://github.com/vllm-project/vllm/blob/main/vllm/config.py#L296). This will ignore the cuda version check.

A temporary workaround:
```
docker run --rm -d -v $PWD/vllm-config.py:/workspace/vllm/config.py --name vllm --gpus "device=8" vllm/vllm-openai:v0.3.0 --model=facebook/opt-125m --kv-cache-dtype=fp8_e5m2
```

The modified code is save in `vllm-config.py`.

Resolving patch:
diff --git a/vllm/config.py b/vllm/config.py
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -310,7 +310,7 @@ def _verify_cache_dtype(self) -> None:
             pass
         elif self.cache_dtype == "fp8_e5m2":
             nvcc_cuda_version = get_nvcc_cuda_version()
-            if nvcc_cuda_version < Version("11.8"):
+            if nvcc_cuda_version and nvcc_cuda_version < Version("11.8"):
                 raise ValueError(
                     "FP8 is not supported when cuda version is lower than 11.8."
                 )
diff --git a/vllm/utils.py b/vllm/utils.py
--- a/vllm/utils.py
+++ b/vllm/utils.py
@@ -181,13 +181,18 @@ def set_cuda_visible_devices(device_ids: List[int]) -> None:
     os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(map(str, device_ids))
 
 
-def get_nvcc_cuda_version() -> Version:
+def get_nvcc_cuda_version() -> Optional[Version]:
     cuda_home = os.environ.get('CUDA_HOME')
     if not cuda_home:
         cuda_home = '/usr/local/cuda'
-        logger.info(
-            f'CUDA_HOME is not found in the environment. Using {cuda_home} as CUDA_HOME.'
-        )
+        if os.path.isfile(cuda_home + '/bin/nvcc'):
+            logger.info(
+                f'CUDA_HOME is not found in the environment. Using {cuda_home} as CUDA_HOME.'
+            )
+        else:
+            logger.warning(
+                f'Not found nvcc in {cuda_home}. Skip cuda version check!')
+            return None
     nvcc_output = subprocess.check_output([cuda_home + "/bin/nvcc", "-V"],
                                           universal_newlines=True)
     output = nvcc_output.split()


Test patch: