Repository: vllm-project/vllm

Issue:
CUDA error: out of memory
I successfully installed vLLM in WSL2, when I was trying to run the sample code, I got error info like this:

```
from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```

INFO 06-21 21:40:02 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 21:40:12 llm_engine.py:128] # GPU blocks: 37375, # CPU blocks: 7281
Traceback (most recent call last):
  File "/mnt/d/01Projects/vllm/prac_1.py", line 11, in <module>
    llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")
  File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in __init__
    self._init_cache()
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
    self._run_workers("init_cache_engine", cache_config=self.cache_config)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
    output = executor(*args, **kwargs)
  File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
    self.cache_engine = CacheEngine(
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in __init__
    self.cpu_cache = self.allocate_cpu_cache()
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
    key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Python: 3.10.11
GPU: RTX 3090 24G
Linux: WSL2, Ubuntu 20.04.6 LTS
Can anyone help to answer this?


Discussion:
The OPT-125M is very small and this cannot happen. While I cannot reproduce the exact error on my side, when you initialize the LLM class, can you try to add the following argument: `gpu_memory_utilization=0.80` or set the utilization to an even lower number? The default utilization upper bound is 0.90.

Additionally, can you set `CUDA_LAUNCH_BLOCKING=1` to see which exact lines causes the error?
I added two lines:
```
import os
os.environ ['CUDA_LAUNCH_BLOCKING'] ='1'
```

and this `gpu_memory_utilization=0.50` .

but it seems the error info is the same:

INFO 06-21 22:53:33 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 22:53:41 llm_engine.py:128] # GPU blocks: 19899, # CPU blocks: 7281
Traceback (most recent call last):
  File "/mnt/d/01Projects/vllm/prac_1.py", line 14, in <module>
    llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m",gpu_memory_utilization=0.50)
  File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in __init__
    self._init_cache()
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
    self._run_workers("init_cache_engine", cache_config=self.cache_config)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
    output = executor(*args, **kwargs)
  File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
    self.cache_engine = CacheEngine(
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in __init__
    self.cpu_cache = self.allocate_cpu_cache()
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
    key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Btw, the text-generation-webui can load the model successfully.
Just to make sure it's not the model's issue, can you run `facebook/opt-125m` model?
I'm running into the same issue. I'm on Ubuntu WSL2, CUDA 11.8, and RTX 2080Ti, which should easily be able to accommodate opt-125m. I've tried other small models, such as cerebras/Cerebras-GPT-111M and distilgpt2, but those give different errors (probably due to architecture incompatibility?). 

_Edit: I just tried EleutherAI/pythia-70M-deduped and it too gives the OOM error._

I've confirmed I can run opt-125m and pythia-70m via standard transformers library...any other ideas what might be causing the issue @zhuohan123  @WoosukKwon ?
> Just to make sure it's not the model's issue, can you run `facebook/opt-125m` model?

Yes, I can use text-generation-webui to load `facebook/opt-125m` and get it work.
Currently having this issue as well. The `CUDA_VISIBLE_DEVICES` environment variable has no effect either, and it only loads the models to GPU 0. I'm running on A100s but still get OOM with 125m OPT. There's something fatally wrong with this repo.

EDIT: `CUDA_VISIBLE_DEVICES` seems to work just fine, but the 125m model is using over 37GB of memory.
@dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?
> @dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

Thanks, I think I understand now. A *somewhat* related question - how is multi-GPU handled? If I load a bigger model, will it split across the available GPUs? Also, if I want more memory to accommodate more users on a single, smaller model, can I have the process use multiple GPUs at once? 
@AlpinDale Good question. You can use the `tensor_parallel_size` argument for multi-GPU inference.
First, initialize your Ray cluster by executing
```bash
$ ray start --head
```

Then, use the `tensor_parallel_size` argument in the LLM class:
```python
llm = LLM(model=<your model>, tensor_parallel_size=2)  # Inference with 2 GPUs 
```




> @dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

I'm pretty sure there's no other process was using my GPU, and here's the screenshot, you can find a burst in both GPU 3D and VRAM, that happended when the code was trying to load the model.
<img width="559" alt="image" src="https://github.com/vllm-project/vllm/assets/34759356/83b75c96-a32b-4630-b772-f3565b16f462">

I'm seeing the same as @SunixLiu. Nothing else is happening on my GPU. Running the code at the top of this issue spikes the GPU and then crashes. 
Any new developments here? @zhuohan123 
@SunixLiu @dcruiz01 We believe this is actually a bug. We are not sure whether this is caused by WSL or vLLM. We don't have a Windows environment with GPUs. How about we set up a 30-min meeting to pair-debug on your environments? Can you send me an email at `zhuohan[at]berkeley.edu` to set up a call? Thanks!
Just pair-debugged with @SunixLiu and we successfully locate the issue. As a temporary fix, please comment out `pin_memory=True` in vLLM code when allocating CPU cache:

https://github.com/vllm-project/vllm/blob/4026a049d3ad510bea8e177bf71722bd510fbb46/vllm/worker/cache_engine.py#L89-L97

`pin_memory` has a limit in WSL ([official doc](https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications)) and the limit [seems to be 2GB](https://www.reddit.com/r/StableDiffusion/comments/y2fca2/does_your_wsl_linux_allow_you_to_pin_more_than/). After commenting out this, vLLM should work properly.
Thank you @zhuohan123 for helping this.
Issue closed.
> Just pair-debugged with @SunixLiu and we successfully locate the issue. As a temporary fix, please comment out `pin_memory=True` in vLLM code when allocating CPU cache:
> 
> https://github.com/vllm-project/vllm/blob/4026a049d3ad510bea8e177bf71722bd510fbb46/vllm/worker/cache_engine.py#L89-L97
> 
> `pin_memory` has a limit in WSL ([official doc](https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications)) and the limit [seems to be 2GB](https://www.reddit.com/r/StableDiffusion/comments/y2fca2/does_your_wsl_linux_allow_you_to_pin_more_than/). After commenting out this, vLLM should work properly.

Hey thank you so much for getting to the bottom of this! Never saw an email, but I'm glad you were able to figure it out with @SunixLiu. Looking forward to testing out the fix.

Resolving patch:
diff --git a/vllm/utils.py b/vllm/utils.py
--- a/vllm/utils.py
+++ b/vllm/utils.py
@@ -1,4 +1,5 @@
 import enum
+from platform import uname
 import uuid
 
 import psutil
@@ -36,3 +37,7 @@ def get_cpu_memory() -> int:
 
 def random_uuid() -> str:
     return str(uuid.uuid4().hex)
+
+def in_wsl() -> bool:
+    # Reference: https://github.com/microsoft/WSL/issues/4071
+    return "microsoft" in " ".join(uname()).lower()
diff --git a/vllm/worker/cache_engine.py b/vllm/worker/cache_engine.py
--- a/vllm/worker/cache_engine.py
+++ b/vllm/worker/cache_engine.py
@@ -5,6 +5,10 @@
 
 from vllm import cache_ops
 from vllm.config import CacheConfig, ModelConfig, ParallelConfig
+from vllm.logger import init_logger
+from vllm.utils import in_wsl
+
+logger = init_logger(__name__)
 
 KVCache = Tuple[torch.Tensor, torch.Tensor]
 
@@ -85,16 +89,22 @@ def allocate_cpu_cache(self) -> List[KVCache]:
         cpu_cache: List[KVCache] = []
         key_block_shape = self.get_key_block_shape()
         value_block_shape = self.get_value_block_shape()
+        pin_memory = not in_wsl()
+        if not pin_memory:
+            # Pinning memory in WSL is not supported.
+            # https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications
+            logger.warn("Using 'pin_memory=False' as WSL is detected. "
+                        "This may slow down the performance.")
         for _ in range(self.num_layers):
             key_blocks = torch.empty(
                 size=(self.num_cpu_blocks, *key_block_shape),
                 dtype=self.dtype,
-                pin_memory=True,
+                pin_memory=pin_memory,
             )
             value_blocks = torch.empty(
                 size=(self.num_cpu_blocks, *value_block_shape),
                 dtype=self.dtype,
-                pin_memory=True,
+                pin_memory=pin_memory,
             )
             cpu_cache.append((key_blocks, value_blocks))
         return cpu_cache


Test patch: