Repository: vllm-project/vllm
Pull request: [Fix] Do not pin memory when in WSL
Fix #188


Problem statement:
CUDA error: out of memory
I successfully installed vLLM in WSL2, when I was trying to run the sample code, I got error info like this:

```
from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```

INFO 06-21 21:40:02 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 21:40:12 llm_engine.py:128] # GPU blocks: 37375, # CPU blocks: 7281
Traceback (most recent call last):
  File "/mnt/d/01Projects/vllm/prac_1.py", line 11, in <module>
    llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")
  File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in __init__
    self._init_cache()
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
    self._run_workers("init_cache_engine", cache_config=self.cache_config)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
    output = executor(*args, **kwargs)
  File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
    self.cache_engine = CacheEngine(
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in __init__
    self.cpu_cache = self.allocate_cpu_cache()
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
    key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Python: 3.10.11
GPU: RTX 3090 24G
Linux: WSL2, Ubuntu 20.04.6 LTS
Can anyone help to answer this?

Hints from issue discussion:
The OPT-125M is very small and this cannot happen. While I cannot reproduce the exact error on my side, when you initialize the LLM class, can you try to add the following argument: `gpu_memory_utilization=0.80` or set the utilization to an even lower number? The default utilization upper bound is 0.90.

Additionally, can you set `CUDA_LAUNCH_BLOCKING=1` to see which exact lines causes the error?
I added two lines:
```
import os
os.environ ['CUDA_LAUNCH_BLOCKING'] ='1'
```

and this `gpu_memory_utilization=0.50` .

but it seems the error info is the same:

INFO 06-21 22:53:33 llm_engine.py:59] Initializing an LLM engine with config: model='/mnt/d/github/text-generation-webui/models/facebook_opt-125m', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 22:53:41 llm_engine.py:128] # GPU blocks: 19899, # CPU blocks: 7281
Traceback (most recent call last):
  File "/mnt/d/01Projects/vllm/prac_1.py", line 14, in <module>
    llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m",gpu_memory_utilization=0.50)
  File "/mnt/d/github/vllm/vllm/entrypoints/llm.py", line 55, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 145, in from_engine_args
    engine = cls(*engine_configs, distributed_init_method, devices,
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 102, in __init__
    self._init_cache()
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 134, in _init_cache
    self._run_workers("init_cache_engine", cache_config=self.cache_config)
  File "/mnt/d/github/vllm/vllm/engine/llm_engine.py", line 307, in _run_workers
    output = executor(*args, **kwargs)
  File "/mnt/d/github/vllm/vllm/worker/worker.py", line 126, in init_cache_engine
    self.cache_engine = CacheEngine(
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 41, in __init__
    self.cpu_cache = self.allocate_cpu_cache()
  File "/mnt/d/github/vllm/vllm/worker/cache_engine.py", line 89, in allocate_cpu_cache
    key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Btw, the text-generation-webui can load the model successfully.
Just to make sure it's not the model's issue, can you run `facebook/opt-125m` model?
I'm running into the same issue. I'm on Ubuntu WSL2, CUDA 11.8, and RTX 2080Ti, which should easily be able to accommodate opt-125m. I've tried other small models, such as cerebras/Cerebras-GPT-111M and distilgpt2, but those give different errors (probably due to architecture incompatibility?). 

_Edit: I just tried EleutherAI/pythia-70M-deduped and it too gives the OOM error._

I've confirmed I can run opt-125m and pythia-70m via standard transformers library...any other ideas what might be causing the issue @zhuohan123  @WoosukKwon ?
> Just to make sure it's not the model's issue, can you run `facebook/opt-125m` model?

Yes, I can use text-generation-webui to load `facebook/opt-125m` and get it work.
Currently having this issue as well. The `CUDA_VISIBLE_DEVICES` environment variable has no effect either, and it only loads the models to GPU 0. I'm running on A100s but still get OOM with 125m OPT. There's something fatally wrong with this repo.

EDIT: `CUDA_VISIBLE_DEVICES` seems to work just fine, but the 125m model is using over 37GB of memory.
@dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?
> @dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

Thanks, I think I understand now. A *somewhat* related question - how is multi-GPU handled? If I load a bigger model, will it split across the available GPUs? Also, if I want more memory to accommodate more users on a single, smaller model, can I have the process use multiple GPUs at once? 
@AlpinDale Good question. You can use the `tensor_parallel_size` argument for multi-GPU inference.
First, initialize your Ray cluster by executing
```bash
$ ray start --head
```

Then, use the `tensor_parallel_size` argument in the LLM class:
```python
llm = LLM(model=<your model>, tensor_parallel_size=2)  # Inference with 2 GPUs 
```




> @dcruiz01 @SunixLiu @AlpinDale vLLM is designed to take almost all of your GPU memory. Could you double-check your GPU is not used by other processes when using vLLM?

I'm pretty sure there's no other process was using my GPU, and here's the screenshot, you can find a burst in both GPU 3D and VRAM, that happended when the code was trying to load the model.
<img width="559" alt="image" src="https://github.com/vllm-project/vllm/assets/34759356/83b75c96-a32b-4630-b772-f3565b16f462">

Commit message: [Fix] Do not pin memory when in WSL

Patch to review:
diff --git a/vllm/utils.py b/vllm/utils.py
index 85fe18778244..868bb9b30c0c 100644
--- a/vllm/utils.py
+++ b/vllm/utils.py
@@ -1,4 +1,5 @@
 import enum
+from platform import uname
 import uuid
 
 import psutil
@@ -36,3 +37,6 @@ def get_cpu_memory() -> int:
 
 def random_uuid() -> str:
     return str(uuid.uuid4().hex)
+
+def in_wsl() -> bool:
+    return "microsoft" in " ".join(uname()).lower()
diff --git a/vllm/worker/cache_engine.py b/vllm/worker/cache_engine.py
index 30b4ec7d79cf..badd24bea289 100644
--- a/vllm/worker/cache_engine.py
+++ b/vllm/worker/cache_engine.py
@@ -5,6 +5,10 @@
 
 from vllm import cache_ops
 from vllm.config import CacheConfig, ModelConfig, ParallelConfig
+from vllm.logger import init_logger
+from vllm.utils import in_wsl
+
+logger = init_logger(__name__)
 
 KVCache = Tuple[torch.Tensor, torch.Tensor]
 
@@ -85,16 +89,21 @@ def allocate_cpu_cache(self) -> List[KVCache]:
         cpu_cache: List[KVCache] = []
         key_block_shape = self.get_key_block_shape()
         value_block_shape = self.get_value_block_shape()
+        pin_memory = not in_wsl()
+        if not pin_memory:
+            # Pinning memory in WSL is not supported.
+            # https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications
+            logger.warn("Running in WSL, not pinning CPU cache memory.")
         for _ in range(self.num_layers):
             key_blocks = torch.empty(
                 size=(self.num_cpu_blocks, *key_block_shape),
                 dtype=self.dtype,
-                pin_memory=True,
+                pin_memory=pin_memory,
             )
             value_blocks = torch.empty(
                 size=(self.num_cpu_blocks, *value_block_shape),
                 dtype=self.dtype,
-                pin_memory=True,
+                pin_memory=pin_memory,
             )
             cpu_cache.append((key_blocks, value_blocks))
         return cpu_cache


Code review comments:
diff_hunk: @@ -85,16 +89,21 @@ def allocate_cpu_cache(self) -> List[KVCache]:
         cpu_cache: List[KVCache] = []
         key_block_shape = self.get_key_block_shape()
         value_block_shape = self.get_value_block_shape()
+        pin_memory = not in_wsl()
+        if not pin_memory:
+            # Pinning memory in WSL is not supported.
+            # https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications
+            logger.warn("Running in WSL, not pinning CPU cache memory.")
path: vllm/worker/cache_engine.py
text: @user1:
nit: Can we provide more information like this? Also, I'd like to avoid the term "CPU cache memory", as it can be misleading to users.
```suggestion
            logger.warn("Using 'pin_memory=False' as WSL is detected. This may slow down the performance.")
```