Repository: vllm-project/vllm

Issue:
Got OOM when using TP
I got this when running OPT-13B on two A100s, with FP16. The error didn't occur when I decreased `gpu_memory_utilization` to 0.9. It seems our memory profiling is somehow inaccurate when using TP.

Find the command and error msg below:
```
$ python benchmarks/benchmark_latency.py --model facebook/opt-13b -tp 2
Namespace(model='facebook/opt-13b', tensor_parallel_size=2, input_len=32, output_len=128, batch_size=8, n=1, use_beam_search=False, num_iters=3, profile=False)
2023-06-09 09:17:47,945 INFO worker.py:1625 -- Started a local Ray instance.
INFO 06-09 09:17:48 llm_server.py:60] Initializing an LLM server with config: model='facebook/opt-13b', dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=2, seed=0)
INFO 06-09 09:18:11 llm_server.py:129] # GPU blocks: 4150, # CPU blocks: 655
Traceback (most recent call last):
  File "/home/gcpuser/workspace/cacheflow/benchmarks/benchmark_latency.py", line 80, in <module>
    main(args)
  File "/home/gcpuser/workspace/cacheflow/benchmarks/benchmark_latency.py", line 17, in main
    llm = LLM(
  File "/home/gcpuser/workspace/cacheflow/cacheflow/entrypoints/llm.py", line 55, in __init__
    self.llm_server = LLMServer.from_server_args(server_args)
  File "/home/gcpuser/workspace/cacheflow/cacheflow/server/llm_server.py", line 146, in from_server_args
    server = cls(*server_configs, distributed_init_method, devices,
  File "/home/gcpuser/workspace/cacheflow/cacheflow/server/llm_server.py", line 103, in __init__
    self._init_cache()
  File "/home/gcpuser/workspace/cacheflow/cacheflow/server/llm_server.py", line 135, in _init_cache
    self._run_workers("init_cache_engine", cache_config=self.cache_config)
  File "/home/gcpuser/workspace/cacheflow/cacheflow/server/llm_server.py", line 312, in _run_workers
    all_outputs = ray.get(all_outputs)
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/ray/_private/worker.py", line 2521, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::Worker.init_cache_engine() (pid=11306, ip=10.128.0.79, repr=<cacheflow.worker.worker.Worker object at 0x7f4aa97596d0>)
  File "/home/gcpuser/workspace/cacheflow/cacheflow/worker/worker.py", line 127, in init_cache_engine
    self.cache_engine = CacheEngine(
  File "/home/gcpuser/workspace/cacheflow/cacheflow/worker/cache_engine.py", line 40, in __init__
    self.gpu_cache = self.allocate_gpu_cache()
  File "/home/gcpuser/workspace/cacheflow/cacheflow/worker/cache_engine.py", line 76, in allocate_gpu_cache
    value_blocks = torch.empty(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 326.00 MiB (GPU 0; 39.41 GiB total capacity; 37.08 GiB already allocated; 188.56 MiB free; 37.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```




Discussion:
@zhuohan123 Can you take a look at this? If this is not trivial to fix, I think we can simply set the utilization to 0.9 for now. 

Resolving patch:
diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py
--- a/vllm/engine/arg_utils.py
+++ b/vllm/engine/arg_utils.py
@@ -21,7 +21,7 @@ class EngineArgs:
     tensor_parallel_size: int = 1
     block_size: int = 16
     swap_space: int = 4  # GiB
-    gpu_memory_utilization: float = 0.95
+    gpu_memory_utilization: float = 0.90
     max_num_batched_tokens: int = 2560
     max_num_seqs: int = 256
     disable_log_stats: bool = False


Test patch: