Repository: vllm-project/vllm
Pull request: [Fix] Add chat completion Example and simplify dependencies
Fix #545 #537

Make fastchat an optional dependency giving the huge dependency list of fastchat and many complaints on installation.

Problem statement:
Slow installation:  install all versions of one package.
I follow the document and it will install all versions of one package.
```
  Downloading fonttools-4.24.0-py3-none-any.whl (853 kB)
     |████████████████████████████████| 853 kB 22.8 MB/s
  Downloading fonttools-4.23.1-py3-none-any.whl (853 kB)
     |████████████████████████████████| 853 kB 24.7 MB/s
  Downloading fonttools-4.23.0-py3-none-any.whl (852 kB)
     |████████████████████████████████| 852 kB 23.7 MB/s
  Downloading fonttools-4.22.1-py3-none-any.whl (850 kB)
     |████████████████████████████████| 850 kB 70.3 MB/s
  Downloading fonttools-4.22.0-py3-none-any.whl (850 kB)
     |████████████████████████████████| 850 kB 45.1 MB/s
INFO: pip is looking at multiple versions of cycler to determine which version is compatible with other requirements. This could take a while.
Collecting cycler>=0.10
```

Hints from issue discussion:
Same here. 

I am running `pip install vllm` from the docker container created with `docker run --gpus all -it --rm --shm-size=8g nvcr.io/nvidia/pytorch:22.12-py3` (and after running `pip uninstall torch`)

```
INFO: pip is looking at multiple versions of contourpy to determine which version is compatible with other requirements. This could take a while.
Collecting contourpy>=1.0.1
  Downloading contourpy-1.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
     |████████████████████████████████| 300 kB 46.6 MB/s
  Downloading contourpy-1.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
     |████████████████████████████████| 300 kB 37.3 MB/s
INFO: pip is looking at multiple versions of cycler to determine which version is compatible with other requirements. This could take a while.
  Downloading contourpy-1.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
     |████████████████████████████████| 295 kB 58.0 MB/s
  Downloading contourpy-1.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (295 kB)
     |████████████████████████████████| 295 kB 47.0 MB/s
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading contourpy-1.0.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (285 kB)
     |████████████████████████████████| 285 kB 64.5 MB/s
  Downloading contourpy-1.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (272 kB)
     |████████████████████████████████| 272 kB 46.6 MB/s
  Downloading contourpy-1.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (270 kB)
     |████████████████████████████████| 270 kB 44.7 MB/s
INFO: pip is looking at multiple versions of contourpy to determine which version is compatible with other requirements. This could take a while.
  Downloading contourpy-1.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (269 kB)
     |████████████████████████████████| 269 kB 45.0 MB/s
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
Collecting matplotlib~=3.0
  Downloading matplotlib-3.7.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
     |████████████████████████████████| 9.2 MB 49.6 MB/s
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading matplotlib-3.7.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
     |████████████████████████████████| 9.2 MB 29.1 MB/s


```
Upgrading pip might help

or ~adding this pip parameter: `--use-deprecated=legacy-resolver`~ (got an error with legacy resolver `ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.`)


https://stackoverflow.com/questions/65122957/resolving-new-pip-backtracking-runtime-issue


> 

Thank you . 
I upgrade to pip23.2, but there is still some problems.
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency
conflicts.
torch-tensorrt 1.3.0a0 requires torch<1.15.0,>=1.13.0dev, but you have torch 2.0.1 which is incompatible.
torchtext 0.13.0a0+fae8e8c requires torch==1.14.0a0+410ce96, but you have torch 2.0.1 which is incompatible.
torchvision 0.15.0a0 requires torch==1.14.0a0+410ce96, but you have torch 2.0.1 which is incompatible.
```
I was able to install (under 10 minutes) and succesfully run vllm following this: 
1. Run from inside a docker container: https://vllm.readthedocs.io/en/latest/getting_started/installation.html
2. pip uninstall torch
3. install miniconda, create a conda environment as shown in the page above
4. Inside conda environment: pip install vllm
> I was able to install (under 10 minutes) and succesfully run vllm following this:
> 
> 1. Run from inside a docker container: https://vllm.readthedocs.io/en/latest/getting_started/installation.html
> 2. pip uninstall torch
> 3. install miniconda, create a conda environment as shown in the page above
> 4. Inside conda environment: pip install vllm

Thank you for your reply. In my practice, I uninstall three packages like torchtext. And it seems to work.

Commit message: fix

Patch to review:
diff --git a/examples/openai_client.py b/examples/openai_client.py
index cf7223d4c143..bcc41065bd86 100644
--- a/examples/openai_client.py
+++ b/examples/openai_client.py
@@ -3,26 +3,48 @@
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai.api_key = "EMPTY"
 openai.api_base = "http://localhost:8000/v1"
-model = "facebook/opt-125m"
 
-# Test list models API
+# List models API
 models = openai.Model.list()
 print("Models:", models)
 
-# Test completion API
-stream = True
+model = models["data"][0]["id"]
+
+# Completion API
+stream = False
 completion = openai.Completion.create(
     model=model,
     prompt="A robot may not injure a human being",
     echo=False,
     n=2,
-    best_of=3,
     stream=stream,
     logprobs=3)
 
-# print the completion
+print("Completion results:")
 if stream:
     for c in completion:
         print(c)
 else:
-    print("Completion result:", completion)
+    print(completion)
+
+# Chat completion API
+chat_completion = openai.ChatCompletion.create(
+    model=model,
+    messages=[{
+        "role": "system",
+        "content": "You are a helpful assistant."
+    }, {
+        "role": "user",
+        "content": "Who won the world series in 2020?"
+    }, {
+        "role":
+        "assistant",
+        "content":
+        "The Los Angeles Dodgers won the World Series in 2020."
+    }, {
+        "role": "user",
+        "content": "Where was it played?"
+    }])
+
+print("Chat completion results:")
+print(chat_completion)
diff --git a/requirements.txt b/requirements.txt
index 42dfbeeb316b..f9f3c787bd32 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -9,4 +9,3 @@ xformers >= 0.0.19
 fastapi
 uvicorn
 pydantic < 2  # Required for OpenAI server.
-fschat  # Required for OpenAI ChatCompletion Endpoint.
diff --git a/vllm/entrypoints/openai/api_server.py b/vllm/entrypoints/openai/api_server.py
index 373c4812264a..b1519f437b42 100644
--- a/vllm/entrypoints/openai/api_server.py
+++ b/vllm/entrypoints/openai/api_server.py
@@ -13,9 +13,6 @@
 from fastapi.exceptions import RequestValidationError
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, StreamingResponse
-from fastchat.conversation import Conversation, SeparatorStyle
-from fastchat.model.model_adapter import get_conversation_template
-
 import uvicorn
 
 from vllm.engine.arg_utils import AsyncEngineArgs
@@ -33,6 +30,13 @@
 from vllm.transformers_utils.tokenizer import get_tokenizer
 from vllm.utils import random_uuid
 
+try:
+    from fastchat.conversation import Conversation, SeparatorStyle
+    from fastchat.model.model_adapter import get_conversation_template
+    _fastchat_available = True
+except ImportError:
+    _fastchat_available = False
+
 TIMEOUT_KEEP_ALIVE = 5  # seconds
 
 logger = init_logger(__name__)
@@ -63,6 +67,9 @@ async def check_model(request) -> Optional[JSONResponse]:
 
 
 async def get_gen_prompt(request) -> str:
+    assert _fastchat_available, (
+        "fastchat is not installed. Please install fastchat to use "
+        "the chat completion and conversation APIs: `$ pip install fschat`")
     conv = get_conversation_template(request.model)
     conv = Conversation(
         name=conv.name,


Code review comments:
diff_hunk: @@ -63,6 +67,9 @@ async def check_model(request) -> Optional[JSONResponse]:
 
 
 async def get_gen_prompt(request) -> str:
+    assert _fastchat_available, (
path: vllm/entrypoints/openai/api_server.py
text: @user1:
nit: As this is user-facing error, I feel raising error is a bit more appropriate.

> assert is used to ensure internal correctness, not to enforce correct usage nor to indicate that some unexpected event occurred. If an exception is desired in the latter cases, use a raise statement.