Repository: vllm-project/vllm
Pull request: Fix the openai benchmarking requests to work with latest OpenAI apis
Resolve #2940 

To test the OpenAI benchmarking script, you can run:

```
python benchmark_serving.py --backend openai --base-url https://api.openai.com --endpoint /v1/chat/completions --num-prompts 1 --model gpt-3.5-turbo --tokenizer openai-community/gpt2 --dataset ShareGPT_V3_unfiltered_cleaned_split.json
```

The previous version will have issues to append text when using streaming API. Besides, the previous version does not support `/v1/chat/completions` apis.

Problem statement:
Benchmarking script for openai chat completion api are not supported
When running vllm with openai chat apis, the benchmarking script will fail as it asserts the backend API of `assert api_url.endswith("v1/completions")`.

```
python benchmark_serving.py --backend openai --model mistralai/Mistral-7B-v0.1 --dataset ShareGPT_V3_unfiltered_cleaned_split.json --save-result
```

The logs are as follows:
```
Namespace(backend='openai', version='N/A', base_url=None, host='localhost', port=8000, endpoint='/generate', dataset='ShareGPT_V3_unfiltered_cleaned_split.json', model='mistralai/Mistral-7B-v0.1', tokenizer=None, best_of=1, use_beam_search=False, num_prompts=1000, request_rate=inf, seed=0, trust_remote_code=False, disable_tqdm=False, save_result=True)
  0%|                                                      | 0/1000 [00:00<?, ?it/s]Traffic request rate: inf
Traceback (most recent call last):
  File "/home/chenw/vllm/benchmarks/benchmark_serving.py", line 387, in <module>
    main(args)
  File "/home/chenw/vllm/benchmarks/benchmark_serving.py", line 259, in main
    benchmark_result = asyncio.run(
  File "/home/chenw/miniconda3/envs/myenv/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/chenw/miniconda3/envs/myenv/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/chenw/vllm/benchmarks/benchmark_serving.py", line 195, in benchmark
    outputs = await asyncio.gather(*tasks)
  File "/home/chenw/vllm/benchmarks/backend_request_func.py", line 223, in async_request_openai_completions
    assert api_url.endswith("v1/completions")
AssertionError
  0%|                                                      | 0/1000 [00:00<?, ?it/s]
```

The `backend_request_func.py` should not only allow chat apis like: `assert api_url.endswith("v1/chat/completions")`.

Hints from issue discussion:

Commit message: fix issue #2940

keep the openai backend url as /v1/completions and add openai-chat backend url as /v1/chat/completions

yapf format

add newline

Patch to review:
diff --git a/benchmarks/backend_request_func.py b/benchmarks/backend_request_func.py
index e7f74e2feaf8..8ca77df86d35 100644
--- a/benchmarks/backend_request_func.py
+++ b/benchmarks/backend_request_func.py
@@ -275,10 +275,80 @@ async def async_request_openai_completions(
     return output
 
 
+async def async_request_openai_chat_completions(
+    request_func_input: RequestFuncInput,
+    pbar: Optional[tqdm] = None,
+) -> RequestFuncOutput:
+    api_url = request_func_input.api_url
+    assert api_url.endswith(
+        "v1/chat/completions"
+    ), "OpenAI Chat API URL must end with'v1/chat/completions'"
+
+    async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session:
+        assert not request_func_input.use_beam_search
+        payload = {
+            "model": request_func_input.model,
+            "messages": [
+                {
+                    "role": "user",
+                    "content": request_func_input.prompt,
+                },
+            ],
+            "temperature": 0.0,
+            "max_tokens": request_func_input.output_len,
+            "stream": True,
+        }
+        headers = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"
+        }
+
+        output = RequestFuncOutput()
+        output.prompt_len = request_func_input.prompt_len
+
+        generated_text = ""
+        ttft = 0
+        st = time.perf_counter()
+        try:
+            async with session.post(url=api_url, json=payload,
+                                    headers=headers) as response:
+                if response.status == 200:
+                    async for chunk in response.content:
+                        if ttft == 0:
+                            ttft = time.perf_counter() - st
+                            output.ttft = ttft
+
+                        chunk = chunk.strip()
+                        if not chunk:
+                            continue
+
+                        chunk = chunk.decode("utf-8").lstrip("data: ")
+                        if chunk == "[DONE]":
+                            latency = time.perf_counter() - st
+                        else:
+                            body = json.loads(chunk)
+                            if "content" in body["choices"][0]["delta"]:
+                                generated_text += body["choices"][0]["delta"][
+                                    "content"]
+
+                    output.generated_text = generated_text
+                    output.success = True
+                    output.latency = latency
+                else:
+                    output.success = False
+        except (aiohttp.ClientOSError, aiohttp.ServerDisconnectedError):
+            output.success = False
+
+    if pbar:
+        pbar.update(1)
+    return output
+
+
 ASYNC_REQUEST_FUNCS = {
     "tgi": async_request_tgi,
     "vllm": async_request_vllm,
     "deepspeed-mii": async_request_deepspeed_mii,
     "openai": async_request_openai_completions,
+    "openai-chat": async_request_openai_chat_completions,
     "tensorrt-llm": async_request_trt_llm,
 }


Code review comments:
diff_hunk: @@ -275,10 +275,80 @@ async def async_request_openai_completions(
     return output
 
 
+async def async_request_openai_chat_completions(
+    request_func_input: RequestFuncInput,
+    pbar: Optional[tqdm] = None,
+) -> RequestFuncOutput:
+    api_url = request_func_input.api_url
+    assert api_url.endswith(
+        "v1/chat/completions"
+    ), "OpenAI Chat API URL must end with'v1/chat/completions'"
path: benchmarks/backend_request_func.py
text: @user1:
nit: 
```suggestion
    ), "OpenAI Chat API URL must end with 'v1/chat/completions'."
```