Repository: modelscope/evalscope (language: python)

Issue:
并发请求感觉并不对
## 问题描述 / Issue Description

在华为国产服务器使用流式压测兼容 openai api 后端（这里使用的是 mindie），无论是使用在python代码还是命令使用，无论参数--rate还是--parallel还是全部使用，启用并发感觉没有完全并发。

代码详见下文。在启用流式，保持总数16的情况下，分别测试rate为8、parallel为8、rate和parallel为8的情况：

rate为8：请求花费106s

parallel为8：请求花费56s

rate和parallel为8：请求花费56s

自己写的并发请求代码（偷懒，没启动流式）：请求花费20s

rate和parallel为8但不启用流式：请求花费56s

## 使用的工具 / Tools Used

- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend 
- [ ] RAGEval backend
- [x] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

```python
#在保持下列command_line_args一样的情况下，分别测试了加入rate为8、parallel为8、rate和parallel为8的情况。
command_line_args = [
            "--url", "http://127.0.0.1:1025/v1/chat/completions",
            "-n", "16",
            "--model", "qwen",
            "--log-every-n-query", "10",
            "--read-timeout", "120",
            "--max-prompt-length", "8192",
            "--api", "openai",
            "--query-template",
            '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}',
            "--dataset", "openqa",
            "--tokenizer-path", "/mnt/data/model/Qwen/Qwen2-7B-Instruct/Qwen/Qwen2-7B-Instruct",
            "--max-tokens", "8192",
            "--stream",
            "--debug"
        ]
        task_cfg = command_line_args_to_dict(command_line_args)
        run_perf_benchmark(task_cfg)
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [ ] Ubuntu
  - [x] 华为闭源欧拉

- Python版本 / Python Version:
  - [ ] 3.11
  - [x] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

版本为1128晚上从modelscope clone并pip install -e .的最新版

四个情况下的请求日志如下
[python_only_rate&para8_no_stream.log](https://github.com/user-attachments/files/17940422/python_only_rate.para8_no_stream.log)
[python_only_para8.log](https://github.com/user-attachments/files/17940423/python_only_para8.log)
[python_only_rate&para8.log](https://github.com/user-attachments/files/17940424/python_only_rate.para8.log)
[python_only_rate8.log](https://github.com/user-attachments/files/17940425/python_only_rate8.log)




Discussion / hints:
请问你自己写的并发请求是怎么实现的呢
> 请问你自己写的并发请求是怎么实现的呢

[参考这个链接](https://blog.csdn.net/arkohut/article/details/139076652)，在这个代码基础上改成读取openqa数据集，并发挨个取openqa数据集。
我自己测试的时候感觉也是有点小问题，一是并非是完全并发，二是TTFT那里计算感觉不太对
> 我自己测试的时候感觉也是有点小问题，一是并非是完全并发，二是TTFT那里计算感觉不太对

并发请求这里我再看一下；TTFT这里只有stream模式才生效

Fix patch:
diff --git a/evalscope/perf/benchmark.py b/evalscope/perf/benchmark.py
index 9b1d0c7..c3ae0bb 100644
--- a/evalscope/perf/benchmark.py
+++ b/evalscope/perf/benchmark.py
@@ -88,16 +88,16 @@ async def dispatch_requests_from_dataset():
 @exception_handler
 async def send_requests_worker(
     task_id,
-    client: AioHttpClient,
     request_queue: asyncio.Queue,
     benchmark_data_queue: asyncio.Queue,
     args: Arguments,
 ):
+    client = AioHttpClient(args)
     async with client:
         while not (query_send_completed_event.is_set() and request_queue.empty()):
             try:
                 # Attempt to get a request from the queue with a timeout
-                request = await asyncio.wait_for(request_queue.get(), timeout=0.001)
+                request = await asyncio.wait_for(request_queue.get(), timeout=0.0001)
                 request_queue.task_done()
             except asyncio.TimeoutError:
                 # If timeout, continue to the next iteration
@@ -189,7 +189,7 @@ async def statistic_benchmark_metric_worker(benchmark_data_queue: asyncio.Queue,
 
 
 @exception_handler
-async def create_client(args: Arguments) -> bool:
+async def start_server(args: Arguments) -> bool:
     if args.api.startswith('local'):
         #  start local server
         server = threading.Thread(target=start_app, args=(copy.deepcopy(args), ), daemon=True)
@@ -201,12 +201,9 @@ async def create_client(args: Arguments) -> bool:
             args.url = 'http://127.0.0.1:8877/v1/chat/completions'
         args.model = os.path.basename(args.model)
 
-    client = AioHttpClient(args)
-    if not await test_connection(client, args):
+    if not await test_connection(args):
         raise TimeoutError('Test connection failed')
 
-    return client
-
 
 @exception_handler
 async def benchmark(args: Arguments) -> None:
@@ -217,20 +214,20 @@ async def benchmark(args: Arguments) -> None:
     request_queue = asyncio.Queue()
     benchmark_data_queue = asyncio.Queue()
 
-    async def create_send_request_tasks(client):
+    async def create_send_request_tasks():
         tasks: List[asyncio.Task] = []
         for idx in range(args.parallel):
-            task = asyncio.create_task(send_requests_worker(idx, client, request_queue, benchmark_data_queue, args))
+            task = asyncio.create_task(send_requests_worker(idx, request_queue, benchmark_data_queue, args))
             tasks.append(task)
         return tasks
 
     async def run_tasks():
-        client = await create_client(args)
+        await start_server(args)
 
         dispatch_task = asyncio.create_task(dispatch_requests_worker(request_queue, args))
         statistic_benchmark_metric_task = asyncio.create_task(
             statistic_benchmark_metric_worker(benchmark_data_queue, args))
-        send_request_tasks = await create_send_request_tasks(client)
+        send_request_tasks = await create_send_request_tasks()
 
         expected_number_of_queries = await dispatch_task
         await request_queue.join()
diff --git a/evalscope/perf/http_client.py b/evalscope/perf/http_client.py
index b81ab6c..cda4cff 100644
--- a/evalscope/perf/http_client.py
+++ b/evalscope/perf/http_client.py
@@ -140,22 +140,24 @@ async def on_response_chunk_received(session, context, params: aiohttp.TraceResp
         logger.debug(f'Request received: <{method=},  {url=}, {truncated_chunk=}>')
 
 
-async def test_connection(client: AioHttpClient, args: Arguments) -> bool:
+async def test_connection(args: Arguments) -> bool:
     is_error = True
-    timeout = args.connect_timeout
     start_time = time.perf_counter()
 
     async def attempt_connection():
-        if 'chat/completions' in args.url:
-            request = {'messages': [{'role': 'user', 'content': 'hello'}], 'model': args.model}
-        else:
-            request = {'prompt': 'hello', 'model': args.model}
-        async for is_error, state_code, response_data in client.post(request):
-            return is_error, state_code, response_data
+        client = AioHttpClient(args)
+        async with client:
+            if 'chat/completions' in args.url:
+                request = {'messages': [{'role': 'user', 'content': 'hello'}], 'model': args.model}
+            else:
+                request = {'prompt': 'hello', 'model': args.model}
+            async for is_error, state_code, response_data in client.post(request):
+                return is_error, state_code, response_data
 
     while True:
         try:
-            is_error, state_code, response_data = await asyncio.wait_for(attempt_connection(), timeout=timeout)
+            is_error, state_code, response_data = await asyncio.wait_for(
+                attempt_connection(), timeout=args.connect_timeout)
             if not is_error:
                 logger.info('Connection successful.')
                 return True
@@ -163,7 +165,7 @@ async def attempt_connection():
         except Exception as e:
             logger.warning(f'Retrying... <{e}>')
 
-        if time.perf_counter() - start_time >= timeout:
+        if time.perf_counter() - start_time >= args.connect_timeout:
             logger.error('Overall connection attempt timed out.')
             return False
 


Test patch:
diff --git a/docs/en/user_guides/stress_test/examples.md b/docs/en/user_guides/stress_test/examples.md
index 8259d11..0a55d82 100644
--- a/docs/en/user_guides/stress_test/examples.md
+++ b/docs/en/user_guides/stress_test/examples.md
@@ -152,8 +152,27 @@ When starting, add the following parameters:
 ## Debugging Requests
 Use the `--debug` option to output the requests and responses.
 
+**Non-`stream` Mode Output Example**
+
 ```text
 2024-11-27 11:25:34,161 - evalscope - http_client.py - on_request_start - 116 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8000/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
 2024-11-27 11:25:34,163 - evalscope - http_client.py - on_request_chunk_sent - 128 - DEBUG - Request sent: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "qwen2.5"}'>
-2024-11-27 11:25:38,172 - evalscope - http_client.py - on_response_chunk_received - 140 - DEBUG - Response received: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"id":"cmpl-a4565eb4fc6b4a5697f38c0adaf9b70b","object":"text_completion","created":1732677934,"model":"qwen2.5","choices":[{"index":0,"text":"Hello, everyone! Today I will tell you a lie. ))\\n\\nToday is a happy day.","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":1,"total_tokens":17,"completion_tokens":16}}'>
+2024-11-27 11:25:38,172 - evalscope - http_client.py - on_response_chunk_received - 140 - DEBUG - Request received: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"id":"cmpl-a4565eb4fc6b4a5697f38c0adaf9b70b","object":"text_completion","created":1732677934,"model":"qwen2.5","choices":[{"index":0,"text":"，everyone！今天我给您撒个谎哦。 ))\\n\\n今天开心的事。","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":1,"total_tokens":17,"completion_tokens":16}}'>
+```
+
+**`stream` Mode Output Example**
+
+```text
+2024-11-27 20:02:24,760 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"重要的"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,803 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,847 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"，以便"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,890 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"及时"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,933 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"得到"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,976 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"帮助"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,023 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"和支持"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,066 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,109 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,111 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"。
"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":50,"completion_tokens":260,"total_tokens":310}}
+2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: [DONE]
 ```
\ No newline at end of file
diff --git a/docs/en/user_guides/stress_test/parameters.md b/docs/en/user_guides/stress_test/parameters.md
index acddc31..1fad90a 100644
--- a/docs/en/user_guides/stress_test/parameters.md
+++ b/docs/en/user_guides/stress_test/parameters.md
@@ -23,8 +23,13 @@ Execute `evalscope perf --help` to get a full parameter description:
 
 ## Request Control
 - `--number`: Number of requests sent, default is None, meaning requests are sent based on the dataset size.
-- `--parallel`: Set the number of concurrent requests, default is 1.
-- `--rate`: Number of requests per second, default is None. If set to -1, all requests are sent at time 0. Otherwise, requests are synthesized using a Poisson process. Mutually exclusive with parallel.
+- `--parallel` sets the number of workers for concurrent requests, with a default of 1.
+- `--rate` specifies the number of requests generated per second (not sent), with a default of -1, indicating that all requests will be generated at time 0 with no interval; otherwise, we use a Poisson process to generate request intervals.
+  ```{tip}
+  In the implementation of this tool, request generation and sending are separated:
+  The `--rate` parameter is used to control the number of requests generated per second, and these requests will be placed in a request queue.
+  The `--parallel` parameter is used to control the number of workers sending requests; workers will take requests from the queue and send them, only sending the next request after receiving a response for the previous one. It is not recommended to set both parameters simultaneously; thus, in this tool, the `--rate` parameter is only effective when `--parallel` is set to 1.
+  ```
 - `--log-every-n-query`: Log every n queries, default is 10.
 - `--stream`: Use SSE stream output, default is False.
 
diff --git a/docs/zh/user_guides/stress_test/examples.md b/docs/zh/user_guides/stress_test/examples.md
index bf3f7d9..8934c45 100644
--- a/docs/zh/user_guides/stress_test/examples.md
+++ b/docs/zh/user_guides/stress_test/examples.md
@@ -157,8 +157,27 @@ pip install wandb
 ## 调试请求
 使用 `--debug` 选项，我们将输出请求和响应，输出示例如下：
 
+**非`stream`模式输出示例**
+
 ```text
 2024-11-27 11:25:34,161 - evalscope - http_client.py - on_request_start - 116 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8000/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
 2024-11-27 11:25:34,163 - evalscope - http_client.py - on_request_chunk_sent - 128 - DEBUG - Request sent: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "qwen2.5"}'>
 2024-11-27 11:25:38,172 - evalscope - http_client.py - on_response_chunk_received - 140 - DEBUG - Request received: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"id":"cmpl-a4565eb4fc6b4a5697f38c0adaf9b70b","object":"text_completion","created":1732677934,"model":"qwen2.5","choices":[{"index":0,"text":"，everyone！今天我给您撒个谎哦。 ))\\n\\n今天开心的事。","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":1,"total_tokens":17,"completion_tokens":16}}'>
+```
+
+**`stream`模式输出示例**
+
+```text
+2024-11-27 20:02:24,760 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"重要的"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,803 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,847 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"，以便"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,890 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"及时"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,933 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"得到"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:24,976 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"帮助"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,023 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"和支持"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,066 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,109 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,111 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"。
"},"finish_reason":null}],"usage":null}
+2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":50,"completion_tokens":260,"total_tokens":310}}
+2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: [DONE]
 ```
\ No newline at end of file
diff --git a/docs/zh/user_guides/stress_test/parameters.md b/docs/zh/user_guides/stress_test/parameters.md
index e1a1ab6..97885eb 100644
--- a/docs/zh/user_guides/stress_test/parameters.md
+++ b/docs/zh/user_guides/stress_test/parameters.md
@@ -24,8 +24,13 @@
 
 ## 请求控制
 - `--number` 发出的请求数量，默认为None，表示基于数据集数量发送请求。
-- `--parallel` 设置并发请求的数量，默认为1。
-- `--rate` 每秒请求的数量，默认为None。如果设置为-1，则所有请求将在时间0发送。否则，我们使用泊松过程合成请求到达时间。与parallel互斥。
+- `--parallel` 设置并发请求的worker数量，默认为1。
+- `--rate` 每秒生成的请求数量（并不发送请求），默认为-1，表示所有请求将在时间0生成，没有间隔；否则，我们使用泊松过程生成请求间隔。
+  ```{tip}
+  在本工具的实现中请求生成与发送是分开的：
+  `--rate`参数用于控制每秒生成的请求数量，请求将放入请求队列。
+  `--parallel`参数用于控制发送请求的worker数量，worker将从请求队列获取请求并发送，且在上一请求回复后才发送下一请求。 两个参数不建议同时设置，也即在本工具中，`--rate`参数仅对`--parallel`为1时有效。
+  ```
 - `--log-every-n-query` 每n个查询记录日志，默认为10。
 - `--stream` 使用SSE流输出，默认为False。