Repository: modelscope/evalscope (language: python)

Issue:
evalscope 5.5 stream.日志不起作用
evalscope perf --url 'http://127.0.0.1:9600/v1/chat/completions' \
--parallel 128 --model 'Qwen2___5-32B-Instruct' --log-every-n-query 10 --read-timeout=360 \
--dataset-path '/home/ai/.cache/evalscope/open_qa.jsonl' -n 2048 --max-prompt-length 128000 \
--tokenizer-path '/home/ai/hub/Qwen2___5-32B-Instruct' \
--api openai --stop '
' --stream --dataset openqa

使用者命令 
<img width="1439" alt="image" src="https://github.com/user-attachments/assets/5d9d0856-7a6d-401b-9264-deb08ea9760e">
一致在输出流式的结果。5.2 是可以的


Discussion / hints:

Fix patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 5bf22aae..9683a9f9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,7 +6,8 @@ repos:
         exclude: |
             (?x)^(
                 thirdparty/|
-                examples/
+                examples/|
+                evalscope/utils/utils.py
             )$
   - repo: https://github.com/PyCQA/isort.git
     rev: 4.3.21
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 00000000..520caecf
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,4 @@
+include README.md
+include requirements.txt
+recursive-include evalscope *
+recursive-include requirements *
diff --git a/README.md b/README.md
index a3811ef0..4c76f053 100644
--- a/README.md
+++ b/README.md
@@ -7,14 +7,15 @@
 </p>
 
 <p align="center">
-<a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
-<a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope">
-</a>
-<a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'>
-    <img src='https://readthedocs.org/projects/evalscope-en/badge/?version=latest' alt='Documentation Status' />
-</a>
-<br>
- <a href="https://evalscope.readthedocs.io/en/latest/">📖 Documents</a>
+  <a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
+  <a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope">
+  </a>
+  <a href="https://github.com/modelscope/evalscope/pulls"><img src="https://img.shields.io/badge/PR-welcome-55EB99.svg"></a>
+  <a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'>
+      <img src='https://readthedocs.org/projects/evalscope-en/badge/?version=latest' alt='Documentation Status' />
+  </a>
+  <br>
+  <a href="https://evalscope.readthedocs.io/en/latest/">📖 Documents</a>
 <p>
 
 > ⭐ If you like this project, please click the "Star" button at the top right to support us. Your support is our motivation to keep going!
@@ -55,6 +56,7 @@ The architecture includes the following modules:
 
 
 ## 🎉 News
+- 🔥 **[2024.11.26]** The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html).
 - 🔥 **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [📖 Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.
 - 🔥 **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.
 - 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
@@ -124,31 +126,47 @@ We recommend using conda to manage your environment and installing dependencies
 ### 1. Simple Evaluation
 To evaluate a model using default settings on specified datasets, follow the process below:
 
-#### Install using pip
-You can execute this command from any directory:
+#### Installation using pip
+
+You can execute this in any directory:
 ```bash
 python -m evalscope.run \
- --model qwen/Qwen2-0.5B-Instruct \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
  --template-type qwen \
- --datasets arc
+ --datasets gsm8k ceval \
+ --limit 10
 ```
 
-#### Install from source
-Execute this command in the `evalscope` directory:
+#### Installation from source
+
+You need to execute this in the `evalscope` directory:
 ```bash
 python evalscope/run.py \
- --model qwen/Qwen2-0.5B-Instruct \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
  --template-type qwen \
- --datasets arc
+ --datasets gsm8k ceval \
+ --limit 10
 ```
 
-If prompted with `Do you wish to run the custom code? [y/N]`, please type `y`.
+> If prompted with `Do you wish to run the custom code? [y/N]`, please type `y`.
+
+**Results (tested with only 10 samples)**
+```text
+Report table:
++-----------------------+--------------------+-----------------+
+| Model                 | ceval              | gsm8k           |
++=======================+====================+=================+
+| Qwen2.5-0.5B-Instruct | (ceval/acc) 0.5577 | (gsm8k/acc) 0.5 |
++-----------------------+--------------------+-----------------+
+```
 
 
 #### Basic Parameter Descriptions
 - `--model`: Specifies the `model_id` of the model on [ModelScope](https://modelscope.cn/), allowing automatic download. For example, see the [Qwen2-0.5B-Instruct model link](https://modelscope.cn/models/qwen/Qwen2-0.5B-Instruct/summary); you can also use a local path, such as `/path/to/model`.
 - `--template-type`: Specifies the template type corresponding to the model. Refer to the `Default Template` field in the [template table](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-datasets.html#llm) for filling in this field.
 - `--datasets`: The dataset name, allowing multiple datasets to be specified, separated by spaces; these datasets will be automatically downloaded. Refer to the [supported datasets list](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html) for available options.
+- `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation.
+
 
 ### 2. Parameterized Evaluation
 If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands:
@@ -188,7 +206,7 @@ In addition to the three [basic parameters](#basic-parameter-descriptions), the
 - `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format, where the key is the dataset name and the value is the parameter; note that these must correspond one-to-one with the values in `--datasets`.
   - `--few_shot_num`: Number of few-shot examples.
   - `--few_shot_random`: Whether to randomly sample few-shot data; if not specified, defaults to `true`.
-- `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation.
+
 
 ### 3. Use the run_task Function to Submit an Evaluation Task
 Using the `run_task` function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields:
@@ -233,6 +251,32 @@ EvalScope supports using third-party evaluation frameworks to initiate evaluatio
 - **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
 - **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
 
+
+## Model Serving Performance Evaluation
+A stress testing tool focused on large language models, which can be customized to support various dataset formats and different API protocol formats.
+
+Reference: Performance Testing [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html)
+
+**Supports wandb for recording results**
+
+![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
+
+**Supports Speed Benchmark**
+
+It supports speed testing and provides speed benchmarks similar to those found in the [official Qwen](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html) reports:
+
+```text
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |      50.69      |      0.97      |
+|     6144      |      51.36      |      1.23      |
+|     14336     |      49.93      |      1.59      |
+|     30720     |      49.56      |      2.34      |
++---------------+-----------------+----------------+
+```
+
 ## Custom Dataset Evaluation
 EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [📖User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html)
 
@@ -247,10 +291,6 @@ The Arena mode allows multiple candidate models to be evaluated through pairwise
 
 Refer to: Arena Mode [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html)
 
-## Model Serving Performance Evaluation
-A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats.
-
-Refer to : Model Serving Performance Evaluation [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html)
 
 
 
diff --git a/README_zh.md b/README_zh.md
index fba70647..e89d2f8b 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -29,7 +29,7 @@
 - [自定义数据集评测](#自定义数据集评测)
 - [离线环境评测](#离线环境评测)
 - [竞技场模式](#竞技场模式)
-- [性能评测工具](#性能评测工具)
+- [性能评测工具](#推理性能评测工具)
 
 
 
@@ -63,6 +63,7 @@ EvalScope包括以下模块：
 
 
 ## 🎉 新闻
+- 🔥 **[2024.11.26]** 模型推理压测工具重构完成：支持本地启动推理服务、支持Speed Benchmark；优化异步调用错误处理，参考[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test.html)
 - 🔥 **[2024.10.31]** 多模态RAG评测最佳实践发布，参考[📖博客](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag)
 - 🔥 **[2024.10.23]** 支持多模态RAG评测，包括[CLIP_Benchmark](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/clip_benchmark.html)评估图文检索器，以及扩展了[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)以支持端到端多模态指标评估。
 - 🔥 **[2024.10.8]** 支持RAG评测，包括使用[MTEB/CMTEB](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html)进行embedding模型和reranker的独立评测，以及使用[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)进行端到端评测。
@@ -138,9 +139,10 @@ pip install -e '.[all]'           # 安装所有 backends (Native, OpenCompass,
 可在任意路径下执行：
 ```bash
 python -m evalscope.run \
- --model qwen/Qwen2-0.5B-Instruct \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
  --template-type qwen \
- --datasets arc
+ --datasets gsm8k ceval \
+ --limit 10
 ```
 
 #### 使用源码安装
@@ -148,18 +150,29 @@ python -m evalscope.run \
 在`evalscope`路径下执行：
 ```bash
 python evalscope/run.py \
- --model qwen/Qwen2-0.5B-Instruct \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
  --template-type qwen \
- --datasets arc
+ --datasets gsm8k ceval \
+ --limit 10
 ```
 
-如遇到 `Do you wish to run the custom code? [y/N]` 请键入 `y`
+> 如遇到 `Do you wish to run the custom code? [y/N]` 请键入 `y`
 
+**运行结果（只使用了10个样例测试）**
+```text
+Report table:
++-----------------------+--------------------+-----------------+
+| Model                 | ceval              | gsm8k           |
++=======================+====================+=================+
+| Qwen2.5-0.5B-Instruct | (ceval/acc) 0.5577 | (gsm8k/acc) 0.5 |
++-----------------------+--------------------+-----------------+
+```
 
 #### 基本参数说明
 - `--model`: 指定了模型在[ModelScope](https://modelscope.cn/)中的`model_id`，可自动下载，例如[Qwen2-0.5B-Instruct模型链接](https://modelscope.cn/models/qwen/Qwen2-0.5B-Instruct/summary)；也可使用模型的本地路径，例如`/path/to/model`
 - `--template-type`: 指定了模型对应的模板类型，参考[模板表格](https://swift.readthedocs.io/zh-cn/latest/Instruction/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html#id4)中的`Default Template`字段填写
 - `--datasets`: 数据集名称，支持输入多个数据集，使用空格分开，数据集将自动下载，支持的数据集参考[数据集列表](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset.html)
+- `--limit`: 每个数据集最大评估数据量，不填写则默认为全部评估，可用于快速验证评估流程
 
 
 ### 2. 带参数评估
@@ -188,7 +201,7 @@ python evalscope/run.py \
 ```
 
 #### 参数说明
-除开上述三个[基本参数](#基本参数说明)，其他参数如下：
+除开上述的[基本参数](#基本参数说明)，其他参数如下：
 - `--model-args`: 模型加载参数，以逗号分隔，key=value形式
 - `--generation-config`: 生成参数，以逗号分隔，key=value形式
   - `do_sample`: 是否使用采样，默认为`false`
@@ -200,7 +213,6 @@ python evalscope/run.py \
 - `--dataset-args`: 评估数据集的设置参数，以json格式传入，key为数据集名称，value为参数，注意需要跟`--datasets`参数中的值一一对应
   - `--few_shot_num`: few-shot的数量
   - `--few_shot_random`: 是否随机采样few-shot数据，如果不设置，则默认为`true`
-- `--limit`: 每个数据集最大评估数据量，不填写则默认为全部评估，可用于快速验证
 
 
 ### 3. 使用run_task函数提交评估任务
@@ -249,6 +261,33 @@ EvalScope支持使用第三方评测框架发起评测任务，我们称之为
 - **RAGEval**：通过EvalScope作为入口，发起RAG评测任务，支持使用[MTEB/CMTEB](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html)进行embedding模型和reranker的独立评测，以及使用[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)进行端到端评测：[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/index.html)
 - **ThirdParty**: 第三方评估任务，如[ToolBench](https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolbench.html)、[LongBench-Write](https://evalscope.readthedocs.io/zh-cn/latest/third_party/longwriter.html)。
 
+## 推理性能评测工具
+一个专注于大型语言模型的压力测试工具，可以自定义以支持各种数据集格式和不同的API协议格式。
+
+参考：性能测试[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test.html)
+
+**支持wandb记录结果**
+
+![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
+
+**支持Speed Benchmark**
+
+支持速度测试，得到类似[Qwen官方](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html)报告的速度基准：
+
+```text
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |      50.69      |      0.97      |
+|     6144      |      51.36      |      1.23      |
+|     14336     |      49.93      |      1.59      |
+|     30720     |      49.56      |      2.34      |
++---------------+-----------------+----------------+
+```
+
+
+
 ## 自定义数据集评测
 EvalScope支持自定义数据集评测，具体请参考：自定义数据集评测[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset.html)
 
@@ -260,11 +299,6 @@ EvalScope支持自定义数据集评测，具体请参考：自定义数据集
 竞技场模式允许多个候选模型通过两两对比(pairwise battle)的方式进行评估，并可以选择借助AI Enhanced Auto-Reviewer（AAR）自动评估流程或者人工评估的方式，最终得到评估报告。参考：竞技场模式[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/arena.html)
 
 
-## 性能评测工具
-一个专注于大型语言模型的压力测试工具，可以自定义以支持各种数据集格式和不同的API协议格式。参考：性能测试[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test.html)
-
-
-
 ## TO-DO List
 - [x] RAG evaluation
 - [x] VLM evaluation
diff --git a/evalscope/cli/start_perf.py b/evalscope/cli/start_perf.py
index a103a4cd..a546c630 100644
--- a/evalscope/cli/start_perf.py
+++ b/evalscope/cli/start_perf.py
@@ -1,20 +1,21 @@
 # Copyright (c) Alibaba, Inc. and its affiliates.
-from abc import abstractmethod
-import os, sys, time
+import os
 from argparse import ArgumentParser
-import subprocess
-
 
 from evalscope.cli.base import CLICommand
-from evalscope.perf.http_client import add_argument, run_perf_benchmark
+from evalscope.perf.arguments import add_argument
+from evalscope.perf.main import run_perf_benchmark
 
 current_path = os.path.dirname(os.path.abspath(__file__))
 root_path = os.path.dirname(current_path)
+
+
 def subparser_func(args):
     """ Function which will be called for a specific sub parser.
     """
     return PerfBenchCMD(args)
-        
+
+
 class PerfBenchCMD(CLICommand):
     name = 'perf'
 
@@ -28,10 +29,6 @@ def define_args(parsers: ArgumentParser):
         parser = parsers.add_parser(PerfBenchCMD.name)
         add_argument(parser)
         parser.set_defaults(func=subparser_func)
-        
+
     def execute(self):
         run_perf_benchmark(self.args)
-        
-        
-        
-        
\ No newline at end of file
diff --git a/evalscope/metrics/rouge_metric.py b/evalscope/metrics/rouge_metric.py
index 6b342805..79b2487b 100644
--- a/evalscope/metrics/rouge_metric.py
+++ b/evalscope/metrics/rouge_metric.py
@@ -9,16 +9,24 @@
 
 from evalscope.constants import MetricsConstant
 from evalscope.metrics.bundled_rouge_score import rouge_scorer
-from evalscope.preprocess.tokenizers.gpt2_tokenizer import DummyTokenizer
+
 from rouge_chinese import Rouge
 import jieba
 
+
+class DummyTokenizer:
+
+    def tokenize(self, text: str):
+        return text.split()
+
+
 HERE = Path(__file__).absolute().parent
 
 logger = logging.getLogger(__name__)
 
-scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'],
-                                  tokenizer=DummyTokenizer())
+scorer = rouge_scorer.RougeScorer(
+    ['rouge1', 'rouge2', 'rougeL'], tokenizer=DummyTokenizer()
+)
 zh_scorer = Rouge()
 
 
@@ -28,6 +36,7 @@ def is_contains_chinese(strs):
             return True
     return False
 
+
 def compute_rouge_score(predict_l, reference_l):
     assert len(predict_l) == len(reference_l)
     if len(predict_l) == 0:
@@ -43,10 +52,14 @@ def compute_rouge_score(predict_l, reference_l):
             result[rouge_key].append(one_sample[rouge_key])
     rlt = {}
     for rouge_key in MetricsConstant.ROUGE_KEYS:
-        rlt[rouge_key] = mean(result[rouge_key]) * 100 if rouge_key in result \
+        rlt[rouge_key] = (
+            mean(result[rouge_key]) * 100
+            if rouge_key in result
             else MetricsConstant.INVALID_VALUE
+        )
     return rlt
 
+
 def compute_rouge_score_one_sample_zh(predict, reference):
     result = dict()
     for p, r in zip(predict, reference):
@@ -63,9 +76,10 @@ def compute_rouge_score_one_sample_zh(predict, reference):
         result['rouge-l-r'] = score['rouge-l']['r']
         result['rouge-l-p'] = score['rouge-l']['p']
         result['rouge-l-f'] = score['rouge-l']['f']
-    
+
     return result
 
+
 def compute_rouge_score_one_sample(predict, reference):
     result = dict()
     for p, r in zip(predict, reference):
@@ -97,11 +111,9 @@ def _to_table(final_result) -> str:
             if not task:
                 continue
             elif task == 'total':
-                row.append(
-                    f'{final_result["total"]["rouge"][rouge_key] :0.2f}')
+                row.append(f'{final_result["total"]["rouge"][rouge_key] :0.2f}')
             else:
-                row.append(
-                    f'{final_result["tasks"][task]["rouge"][rouge_key] :0.2f}')
+                row.append(f'{final_result["tasks"][task]["rouge"][rouge_key] :0.2f}')
         table.append('\t'.join(row))
 
     return '\n'.join(table)
@@ -111,19 +123,22 @@ def run_rouge_eval(data_l, md_level=2, report_metric_key='rouge-l-f'):
     print(f"{'#' * md_level} Rouge Eval")
     for data in tqdm(data_l):
         data['rouge'] = compute_rouge_score_one_sample(
-            data['gen_tok_str'], data['reference_tok_str'])
+            data['gen_tok_str'], data['reference_tok_str']
+        )
     task_data_d = defaultdict(list)
     for data in data_l:
         for task in data['task_tags']:
             task_data_d[task].append(data)
 
     total_rouge = mean([data['rouge'][report_metric_key] for data in data_l])
-    print(f'[total], count: {len(data_l)}, {report_metric_key}: '
-          f'{total_rouge * 100:0.2f}%')
+    print(
+        f'[total], count: {len(data_l)}, {report_metric_key}: '
+        f'{total_rouge * 100:0.2f}%'
+    )
 
     for task, task_data in task_data_d.items():
-        task_rouge = mean(
-            [data['rouge'][report_metric_key] for data in task_data])
+        task_rouge = mean([data['rouge'][report_metric_key] for data in task_data])
         print(
             f'[{task}], count: {len(task_data_d[task])}, {report_metric_key}: '
-            f'{task_rouge * 100:0.2f}%')
+            f'{task_rouge * 100:0.2f}%'
+        )
diff --git a/evalscope/perf/README.md b/evalscope/perf/README.md
deleted file mode 100644
index b039012a..00000000
--- a/evalscope/perf/README.md
+++ /dev/null
@@ -1,302 +0,0 @@
-# Quick Start
-A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats.
-## Usage
-
-### Command line  
-```bash
-evalscope perf --help
-usage: evalscope <command> [<args>] perf [-h] --model MODEL [--url URL] [--connect-timeout CONNECT_TIMEOUT] [--read-timeout READ_TIMEOUT] [-n NUMBER] [--parallel PARALLEL] [--rate RATE]
-                                       [--log-every-n-query LOG_EVERY_N_QUERY] [--headers KEY1=VALUE1 [KEY1=VALUE1 ...]] [--wandb-api-key WANDB_API_KEY] [--name NAME] [--debug] [--tokenizer-path TOKENIZER_PATH]
-                                       [--api API] [--max-prompt-length MAX_PROMPT_LENGTH] [--min-prompt-length MIN_PROMPT_LENGTH] [--prompt PROMPT] [--query-template QUERY_TEMPLATE] [--dataset DATASET]
-                                       [--dataset-path DATASET_PATH] [--frequency-penalty FREQUENCY_PENALTY] [--logprobs] [--max-tokens MAX_TOKENS] [--n-choices N_CHOICES] [--seed SEED] [--stop STOP] [--stream]
-                                       [--temperature TEMPERATURE] [--top-p TOP_P]
-
-options:
-  -h, --help            show this help message and exit
-  --model MODEL         The test model name.
-  --url URL
-  --connect-timeout CONNECT_TIMEOUT
-                        The network connection timeout
-  --read-timeout READ_TIMEOUT
-                        The network read timeout
-  -n NUMBER, --number NUMBER
-                        How many requests to be made, if None, will will send request base dataset or prompt.
-  --parallel PARALLEL   Set number of concurrency request, default 1
-  --rate RATE           Number of requests per second. default None, if it set to -1,then all the requests are sent at time 0. Otherwise, we use Poisson process to synthesize the request arrival times. Mutual exclusion
-                        with parallel
-  --log-every-n-query LOG_EVERY_N_QUERY
-                        Logging every n query.
-  --headers KEY1=VALUE1 [KEY1=VALUE1 ...]
-                        Extra http headers accepts by key1=value1 key2=value2. The headers will be use for each query.You can use this parameter to specify http authorization and other header.
-  --wandb-api-key WANDB_API_KEY
-                        The wandb api key, if set the metric will be saved to wandb.
-  --name NAME           The wandb db result name and result db name, default: {model_name}_{current_time}
-  --debug               Debug request send.
-  --tokenizer-path TOKENIZER_PATH
-                        Specify the tokenizer weight path, used to calculate the number of input and output tokens,usually in the same directory as the model weight.
-  --api API             Specify the service api, current support [openai|dashscope]you can define your custom parser with python, and specify the python file path, reference api_plugin_base.py,
-  --max-prompt-length MAX_PROMPT_LENGTH
-                        Maximum input prompt length
-  --min-prompt-length MIN_PROMPT_LENGTH
-                        Minimum input prompt length.
-  --prompt PROMPT       Specified the request prompt, all the query will use this prompt, You can specify local file via @file_path, the prompt will be the file content.
-  --query-template QUERY_TEMPLATE
-                        Specify the query template, should be a json string, or local file,with local file, specified with @local_file_path,will will replace model and prompt in the template.
-  --dataset DATASET     Specify the dataset [openqa|longalpaca|line_by_line]you can define your custom dataset parser with python, and specify the python file path, reference dataset_plugin_base.py,
-  --dataset-path DATASET_PATH
-                        Path to the dataset file, Used in conjunction with dataset. If dataset is None, each line defaults to a prompt.
-  --frequency-penalty FREQUENCY_PENALTY
-                        The frequency_penalty value.
-  --logprobs            The logprobs.
-  --max-tokens MAX_TOKENS
-                        The maximum number of tokens can be generated.
-  --n-choices N_CHOICES
-                        How may chmpletion choices to generate.
-  --seed SEED           The random seed.
-  --stop STOP           The stop generating tokens.
-  --stop-token-ids      Set the stop token ids.
-  --stream              Stream output with SSE.
-  --temperature TEMPERATURE
-                        The sample temperature.
-  --top-p TOP_P         Sampling top p.
-
-```
-### The result:
-```bash
- Total requests: 10
- Succeed requests: 10
- Failed requests: 0
- Average QPS: 0.256
- Average latency: 3.859
- Throughput(average output tokens per second): 23.317
- Average time to first token: 0.007
- Average input tokens per request: 21.800
- Average output tokens per request: 91.100
- Average time per output token: 0.04289
- Average package per request: 93.100
- Average package latency: 0.042
- Percentile of time to first token: 
-     p50: 0.0021
-     p66: 0.0023
-     p75: 0.0025
-     p80: 0.0030
-     p90: 0.0526
-     p95: 0.0526
-     p98: 0.0526
-     p99: 0.0526
- Percentile of request latency: 
-     p50: 3.9317
-     p66: 3.9828
-     p75: 4.0153
-     p80: 7.2801
-     p90: 7.7003
-     p95: 7.7003
-     p98: 7.7003
-     p99: 7.7003
-```
-
-#### Metrics Description
-| **Metrics**                                     | **Description**                       | **Values**        |
-|------------------------------------------|-----------------------------|-----------------|
-| Total requests                          | Total requests                     | 10              |
-| Succeed requests                        | Succeed requests                 | 10              |
-| Failed requests                         | Failed requests                  | 0               |
-| Average QPS                            | Average queries per second             | 0.256           |
-| Average latency                         | Average latency for total requests                    | 3.859           |
-| Throughput                              | Output tokens per second        | 23.317          |
-| Average time to first token            | Average time to first token            | 0.007           |
-| Average input tokens per request        | Average input tokens per request    | 21.800          |
-| Average output tokens per request       | Average output tokens per request    | 91.100          |
-| Average time per output token           | Average time per output token      | 0.04289         |
-| Average package per request             | Average package per request         | 93.100          |
-| Average package latency                  | Average package latency                   | 0.042           |
-| Percentile of time to first token (p50, ..., p99)     | Percentile of time to first token      |   p50=0.0021, ..., p99=0.0526         |
-| Percentile of request latency (p50, ..., p99)          | Percentile of request latency          |  p50=3.9317, ..., p99=7.7003         |
-
-
-### Request parameter  
-You can set request parameter's in query-template and with (--stop,--stream,--temperature, etc), the argument parameter will replace or add to the request.
-#### request with parameters
-Sample request llama3 vllm openai compatible interface.
-```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '
' --dataset openqa --debug
-```
-evalscope perf 'http://host:port/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 -n 10000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
-
-#### query-template usage.
-When you need to process more complex requests, you can use templates to simplify the command line.
-If both the template and the parameter are present, the value in the parameter will prevail.
-Query-template example:
-```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 12 --model 'llama3' --log-every-n-query 10 --read-timeout=120 -n 1 --max-prompt-length 128000 --api openai --query-template '{"model": "%m", "messages": [], "stream": true, "stream_options":{"include_usage": true},"n": 3, "stop_token_ids": [128001, 128009]}' --dataset openqa --dataset-path './datasets/open_qa.jsonl'
-```
-For messages, the dataset processor message will replace messages in the query-template.
-
-#### Start the client
-```bash
-# test openai service
-evalscope perf --url 'https://api.openai.com/v1/chat/completions' --parallel 1 --headers 'Authorization=Bearer YOUR_OPENAI_API_KEY' --model 'gpt-4o' --dataset-path 'THE_DATA_TO/open_qa.jsonl'  --log-every-n-query 10 --read-timeout=120  -n 100 --max-prompt-length 128000 --api openai --stream --dataset openqa
-```
-
-```bash
-##### open qa dataset and 
-#### dataset address: https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl
-evalscope perf --url 'http://IP:PORT/v1/chat/completions' --parallel 1 --model 'qwen' --log-every-n-query 1 --read-timeout=120 -n 1000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
-```
-
-#### How to log metrics to wandb
---wandb-api-key 'your_wandb_api_key'  --name 'name_of_wandb_and_result_db'  
-
-![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
-
-#### How to debug
---debug 
-with --debug option, we will output the request and response.
-
-#### How to analysis result.
-The tool will save all data during the test to the sqlite3 database, including requests and responses. You can analyze the test data after the test.
-```python
-import sqlite3
-import base64
-import pickle
-import json
-result_db_path = 'db_name.db'
-con = sqlite3.connect(result_db_path)
-query_sql = "SELECT request, response_messages, prompt_tokens, completion_tokens \
-                FROM result WHERE success='True'"
-# how to save base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"), 
-with con:
-    rows = con.execute(query_sql).fetchall()
-    if len(rows) > 0:
-        for row in rows:
-            request = row[0]
-            responses = row[1]
-            request = base64.b64decode(request)
-            request = pickle.loads(request)
-            responses = base64.b64decode(responses)
-            responses = pickle.loads(responses)
-            response_content = ''
-            for response in responses:
-                response = json.loads(response)
-                response_content += response['choices'][0]['delta']['content']
-            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' % (request['messages'][0]['content'], row[2], response_content, row[3]))
-```
-
-#### Api supported
-Currently supports openai, dashscope, zhipu API request. You can specify api with --api.
-You can custom your request with --query-template, you can specify a json string as:
-'{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}'
-or a local file with @to_query_template_path. We will replace %m with model and %p with prompt.
-
-#### How to extend API
-To extend api you can create sub class of `ApiPluginBase`, annotation with @register_api("name_of_api")
-with build_request build request via model, prompt, and
-query_template. you can reference opanai_api.py
-parse_responses return number_of_prompt_tokens and number_of_completion_tokens.
-```python
-class ApiPluginBase:
-    def __init__(self, model_path: str) -> None:
-        self.model_path = model_path
-        
-    @abstractmethod
-    def build_request(self, messages: List[Dict], param: QueryParameters)->Dict:
-        """Build a api request body.
-
-        Args:
-            messages (List[Dict]): The messages generated by dataset.
-            param (QueryParameters): The query parameters.
-
-        Raises:
-            NotImplementedError: Not implemented.
-
-        Returns:
-            Dict: The api request body.
-        """
-        raise NotImplementedError
-    
-    @abstractmethod
-    def parse_responses(self, 
-                        responses: List, 
-                        request: Any=None,
-                        **kwargs:Any) -> Tuple[int, int]:
-        """Parser responses and return number of request and response tokens.
-
-        Args:
-            responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, each is bytes, for general only one. 
-            request (Any): The request body.
-
-        Returns:
-            Tuple: (Number of prompt_tokens and number of completion_tokens).
-        """
-        raise NotImplementedError  
-```
-
-#### Dataset supported
-Currently supports line by line, longalpaca and openqa data set.
-line by line with each line as a prompt.
-longalpaca will get item['instruction'] as prompt.
-openqa will get item['question'] as prompt.
-
-#### How to extension dataset.
-To extend api you can create sub class of `DatasetPluginBase`, annotation with @register_dataset('name_of_dataset')
-implement build_prompt api return a prompt.
-```python
-class DatasetPluginBase:
-    def __init__(self, query_parameters: QueryParameters):
-        """Build data set plugin
-
-        Args:
-            dataset_path (str, optional): The input dataset path. Defaults to None.
-        """
-        self.query_parameters = query_parameters
-
-    def __next__(self):
-        for item in self.build_messages():
-            yield item
-        raise StopIteration
-
-    def __iter__(self):
-        return self.build_messages()
-    
-    @abstractmethod
-    def build_messages(self)->Iterator[List[Dict]]:
-        """Build the request.
-
-        Raises:
-            NotImplementedError: The request is not impletion.
-
-        Yields:
-            Iterator[List[Dict]]: Yield request messages.
-        """
-        raise NotImplementedError
-    
-    def dataset_line_by_line(self, dataset: str)->Iterator[str]:
-        """Get content line by line of dataset.
-
-        Args:
-            dataset (str): The dataset path.
-
-        Yields:
-            Iterator[str]: Each line of file.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            for line in f:
-                yield line
-    
-    def dataset_json_list(self, dataset: str)->Iterator[Dict]:
-        """Read data from file which is list of requests.
-           Sample: https://huggingface.co/datasets/Yukang/LongAlpaca-12k
-
-        Args:
-            dataset (str): The dataset path.
-
-        Yields:
-            Iterator[Dict]: The each request object.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            content = f.read()
-        data = json.loads(content)
-        for item in data:
-            yield item      
-```
\ No newline at end of file
diff --git a/evalscope/perf/_logging.py b/evalscope/perf/_logging.py
deleted file mode 100644
index 9e84ed6a..00000000
--- a/evalscope/perf/_logging.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import logging
-import os
-
-
-logger = logging.getLogger('perf')
-
-
-def enable_logging():
-    level = os.environ.get('LOGGING_LEVEL', 'info')
-    if level is not None:  # set logging level.
-        if level not in ['info', 'debug']:
-            # set logging level env, but invalid value, use default.
-            level = 'info'
-        if level == 'info':
-            logger.setLevel(logging.INFO)
-        else:
-            logger.setLevel(logging.DEBUG)
-        # set default logging handler
-        console_handler = logging.StreamHandler()
-        formatter = logging.Formatter(
-            '%(asctime)s - %(name)s - %(filename)s - %(funcName)s - %(lineno)d - %(levelname)s - %(message)s'  # noqa E501
-        )
-        #formatter = logging.Formatter(
-        #    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'  # noqa E501
-        #)
-        console_handler.setFormatter(formatter)
-        logger.addHandler(console_handler)
-
-
-# in release disable dashscope log
-# you can enable dashscope log for debugger.
-enable_logging()
diff --git a/evalscope/perf/arguments.py b/evalscope/perf/arguments.py
new file mode 100644
index 00000000..d85fc6cc
--- /dev/null
+++ b/evalscope/perf/arguments.py
@@ -0,0 +1,179 @@
+import argparse
+import sys
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+import json
+
+
+@dataclass
+class Arguments:
+    # Model and API
+    model: str  # Model identifier
+    attn_implementation: Optional[str] = None  # Attention implementaion, only for local inference
+    api: str = 'openai'  # API to be used (default: 'openai')
+    tokenizer_path: Optional[str] = None  # Path to the tokenizer
+
+    # Connection settings
+    url: str = 'http://127.0.0.1:8877/v1/chat/completions'  # URL for the API connection
+    headers: Dict[str, Any] = field(default_factory=dict)  # Custom headers
+    connect_timeout: int = 120  # Connection timeout in seconds
+    read_timeout: int = 120  # Read timeout in seconds
+    api_key: str = 'EMPTY'
+
+    # Performance and parallelism
+    number: Optional[int] = None  # Number of requests to be made
+    parallel: int = 1  # Number of parallel requests
+    rate: int = -1  # Rate limit for requests (default: -1, no limit)
+
+    # Logging and debugging
+    log_every_n_query: int = 10  # Log every N queries
+    debug: bool = False  # Debug mode
+    wandb_api_key: Optional[str] = None  # WandB API key for logging
+    name: Optional[str] = None  # Name for the run
+
+    # Prompt settings
+    max_prompt_length: int = sys.maxsize  # Maximum length of the prompt
+    min_prompt_length: int = 0  # Minimum length of the prompt
+    prompt: Optional[str] = None  # The prompt text
+    query_template: Optional[str] = None  # Template for the query
+
+    # Dataset settings
+    dataset: str = 'openqa'  # Dataset type (default: 'line_by_line')
+    dataset_path: Optional[str] = None  # Path to the dataset
+
+    # Response settings
+    frequency_penalty: Optional[float] = None  # Frequency penalty for the response
+    logprobs: Optional[bool] = None  # Whether to log probabilities
+    max_tokens: Optional[int] = 2048  # Maximum number of tokens in the response
+    min_tokens: Optional[int] = None  # Minimum number of tokens in the response
+    n_choices: Optional[int] = None  # Number of response choices
+    seed: Optional[int] = 42  # Random seed for reproducibility
+    stop: Optional[List[str]] = field(default_factory=list)  # Stop sequences for the response
+    stop_token_ids: Optional[List[str]] = field(default_factory=list)  # Stop token IDs for the response
+    stream: Optional[bool] = None  # Whether to stream the response
+    temperature: Optional[float] = None  # Temperature setting for the response
+    top_p: Optional[float] = None  # Top-p (nucleus) sampling setting for the response
+
+    @staticmethod
+    def from_args(args):
+
+        return Arguments(
+            model=args.model,
+            attn_implementation=args.attn_implementation,
+            url=args.url,
+            api_key=args.api_key,
+            connect_timeout=args.connect_timeout,
+            read_timeout=args.read_timeout,
+            number=args.number,
+            parallel=args.parallel,
+            rate=args.rate,
+            log_every_n_query=args.log_every_n_query,
+            headers=args.headers,
+            wandb_api_key=args.wandb_api_key,
+            name=args.name,
+            debug=args.debug,
+            tokenizer_path=args.tokenizer_path,
+            api=args.api,
+            max_prompt_length=args.max_prompt_length,
+            min_prompt_length=args.min_prompt_length,
+            prompt=args.prompt,
+            query_template=args.query_template,
+            dataset=args.dataset,
+            dataset_path=args.dataset_path,
+            frequency_penalty=args.frequency_penalty,
+            logprobs=args.logprobs,
+            max_tokens=args.max_tokens,
+            min_tokens=args.min_tokens,
+            n_choices=args.n_choices,
+            seed=args.seed,
+            stop=args.stop,
+            stop_token_ids=args.stop_token_ids,
+            stream=args.stream,
+            temperature=args.temperature,
+            top_p=args.top_p)
+
+    def __post_init__(self):
+        self.headers = self.headers or {}  # Default to empty dictionary
+        if self.api_key:
+            # Assuming the API key is used as a Bearer token
+            self.headers['Authorization'] = f'Bearer {self.api_key}'
+
+    def __str__(self):
+        return json.dumps(self.to_dict(), indent=4, default=str, ensure_ascii=False)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return self.__dict__
+
+
+class ParseKVAction(argparse.Action):
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        if not values:
+            setattr(namespace, self.dest, {})
+        else:
+            try:
+                kv_dict = dict(kv.split('=') for kv in values)
+                setattr(namespace, self.dest, kv_dict)
+            except ValueError as e:
+                parser.error(f'Error parsing key-value pairs: {e}')
+
+
+def add_argument(parser: argparse.ArgumentParser):
+    # yapf: disable
+    # Model and API
+    parser.add_argument('--model', type=str, required=True, help='The test model name.')
+    parser.add_argument('--attn-implementation', required=False, default=None, help='Attention implementaion')
+    parser.add_argument('--api', type=str, default='openai', help='Specify the service API')
+    parser.add_argument(
+        '--tokenizer-path', type=str, required=False, default=None, help='Specify the tokenizer weight path')
+
+    # Connection settings
+    parser.add_argument('--url', type=str, default='http://127.0.0.1:8877/v1/chat/completions')
+    parser.add_argument('--headers', nargs='+', dest='headers', action=ParseKVAction, help='Extra HTTP headers')
+    parser.add_argument('--api-key', type=str, required=False, default='EMPTY', help='The API key for authentication')
+    parser.add_argument('--connect-timeout', type=int, default=120, help='The network connection timeout')
+    parser.add_argument('--read-timeout', type=int, default=120, help='The network read timeout')
+
+    # Performance and parallelism
+    parser.add_argument('-n', '--number', type=int, default=None, help='How many requests to be made')
+    parser.add_argument('--parallel', type=int, default=1, help='Set number of concurrency requests, default 1')
+    parser.add_argument('--rate', type=int, default=-1, help='Number of requests per second. default None')
+
+    # Logging and debugging
+    parser.add_argument('--log-every-n-query', type=int, default=10, help='Logging every n query')
+    parser.add_argument('--debug', action='store_true', default=False, help='Debug request send')
+    parser.add_argument('--wandb-api-key', type=str, default=None, help='The wandb API key')
+    parser.add_argument('--name', type=str, help='The wandb db result name and result db name')
+
+    # Prompt settings
+    parser.add_argument('--max-prompt-length', type=int, default=sys.maxsize, help='Maximum input prompt length')
+    parser.add_argument('--min-prompt-length', type=int, default=0, help='Minimum input prompt length')
+    parser.add_argument('--prompt', type=str, required=False, default=None, help='Specified the request prompt')
+    parser.add_argument('--query-template', type=str, default=None, help='Specify the query template')
+
+    # Dataset settings
+    parser.add_argument('--dataset', type=str, default='openqa', help='Specify the dataset')
+    parser.add_argument('--dataset-path', type=str, required=False, help='Path to the dataset file')
+
+    # Response settings
+    parser.add_argument('--frequency-penalty', type=float, help='The frequency_penalty value', default=None)
+    parser.add_argument('--logprobs', action='store_true', help='The logprobs', default=None)
+    parser.add_argument(
+        '--max-tokens', type=int, help='The maximum number of tokens that can be generated', default=2048)
+    parser.add_argument(
+        '--min-tokens', type=int, help='The minimum number of tokens that can be generated', default=None)
+    parser.add_argument('--n-choices', type=int, help='How many completion choices to generate', default=None)
+    parser.add_argument('--seed', type=int, help='The random seed', default=42)
+    parser.add_argument('--stop', nargs='*', help='The stop tokens', default=None)
+    parser.add_argument('--stop-token-ids', nargs='*', help='Set the stop token IDs', default=None)
+    parser.add_argument('--stream', action='store_true', help='Stream output with SSE', default=None)
+    parser.add_argument('--temperature', type=float, help='The sample temperature', default=None)
+    parser.add_argument('--top-p', type=float, help='Sampling top p', default=None)
+    # yapf: enable
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='Benchmark LLM service performance.')
+    add_argument(parser)
+    return parser.parse_args()
diff --git a/evalscope/perf/benchmark.py b/evalscope/perf/benchmark.py
new file mode 100644
index 00000000..9b1d0c7f
--- /dev/null
+++ b/evalscope/perf/benchmark.py
@@ -0,0 +1,248 @@
+import asyncio
+import copy
+import os
+import platform
+import sqlite3
+import threading
+import time
+from http import HTTPStatus
+from typing import List
+
+import json
+import numpy as np
+from tqdm import tqdm
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.http_client import AioHttpClient, test_connection
+from evalscope.perf.plugin.registry import ApiRegistry, DatasetRegistry
+from evalscope.perf.utils.benchmark_util import BenchmarkData, BenchmarkMetrics
+from evalscope.perf.utils.db_util import create_result_table, get_result_db_path, insert_benchmark_data, summary_result
+from evalscope.perf.utils.handler import add_signal_handlers, exception_handler
+from evalscope.perf.utils.local_server import start_app
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+query_send_completed_event = asyncio.Event()
+data_process_completed_event = asyncio.Event()
+
+
+@exception_handler
+async def dispatch_requests_worker(request_queue: asyncio.Queue, args: Arguments):
+    query_generator_class = ApiRegistry(args.api)
+    query_generator = query_generator_class(args.tokenizer_path)
+
+    def load_prompt(prompt_path_or_text):
+        """Load the prompt from a file or directly from the input text."""
+        if prompt_path_or_text.startswith('@'):
+            with open(prompt_path_or_text[1:], 'r', encoding='utf-8') as file:
+                return file.read()
+        return prompt_path_or_text
+
+    async def dispatch_request(request):
+        """Dispatch a single request with optional rate limiting."""
+        await request_queue.put(request)
+        if args.rate != -1:
+            interval = np.random.exponential(1.0 / args.rate)
+            await asyncio.sleep(interval)
+
+    async def dispatch_requests_from_prompt(messages):
+        """Generate and dispatch requests based on the given prompt."""
+        request = query_generator.build_request(messages, args)
+        if args.number is None:
+            await dispatch_request(request)
+            return 1
+        for _ in range(args.number):
+            await dispatch_request(request)
+        return args.number
+
+    async def dispatch_requests_from_dataset():
+        """Generate and dispatch requests based on the dataset."""
+        total_query_count = 0
+        message_generator_class = DatasetRegistry(args.dataset)
+        message_generator = message_generator_class(args)
+
+        for messages in message_generator:
+            request = query_generator.build_request(messages, args)
+            if request is None:
+                continue
+            await dispatch_request(request)
+            total_query_count += 1
+            if args.number and total_query_count >= args.number:
+                break
+
+        return total_query_count
+
+    # Load prompt or dataset and dispatch requests accordingly
+    if args.prompt:
+        prompt = load_prompt(args.prompt)
+        messages = [{'role': 'user', 'content': prompt}]
+        total_queries = await dispatch_requests_from_prompt(messages)
+    elif args.dataset:
+        total_queries = await dispatch_requests_from_dataset()
+    else:
+        raise Exception('Either prompt or dataset is required!')
+
+    return total_queries
+
+
+@exception_handler
+async def send_requests_worker(
+    task_id,
+    client: AioHttpClient,
+    request_queue: asyncio.Queue,
+    benchmark_data_queue: asyncio.Queue,
+    args: Arguments,
+):
+    async with client:
+        while not (query_send_completed_event.is_set() and request_queue.empty()):
+            try:
+                # Attempt to get a request from the queue with a timeout
+                request = await asyncio.wait_for(request_queue.get(), timeout=0.001)
+                request_queue.task_done()
+            except asyncio.TimeoutError:
+                # If timeout, continue to the next iteration
+                continue
+
+            # Initialize benchmark data for the current request
+            benchmark_data = BenchmarkData(request=request)
+            collected_messages = []
+            try:
+                # Send the request and process the response
+                async for is_error, state_code, response_data in client.post(request):
+                    if is_error or state_code != HTTPStatus.OK:
+                        logger.error(f'Request: {request} failed, state_code: {state_code}, data: {response_data}')
+                        benchmark_data.success = False
+                        break
+                    if response_data:
+                        collected_messages.append(response_data)
+                        benchmark_data.chunk_times.append(time.perf_counter())
+                        benchmark_data.success = True
+                        benchmark_data.update_gpu_usage()
+            except Exception as e:
+                if response_data:
+                    collected_messages.append(response_data)
+                benchmark_data.success = False
+                logger.exception(e)
+                logger.error(f'Request query: {request} exception')
+            finally:
+                # Record completion time and collected messages
+                benchmark_data.completed_time = time.perf_counter()
+                benchmark_data.response_messages = collected_messages
+                await benchmark_data_queue.put(benchmark_data)
+
+
+@exception_handler
+async def statistic_benchmark_metric_worker(benchmark_data_queue: asyncio.Queue, args: Arguments):
+    metrics = BenchmarkMetrics(concurrency=args.parallel)
+
+    api_plugin_class = ApiRegistry(args.api)
+    api_plugin = api_plugin_class(args.tokenizer_path)
+
+    result_db_path = get_result_db_path(args.name, args.model)
+    # Initialize wandb
+    if args.wandb_api_key:
+        import wandb
+        import datetime
+        os.environ['WANDB_SILENT'] = 'true'
+        os.environ['WANDB_DIR'] = './outputs'
+
+        wandb.login(key=args.wandb_api_key)
+        current_time = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
+        name = args.name if args.name else f'{args.model}_{current_time}'
+        wandb.init(project='perf_benchmark', name=name, config=args.to_dict())
+
+    with sqlite3.connect(result_db_path) as con:
+        cursor = con.cursor()
+        create_result_table(cursor)
+        with tqdm(desc='Processing') as pbar:
+            while not (data_process_completed_event.is_set() and benchmark_data_queue.empty()):
+                try:
+                    # Attempt to get benchmark data from the queue with a timeout
+                    benchmark_data = await asyncio.wait_for(benchmark_data_queue.get(), timeout=1)
+                    benchmark_data_queue.task_done()
+                except asyncio.TimeoutError:
+                    # If timeout, continue to the next iteration
+                    continue
+
+                # Update metrics based on the benchmark data
+                metrics.update_metrics(benchmark_data, api_plugin)
+
+                # Insert benchmark data into the database and commit the transaction
+                insert_benchmark_data(cursor, benchmark_data)
+                con.commit()
+
+                # Create a message with the updated metrics
+                message = metrics.create_message()
+
+                # Log the message to wandb if the api key is provided
+                if args.wandb_api_key:
+                    wandb.log(message)
+
+                # Log the message to the logger every n queries
+                if int(metrics.n_total_queries) % args.log_every_n_query == 0:
+                    msg = json.dumps(message, ensure_ascii=False, indent=2)
+                    logger.info(msg)
+
+                pbar.update(1)  # Update the progress bar
+
+    return metrics, result_db_path
+
+
+@exception_handler
+async def create_client(args: Arguments) -> bool:
+    if args.api.startswith('local'):
+        #  start local server
+        server = threading.Thread(target=start_app, args=(copy.deepcopy(args), ), daemon=True)
+        server.start()
+
+        if args.dataset.startswith('speed_benchmark'):
+            args.url = 'http://127.0.0.1:8877/v1/completions'
+        else:
+            args.url = 'http://127.0.0.1:8877/v1/chat/completions'
+        args.model = os.path.basename(args.model)
+
+    client = AioHttpClient(args)
+    if not await test_connection(client, args):
+        raise TimeoutError('Test connection failed')
+
+    return client
+
+
+@exception_handler
+async def benchmark(args: Arguments) -> None:
+    if platform.system() != 'Windows':
+        loop = asyncio.get_running_loop()
+        add_signal_handlers(loop)
+
+    request_queue = asyncio.Queue()
+    benchmark_data_queue = asyncio.Queue()
+
+    async def create_send_request_tasks(client):
+        tasks: List[asyncio.Task] = []
+        for idx in range(args.parallel):
+            task = asyncio.create_task(send_requests_worker(idx, client, request_queue, benchmark_data_queue, args))
+            tasks.append(task)
+        return tasks
+
+    async def run_tasks():
+        client = await create_client(args)
+
+        dispatch_task = asyncio.create_task(dispatch_requests_worker(request_queue, args))
+        statistic_benchmark_metric_task = asyncio.create_task(
+            statistic_benchmark_metric_worker(benchmark_data_queue, args))
+        send_request_tasks = await create_send_request_tasks(client)
+
+        expected_number_of_queries = await dispatch_task
+        await request_queue.join()
+        query_send_completed_event.set()
+
+        await asyncio.gather(*send_request_tasks, return_exceptions=True)
+        await benchmark_data_queue.join()
+        data_process_completed_event.set()
+
+        metrics, result_db_path = await statistic_benchmark_metric_task
+        summary_result(args, metrics, expected_number_of_queries, result_db_path)
+
+        await asyncio.sleep(0.250)
+
+    await run_tasks()
diff --git a/evalscope/perf/datasets/longalpaca_12k.py b/evalscope/perf/datasets/longalpaca_12k.py
deleted file mode 100644
index 0f42df56..00000000
--- a/evalscope/perf/datasets/longalpaca_12k.py
+++ /dev/null
@@ -1,20 +0,0 @@
-import sys
-from typing import Any, Dict, Iterator, List
-from evalscope.perf.dataset_plugin_base import DatasetPluginBase
-
-from evalscope.perf.plugin_registry import register_dataset
-from evalscope.perf.query_parameters import QueryParameters
-
-@register_dataset('longalpaca')
-class LongAlpacaDatasetPlugin(DatasetPluginBase):
-    """Read data from file which is list of requests.
-           Sample: https://huggingface.co/datasets/Yukang/LongAlpaca-12k
-    """
-    def __init__(self, query_parameters: QueryParameters):
-        super().__init__(query_parameters)
-
-    def build_messages(self) -> Iterator[List[Dict]]:
-        for item in self.dataset_json_list(self.query_parameters.dataset_path):
-            prompt = item['instruction'].strip()
-            if len(prompt) > self.query_parameters.min_prompt_length and len(prompt) < self.query_parameters.max_prompt_length:
-                yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/datasets/openqa.py b/evalscope/perf/datasets/openqa.py
deleted file mode 100644
index b1932313..00000000
--- a/evalscope/perf/datasets/openqa.py
+++ /dev/null
@@ -1,22 +0,0 @@
-from sys import maxsize
-import sys
-from typing import Any, Dict, Iterator, List
-import json
-from evalscope.perf.dataset_plugin_base import DatasetPluginBase
-from evalscope.perf.plugin_registry import register_dataset
-from evalscope.perf.query_parameters import QueryParameters
-
-@register_dataset('openqa')
-class OpenqaDatasetPlugin(DatasetPluginBase):
-    """Read dataset and return prompt.
-        Datasets: https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl
-    """
-    def __init__(self, query_parameters: QueryParameters):
-        super().__init__(query_parameters)
-        
-    def build_messages(self) -> Iterator[List[Dict]]:
-        for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
-            item = json.loads(item)
-            prompt = item['question'].strip()
-            if len(prompt) > self.query_parameters.min_prompt_length and len(prompt) < self.query_parameters.max_prompt_length:
-                yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/http_client.py b/evalscope/perf/http_client.py
index 651b66ac..b81ab6c1 100644
--- a/evalscope/perf/http_client.py
+++ b/evalscope/perf/http_client.py
@@ -1,94 +1,46 @@
-"""LLM performance benchmark client.
-"""
-import argparse
 import asyncio
-from dataclasses import dataclass
-import functools
-import json
 import logging
-import platform
-import signal
-import sqlite3
-import os
 import time
-import base64
-import pickle
-import importlib.util
-import sys
-import platform
-from typing import List, Dict, Optional
-from datetime import datetime, timezone
-import aiohttp
 from http import HTTPStatus
-import aiohttp
-import numpy as np
-from evalscope.perf.plugin_registry import api_registry, dataset_registry
-from evalscope.perf.query_parameters import QueryParameters
-from evalscope.perf.server_sent_event import ServerSentEvent
-# for plugin registry
-from evalscope.perf.dashscope_api import DashScopeApiPlugin
-from evalscope.perf.openai_api import OpenaiPlugin
-from evalscope.perf.datasets.line_by_line import LineByLineDatasetPlugin
-from evalscope.perf.datasets.longalpaca_12k import LongAlpacaDatasetPlugin
-from evalscope.perf.datasets.openqa import OpenqaDatasetPlugin
-from evalscope.perf.custom_api import CustomPlugin
-from evalscope.perf._logging import logger
-
-__all__ = [
-    DashScopeApiPlugin,
-    OpenaiPlugin,
-    CustomPlugin,
-    LineByLineDatasetPlugin,
-    LongAlpacaDatasetPlugin,
-    OpenqaDatasetPlugin,
-]
-
-_query_send_completed = False
-_data_process_completed = False
-_table_name = "result"
-
-UNLIMITED_RATE = -1
-
-
-async def on_request_start(session, context, params):
-    logger.info(f'Starting request: <{params}>')
-
+from typing import AsyncGenerator, Dict, List, Tuple
 
-async def on_request_chunk_sent(session, context, params):
-    logger.info(f'Request body: {params}')
+import aiohttp
+import json
 
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.utils.local_server import ServerSentEvent
+from evalscope.utils.logger import get_logger
 
-async def on_response_chunk_received(session, context, params):
-    logger.info(f'Response info: <{params}>')
+logger = get_logger()
 
 
 class AioHttpClient:
-    def __init__(self,
-                 url: str,
-                 conn_timeout: int = 120,
-                 read_timeout: int = 120,
-                 headers: Dict = None,
-                 debug: bool = False):
-        # one client only has one connection
-        client_timeout = aiohttp.ClientTimeout(total=read_timeout + conn_timeout,
-                                               connect=conn_timeout,
-                                               sock_read=read_timeout)
-        self.debug = debug
-        if debug:
-            logger.setLevel(level=logging.DEBUG)
-            trace_config = aiohttp.TraceConfig()
-            trace_config.on_request_start.append(on_request_start)
-            trace_config.on_request_chunk_sent.append(on_request_chunk_sent)
-            # not support server sent event(stream=true)
-            trace_config.on_response_chunk_received.append(on_response_chunk_received)
-        self.client = aiohttp.ClientSession(trace_configs=[trace_config] if debug else [],
-                                            connector=aiohttp.TCPConnector(limit=1),
-                                            timeout=client_timeout)
-        ua = "modelscope_bench"
-        self.headers = {"user-agent": ua}
-        if headers:
-            self.headers.update(headers)
-        self.url = url
+
+    def __init__(
+        self,
+        args: Arguments,
+    ):
+        self.url = args.url
+        self.debug = args.debug
+        self.headers = {'user-agent': 'modelscope_bench', **(args.headers or {})}
+        self.read_timeout = args.read_timeout
+        self.connect_timeout = args.connect_timeout
+        self.client = aiohttp.ClientSession(
+            timeout=aiohttp.ClientTimeout(
+                total=self.read_timeout + self.connect_timeout,
+                connect=self.connect_timeout,
+                sock_read=self.read_timeout),
+            connector=aiohttp.TCPConnector(limit=1),
+            trace_configs=[self._create_trace_config()] if self.debug else [])
+        if self.debug:
+            get_logger(log_level=logging.DEBUG)
+
+    def _create_trace_config(self):
+        trace_config = aiohttp.TraceConfig()
+        trace_config.on_request_start.append(self.on_request_start)
+        trace_config.on_request_chunk_sent.append(self.on_request_chunk_sent)
+        trace_config.on_response_chunk_received.append(self.on_response_chunk_received)
+        return trace_config
 
     async def __aenter__(self):
         pass
@@ -96,661 +48,123 @@ async def __aenter__(self):
     async def __aexit__(self, exc_type, exc, tb):
         await self.client.close()
 
-    async def aio_call(self):
-        response = self._handle_request()
-        if self.stream:
-            return (item async for item in response)
-        else:
-            result = await response.__anext__()
-            try:
-                await response.__anext__()
-            except StopAsyncIteration:
-                pass
-            return result
-
-    async def _handle_stream(self, response):
+    async def _handle_stream(self, response: aiohttp.ClientResponse):
         is_error = False
-        status_code = response.status
         async for line in response.content:
-            if line:
-                line = line.decode("utf8")
-                line = line.rstrip("\n").rstrip("\r")
-                if self.debug:
-                    logger.info(line)
-                sse_msg = ServerSentEvent.decode(line)
-                if not sse_msg:
-                    continue
-                if sse_msg.event and sse_msg.event == "error":  # dashscope error
+            line = line.decode('utf8').rstrip('\n\r')
+            sse_msg = ServerSentEvent.decode(line)
+            if sse_msg:
+                logger.debug(f'Response recevied: {line}')
+                if sse_msg.event == 'error':
                     is_error = True
-
                 if sse_msg.data:
-                    if sse_msg.data.startswith("[DONE]"):  # openai api completed
+                    if sse_msg.data.startswith('[DONE]'):
                         break
-                    yield (is_error, status_code, sse_msg.data)
-                    # yield data
-
-    async def _handle_response(self, response: aiohttp.ClientResponse):
-        if (response.status == HTTPStatus.OK and "text/event-stream" in response.content_type):
-            async for is_error, status_code, data in self._handle_stream(response):
-                yield (is_error, status_code, data)
-        elif response.status == HTTPStatus.OK and "application/json" in response.content_type:
-            content = await response.json()
-            if 'object' in content and content['object'] == 'error':
-                yield(True, content['code'], content['message'])
+                    yield is_error, response.status, sse_msg.data
+
+    async def _handle_response(self, response: aiohttp.ClientResponse) -> AsyncGenerator[Tuple[bool, int, str], None]:
+        response_status = response.status
+        response_content_type = response.content_type
+        content_type_json = 'application/json'
+        content_type_event_stream = 'text/event-stream'
+        is_success = response_status == HTTPStatus.OK
+
+        if is_success:
+            # Handle successful response with 'text/event-stream' content type
+            if content_type_event_stream in response_content_type:
+                async for is_error, response_status, content in self._handle_stream(response):
+                    yield (is_error, response_status, content)
+            # Handle successful response with 'application/json' content type
+            elif content_type_json in response_content_type:
+                content = await response.json()
+                if content.get('object') == 'error':
+                    yield (True, content.get('code'), content.get('message'))  # DashScope
+                else:
+                    yield (False, response_status, json.dumps(content, ensure_ascii=False))
+            # Handle other successful responses
             else:
-                yield (False, HTTPStatus.OK, json.dumps(content))            
-        elif response.status == HTTPStatus.OK:
-            content = await response.read()
-            yield (False, HTTPStatus.OK, content)
+                content = await response.read()
+                yield (False, response_status, content)
         else:
-            if "application/json" in response.content_type:
+            # Handle error response with 'application/json' content type
+            if content_type_json in response_content_type:
                 error = await response.json()
-                yield (True, response.status, json.dumps(error))
-            elif "text/event-stream" in response.content_type:
+                yield (True, response_status, json.dumps(error, ensure_ascii=False))
+            # Handle error response with 'text/event-stream' content type
+            elif content_type_event_stream in response_content_type:
                 async for _, _, data in self._handle_stream(response):
                     error = json.loads(data)
-                yield (True, response.status, error)
+                    yield (True, response_status, json.dumps(error, ensure_ascii=False))
+            # Handle other error responses
             else:
                 msg = await response.read()
-                yield (True, response.status, msg.decode('utf-8'))
+                yield (True, response_status, msg.decode('utf-8'))
 
     async def post(self, body):
+        headers = {'Content-Type': 'application/json', **self.headers}
         try:
-            headers = {"Content-Type": "application/json", **self.headers}
-            response = await self.client.request("POST",
-                                                 url=self.url,
-                                                 json=body,
-                                                 headers=headers)
-            async with response:
+            data = json.dumps(body, ensure_ascii=False)  # serialize to JSON
+            async with self.client.request('POST', url=self.url, data=data, headers=headers) as response:
                 async for rsp in self._handle_response(response):
                     yield rsp
-        except aiohttp.ClientConnectorError as e:
+        except (aiohttp.ClientConnectorError, Exception) as e:
             logger.error(e)
-            raise e
-        except Exception as e:
-            logger.error(e)
-            raise e
-
-
-def dynamic_import_module(dynamic_module_file_path: str):
-    """Dynamic import input output process python file.
-
-    Args:
-        dynamic_module_file_path (str): The absolute path of the 
-            input output process python path, or name of the format, 
-            system support openai, dashscope format.
-    """
-    module_name = 'module_request_response_parser'
-
-    dynamic_module_spec = importlib.util.spec_from_file_location(module_name, dynamic_module_file_path)
-    dynamic_module = importlib.util.module_from_spec(dynamic_module_spec)
-    sys.modules[module_name] = dynamic_module
-    dynamic_module_spec.loader.exec_module(dynamic_module)
-    return dynamic_module
-
-def get_query_template(args):
-    if args.query_template.startswith('@'):
-        # read from file
-        with open(args.query_template[1:], 'r') as f:
-            content = f.read()
-            return content.strip()
-    return args.query_template.strip()
-
-async def dispatch_requests_worker(request_queue: asyncio.Queue, args):
-    query_generator_class = api_registry(args.api)
-    if not query_generator_class:
-        print('Can not find query generator: %s'%args.api)
-    query_generator = query_generator_class(args.tokenizer_path)
-    total_query_counter = 0
-    query_parameters = QueryParameters(args)
-    if args.prompt is not None:
-        if args.prompt.startswith("@"):  # read local as prompt, same as curl --data
-            with open(args.prompt, 'r', encoding='utf-8') as f:
-                prompt = f.read()
+            yield (True, None, e)
+
+    @staticmethod
+    async def on_request_start(session, context, params: aiohttp.TraceRequestStartParams):
+        logger.debug(f'Starting request: <{params}>')
+
+    @staticmethod
+    async def on_request_chunk_sent(session, context, params: aiohttp.TraceRequestChunkSentParams):
+        method = params.method
+        url = params.url
+        chunk = params.chunk.decode('utf-8')
+        max_length = 100
+        if len(chunk) > 2 * max_length:
+            truncated_chunk = f'{chunk[:max_length]}...{chunk[-max_length:]}'
         else:
-            prompt = args.prompt
-        messages = [{'role': 'user', 'content': prompt}]
-        request = query_generator.build_request(messages, query_parameters)
-        if args.number is None:
-            await request_queue.put(request)
+            truncated_chunk = chunk
+        logger.debug(f'Request sent: <{method=},  {url=}, {truncated_chunk=}>')
+
+    @staticmethod
+    async def on_response_chunk_received(session, context, params: aiohttp.TraceResponseChunkReceivedParams):
+        method = params.method
+        url = params.url
+        chunk = params.chunk.decode('utf-8')
+        max_length = 200
+        if len(chunk) > 2 * max_length:
+            truncated_chunk = f'{chunk[:max_length]}...{chunk[-max_length:]}'
         else:
-            for i in range(args.number):
-                if args.rate == UNLIMITED_RATE:
-                    await request_queue.put(request)
-                else:
-                    interval = np.random.exponential(1.0 / args.rate)
-                    # The next request will be sent after the interval.
-                    await asyncio.sleep(interval)
-                    await request_queue.put(request)
-    elif args.dataset_path is not None:
-        # Ensure sufficient quantity of queries.
-        while True:
-            message_generator_class = dataset_registry.get_class(args.dataset)
-            if not message_generator_class:
-                print('Can not find dataset: %s plugin.'%(args.dataset))
-                sys.exit(1)
-            message_generator = message_generator_class(query_parameters)
-            for messages in message_generator:
-                request = query_generator.build_request(messages, query_parameters)
-                if request is None:
-                    continue
-                await request_queue.put(request)
-                total_query_counter += 1
-                if args.number is not None:
-                    if total_query_counter >= args.number:
-                        break
-                if args.rate == UNLIMITED_RATE:  # on rate limit
-                    continue
-                # Sample the request interval from the exponential distribution.
-                # from vllm
-                interval = np.random.exponential(1.0 / args.rate)
-                # The next request will be sent after the interval.
-                await asyncio.sleep(interval)
-            if args.number is None:
-                break
-            elif total_query_counter >= args.number:
-                break
-    else:
-        raise Exception("Prompt or dataset is required!")
-    return total_query_counter
-
-
-class BenchmarkData(dict):
-    """Benchmark info, two parts
-    1. query info.
-       prompt length
-    2. response info
-       start send time
-       list of package_receive_time
-           package info.
-       response complete time
-           total response info(response tokens)       
-    """
-
-    def __init__(self):
-        pass
-
-
-def calculate_query_stream_metric(benchmark_data):
-    first_chunk_latency = benchmark_data["chunk_times"][0] - benchmark_data["start_time"]  # the first chunk latency
-    n_chunks = len(benchmark_data["chunk_times"]) - 2  # minus first and last chunk.
-    n_chunks_time = benchmark_data["chunk_times"][-2] - benchmark_data["chunk_times"][0]  # -2 to last chunk
-    return (first_chunk_latency, n_chunks, n_chunks_time)
+            truncated_chunk = chunk
+        logger.debug(f'Request received: <{method=},  {url=}, {truncated_chunk=}>')
 
 
-async def statistic_benchmark_metric_worker(benchmark_data_queue: asyncio.Queue, args):
-    """Statistics of performance metrics based on performance data
-    """
-    n_succeed_queries = 0
-    n_failed_queries = 0
-    total_first_chunk_latency = 0
-    total_latency = 0.0
-    n_total_chunks = 0
-    n_total_prompt_tokens = 0
-    n_total_completion_tokens = 0
-    qps = 0
-    concurrency = args.parallel
-    start_time = None
-    total_chunks_time = 0.0
-    avg_latency = -1
-    avg_first_chunk_latency = -1
-    avg_token_per_seconds = -1
-    avg_time_per_token = -1
-    n_avg_chunks = -1
-    avg_chunk_time = -1
-    avg_prompt_tokens = -1
-    avg_completion_tokens = -1
-    total_time = 1  # avoid divide by zero
-    n_total_queries = 0
-    # avg generate tps generated tokens / time
-    # avg chunk time, first latency - avg_chunk_time == first latency, 去掉第一个和最后一个，第一个和prefill合并了，最后一个生成token可能比较短
-    # avg prefill tps
-    # prefill time = 首包时间-avg_chunk_time
-    # n-tokens-per-trunk
-    n_benchmark_result = 0
-    api_plugin_class = api_registry(args.api)
-    if not api_plugin_class:
-        print('Can not find query generator: %s'%args.api)
-    api_plugin = api_plugin_class(args.tokenizer_path)
-    utc_dt = datetime.now(timezone.utc)
-    current_time = utc_dt.astimezone().strftime("%Y_%m_%d_%H_%M_%S_%f")
-    if args.name:
-        result_db_path = os.path.join('./', args.name)
-    else:
-        result_db_path = os.path.join('./', "%s_benchmark_%s.db" % (args.model, current_time))
-    result_db_path_split = result_db_path.split('/')[1:]
-    if len(result_db_path_split) > 2:
-        result_db_path_split = result_db_path_split[-2:]
-    result_db_path = os.path.join(os.getcwd(), "/".join(result_db_path_split))
-    result_db_dir = os.path.split(result_db_path)[0]
-    if not os.path.exists(result_db_dir):
-        os.makedirs(result_db_dir, exist_ok=True)
-    print('Save the result to : %s'%result_db_path)
-    if os.path.exists(result_db_path):
-        print('The db file exist, delete it and start again!.')
-        sys.exit(1)
-        
-    con = sqlite3.connect(result_db_path)
+async def test_connection(client: AioHttpClient, args: Arguments) -> bool:
+    is_error = True
+    timeout = args.connect_timeout
+    start_time = time.perf_counter()
 
-    db_cur = con.cursor()
-    # create table
-    # TPS output tokens per second
-    # tpot Time per ooutput token
-    db_cur.execute("CREATE TABLE %s(request, start_time, chunk_times, success, \
-                   response_messages, completed_time, latency, first_chunk_latency, \
-                   n_chunks, chunk_time, prompt_tokens, completion_tokens)" % _table_name)
-    if args.wandb_api_key is not None:
-        import wandb
-        name = args.name if args.name is not None else '%s_%s' % (args.model, current_time)
-        wandb.init(
-            project="perf_benchmark",
-            name=name,
-            # track run metadata
-            config={
-                "model": args.model,
-                "time": current_time
-            })
-        os.environ["WANDB_SILENT"] = "true"
-
-    while True:
-        try:
-            benchmark_data = benchmark_data_queue.get_nowait()
-            benchmark_data_queue.task_done()
-            n_benchmark_result += 1
-        except asyncio.QueueEmpty as e:
-            if _data_process_completed:
-                break
-            await asyncio.sleep(1)
-            continue
-        if start_time is None:
-            start_time = benchmark_data["start_time"]  # start time with first request start time
-        # total requests
-        total_time = time.perf_counter() - start_time
-
-        if benchmark_data["success"]:
-            n_succeed_queries += 1
-            n_query_trunks = len(benchmark_data["chunk_times"])
-            query_latency = benchmark_data["completed_time"] - benchmark_data["start_time"]
-            if n_query_trunks > 1:
-                query_first_chunk_latency, query_n_chunks, query_n_chunks_time = calculate_query_stream_metric(
-                    benchmark_data)
-            else:
-                query_first_chunk_latency = query_latency     # not stream mode, query latency is equal total latency
-                query_n_chunks = 1
-                query_n_chunks_time = query_latency
-
-            n_query_prompt_tokens, n_query_completion_tokens = api_plugin.parse_responses(
-                benchmark_data["response_messages"], 
-                request=benchmark_data["request"])
-            n_total_prompt_tokens += n_query_prompt_tokens
-            n_total_completion_tokens += n_query_completion_tokens
-
-            total_first_chunk_latency += query_first_chunk_latency
-            total_latency += query_latency
-            n_total_chunks += query_n_chunks
-            total_chunks_time += query_n_chunks_time
-
-            # calc average
-            avg_first_chunk_latency = total_first_chunk_latency / n_succeed_queries
-            # average latency
-            avg_latency = total_latency / n_succeed_queries
-            # average generate chunks
-            if n_query_trunks > 1:
-                n_avg_chunks = n_total_chunks / n_succeed_queries + 2     # we remove the frist and last chunk.
-            else:
-                n_avg_chunks = n_total_chunks / n_succeed_queries
-            avg_chunk_time = total_chunks_time / n_total_chunks
-            avg_prompt_tokens = n_total_prompt_tokens / n_succeed_queries
-            avg_completion_tokens = n_total_completion_tokens / n_succeed_queries
-            # avg generate tps generated tokens / time
-            avg_token_per_seconds = n_total_completion_tokens / total_time
-            avg_time_per_token = total_time / n_total_completion_tokens
-            # save the benchmark data to database.
-            # save data to dist.
-            insert_sql = "INSERT INTO %s VALUES('%s', %s, '%s', '%s', '%s', %s, %s, %s, %s, %s, %s, %s)" % (
-                _table_name,
-                base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"),
-                benchmark_data["start_time"],
-                json.dumps(benchmark_data["chunk_times"]),
-                benchmark_data["success"],
-                base64.b64encode(pickle.dumps(benchmark_data["response_messages"])).decode("ascii"),
-                benchmark_data["completed_time"],
-                query_latency,
-                query_first_chunk_latency,
-                query_n_chunks,
-                query_n_chunks_time,
-                n_query_prompt_tokens,
-                n_query_completion_tokens
-            )
-        else:
-            n_failed_queries += 1
-            # save the benchmark data to database.
-            # save data to dist.
-            insert_sql = "INSERT INTO %s(request, start_time, chunk_times, success, response_messages, completed_time)\
-                VALUES('%s', %s, '%s', '%s', '%s', %s)" % (
-                _table_name,
-                base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"),
-                benchmark_data["start_time"],
-                json.dumps(benchmark_data["chunk_times"]),
-                benchmark_data["success"],
-                base64.b64encode(pickle.dumps(benchmark_data["response_messages"])).decode("ascii"),
-                benchmark_data["completed_time"]
-            )
-        n_total_queries = float(n_succeed_queries + n_failed_queries)  # float for calc
-        qps = n_succeed_queries / total_time
-        db_cur.execute(insert_sql)
-        con.commit()
-        default_ndigits = 3
-        message = {"Time": round(total_time, default_ndigits),
-                   "concurrency": concurrency,
-                   "completed": int(n_total_queries),
-                   "succeed": n_succeed_queries,
-                   "failed": n_failed_queries,
-                   "qps": round(qps, default_ndigits),
-                   "latency": round(avg_latency, default_ndigits),
-                   "time to first token": round(avg_first_chunk_latency, default_ndigits),
-                   "throughput(output tokens per second)": round(avg_token_per_seconds, default_ndigits),
-                   "time per output token": round(avg_time_per_token, 5),
-                   "package per request": round(n_avg_chunks, default_ndigits),
-                   "time per package": round(avg_chunk_time, default_ndigits),
-                   "input tokens per request": round(avg_prompt_tokens, default_ndigits),
-                   "output tokens per request": round(avg_completion_tokens, default_ndigits)}
-        if args.wandb_api_key is not None:
-            wandb.log(message)
-        if int(n_total_queries) % args.log_every_n_query == 0:
-            msg = json.dumps(message)
-            msg = msg[1:-1].replace('"', '')
-            logger.info(msg)
-    con.commit()
-    con.close()
-    return (total_time, n_total_queries,
-            n_succeed_queries, n_failed_queries,
-            qps, avg_latency, avg_first_chunk_latency,
-            n_avg_chunks, avg_chunk_time,
-            avg_prompt_tokens, avg_completion_tokens,
-            avg_token_per_seconds, avg_time_per_token,
-            result_db_path)
-
-
-def summary_result(expected_number_of_queries,
-                   total_time,
-                   n_total_queries,
-                   n_succeed_queries,
-                   n_failed_queries,
-                   qps,
-                   avg_latency,
-                   avg_first_chunk_latency,
-                   n_avg_chunks,
-                   avg_chunk_time,
-                   avg_prompt_tokens,
-                   avg_completion_tokens,
-                   avg_token_per_seconds,
-                   avg_time_per_token,
-                   result_db_path, args):
-
-    print("Benchmarking summary: ")
-    print(" Time taken for tests: %.3f seconds" % total_time)
-    print(" Expected number of requests: %s" % expected_number_of_queries)
-    print(" Number of concurrency: %d" % args.parallel)
-    print(" Total requests: %d" % n_total_queries)
-    print(" Succeed requests: %d" % n_succeed_queries)
-    print(" Failed requests: %d" % n_failed_queries)
-    print(" Average QPS: %.3f" % qps)
-    print(" Average latency: %.3f" % avg_latency)
-    print(" Throughput(average output tokens per second): %.3f" % avg_token_per_seconds)
-    print(" Average time to first token: %.3f" % avg_first_chunk_latency)
-    print(" Average input tokens per request: %.3f" % avg_prompt_tokens)
-    print(" Average output tokens per request: %.3f" % avg_completion_tokens)
-    print(" Average time per output token: %.5f" % avg_time_per_token)
-    print(" Average package per request: %.3f" % n_avg_chunks)
-    print(" Average package latency: %.3f" % avg_chunk_time)
-
-    con = sqlite3.connect(result_db_path)
-    query_sql = "SELECT start_time, chunk_times, success, \
-                   completed_time, latency, first_chunk_latency, \
-                   n_chunks, chunk_time, prompt_tokens, completion_tokens \
-                       FROM %s WHERE success='True' ORDER BY first_chunk_latency ASC" % _table_name
-
-    percentiles = [50, 66, 75, 80, 90, 95, 98, 99]
-    with con:
-        rows = con.execute(query_sql).fetchall()
-        n_success_queries = len(rows)
-        if len(rows) > len(percentiles):
-            print(" Percentile of time to first token: ")
-            for percentile in percentiles:
-                idx = (int)(n_success_queries * percentile / 100)
-                row = rows[idx]
-                print("     p%s: %.4f" % (percentile, row[5] if row[5] is not None else float("inf")))
-                # print(row)
-            print(" Percentile of request latency: ")
-            latency_index = 4
-            rows.sort(key=lambda x: x[latency_index])
-            for percentile in percentiles:
-                idx = (int)(n_success_queries * percentile / 100)
-                row = rows[idx]
-                print("     p%s: %.4f" % (percentile, row[latency_index]
-                      if row[latency_index] is not None else float("inf")))
+    async def attempt_connection():
+        if 'chat/completions' in args.url:
+            request = {'messages': [{'role': 'user', 'content': 'hello'}], 'model': args.model}
         else:
-            print(" Too little data to calculate quantiles!")
-    con.close()
-
+            request = {'prompt': 'hello', 'model': args.model}
+        async for is_error, state_code, response_data in client.post(request):
+            return is_error, state_code, response_data
 
-async def send_requests_worker(task_id, request_queue: asyncio.Queue, benchmark_data_queue: asyncio.Queue, args):
-    client = AioHttpClient(args.url,
-                           conn_timeout=args.connect_timeout,
-                           read_timeout=args.read_timeout,
-                           headers=args.headers,
-                           debug=args.debug)
-    async with client:
-        while True:
-            # Get a request out of the queue.
-            try:
-                request = request_queue.get_nowait()
-                request_queue.task_done()
-            except asyncio.QueueEmpty as e:
-                if _query_send_completed:
-                    break
-                await asyncio.sleep(0.01)
-                continue    # keep polling querys
-            benchmark_data = BenchmarkData()
-            benchmark_data["request"] = request
-            benchmark_data["start_time"] = time.perf_counter()
-            benchmark_data["chunk_times"] = []
-            benchmark_data["success"] = False
-            collected_messages = []
-            try:
-                async for (is_error, state_code, response_data) in client.post(request):
-                    if is_error or state_code != HTTPStatus.OK:
-                        logger.error("Request: %s failed, state_code: %s, data: %s" %
-                                     (request, state_code, response_data))
-                        break
-                    else:
-                        if response_data:
-                            collected_messages.append(response_data)  # save the message
-                            logger.info(response_data)
-                            benchmark_data["chunk_times"].append(time.perf_counter())
-
-                benchmark_data["response_messages"] = collected_messages
-                benchmark_data["completed_time"] = time.perf_counter()
-                benchmark_data["success"] = not is_error
-                await benchmark_data_queue.put(benchmark_data)
-            except BaseException as e:
-                if response_data:
-                    collected_messages.append(response_data)  # save the message
-                benchmark_data["response_messages"] = collected_messages
-                benchmark_data["completed_time"] = time.perf_counter()
-                await benchmark_data_queue.put(benchmark_data)
-                logger.error("Request query: %s exception, response: %s" % (request, response_data))
-                logger.exception(e)
-
-def signal_handler(signal_name, loop):
-    print("Got signal %s: exit" % signal_name)
-    loop.stop()
-    
-
-async def benchmark(args) -> None:
-    # Check if the current platform is Windows
-    if platform.system() != 'Windows':
-        # add SIGINT and SIGTERM handler
-        loop = asyncio.get_running_loop()
-        for signal_name in {'SIGINT', 'SIGTERM'}:
-            loop.add_signal_handler(
-                getattr(signal, signal_name),
-                functools.partial(signal_handler, signal_name, loop))
-
-    request_tasks: List[asyncio.Task] = []
-    # Queues can be used to distribute workload between several concurrent tasks
-    # Create a queue that we will use to store our "workload".
-    request_queue = asyncio.Queue()
-    benchmark_data_queue = asyncio.Queue()
-    dispatch_task = asyncio.create_task(dispatch_requests_worker(request_queue, args))
-    statistic_benchmark_metric_task = asyncio.create_task(statistic_benchmark_metric_worker(benchmark_data_queue, args))
-    for idx, task in enumerate(range(args.parallel)):
-        task = asyncio.create_task(send_requests_worker(idx, request_queue, benchmark_data_queue, args))
-        request_tasks.append(task)
-
-    expected_number_of_queries = await dispatch_task  # wait for dispatch task complete
-    await request_queue.join()
-    global _query_send_completed
-    _query_send_completed = True
-    await asyncio.gather(*request_tasks, return_exceptions=True)
-    await benchmark_data_queue.join()  # wait for all query is processed
-    global _data_process_completed
-    _data_process_completed = True
-    (total_time, n_total_queries,
-     n_succeed_queries, n_failed_queries,
-     qps, avg_latency,
-     avg_first_chunk_latency, n_avg_chunks,
-     avg_chunk_time, avg_prompt_tokens,
-     avg_completion_tokens, avg_token_per_seconds,
-     avg_time_per_token, result_db_path) = await statistic_benchmark_metric_task
-
-    summary_result(expected_number_of_queries, total_time, n_total_queries, n_succeed_queries,
-                   n_failed_queries, qps, avg_latency, avg_first_chunk_latency,
-                   n_avg_chunks, avg_chunk_time,
-                   avg_prompt_tokens, avg_completion_tokens,
-                   avg_token_per_seconds, avg_time_per_token,
-                   result_db_path, args)
-    await asyncio.sleep(0.250)
-
-
-def process_number(input):
-    try:
-        return int(input)
-    except ValueError:
+    while True:
         try:
-            return float(input)
-        except ValueError:
-            return input
-
-
-# from: https://gist.github.com/vadimkantorov/37518ff88808af840884355c845049ea
-class ParseKVAction(argparse.Action):
-    def __call__(self, parser, namespace, values, option_string=None):
-        setattr(namespace, self.dest, dict())
-        for each in values:
-            try:
-                key, value = each.split("=")
-                if value.lower() == 'bool_true':
-                    value = True
-                if value.lower() == 'bool_false':
-                    value = False
-                
-                value = process_number(value)
-                getattr(namespace, self.dest)[key] = value
-            except ValueError as ex:
-                message = "\nTraceback: {}".format(ex)
-                message += "\nError on '{}' || It should be 'key=value'".format(
-                    each)
-                raise argparse.ArgumentError(self, str(message))
-
-
-def run_perf_benchmark(args):
-    asyncio.run(benchmark(args))
-
+            is_error, state_code, response_data = await asyncio.wait_for(attempt_connection(), timeout=timeout)
+            if not is_error:
+                logger.info('Connection successful.')
+                return True
+            logger.warning(f'Retrying...  <{state_code}> {response_data}')
+        except Exception as e:
+            logger.warning(f'Retrying... <{e}>')
 
-def add_argument(parser: argparse.ArgumentParser):
-    parser.add_argument("--model", type=str, required=True,
-                        help="The test model name.")
-    parser.add_argument("--url", type=str, default="localhost")
-    parser.add_argument("--connect-timeout", type=int, default=120,
-                        help="The network connection timeout")
-    parser.add_argument("--read-timeout", type=int, default=120,
-                        help="The network read timeout")
-    parser.add_argument("-n", "--number", type=int, default=None,
-                        help="How many requests to be made, if None, "
-                        "will will send request base dataset or prompt.")
-    parser.add_argument("--parallel", type=int, default=1,
-                        help="Set number of concurrency request, default 1")
-    parser.add_argument("--rate", type=int, default=UNLIMITED_RATE,
-                        help="Number of requests per second. default None, if it set to -1,"
-                        "then all the requests are sent at time 0. "
-                             "Otherwise, we use Poisson process to synthesize "
-                             "the request arrival times.  Mutual exclusion with parallel")
-    parser.add_argument("--log-every-n-query", type=int, default=10,
-                        help="Logging every n query.")
-    parser.add_argument("--headers", nargs="+", dest="headers",
-                        action=ParseKVAction,
-                        help="Extra http headers accepts by key1=value1 key2=value2. "
-                             "The headers will be use for each query."
-                             "You can use this parameter to specify http authorization and other header.",
-                        metavar="KEY1=VALUE1")
-    parser.add_argument("--wandb-api-key", type=str, default=None,
-                        help="The wandb api key, if set the metric will be saved to wandb.")
-    parser.add_argument("--name", type=str,
-                        help="The wandb db result name and result db name, default: {model_name}_{current_time}")
-    parser.add_argument("--debug", action='store_true', default=False,
-                        help='Debug request send.')
-    parser.add_argument("--tokenizer-path", type=str, required=False, default=None,
-                        help="Specify the tokenizer weight path, used to calculate the number of input and output tokens,"
-                        "usually in the same directory as the model weight. If service return usage will use usage info.")
-    parser.add_argument("--api",
-                        type=str,
-                        default="openai",
-                        help="Specify the service api, current support [openai|dashscope]"
-                             "you can define your custom parser with python, and specify the python file path, "
-                             "reference api_plugin_base.py,")
-    parser.add_argument("--max-prompt-length", type=int, default=sys.maxsize,
-                        help="Maximum input prompt length")
-    parser.add_argument("--min-prompt-length", type=int, default=0,
-                        help="Minimum input prompt length.")
-    parser.add_argument("--prompt", type=str, required=False, default=None,
-                        help="Specified the request prompt, all the query will use this prompt, "
-                        "You can specify local file via @file_path, the prompt will be "
-                        "the file content.")
-    parser.add_argument("--query-template",
-                        type=str,
-                        default=None,
-                        help="Specify the query template, should be a json string, or local file,"
-                             "with local file, specified with @local_file_path,"
-                             "will will replace model and prompt in the template.")
-    parser.add_argument("--dataset",
-                        type=str,
-                        default='line_by_line',
-                        help="Specify the dataset [openqa|longalpaca|line_by_line]"
-                             "you can define your custom dataset parser with python, and specify the python file path, "
-                             "reference dataset_plugin_base.py,")
-    parser.add_argument("--dataset-path", type=str, required=False,
-                        help="Path to the dataset file, Used in conjunction with dataset. "
-                             "If dataset is None, each line defaults to a prompt.")
-    
-    parser.add_argument("--frequency-penalty", type=float, help="The frequency_penalty value.", default= None)
-    parser.add_argument("--logprobs", action='store_true', help="The logprobs.", default=None)
-    parser.add_argument("--max-tokens", type=int, help="The maximum number of tokens can be generated.", default=None)
-    parser.add_argument("--n-choices", type=int, help="How may chmpletion choices to generate.", default=None)
-    parser.add_argument("--seed", type=int, help="Rhe random seed.", default=None)
-    parser.add_argument("--stop", nargs='*', help="The stop tokens.", default=None)
-    parser.add_argument("--stop-token-ids", nargs='*', help="Set the stop token ids.", default=None)
-    parser.add_argument("--stream", action='store_true', help="Stream output with SSE, Automatically add stream_option.include_usage with openai interface.", default=None)
-    parser.add_argument("--temperature", type=float, help="The sample temperature.", default=None)
-    parser.add_argument("--top-p", type=float, help="Sampling top p.", default=None)
+        if time.perf_counter() - start_time >= timeout:
+            logger.error('Overall connection attempt timed out.')
+            return False
 
-if __name__ == "__main__":
-    # for windows raise RuntimeError: Event loop is closed
-    if platform.system() == 'Windows':
-        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
-    parser = argparse.ArgumentParser(
-        description="Benchmark LLM service performance.")
-    add_argument(parser)
-    args = parser.parse_args()
-    run_perf_benchmark(args)
+        await asyncio.sleep(10)
diff --git a/evalscope/perf/main.py b/evalscope/perf/main.py
new file mode 100644
index 00000000..c69f3ac8
--- /dev/null
+++ b/evalscope/perf/main.py
@@ -0,0 +1,35 @@
+import asyncio
+import platform
+from argparse import Namespace
+
+from evalscope.perf.arguments import Arguments, parse_args
+from evalscope.perf.benchmark import benchmark
+from evalscope.perf.utils.handler import add_signal_handlers
+from evalscope.utils.logger import get_logger
+from evalscope.utils.utils import seed_everything
+
+logger = get_logger()
+
+
+def run_perf_benchmark(args):
+    if isinstance(args, dict):
+        args = Arguments(**args)
+    elif isinstance(args, Namespace):
+        args = Arguments.from_args(args)
+    seed_everything(args.seed)
+
+    logger.info('Starting benchmark...')
+    logger.info(args)
+
+    if platform.system() == 'Windows':
+        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
+
+    loop = asyncio.get_event_loop()
+    if platform.system() != 'Windows':
+        add_signal_handlers(loop)
+    loop.run_until_complete(benchmark(args))
+
+
+if __name__ == '__main__':
+    args = Arguments.from_args(parse_args())
+    run_perf_benchmark(args)
diff --git a/evalscope/perf/plugin/__init__.py b/evalscope/perf/plugin/__init__.py
new file mode 100644
index 00000000..7f040eaf
--- /dev/null
+++ b/evalscope/perf/plugin/__init__.py
@@ -0,0 +1,2 @@
+from evalscope.perf.plugin.api import *
+from evalscope.perf.plugin.datasets import *
diff --git a/evalscope/perf/plugin/api/__init__.py b/evalscope/perf/plugin/api/__init__.py
new file mode 100644
index 00000000..8bc24467
--- /dev/null
+++ b/evalscope/perf/plugin/api/__init__.py
@@ -0,0 +1,3 @@
+from evalscope.perf.plugin.api.custom_api import CustomPlugin
+from evalscope.perf.plugin.api.dashscope_api import DashScopeApiPlugin
+from evalscope.perf.plugin.api.openai_api import OpenaiPlugin
diff --git a/evalscope/perf/api_plugin_base.py b/evalscope/perf/plugin/api/base.py
similarity index 64%
rename from evalscope/perf/api_plugin_base.py
rename to evalscope/perf/plugin/api/base.py
index 309ba6b1..f242f6d8 100644
--- a/evalscope/perf/api_plugin_base.py
+++ b/evalscope/perf/plugin/api/base.py
@@ -1,14 +1,16 @@
 from abc import abstractmethod
 from typing import Any, Dict, List, Tuple
 
-from evalscope.perf.query_parameters import QueryParameters
+from evalscope.perf.arguments import Arguments
+
 
 class ApiPluginBase:
+
     def __init__(self, model_path: str) -> None:
         self.model_path = model_path
-        
+
     @abstractmethod
-    def build_request(self, messages: List[Dict], param: QueryParameters)->Dict:
+    def build_request(self, messages: List[Dict], param: Arguments) -> Dict:
         """Build a api request body.
 
         Args:
@@ -22,39 +24,36 @@ def build_request(self, messages: List[Dict], param: QueryParameters)->Dict:
             Dict: The api request body.
         """
         raise NotImplementedError
-    
+
     @abstractmethod
-    def parse_responses(self, 
-                        responses: List, 
-                        request: Any=None,
-                        **kwargs:Any) -> Tuple[int, int]:
+    def parse_responses(self, responses: List, request: Any = None, **kwargs: Any) -> Tuple[int, int]:
         """Parser responses and return number of request and response tokens.
 
         Args:
             responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, each is bytes, for general only one. 
+                there are multiple responses, each is bytes, for general only one.
             request (Any): The request body.
 
         Returns:
             Tuple: (Number of prompt_tokens and number of completion_tokens).
         """
-        raise NotImplementedError  
+        raise NotImplementedError
 
     @staticmethod
     def replace_values(input_json: Any, model: str, prompt: str):
-        if isinstance(input_json, dict):  
+        if isinstance(input_json, dict):
             for key, value in input_json.items():
                 if isinstance(value, str):
-                    input_json[key] = value.replace("%m", model).replace("%p", prompt)
-                else:                    
-                    ApiPluginBase.replace_values(value, model, prompt)  
-        elif isinstance(input_json, list): 
+                    input_json[key] = value.replace('%m', model).replace('%p', prompt)
+                else:
+                    ApiPluginBase.replace_values(value, model, prompt)
+        elif isinstance(input_json, list):
             for idx, item in enumerate(input_json):
                 if isinstance(item, str):
-                    input_json[idx] = item.replace("%m", model).replace("%p", prompt)
+                    input_json[idx] = item.replace('%m', model).replace('%p', prompt)
                 else:
                     ApiPluginBase.replace_values(item, model, prompt)
         elif isinstance(input_json, str):
-            input_json = input_json.replace("%m", model).replace("%p", prompt)
+            input_json = input_json.replace('%m', model).replace('%p', prompt)
         else:
-            pass
\ No newline at end of file
+            pass
diff --git a/evalscope/perf/custom_api.py b/evalscope/perf/plugin/api/custom_api.py
similarity index 78%
rename from evalscope/perf/custom_api.py
rename to evalscope/perf/plugin/api/custom_api.py
index b00e984f..9954254f 100644
--- a/evalscope/perf/custom_api.py
+++ b/evalscope/perf/plugin/api/custom_api.py
@@ -1,19 +1,26 @@
 from typing import Any, Dict, Iterator, List
+
 import json
-from evalscope.perf.api_plugin_base import ApiPluginBase
 from transformers import AutoTokenizer
-from evalscope.perf.plugin_registry import register_api
-from evalscope.perf.query_parameters import QueryParameters
 
-@register_api("custom")
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.api.base import ApiPluginBase
+from evalscope.perf.plugin.registry import register_api
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+@register_api('custom')
 class CustomPlugin(ApiPluginBase):
     """Support tensorrt-llm triton server
     """
+
     def __init__(self, mode_path: str):
         """Init the plugin
 
         Args:
-            mode_path (str): The model path, we use the tokenizer 
+            mode_path (str): The model path, we use the tokenizer
                 weight in the model to calculate the number of the
                 input and output tokens.
         """
@@ -23,12 +30,12 @@ def __init__(self, mode_path: str):
         else:
             self.tokenizer = None
 
-    def build_request(self, messages: List[Dict], param: QueryParameters) -> Dict:
+    def build_request(self, messages: List[Dict], param: Arguments) -> Dict:
         """Build the openai format request based on prompt, dataset
 
         Args:
             message (Dict): The basic message to generator query.
-            param (QueryParameters): The query parameters.
+            param (Arguments): The query parameters.
 
         Raises:
             Exception: NotImplemented
@@ -41,8 +48,8 @@ def build_request(self, messages: List[Dict], param: QueryParameters) -> Dict:
             ApiPluginBase.replace_values(query, param.model, messages[0]['content'])
             return query
         except Exception as e:
-            print(e)
-            print('Prompt: %s invalidate!'%messages)
+            logger.exception(e)
+            logger.error('Prompt: %s invalidate!' % messages)
             return None
 
     def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
@@ -53,7 +60,7 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
 
         Args:
             responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, for general only one. 
+                there are multiple responses, for general only one.
             kwargs: (Any): The command line --parameter content.
         Returns:
             Tuple: Return number of prompt token and number of completion tokens.
@@ -63,15 +70,15 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
         input_tokens = None
         output_tokens = None
         for response in responses:
-            js = json.loads(response)
+            data = json.loads(response)
             # {"context_logits":0.0,"cum_log_probs":0.0,"generation_logits":0.0,"model_name":"ensemble",
             # "model_version":"1","output_log_probs":[0.0,0.0,0.0,0.0,0.0],"sequence_end":false,"sequence_id":0,"sequence_start":false,"text_output":"性"}
-            if 'text_output' in js:
+            if 'text_output' in data:
                 if 0 in delta_contents:
-                    delta_contents[0].append(js['text_output'])
+                    delta_contents[0].append(data['text_output'])
                 else:
-                    delta_contents[0] = [js['text_output']]     
-        if input_tokens is None and output_tokens is None and self.tokenizer is not None:                
+                    delta_contents[0] = [data['text_output']]
+        if input_tokens is None and output_tokens is None and self.tokenizer is not None:
             input_tokens = 0
             output_tokens = 0
             for _, choice_contents in delta_contents.items():
@@ -80,8 +87,7 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
                 output_tokens += len(self.tokenizer.encode(full_response_content))
         elif input_tokens is None and output_tokens is None:  # no usage info get.
             input_tokens = 0
-            output_tokens = 0            
-        
+            output_tokens = 0
+            logger.warning('No usage info get.')
+
         return input_tokens, output_tokens
-        
-        
diff --git a/evalscope/perf/dashscope_api.py b/evalscope/perf/plugin/api/dashscope_api.py
similarity index 77%
rename from evalscope/perf/dashscope_api.py
rename to evalscope/perf/plugin/api/dashscope_api.py
index 811ded77..98de5715 100644
--- a/evalscope/perf/dashscope_api.py
+++ b/evalscope/perf/plugin/api/dashscope_api.py
@@ -1,26 +1,30 @@
-
-from sys import maxsize
-import sys
+import os
 from typing import Any, Dict, Iterator, List
+
 import json
-from evalscope.perf.api_plugin_base import ApiPluginBase
 
-from evalscope.perf.plugin_registry import register_api
-from evalscope.perf.query_parameters import QueryParameters
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.api.base import ApiPluginBase
+from evalscope.perf.plugin.registry import register_api
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
 
-@register_api("dashscope")
+@register_api('dashscope')
 class DashScopeApiPlugin(ApiPluginBase):
+
     def __init__(self, mode_path: str):
         """Init the plugin
 
         Args:
-            mode_path (str): The model path, we use the tokenizer 
+            mode_path (str): The model path, we use the tokenizer
                 weight in the model to calculate the number of the
                 input and output tokens.
         """
         super().__init__(model_path=mode_path)
-        
-    def build_request(self,messages: List[Dict], param: QueryParameters) -> Dict:
+
+    def build_request(self, messages: List[Dict], param: Arguments) -> Dict:
         """Build the openai format request based on prompt, dataset
 
         Args:
@@ -35,16 +39,26 @@ def build_request(self,messages: List[Dict], param: QueryParameters) -> Dict:
         """
         try:
             if param.query_template is not None:
-                query = json.loads(param.query_template)
+                if param.query_template.startswith('@'):
+                    file_path = param.query_template[1:]
+                    if os.path.exists(file_path):
+                        with open(file_path, 'r') as file:
+                            query = json.load(file)
+                    else:
+                        raise FileNotFoundError(f'{file_path}')
+                else:
+                    query = json.loads(param.query_template)
+
                 query['input']['messages'] = messages  # replace template content with message.
                 return self.__compose_query_from_parameter(query, param)
             else:
                 query = {'messages': messages}
                 return self.__compose_query_from_parameter(query, param)
         except Exception as e:
-            print(e)
+            logger.exception(e)
             return None
-    def __compose_query_from_parameter(self, payload: Dict, param: QueryParameters):
+
+    def __compose_query_from_parameter(self, payload: Dict, param: Arguments):
         payload['model'] = param.model
         if 'parameters' not in payload:
             payload['parameters'] = {}
@@ -73,7 +87,7 @@ def parse_responses(self, responses, **kwargs) -> Dict:
 
         Args:
             responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, for general only one. 
+                there are multiple responses, for general only one.
             kwargs: (Any): The command line --parameter content.
 
         Returns:
diff --git a/evalscope/perf/openai_api.py b/evalscope/perf/plugin/api/openai_api.py
similarity index 71%
rename from evalscope/perf/openai_api.py
rename to evalscope/perf/plugin/api/openai_api.py
index 93573168..c8674aab 100644
--- a/evalscope/perf/openai_api.py
+++ b/evalscope/perf/plugin/api/openai_api.py
@@ -1,19 +1,26 @@
+import os
 from typing import Any, Dict, Iterator, List
+
 import json
-from evalscope.perf.api_plugin_base import ApiPluginBase
 from transformers import AutoTokenizer
-from evalscope.perf.plugin_registry import register_api
-from evalscope.perf.query_parameters import QueryParameters
 
-@register_api("openai")
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.api.base import ApiPluginBase
+from evalscope.perf.plugin.registry import register_api
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+@register_api(['openai', 'local_vllm', 'local'])
 class OpenaiPlugin(ApiPluginBase):
-    """Base of openai interface.
-    """
+    """Base of openai interface."""
+
     def __init__(self, mode_path: str):
         """Init the plugin
 
         Args:
-            mode_path (str): The model path, we use the tokenizer 
+            mode_path (str): The model path, we use the tokenizer
                 weight in the model to calculate the number of the
                 input and output tokens.
         """
@@ -23,11 +30,11 @@ def __init__(self, mode_path: str):
         else:
             self.tokenizer = None
 
-    def build_request(self, messages: List[Dict], param: QueryParameters) -> Dict:
+    def build_request(self, messages: List[Dict] | str, param: Arguments) -> Dict:
         """Build the openai format request based on prompt, dataset
 
         Args:
-            message (Dict): The basic message to generator query.
+            message (List[Dict] | str): The basic message to generator query.
             param (QueryParameters): The query parameters.
 
         Raises:
@@ -38,22 +45,35 @@ def build_request(self, messages: List[Dict], param: QueryParameters) -> Dict:
         """
         try:
             if param.query_template is not None:
-                query = json.loads(param.query_template)
+                if param.query_template.startswith('@'):
+                    file_path = param.query_template[1:]
+                    if os.path.exists(file_path):
+                        with open(file_path, 'r') as file:
+                            query = json.load(file)
+                    else:
+                        raise FileNotFoundError(f'{file_path}')
+                else:
+                    query = json.loads(param.query_template)
+
                 if 'stream' in query.keys():
                     param.stream = query['stream']
-                query['messages'] = messages   # replace template messages with input messages.
-                return self.__compose_query_from_parameter(query, param)
+                # replace template messages with input messages.
+                query['messages'] = messages
+            elif isinstance(messages, str):
+                query = {'prompt': messages}
             else:
                 query = {'messages': messages}
-                return self.__compose_query_from_parameter(query, param)
+            return self.__compose_query_from_parameter(query, param)
         except Exception as e:
-            print(e)
+            logger.exception(e)
             return None
-        
-    def __compose_query_from_parameter(self, payload: Dict, param: QueryParameters):
+
+    def __compose_query_from_parameter(self, payload: Dict, param: Arguments):
         payload['model'] = param.model
         if param.max_tokens is not None:
             payload['max_tokens'] = param.max_tokens
+        if param.min_tokens is not None:
+            payload['min_tokens'] = param.min_tokens
         if param.frequency_penalty is not None:
             payload['frequency_penalty'] = param.frequency_penalty
         if param.logprobs is not None:
@@ -66,7 +86,7 @@ def __compose_query_from_parameter(self, payload: Dict, param: QueryParameters):
             payload['stop'] = param.stop
         if param.stream is not None and param.stream:
             payload['stream'] = param.stream
-            payload['stream_options'] = {"include_usage": True}
+            payload['stream_options'] = {'include_usage': True}
         if param.stop_token_ids is not None:
             payload['stop_token_ids'] = param.stop_token_ids
         if param.temperature is not None:
@@ -83,7 +103,7 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
 
         Args:
             responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, for general only one. 
+                there are multiple responses, for general only one.
             kwargs: (Any): The command line --parameter content.
         Returns:
             Tuple: Return number of prompt token and number of completion tokens.
@@ -96,10 +116,15 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
             js = json.loads(response)
             if js['object'] == 'chat.completion':
                 for choice in js['choices']:
-                    delta_contents[choice['index']] = [choice['message']['content']]     
+                    delta_contents[choice['index']] = [choice['message']['content']]
+                input_tokens = js['usage']['prompt_tokens']
+                output_tokens = js['usage']['completion_tokens']
+            elif js['object'] == 'text_completion':
+                for choice in js['choices']:
+                    delta_contents[choice['index']] = [choice['text']]
                 input_tokens = js['usage']['prompt_tokens']
-                output_tokens = js['usage']['completion_tokens']                 
-            else:  # 'object' == "chat.completion.chunk":
+                output_tokens = js['usage']['completion_tokens']
+            elif js['object'] == 'chat.completion.chunk':
                 if 'choices' in js:
                     for choice in js['choices']:
                         if 'delta' in choice and 'index' in choice:
@@ -115,8 +140,8 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
                 # "choices":[],"usage":{"prompt_tokens":32,"total_tokens":384,"completion_tokens":352}}
                 if 'usage' in js and js['usage']:
                     input_tokens = js['usage']['prompt_tokens']
-                    output_tokens = js['usage']['completion_tokens']     
-        if input_tokens is None and output_tokens is None and self.tokenizer is not None:                
+                    output_tokens = js['usage']['completion_tokens']
+        if (input_tokens is None and output_tokens is None and self.tokenizer is not None):
             input_tokens = 0
             output_tokens = 0
             for idx, choice_contents in delta_contents.items():
@@ -125,8 +150,7 @@ def parse_responses(self, responses, request: Any = None, **kwargs) -> Dict:
                 output_tokens += len(self.tokenizer.encode(full_response_content))
         elif input_tokens is None and output_tokens is None:  # no usage info get.
             input_tokens = 0
-            output_tokens = 0            
-        
+            output_tokens = 0
+            logger.warning('No usage info get.')
+
         return input_tokens, output_tokens
-        
-        
diff --git a/evalscope/perf/plugin/datasets/__init__.py b/evalscope/perf/plugin/datasets/__init__.py
new file mode 100644
index 00000000..09f18d35
--- /dev/null
+++ b/evalscope/perf/plugin/datasets/__init__.py
@@ -0,0 +1,5 @@
+from evalscope.perf.plugin.datasets.custom import CustomDatasetPlugin
+from evalscope.perf.plugin.datasets.line_by_line import LineByLineDatasetPlugin
+from evalscope.perf.plugin.datasets.longalpaca import LongAlpacaDatasetPlugin
+from evalscope.perf.plugin.datasets.openqa import OpenqaDatasetPlugin
+from evalscope.perf.plugin.datasets.speed_benchmark import SpeedBenchmarkDatasetPlugin, SpeedBenchmarkLongDatasetPlugin
diff --git a/evalscope/perf/dataset_plugin_base.py b/evalscope/perf/plugin/datasets/base.py
similarity index 81%
rename from evalscope/perf/dataset_plugin_base.py
rename to evalscope/perf/plugin/datasets/base.py
index 8a5e2902..e5bd5435 100644
--- a/evalscope/perf/dataset_plugin_base.py
+++ b/evalscope/perf/plugin/datasets/base.py
@@ -1,12 +1,15 @@
-from abc import abstractmethod
 import sys
+from abc import abstractmethod
 from typing import Any, Dict, Iterator, List, Tuple
+
 import json
 
-from evalscope.perf.query_parameters import QueryParameters
+from evalscope.perf.arguments import Arguments
+
 
 class DatasetPluginBase:
-    def __init__(self, query_parameters: QueryParameters):
+
+    def __init__(self, query_parameters: Arguments):
         """Build data set plugin
 
         Args:
@@ -21,9 +24,9 @@ def __next__(self):
 
     def __iter__(self):
         return self.build_messages()
-    
+
     @abstractmethod
-    def build_messages(self)->Iterator[List[Dict]]:
+    def build_messages(self) -> Iterator[List[Dict]]:
         """Build the request.
 
         Raises:
@@ -33,8 +36,8 @@ def build_messages(self)->Iterator[List[Dict]]:
             Iterator[List[Dict]]: Yield request messages.
         """
         raise NotImplementedError
-    
-    def dataset_line_by_line(self, dataset: str)->Iterator[str]:
+
+    def dataset_line_by_line(self, dataset: str) -> Iterator[str]:
         """Get content line by line of dataset.
 
         Args:
@@ -46,8 +49,8 @@ def dataset_line_by_line(self, dataset: str)->Iterator[str]:
         with open(dataset, 'r', encoding='utf-8') as f:
             for line in f:
                 yield line
-    
-    def dataset_json_list(self, dataset: str)->Iterator[Dict]:
+
+    def dataset_json_list(self, dataset: str) -> Iterator[Dict]:
         """Read data from file which is list of requests.
            Sample: https://huggingface.co/datasets/Yukang/LongAlpaca-12k
 
@@ -61,4 +64,4 @@ def dataset_json_list(self, dataset: str)->Iterator[Dict]:
             content = f.read()
         data = json.loads(content)
         for item in data:
-            yield item      
\ No newline at end of file
+            yield item
diff --git a/evalscope/perf/plugin/datasets/custom.py b/evalscope/perf/plugin/datasets/custom.py
new file mode 100644
index 00000000..4d278e00
--- /dev/null
+++ b/evalscope/perf/plugin/datasets/custom.py
@@ -0,0 +1,21 @@
+from typing import Dict, Iterator, List
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
+
+@register_dataset('custom')
+class CustomDatasetPlugin(DatasetPluginBase):
+    """Read dataset and return prompt.
+    """
+
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
+
+    def build_messages(self) -> Iterator[List[Dict]]:
+        for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
+            prompt = item.strip()
+            if len(prompt) > self.query_parameters.min_prompt_length and len(
+                    prompt) < self.query_parameters.max_prompt_length:
+                yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/datasets/line_by_line.py b/evalscope/perf/plugin/datasets/line_by_line.py
similarity index 62%
rename from evalscope/perf/datasets/line_by_line.py
rename to evalscope/perf/plugin/datasets/line_by_line.py
index 466276ec..51cb8808 100644
--- a/evalscope/perf/datasets/line_by_line.py
+++ b/evalscope/perf/plugin/datasets/line_by_line.py
@@ -1,18 +1,22 @@
 import sys
 from typing import Dict, Iterator, List
-from evalscope.perf.dataset_plugin_base import DatasetPluginBase
-from evalscope.perf.plugin_registry import register_dataset
-from evalscope.perf.query_parameters import QueryParameters
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
 
 @register_dataset('line_by_line')
 class LineByLineDatasetPlugin(DatasetPluginBase):
     """Read dataset and return prompt.
     """
-    def __init__(self, query_parameters: QueryParameters):
+
+    def __init__(self, query_parameters: Arguments):
         super().__init__(query_parameters)
 
     def build_messages(self) -> Iterator[List[Dict]]:
         for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
             prompt = item.strip()
-            if len(prompt) > self.query_parameters.min_prompt_length and len(prompt) < self.query_parameters.max_prompt_length:
+            if len(prompt) > self.query_parameters.min_prompt_length and len(
+                    prompt) < self.query_parameters.max_prompt_length:
                 yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/plugin/datasets/longalpaca.py b/evalscope/perf/plugin/datasets/longalpaca.py
new file mode 100644
index 00000000..d9e12c19
--- /dev/null
+++ b/evalscope/perf/plugin/datasets/longalpaca.py
@@ -0,0 +1,28 @@
+from typing import Any, Dict, Iterator, List
+
+from modelscope import MsDataset
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
+
+@register_dataset('longalpaca')
+class LongAlpacaDatasetPlugin(DatasetPluginBase):
+    """Read data from file which is list of requests.
+           Sample: https://www.modelscope.cn/datasets/AI-ModelScope/LongAlpaca-12k/files
+    """
+
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
+
+    def build_messages(self) -> Iterator[List[Dict]]:
+        if not self.query_parameters.dataset_path:
+            ds = MsDataset.load('AI-ModelScope/LongAlpaca-12k', subset_name='default', split='train')
+        else:
+            ds = self.dataset_json_list(self.query_parameters.dataset_path)
+        for item in ds:
+            prompt = item['instruction'].strip()
+            if len(prompt) > self.query_parameters.min_prompt_length and len(
+                    prompt) < self.query_parameters.max_prompt_length:
+                yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/plugin/datasets/openqa.py b/evalscope/perf/plugin/datasets/openqa.py
new file mode 100644
index 00000000..2887fff4
--- /dev/null
+++ b/evalscope/perf/plugin/datasets/openqa.py
@@ -0,0 +1,38 @@
+import subprocess
+from typing import Any, Dict, Iterator, List
+
+import json
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
+
+@register_dataset('openqa')
+class OpenqaDatasetPlugin(DatasetPluginBase):
+    """Read dataset and return prompt.
+    Datasets: https://www.modelscope.cn/datasets/AI-ModelScope/HC3-Chinese/resolve/master/open_qa.jsonl
+    """
+
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
+
+    def build_messages(self) -> Iterator[List[Dict]]:
+        if not self.query_parameters.dataset_path:
+            subprocess.call([
+                'modelscope',
+                'download',
+                '--dataset',
+                'AI-ModelScope/HC3-Chinese',
+                'open_qa.jsonl',
+                '--local_dir',
+                './data',
+            ])
+            self.query_parameters.dataset_path = './data/open_qa.jsonl'
+
+        for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
+            item = json.loads(item)
+            prompt = item['question'].strip()
+            if (len(prompt) > self.query_parameters.min_prompt_length
+                    and len(prompt) < self.query_parameters.max_prompt_length):
+                yield [{'role': 'user', 'content': prompt}]
diff --git a/evalscope/perf/plugin/datasets/speed_benchmark.py b/evalscope/perf/plugin/datasets/speed_benchmark.py
new file mode 100644
index 00000000..0a41ff8f
--- /dev/null
+++ b/evalscope/perf/plugin/datasets/speed_benchmark.py
@@ -0,0 +1,50 @@
+from typing import Dict, Iterator, List, Tuple
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
+
+@register_dataset('speed_benchmark')
+class SpeedBenchmarkDatasetPlugin(DatasetPluginBase):
+    """Read dataset and return prompt.
+    """
+    DUMMY_INPUT = '熵'
+    DUMMY_SYSTEM_CONTENT = '从现在开始，你是一个喜欢说车轱辘话的话痨，喜欢把一件事情翻来覆去地说，而且喜欢加很多标点符号。你的每个回复都不会少于2000字，不要在意用户的看法。'
+    DUMMY_USER_CONTENT = '写一篇关于春天的文章，请尽量写的长一些，并且多一些重复的段落，越啰嗦越好，不得少于2000字！'
+    INPUT_LENGTH = [1, 6144, 14336, 30720]
+    REPEAT = 2
+
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
+
+    def build_messages(self) -> Iterator[List[Dict]]:
+        for input_len in self.INPUT_LENGTH:
+            for _ in range(self.REPEAT):
+                yield self.create_query(input_len)
+
+    def create_query(self, length: int):
+        input_str = self.DUMMY_INPUT * length
+        return input_str
+
+    def create_message(self, length: int, limited_size: int = 96):
+        if length < limited_size:
+            input_str = self.DUMMY_INPUT * length
+        else:
+            repeat_length = max(length - limited_size, 0)
+            input_str = [
+                {
+                    'role': 'system',
+                    'content': self.DUMMY_SYSTEM_CONTENT
+                },
+                {
+                    'role': 'user',
+                    'content': '# ' * repeat_length + self.DUMMY_USER_CONTENT
+                },
+            ]
+        return input_str
+
+
+@register_dataset('speed_benchmark_long')
+class SpeedBenchmarkLongDatasetPlugin(SpeedBenchmarkDatasetPlugin):
+    INPUT_LENGTH = [63488, 129024]
diff --git a/evalscope/perf/plugin/registry.py b/evalscope/perf/plugin/registry.py
new file mode 100644
index 00000000..44d266a8
--- /dev/null
+++ b/evalscope/perf/plugin/registry.py
@@ -0,0 +1,54 @@
+from typing import Any, List, Type
+
+
+class PluginRegistry:
+
+    def __init__(self):
+        self._registry = {}
+
+    def register(self, name, cls):
+        self._registry[name] = cls
+        return cls
+
+    def get_class(self, name):
+        return self._registry[name]
+
+    def all_classes(self):
+        return list(self._registry.keys())
+
+    def __call__(self, name: str) -> Any:
+        return self.get_class(name)
+
+
+def register_dataset(name: str | List[str]):
+
+    def class_decorator(cls: Type):
+        if isinstance(name, str):
+            DatasetRegistry.register(name, cls)
+        elif isinstance(name, list):
+            for n in name:
+                DatasetRegistry.register(n, cls)
+        else:
+            raise TypeError('name must be a string or a list of strings')
+        return cls
+
+    return class_decorator
+
+
+def register_api(name: str | List[str]):
+
+    def class_decorator(cls: Type):
+        if isinstance(name, str):
+            ApiRegistry.register(name, cls)
+        elif isinstance(name, list):
+            for n in name:
+                ApiRegistry.register(n, cls)
+        else:
+            raise TypeError('name must be a string or a list of strings')
+        return cls
+
+    return class_decorator
+
+
+DatasetRegistry = PluginRegistry()
+ApiRegistry = PluginRegistry()
diff --git a/evalscope/perf/plugin_registry.py b/evalscope/perf/plugin_registry.py
deleted file mode 100644
index a07b1a2f..00000000
--- a/evalscope/perf/plugin_registry.py
+++ /dev/null
@@ -1,35 +0,0 @@
-
-from typing import Any
-
-
-class PluginRegistry:
-    def __init__(self):
-        self._registry = {}
-
-    def register(self, name, cls):
-        self._registry[name] = cls
-        return cls
-
-    def get_class(self, name):
-        return self._registry[name]
-
-    def all_classes(self):
-        return list(self._registry.keys())
-    
-    def __call__(self, name: str) -> Any:
-        return self.get_class(name)
-
-dataset_registry = PluginRegistry()
-api_registry = PluginRegistry()
-
-def register_dataset(name: str):
-    def class_decorator(cls):
-        dataset_registry.register(name, cls)
-        return cls
-    return class_decorator
-
-def register_api(name: str):
-    def class_decorator(cls):
-        api_registry.register(name, cls)
-        return cls
-    return class_decorator
\ No newline at end of file
diff --git a/evalscope/perf/query_parameters.py b/evalscope/perf/query_parameters.py
deleted file mode 100644
index 3fb17036..00000000
--- a/evalscope/perf/query_parameters.py
+++ /dev/null
@@ -1,42 +0,0 @@
-from dataclasses import dataclass
-from typing import Optional
-
-
-@dataclass
-class QueryParameters:
-    model: str
-    prompt: Optional[str]
-    dataset: Optional[str]
-    query_template: Optional[str]
-    dataset_path: Optional[str]
-    frequency_penalty: Optional[float]
-    logprobs: Optional[bool]
-    max_tokens: Optional[int]
-    n_choices: Optional[int]
-    seed: Optional[int]
-    stop: Optional[str]
-    stream: Optional[bool]
-    temperature: Optional[float]
-    top_p: Optional[float]
-    max_prompt_length: Optional[int]
-    min_prompt_length: Optional[int]    
-    include_usage: Optional[bool]
-    
-    def __init__(self, args):
-        self.model = args.model
-        self.prompt = args.prompt
-        self.dataset = args.dataset
-        self.query_template = args.query_template
-        self.dataset_path = args.dataset_path
-        self.frequency_penalty = args.frequency_penalty
-        self.logprobs = args.logprobs
-        self.max_tokens = args.max_tokens
-        self.n_choices = args.n_choices
-        self.seed = args.seed
-        self.stop = args.stop
-        self.stream = args.stream
-        self.temperature = args.temperature
-        self.top_p = args.top_p
-        self.max_prompt_length = args.max_prompt_length
-        self.min_prompt_length = args.min_prompt_length
-        self.stop_token_ids = args.stop_token_ids
\ No newline at end of file
diff --git a/evalscope/perf/sample_prompt_line_by_line.txt b/evalscope/perf/sample_prompt_line_by_line.txt
deleted file mode 100644
index bc1df7fe..00000000
--- a/evalscope/perf/sample_prompt_line_by_line.txt
+++ /dev/null
@@ -1,19 +0,0 @@
-保持健康的三个提示。
-描述一次你不得不做出困难决定的经历。
-解释为什么以下分数等同于1/4
-渲染一座房子的三维模型
-朱利叶斯·凯撒是如何死亡的？
-自然语言处理中词嵌入的用途。
-逆向工程此代码以创建新版本。
-提出数据隐私问题的道德解决方案
-生成三个与“道歉”意思相同的动词。
-比较古巴导弹危机和越南战争。
-把关于滑铁卢历史的录音转录成文字。
-写一个关于主人公必须做出重要职业决定的第三人称叙述的短故事。
-生成一个人们去野营旅行可能需要的十件物品的清单。
-将以下内容分类为动物、植物和矿物质：橡树、铜矿石、大象。
-变量x定义为“4x + 2y = 10”。找出x的值。
-写一段关于使用可再生能源的重要性的简短段落。
-为了赶上即将到达的公交车，他跑向公交车站。 
-为以下陈述构建一个论点：替代能源对于解决气候危机至关重要。
-餐厅食物很好，但价格有点贵。修改句子以提高清晰度和流畅性。
\ No newline at end of file
diff --git a/evalscope/perf/server_sent_event.py b/evalscope/perf/server_sent_event.py
deleted file mode 100644
index cd994090..00000000
--- a/evalscope/perf/server_sent_event.py
+++ /dev/null
@@ -1,43 +0,0 @@
-from dataclasses import dataclass
-
-@dataclass
-class ServerSentEvent(object):
-    def __init__(self, data='', event=None, id=None, retry=None):
-        self.data = data
-        self.event = event
-        self.id = id 
-        self.retry = retry
-
-    @classmethod
-    def decode(cls, line):
-        """ Decode line to ServerSentEvent
-
-
-        Args:
-            line (str): The line.
-
-        Return:
-            ServerSentEvent (obj:`ServerSentEvent`): The ServerSentEvent object.
-
-        """
-        if not line:
-            return None
-        sse_msg = cls()
-        # format data:xxx
-        field_type, _, field_value = line.partition(":")
-        if field_value.startswith(" "): # compatible with openai api
-            field_value = field_value[1:]
-        if field_type == "event":
-            sse_msg.event = field_value
-        elif field_type == "data":
-            field_value = field_value.rstrip()
-            sse_msg.data = field_value
-        elif field_type == "id":
-            sse_msg.id = field_value
-        elif field_type == "retry":
-            sse_msg.retry = field_value
-        else:
-            pass
-
-        return sse_msg
-
diff --git a/evalscope/perf/datasets/__init__.py b/evalscope/perf/utils/__init__.py
similarity index 100%
rename from evalscope/perf/datasets/__init__.py
rename to evalscope/perf/utils/__init__.py
diff --git a/evalscope/perf/how_to_analysis_result.py b/evalscope/perf/utils/analysis_result.py
similarity index 70%
rename from evalscope/perf/how_to_analysis_result.py
rename to evalscope/perf/utils/analysis_result.py
index b93869ee..583bb02a 100644
--- a/evalscope/perf/how_to_analysis_result.py
+++ b/evalscope/perf/utils/analysis_result.py
@@ -1,12 +1,15 @@
-import sqlite3
 import base64
 import pickle
+import sqlite3
+
 import json
-result_db_path = 'db_name.db'
+
+result_db_path = '/mnt/data/data/user/maoyunlin.myl/eval-scope/outputs/qwen2.5_benchmark_20241111_160543.db'
 con = sqlite3.connect(result_db_path)
 query_sql = "SELECT request, response_messages, prompt_tokens, completion_tokens \
-                FROM result WHERE success='True'"
-# how to save base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"), 
+                FROM result WHERE success='1'"
+
+# how to save base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"),
 with con:
     rows = con.execute(query_sql).fetchall()
     if len(rows) > 0:
@@ -20,5 +23,8 @@
             response_content = ''
             for response in responses:
                 response = json.loads(response)
+                if not response['choices']:
+                    continue
                 response_content += response['choices'][0]['delta']['content']
-            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' % (request['messages'][0]['content'], row[2], response_content, row[3]))
+            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' %
+                  (request['messages'][0]['content'], row[2], response_content, row[3]))
diff --git a/evalscope/perf/utils/benchmark_util.py b/evalscope/perf/utils/benchmark_util.py
new file mode 100644
index 00000000..cc3fd48d
--- /dev/null
+++ b/evalscope/perf/utils/benchmark_util.py
@@ -0,0 +1,135 @@
+import time
+from dataclasses import dataclass, field
+from typing import Any, List, Optional, Tuple
+
+import torch
+
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+@dataclass
+class BenchmarkData:
+    request: Any = None
+    start_time: float = field(default_factory=time.perf_counter)
+    completed_time: float = 0.0
+    chunk_times: List[float] = field(default_factory=list)
+    success: bool = False
+    response_messages: List[Any] = field(default_factory=list)
+
+    # late init
+    query_latency: float = 0.0
+    first_chunk_latency: float = 0.0
+    n_chunks: int = 0
+    n_chunks_time: float = 0.0
+    max_gpu_memory_cost = 0
+
+    prompt_tokens = None
+    completion_tokens = None
+
+    def _calculate_query_stream_metric(self) -> Tuple[float, int, float]:
+        self.query_latency = self.completed_time - self.start_time
+        if len(self.chunk_times) > 1:
+            self.first_chunk_latency = self.chunk_times[0] - self.start_time
+            self.n_chunks = len(self.chunk_times) - 2
+            self.n_chunks_time = self.chunk_times[-2] - self.chunk_times[0]
+        else:
+            self.first_chunk_latency = self.query_latency
+            self.n_chunks = 1
+            self.n_chunks_time = self.query_latency
+
+    def _calculate_tokens(self, api_plugin):
+        self.prompt_tokens, self.completion_tokens = \
+            api_plugin.parse_responses(self.response_messages, request=self.request)
+
+    def update_gpu_usage(self):
+        total_memory = 0
+        for i in range(torch.cuda.device_count()):
+            total_memory += (torch.cuda.max_memory_allocated(i) / 2**30)  # GB
+        self.max_gpu_memory_cost = max(self.max_gpu_memory_cost, total_memory)
+
+
+@dataclass
+class BenchmarkMetrics:
+    concurrency: int = 0
+    n_succeed_queries: int = 0
+    n_failed_queries: int = 0
+    total_first_chunk_latency: float = 0.0
+    total_latency: float = 0.0
+    n_total_chunks: int = 0
+    n_total_prompt_tokens: int = 0
+    n_total_completion_tokens: int = 0
+    total_chunks_time: float = 0.0
+    start_time: Optional[float] = None
+    total_time: float = 1.0
+    n_total_queries: int = 0
+
+    avg_first_chunk_latency: float = -1
+    avg_latency: float = -1
+    n_avg_chunks: float = -1
+    avg_chunk_time: float = -1
+    avg_prompt_tokens: float = -1
+    avg_completion_tokens: float = -1
+    avg_token_per_seconds: float = -1
+    avg_time_per_token: float = -1
+    qps: float = -1
+
+    def update_metrics(self, benchmark_data: BenchmarkData, api_plugin):
+        self.n_total_queries += 1
+        if self.start_time is None:
+            self.start_time = benchmark_data.start_time
+        self.total_time = time.perf_counter() - self.start_time
+
+        if benchmark_data.success:
+            self.n_succeed_queries += 1
+
+            benchmark_data._calculate_tokens(api_plugin)
+            self.n_total_prompt_tokens += benchmark_data.prompt_tokens
+            self.n_total_completion_tokens += benchmark_data.completion_tokens
+
+            benchmark_data._calculate_query_stream_metric()
+            self.total_latency += benchmark_data.query_latency
+            self.total_first_chunk_latency += benchmark_data.first_chunk_latency
+            self.n_total_chunks += benchmark_data.n_chunks
+            self.total_chunks_time += benchmark_data.n_chunks_time
+        else:
+            self.n_failed_queries += 1
+
+        self.calculate_averages()
+
+    def calculate_averages(self):
+        if self.n_succeed_queries == 0:
+            return
+        try:
+            self.avg_first_chunk_latency = self.total_first_chunk_latency / self.n_succeed_queries
+            self.avg_latency = self.total_latency / self.n_succeed_queries
+            self.n_avg_chunks = self.n_total_chunks / self.n_succeed_queries
+            self.avg_chunk_time = self.total_chunks_time / self.n_total_chunks
+            self.avg_prompt_tokens = self.n_total_prompt_tokens / self.n_succeed_queries
+            self.avg_completion_tokens = self.n_total_completion_tokens / self.n_succeed_queries
+            self.avg_token_per_seconds = self.n_total_completion_tokens / self.total_time
+            self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
+            self.qps = self.n_succeed_queries / self.total_time
+        except ZeroDivisionError as e:
+            logger.exception(e)
+            return
+
+    def create_message(self, default_ndigits=3):
+        message = {
+            'Time taken for tests (senconds)': round(self.total_time, default_ndigits),
+            'Number of concurrency': self.concurrency,
+            'Total requests': int(self.n_total_queries),
+            'Succeed requests': self.n_succeed_queries,
+            'Failed requests': self.n_failed_queries,
+            'Average QPS': round(self.qps, default_ndigits),
+            'Average latency (s)': round(self.avg_latency, default_ndigits),
+            'Average time to first token (s)': round(self.avg_first_chunk_latency, default_ndigits),
+            'Average time per output token (s)': round(self.avg_time_per_token, 5),
+            'Average package latency (s)': round(self.avg_chunk_time, default_ndigits),
+            'Average package per request': round(self.n_avg_chunks, default_ndigits),
+            'Throughput(average output tokens per second)': round(self.avg_token_per_seconds, default_ndigits),
+            'Average input tokens per request': round(self.avg_prompt_tokens, default_ndigits),
+            'Average output tokens per request': round(self.avg_completion_tokens, default_ndigits),
+        }
+        return message
diff --git a/evalscope/perf/utils/chat_service.py b/evalscope/perf/utils/chat_service.py
new file mode 100644
index 00000000..b724e4ef
--- /dev/null
+++ b/evalscope/perf/utils/chat_service.py
@@ -0,0 +1,252 @@
+import os
+import time
+from contextlib import contextmanager
+from functools import partial
+from threading import Thread
+from typing import List, Literal, Optional, Union
+
+import torch
+from modelscope import AutoModelForCausalLM, AutoTokenizer
+from pydantic import BaseModel, Field
+from transformers import TextIteratorStreamer
+
+
+class Usage(BaseModel):
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    total_tokens: int = 0
+
+
+class ModelCard(BaseModel):
+    id: str
+    object: str = 'model'
+    created: int = Field(default_factory=lambda: int(time.time()))
+    owned_by: str = 'owner'
+    root: Optional[str] = None
+    parent: Optional[str] = None
+    permission: Optional[list] = None
+
+
+class ModelList(BaseModel):
+    object: str = 'list'
+    data: List[ModelCard] = []
+
+
+class ChatMessage(BaseModel):
+    role: Literal['user', 'assistant', 'system']
+    content: str
+
+
+class DeltaMessage(BaseModel):
+    role: Optional[Literal['user', 'assistant', 'system']] = None
+    content: Optional[str] = None
+
+
+class ChatCompletionRequest(BaseModel):
+    model: str
+    messages: List[ChatMessage] | str
+    temperature: Optional[float] = None
+    top_p: Optional[float] = None
+    max_tokens: Optional[int] = 2048
+    min_tokens: Optional[int] = None
+    stream: Optional[bool] = False
+
+
+class ChatCompletionResponseChoice(BaseModel):
+    index: int
+    message: ChatMessage
+    finish_reason: Literal['stop', 'length']
+
+
+class ChatCompletionResponseStreamChoice(BaseModel):
+    index: int
+    delta: DeltaMessage
+    finish_reason: Optional[Literal['stop', 'length']]
+
+
+class ChatCompletionResponse(BaseModel):
+    model: str
+    object: Literal['chat.completion', 'chat.completion.chunk']
+    choices: List[Union[ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice]]
+    created: Optional[int] = Field(default_factory=lambda: int(time.time()))
+    usage: Optional[Usage]
+
+
+class TextCompletionRequest(BaseModel):
+    model: str
+    prompt: str
+    temperature: Optional[float] = None
+    max_tokens: Optional[int] = 2048
+    min_tokens: Optional[int] = None
+
+
+class TextCompletionResponseChoice(BaseModel):
+    index: int
+    text: str
+    finish_reason: Literal['stop', 'length']
+
+
+class TextCompletionResponse(BaseModel):
+    model: str
+    object: Literal['text_completion']
+    created: Optional[int] = Field(default_factory=lambda: int(time.time()))
+    choices: List[TextCompletionResponseChoice]
+    usage: Optional[Usage]
+
+
+class ChatService:
+
+    def __init__(self, model_path, attn_implementation):
+        self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_path,
+            trust_remote_code=True,
+            device_map='auto',
+            torch_dtype='auto',
+            attn_implementation=attn_implementation,
+        ).eval()
+        self.streamer = TextIteratorStreamer(self.tokenizer, skip_prompt=True)
+        self.model_id = os.path.basename(model_path)
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+    def count_tokens(self, text: str) -> int:
+        # Use the tokenizer to count the number of tokens
+        return len(self.tokenizer.encode(text, add_special_tokens=False))
+
+    async def list_models(self):
+        model_card = ModelCard(id=self.model_id)
+        return ModelList(data=[model_card])
+
+    async def _chat(self, request: ChatCompletionRequest):
+        formatted_prompt, inputs, prompt_tokens = self._prepare_chat_inputs(request)
+        outputs = self.model.generate(
+            **inputs,
+            max_new_tokens=request.max_tokens,
+            min_new_tokens=request.min_tokens,
+            temperature=request.temperature,
+        )
+        outputs = outputs[0][prompt_tokens:]  # remove prompt
+        completion_tokens = len(outputs)
+        response = self.tokenizer.decode(outputs, skip_special_tokens=True)
+
+        choice_data = ChatCompletionResponseChoice(
+            index=0,
+            message=ChatMessage(role='assistant', content=response),
+            finish_reason='stop',
+        )
+        return ChatCompletionResponse(
+            model=self.model_id,
+            choices=[choice_data],
+            object='chat.completion',
+            usage=Usage(
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=prompt_tokens + completion_tokens,
+            ),
+        )
+
+    async def _text_completion(self, request: TextCompletionRequest):
+        inputs, prompt_tokens = self._prepare_text_inputs(request)
+        outputs = self.model.generate(
+            **inputs,
+            max_new_tokens=request.max_tokens,
+            min_new_tokens=request.min_tokens,
+            temperature=request.temperature,
+        )
+        outputs = outputs[0][prompt_tokens:]  # remove prompt
+        completion_tokens = len(outputs)
+        response = self.tokenizer.decode(outputs, skip_special_tokens=True)
+
+        choice_data = TextCompletionResponseChoice(
+            index=0,
+            text=response,
+            finish_reason='stop',
+        )
+        return TextCompletionResponse(
+            model=self.model_id,
+            choices=[choice_data],
+            object='text_completion',
+            usage=Usage(
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=prompt_tokens + completion_tokens,
+            ),
+        )
+
+    def _prepare_text_inputs(self, request: TextCompletionRequest):
+        inputs = self.tokenizer(request.prompt, return_tensors='pt', padding=True).to(self.device)
+        prompt_tokens = len(inputs['input_ids'][0])
+        return inputs, prompt_tokens
+
+    def _stream_chat(self, request: ChatCompletionRequest):
+        formatted_prompt, inputs, prompt_tokens = self._prepare_chat_inputs(request)
+        completion_tokens = 0
+
+        yield self._create_initial_chunk()
+
+        generation_kwargs = dict(
+            **inputs,
+            streamer=self.streamer,
+            max_new_tokens=request.max_tokens,
+            min_new_tokens=request.min_tokens,
+            temperature=request.temperature,
+        )
+        generate_partial = partial(self.model.generate, **generation_kwargs)
+
+        with self._start_generation_thread(generate_partial):
+            for new_text in self.streamer:
+                yield self._create_chunk(new_text)
+                completion_tokens += self.count_tokens(new_text)
+
+        yield self._create_final_chunk(prompt_tokens, completion_tokens)
+        yield '[DONE]'
+
+    def _prepare_chat_inputs(self, request: ChatCompletionRequest):
+        formatted_prompt = self.tokenizer.apply_chat_template(
+            request.messages, tokenize=False, add_generation_prompt=True)
+        inputs = self.tokenizer(formatted_prompt, return_tensors='pt', padding=True).to(self.device)
+        prompt_tokens = len(inputs['input_ids'][0])
+        return formatted_prompt, inputs, prompt_tokens
+
+    @contextmanager
+    def _start_generation_thread(self, generate_partial):
+        thread = Thread(target=generate_partial)
+        thread.start()
+        try:
+            yield
+        finally:
+            thread.join()
+
+    def _create_initial_chunk(self):
+        choice_data = ChatCompletionResponseStreamChoice(index=0, delta={'role': 'assistant'}, finish_reason=None)
+        chunk = ChatCompletionResponse(
+            model=self.model_id,
+            choices=[choice_data],
+            object='chat.completion.chunk',
+            usage=None,
+        )
+        return chunk.model_dump_json(exclude_unset=True)
+
+    def _create_chunk(self, new_text):
+        choice_data = ChatCompletionResponseStreamChoice(index=0, delta={'content': new_text}, finish_reason=None)
+        chunk = ChatCompletionResponse(
+            model=self.model_id,
+            choices=[choice_data],
+            object='chat.completion.chunk',
+            usage=None,
+        )
+        return chunk.model_dump_json(exclude_unset=True)
+
+    def _create_final_chunk(self, prompt_tokens, completion_tokens):
+        choice_data = ChatCompletionResponseStreamChoice(index=0, delta={}, finish_reason='stop')
+        chunk = ChatCompletionResponse(
+            model=self.model_id,
+            choices=[choice_data],
+            object='chat.completion.chunk',
+            usage=Usage(
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=prompt_tokens + completion_tokens,
+            ),
+        )
+        return chunk.model_dump_json(exclude_unset=True)
diff --git a/evalscope/perf/utils/db_util.py b/evalscope/perf/utils/db_util.py
new file mode 100644
index 00000000..d045434d
--- /dev/null
+++ b/evalscope/perf/utils/db_util.py
@@ -0,0 +1,200 @@
+import base64
+import os
+import pickle
+import sqlite3
+import sys
+from datetime import datetime
+
+import json
+from tabulate import tabulate
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.utils.benchmark_util import BenchmarkData, BenchmarkMetrics
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def encode_data(data) -> str:
+    """Encodes data using base64 and pickle."""
+    return base64.b64encode(pickle.dumps(data)).decode('utf-8')
+
+
+def write_json_file(data, output_path):
+    with open(output_path, 'w') as f:
+        json.dump(data, f, indent=4, ensure_ascii=False)
+
+
+def transpose_results(data):
+    headers = data.keys()
+    rows = zip(*data.values())
+
+    return [dict(zip(headers, row)) for row in rows]
+
+
+def create_result_table(cursor):
+    cursor.execute('''CREATE TABLE IF NOT EXISTS result(
+                      request TEXT,
+                      start_time REAL,
+                      chunk_times TEXT,
+                      success INTEGER,
+                      response_messages TEXT,
+                      completed_time REAL,
+                      latency REAL,
+                      first_chunk_latency REAL,
+                      n_chunks INTEGER,
+                      chunk_time REAL,
+                      prompt_tokens INTEGER,
+                      completion_tokens INTEGER,
+                      max_gpu_memory_cost REAL)''')
+
+
+def insert_benchmark_data(cursor: sqlite3.Cursor, benchmark_data: BenchmarkData):
+    request = encode_data(benchmark_data.request)
+    chunk_times = json.dumps(benchmark_data.chunk_times)
+    response_messages = encode_data(benchmark_data.response_messages)
+
+    # Columns common to both success and failure cases
+    common_columns = (
+        request,
+        benchmark_data.start_time,
+        chunk_times,
+        benchmark_data.success,
+        response_messages,
+        benchmark_data.completed_time,
+    )
+
+    if benchmark_data.success:
+        # Add additional columns for success case
+        additional_columns = (
+            benchmark_data.query_latency,
+            benchmark_data.first_chunk_latency,
+            benchmark_data.n_chunks,
+            benchmark_data.n_chunks_time,
+            benchmark_data.prompt_tokens,
+            benchmark_data.completion_tokens,
+            benchmark_data.max_gpu_memory_cost,
+        )
+        query = """INSERT INTO result(
+                      request, start_time, chunk_times, success, response_messages,
+                      completed_time, latency, first_chunk_latency,
+                      n_chunks, chunk_time, prompt_tokens, completion_tokens, max_gpu_memory_cost
+                   ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"""
+        cursor.execute(query, common_columns + additional_columns)
+    else:
+        query = """INSERT INTO result(
+                      request, start_time, chunk_times, success, response_messages, completed_time
+                   ) VALUES (?, ?, ?, ?, ?, ?)"""
+        cursor.execute(query, common_columns)
+
+
+def get_result_db_path(name, model):
+    current_time = datetime.now().strftime('%Y%m%d_%H%M%S')
+    output_dir = './outputs'
+    result_db_path = os.path.join(output_dir, f'{name or model}_perf', current_time, 'benchmark_data.db')
+
+    if not os.path.exists(os.path.dirname(result_db_path)):
+        os.makedirs(os.path.dirname(result_db_path), exist_ok=True)
+
+    logger.info(f'Save the result to: {result_db_path}')
+    if os.path.exists(result_db_path):
+        logger.warning('The db file exists, delete it and start again!.')
+        sys.exit(1)
+
+    return result_db_path
+
+
+def get_percentile_results(result_db_path: str):
+
+    def percentile_results(rows, index, percentiles):
+        results = {}
+        n_success_queries = len(rows)
+        for percentile in percentiles:
+            idx = int(n_success_queries * percentile / 100)
+            row = rows[idx]
+            value = row[index] if row[index] is not None else float('inf')
+            results[percentile] = round(value, 4)
+        return results
+
+    query_sql = ('SELECT start_time, chunk_times, success, completed_time, latency, first_chunk_latency, '
+                 'n_chunks, chunk_time, prompt_tokens, completion_tokens '
+                 'FROM result WHERE success=1 ORDER BY first_chunk_latency ASC')
+    percentiles = [10, 25, 50, 66, 75, 80, 90, 95, 98, 99]
+
+    with sqlite3.connect(result_db_path) as con:
+        rows = con.execute(query_sql).fetchall()
+
+    if len(rows) <= len(percentiles):
+        logger.info('Too little data to calculate quantiles!')
+        return {}
+
+    # Calculate percentiles for first chunk latency and latency
+    first_chunk_latency_index = 5
+    latency_index = 4
+
+    first_chunk_latency_results = percentile_results(rows, first_chunk_latency_index, percentiles)
+    rows.sort(key=lambda x: x[latency_index])
+    latency_results = percentile_results(rows, latency_index, percentiles)
+
+    # Prepare data for tabulation
+    return {
+        'Percentile': [f'{p}%' for p in percentiles],
+        'First Chunk Latency (s)': [first_chunk_latency_results[p] for p in percentiles],
+        'Latency (s)': [latency_results[p] for p in percentiles]
+    }
+
+
+def summary_result(args: Arguments, metrics: BenchmarkMetrics, expected_number_of_queries: int, result_db_path: str):
+    result_path = os.path.dirname(result_db_path)
+    write_json_file(args.to_dict(), os.path.join(result_path, 'benchmark_args.json'))
+
+    data = metrics.create_message()
+    data.update({'Expected number of requests': expected_number_of_queries, 'Result DB path': result_db_path})
+    write_json_file(data, os.path.join(result_path, 'benchmark_summary.json'))
+
+    # Print summary in a table
+    table = tabulate(list(data.items()), headers=['Key', 'Value'], tablefmt='grid')
+    logger.info('\nBenchmarking summary:\n' + table)
+
+    # Get percentile results
+    percentile_result = get_percentile_results(result_db_path)
+    if percentile_result:
+        write_json_file(transpose_results(percentile_result), os.path.join(result_path, 'benchmark_percentile.json'))
+        # Print percentile results in a table
+        table = tabulate(percentile_result, headers='keys', tablefmt='pretty')
+        logger.info('\nPercentile results:\n' + table)
+
+    if args.dataset.startswith('speed_benchmark'):
+        speed_benchmark_result(result_db_path)
+
+
+def speed_benchmark_result(result_db_path: str):
+    query_sql = """
+        SELECT
+            prompt_tokens,
+            ROUND(AVG(completion_tokens / latency), 2) AS avg_completion_token_per_second,
+            ROUND(AVG(max_gpu_memory_cost), 2)
+        FROM
+            result
+        WHERE
+            success = 1 AND latency > 0
+        GROUP BY
+            prompt_tokens
+    """
+
+    with sqlite3.connect(result_db_path) as con:
+        cursor = con.cursor()
+        cursor.execute(query_sql)
+        rows = cursor.fetchall()
+
+    # Prepare data for tabulation
+    headers = ['Prompt Tokens', 'Speed(tokens/s)', 'GPU Memory(GB)']
+    data = [dict(zip(headers, row)) for row in rows]
+
+    # Print results in a table
+    table = tabulate(data, headers='keys', tablefmt='pretty')
+    logger.info('\nSpeed Benchmark Results:\n' + table)
+
+    # Write results to JSON file
+    result_path = os.path.dirname(result_db_path)
+    write_json_file(data, os.path.join(result_path, 'speed_benchmark.json'))
diff --git a/evalscope/perf/utils/handler.py b/evalscope/perf/utils/handler.py
new file mode 100644
index 00000000..20729834
--- /dev/null
+++ b/evalscope/perf/utils/handler.py
@@ -0,0 +1,46 @@
+import asyncio
+import functools
+import signal
+import sys
+
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def exception_handler(func):
+    if asyncio.iscoroutinefunction(func):
+
+        @functools.wraps(func)
+        async def async_wrapper(*args, **kwargs):
+            try:
+                return await func(*args, **kwargs)
+            except Exception as e:
+                logger.exception(f"Exception in async function '{func.__name__}': {e}")
+                sys.exit(1)
+
+        return async_wrapper
+    else:
+
+        @functools.wraps(func)
+        def sync_wrapper(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except Exception as e:
+                logger.exception(f"Exception in function '{func.__name__}': {e}")
+                sys.exit(1)
+
+        return sync_wrapper
+
+
+def signal_handler(signal_name, loop):
+    logger.info('Got signal %s: exit' % signal_name)
+    loop.stop()
+
+
+def add_signal_handlers(loop):
+    for signal_name in {'SIGINT', 'SIGTERM'}:
+        loop.add_signal_handler(
+            getattr(signal, signal_name),
+            functools.partial(signal_handler, signal_name, loop),
+        )
diff --git a/evalscope/perf/utils/local_server.py b/evalscope/perf/utils/local_server.py
new file mode 100644
index 00000000..fdb5f2c0
--- /dev/null
+++ b/evalscope/perf/utils/local_server.py
@@ -0,0 +1,139 @@
+import os
+import subprocess
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
+
+import torch
+import uvicorn
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from sse_starlette.sse import EventSourceResponse
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.utils.chat_service import ChatCompletionRequest, ChatService, ModelList, TextCompletionRequest
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+@dataclass
+class ServerSentEvent(object):
+
+    def __init__(self, data='', event=None, id=None, retry=None):
+        self.data = data
+        self.event = event
+        self.id = id
+        self.retry = retry
+
+    @classmethod
+    def decode(cls, line):
+        """Decode line to ServerSentEvent
+
+
+        Args:
+            line (str): The line.
+
+        Return:
+            ServerSentEvent (obj:`ServerSentEvent`): The ServerSentEvent object.
+
+        """
+        if not line:
+            return None
+        sse_msg = cls()
+        # format data:xxx
+        field_type, _, field_value = line.partition(':')
+        if field_value.startswith(' '):  # compatible with openai api
+            field_value = field_value[1:]
+        if field_type == 'event':
+            sse_msg.event = field_value
+        elif field_type == 'data':
+            field_value = field_value.rstrip()
+            sse_msg.data = field_value
+        elif field_type == 'id':
+            sse_msg.id = field_value
+        elif field_type == 'retry':
+            sse_msg.retry = field_value
+        else:
+            pass
+
+        return sse_msg
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    yield
+    if torch.cuda.is_available():
+        torch.cuda.empty_cache()
+
+
+def create_app(args) -> FastAPI:
+    app = FastAPI(lifespan=lifespan)
+    chat_service = ChatService(model_path=args.model, attn_implementation=args.attn_implementation)
+
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=['*'],
+        allow_credentials=True,
+        allow_methods=['*'],
+        allow_headers=['*'],
+    )
+
+    @app.get('/v1/models', response_model=ModelList)
+    async def list_models():
+        return await chat_service.list_models()
+
+    @app.post('/v1/completions')
+    async def create_text_completion(request: TextCompletionRequest):
+        return await chat_service._text_completion(request)
+
+    @app.post('/v1/chat/completions')
+    async def create_chat_completion(request: ChatCompletionRequest):
+        if request.stream:
+            return EventSourceResponse(chat_service._stream_chat(request))
+        else:
+            return await chat_service._chat(request)
+
+    return app
+
+
+def start_app(args: Arguments):
+    if args.api == 'local':
+        app = create_app(args)
+        uvicorn.run(app, host='0.0.0.0', port=8877, workers=1)
+
+    elif args.api == 'local_vllm':
+        os.environ['VLLM_USE_MODELSCOPE'] = 'True'
+
+        proc = subprocess.Popen([
+            'python', '-m', 'vllm.entrypoints.openai.api_server', '--model', args.model, '--served-model-name',
+            os.path.basename(args.model), '--tensor-parallel-size',
+            str(torch.cuda.device_count()), '--max-model-len', '32768', '--gpu-memory-utilization', '0.9', '--host',
+            '0.0.0.0', '--port', '8877', '--disable-log-requests', '--disable-log-stats'
+        ])
+        import atexit
+
+        def on_exit():
+            if proc.poll() is None:
+                logger.info('Terminating the child process...')
+                proc.terminate()
+                try:
+                    proc.wait(timeout=10)
+                except subprocess.TimeoutExpired:
+                    logger.warning('Child process did not terminate within the timeout, killing it forcefully...')
+                    proc.kill()
+                    proc.wait()
+                logger.info('Child process terminated.')
+            else:
+                logger.info('Child process has already terminated.')
+
+        atexit.register(on_exit)
+    else:
+        raise ValueError(f'Unknown API type: {args.api}')
+
+
+if __name__ == '__main__':
+    from collections import namedtuple
+
+    args = namedtuple('Args', ['model', 'attn_implementation', 'api'])
+
+    start_app(args(model='Qwen/Qwen2.5-0.5B-Instruct', attn_implementation=None, api='local_vllm'))
diff --git a/evalscope/perf/zhipu/generate_zhipu_token.py b/evalscope/perf/zhipu/generate_zhipu_token.py
deleted file mode 100644
index 253a90ab..00000000
--- a/evalscope/perf/zhipu/generate_zhipu_token.py
+++ /dev/null
@@ -1,34 +0,0 @@
-
-import time
-import jwt
-import argparse
-# pip install PyJWT
-def generate_token(apikey: str, exp_seconds: int):
-    try:
-        id, secret = apikey.split(".")
-    except Exception as e:
-        raise Exception("invalid apikey", e)
- 
-    payload = {
-        "api_key": id,
-        "exp": int(round(time.time() * 1000)) + exp_seconds * 1000,
-        "timestamp": int(round(time.time() * 1000)),
-    }
- 
-    return jwt.encode(
-        payload,
-        secret,
-        algorithm="HS256",
-        headers={"alg": "HS256", "sign_type": "SIGN"},
-    )
-    
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description="Generate zhipu glm api token, you need install PyJWT with: pip install PyJWT")
-    parser.add_argument("--api-key", type=str, required=True, 
-                        help="The zhipu api key.")
-    parser.add_argument("--exp-seconds", type=int, default=24*60*60,
-                        help='The token expiration seconds')
-    args = parser.parse_args()
-    print('Token generated: %s' % generate_token(args.api_key, args.exp_seconds))
-    
diff --git a/evalscope/preprocess/tokenizers/__init__.py b/evalscope/preprocess/tokenizers/__init__.py
deleted file mode 100644
index e69de29b..00000000
diff --git a/evalscope/preprocess/tokenizers/gpt2_tokenizer.py b/evalscope/preprocess/tokenizers/gpt2_tokenizer.py
deleted file mode 100644
index 46166255..00000000
--- a/evalscope/preprocess/tokenizers/gpt2_tokenizer.py
+++ /dev/null
@@ -1,221 +0,0 @@
-import logging
-import sys
-from functools import lru_cache
-from typing import Sequence
-
-import json
-import regex as re
-
-logger = logging.getLogger(__name__)
-
-
-def get_pairs(word):
-    """Return set of symbol pairs in a word.
-
-    Word is represented as tuple of symbols (symbols being variable-length strings).
-    """
-    pairs = set()
-    prev_char = word[0]
-    for char in word[1:]:
-        pairs.add((prev_char, char))
-        prev_char = char
-    return pairs
-
-
-@lru_cache()
-def bytes_to_unicode():
-    """
-    Returns list of utf-8 byte and a corresponding list of unicode strings.
-    The reversible bpe codes work on unicode strings.
-    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
-    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
-    This is a signficant percentage of your normal, say, 32K bpe vocab.
-    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
-    And avoids mapping to whitespace/control characters the bpe code barfs on.
-    """
-    bs = list(range(ord('!'),
-                    ord('~') + 1)) + list(range(
-                        ord('¡'),
-                        ord('¬') + 1)) + list(range(ord('®'),
-                                                    ord('ÿ') + 1))
-    cs = bs[:]
-    n = 0
-    for b in range(2**8):
-        if b not in bs:
-            bs.append(b)
-            cs.append(2**8 + n)
-            n += 1
-    cs = [chr(n) for n in cs]
-    return dict(zip(bs, cs))
-
-
-class GPT2Tokenizer(object):
-    """
-    GPT-2 BPE tokenizer. Peculiarities:
-        - Byte-level BPE
-
-    [NOTE]: Copied from megatron.tokenizer.gpt2_tokenization.GPT2Tokenizer.
-    """
-
-    def __init__(self,
-                 vocab_file,
-                 merges_file,
-                 errors='replace',
-                 special_tokens=None,
-                 max_len=None):
-        assert sys.version_info[0] != 2
-
-        self.max_len = max_len if max_len is not None else int(1e12)
-        self.encoder = json.load(open(vocab_file))
-        self.decoder = {v: k for k, v in self.encoder.items()}
-        self.errors = errors  # how to handle errors in decoding
-        self.byte_encoder = bytes_to_unicode()
-        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
-        bpe_data = open(merges_file, encoding='utf-8').read().split('\n')[1:-1]
-        bpe_merges = [tuple(merge.split()) for merge in bpe_data]
-        self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
-        self.cache = {}
-
-        # Should haved added re.IGNORECASE so BPE merges can happen for
-        # capitalized versions of contractions
-        self.pat = re.compile(
-            r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
-        )
-
-        self.special_tokens = {}
-        self.special_tokens_decoder = {}
-        self.set_special_tokens(special_tokens)
-
-    def __len__(self):
-        return len(self.encoder) + len(self.special_tokens)
-
-    def set_special_tokens(self, special_tokens):
-        """ Add a list of additional tokens to the encoder.
-            The additional tokens are indexed starting from the last index of the
-            current vocabulary in the order of the `special_tokens` list.
-        """
-        if not special_tokens:
-            self.special_tokens = {}
-            self.special_tokens_decoder = {}
-            return
-        self.special_tokens = dict((tok, len(self.encoder) + i)
-                                   for i, tok in enumerate(special_tokens))
-        self.special_tokens_decoder = {
-            v: k
-            for k, v in self.special_tokens.items()
-        }
-        logger.info('Special tokens {}'.format(self.special_tokens))
-
-    def bpe(self, token):
-        if token in self.cache:
-            return self.cache[token]
-        word = tuple(token)
-        pairs = get_pairs(word)
-
-        if not pairs:
-            return token
-
-        while True:
-            bigram = min(
-                pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf')))
-            if bigram not in self.bpe_ranks:
-                break
-            first, second = bigram
-            new_word = []
-            i = 0
-            while i < len(word):
-                try:
-                    j = word.index(first, i)
-                    new_word.extend(word[i:j])
-                    i = j
-                except BaseException:
-                    new_word.extend(word[i:])
-                    break
-
-                if word[i] == first and i < len(word) - 1 and word[
-                        i + 1] == second:
-                    new_word.append(first + second)
-                    i += 2
-                else:
-                    new_word.append(word[i])
-                    i += 1
-            new_word = tuple(new_word)
-            word = new_word
-            if len(word) == 1:
-                break
-            else:
-                pairs = get_pairs(word)
-        word = ' '.join(word)
-        self.cache[token] = word
-        return word
-
-    def tokenize(self, text):
-        """ Tokenize a string. """
-        bpe_tokens = []
-        for token in re.findall(self.pat, text):
-            if sys.version_info[0] == 2:
-                token = ''.join(self.byte_encoder[ord(b)] for b in token)
-            else:
-                token = ''.join(self.byte_encoder[b]
-                                for b in token.encode('utf-8'))
-            bpe_tokens.extend(
-                bpe_token for bpe_token in self.bpe(token).split(' '))
-        return bpe_tokens
-
-    def convert_tokens_to_ids(self, tokens):
-        """ Converts a sequence of tokens into ids using the vocab. """
-        ids = []
-        if isinstance(tokens, str):
-            if tokens in self.special_tokens:
-                return self.special_tokens[tokens]
-            else:
-                return self.encoder.get(tokens, 0)
-        for token in tokens:
-            if token in self.special_tokens:
-                ids.append(self.special_tokens[token])
-            else:
-                ids.append(self.encoder.get(token, 0))
-        if len(ids) > self.max_len:
-            logger.warning(
-                'Token indices sequence length is longer than the specified maximum '
-                ' sequence length for this OpenAI GPT model ({} > {}). Running this'
-                ' sequence through the model will result in indexing errors'.
-                format(len(ids), self.max_len))
-        return ids
-
-    def convert_ids_to_tokens(self, ids, skip_special_tokens=False):
-        """Converts a sequence of ids in BPE tokens using the vocab."""
-        tokens = []
-        for i in ids:
-            if i in self.special_tokens_decoder:
-                if not skip_special_tokens:
-                    tokens.append(self.special_tokens_decoder[i])
-            else:
-                tokens.append(self.decoder[i])
-        return tokens
-
-    def encode(self, text):
-        return self.convert_tokens_to_ids(self.tokenize(text))
-
-    def decode(self, tokens):
-        text = ''.join([self.decoder[token] for token in tokens])
-        text = bytearray([self.byte_decoder[c] for c in text]).decode(
-            'utf-8', errors=self.errors)
-        return text
-
-
-class DummyTokenizer:
-
-    def tokenize(self, text: str):
-        return text.split()
-
-
-def get_tokenized_string(tokenizer: GPT2Tokenizer, text_list: Sequence[str]):
-    token_ids_list, tokenized_string_list = [], []
-    for text in text_list:
-        assert tokenizer is not None
-        token_ids = tokenizer.encode(text)
-        tokenized_string = ' '.join(tokenizer.convert_ids_to_tokens(token_ids))
-        token_ids_list.append(token_ids)
-        tokenized_string_list.append(tokenized_string)
-    return token_ids_list, tokenized_string_list
diff --git a/evalscope/utils/logger.py b/evalscope/utils/logger.py
index db87cc47..5738cb91 100644
--- a/evalscope/utils/logger.py
+++ b/evalscope/utils/logger.py
@@ -1,19 +1,19 @@
-# Copyright (c) Alibaba, Inc. and its affiliates.
-
 import importlib.util as iutil
 import logging
 from typing import Optional
 
 init_loggers = {}
-format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
-formatter = logging.Formatter(format)
 
-logging.basicConfig(format=format, level=logging.INFO)
+detailed_format = '%(asctime)s - %(name)s - %(filename)s - %(funcName)s - %(lineno)d - %(levelname)s - %(message)s'
+simple_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+
+detailed_formatter = logging.Formatter(detailed_format)
+simple_formatter = logging.Formatter(simple_format)
+
+logging.basicConfig(format=simple_format, level=logging.INFO)
 
 
-def get_logger(
-    log_file: Optional[str] = None, log_level: int = logging.INFO, file_mode: str = "w"
-):
+def get_logger(log_file: Optional[str] = None, log_level: int = logging.INFO, file_mode: str = 'w'):
     """Get logging logger
 
     Args:
@@ -24,25 +24,23 @@ def get_logger(
             specified (if filemode is unspecified, it defaults to 'w').
     """
 
-    logger_name = __name__.split(".")[0]
+    logger_name = __name__.split('.')[0]
     logger = logging.getLogger(logger_name)
     logger.propagate = False
+
     if logger_name in init_loggers:
-        add_file_handler_if_needed(logger, log_file, file_mode, log_level)
         if logger.level != log_level:
             logger.setLevel(log_level)
+        add_file_handler_if_needed(logger, log_file, file_mode, log_level)
+        for handler in logger.handlers:
+            handler.setLevel(log_level)
+            handler.setFormatter(detailed_formatter if log_level == logging.DEBUG else simple_formatter)
         return logger
 
     # handle duplicate logs to the console
-    # Starting in 1.8.0, PyTorch DDP attaches a StreamHandler <stderr> (NOTSET)
-    # to the root logger. As logger.propagate is True by default, this root
-    # level handler causes logging messages from rank>0 processes to
-    # unexpectedly show up on the console, creating much unwanted clutter.
-    # To fix this issue, we set the root logger's StreamHandler, if any, to log
-    # at the ERROR level.
     torch_dist = False
     is_worker0 = True
-    if iutil.find_spec("torch") is not None:
+    if iutil.find_spec('torch') is not None:
         from modelscope.utils.torch_utils import is_dist, is_master
 
         torch_dist = is_dist()
@@ -61,7 +59,7 @@ def get_logger(
         handlers.append(file_handler)
 
     for handler in handlers:
-        handler.setFormatter(formatter)
+        handler.setFormatter(detailed_formatter if log_level == logging.DEBUG else simple_formatter)
         handler.setLevel(log_level)
         logger.addHandler(handler)
 
@@ -80,7 +78,7 @@ def add_file_handler_if_needed(logger, log_file, file_mode, log_level):
         if isinstance(handler, logging.FileHandler):
             return
 
-    if iutil.find_spec("torch") is not None:
+    if iutil.find_spec('torch') is not None:
         from modelscope.utils.torch_utils import is_master
 
         is_worker0 = is_master()
@@ -89,6 +87,6 @@ def add_file_handler_if_needed(logger, log_file, file_mode, log_level):
 
     if is_worker0 and log_file is not None:
         file_handler = logging.FileHandler(log_file, file_mode)
-        file_handler.setFormatter(formatter)
+        file_handler.setFormatter(detailed_formatter if log_level == logging.DEBUG else simple_formatter)
         file_handler.setLevel(log_level)
         logger.addHandler(file_handler)
diff --git a/evalscope/utils/utils.py b/evalscope/utils/utils.py
index b2c183b4..dba27605 100644
--- a/evalscope/utils/utils.py
+++ b/evalscope/utils/utils.py
@@ -2,18 +2,20 @@
 # Copyright (c) OpenCompass.
 
 import functools
+import hashlib
 import importlib
 import importlib.util
 import os
-import re
-import json
 import random
+import re
 import sys
-from typing import Any, Union, Dict, Tuple, List
-import hashlib
-import torch.nn.functional as F
+from typing import Any, Dict, List, Tuple, Union
 
+import json
 import jsonlines as jsonl
+import numpy as np
+import torch
+import torch.nn.functional as F
 import yaml
 
 from evalscope.constants import DumpMode, OutputsStructure
@@ -30,9 +32,7 @@
 def test_level_list():
     global TEST_LEVEL_LIST
     if TEST_LEVEL_LIST_STR in os.environ:
-        TEST_LEVEL_LIST = [
-            int(x) for x in os.environ[TEST_LEVEL_LIST_STR].split(',')
-        ]
+        TEST_LEVEL_LIST = [int(x) for x in os.environ[TEST_LEVEL_LIST_STR].split(',')]
 
     return TEST_LEVEL_LIST
 
@@ -49,8 +49,7 @@ def jsonl_to_list(jsonl_file):
     """
     res_list = []
     with jsonl.open(jsonl_file, mode='r') as reader:
-        for line in reader.iter(
-                type=dict, allow_none=True, skip_invalid=False):
+        for line in reader.iter(type=dict, allow_none=True, skip_invalid=False):
             res_list.append(line)
     return res_list
 
@@ -283,12 +282,12 @@ def parse_first_option(text: str) -> str:
             text: The text to parse.
         """
         patterns = [
-            r"[Aa]nswer:\s*(\w+)",
-            r"[Tt]he correct answer is:\s*(\w+)",
-            r"[Tt]he correct answer is:\n\s*(\w+)",
-            r"[Tt]he correct answer is:\n\n-\s*(\w+)",
-            r"[Tt]he answer might be:\n\n-\s*(\w+)",
-            r"[Tt]he answer is \s*(\w+)",
+            r'[Aa]nswer:\s*(\w+)',
+            r'[Tt]he correct answer is:\s*(\w+)',
+            r'[Tt]he correct answer is:\n\s*(\w+)',
+            r'[Tt]he correct answer is:\n\n-\s*(\w+)',
+            r'[Tt]he answer might be:\n\n-\s*(\w+)',
+            r'[Tt]he answer is \s*(\w+)',
         ]
 
         regexes = [re.compile(pattern) for pattern in patterns]
@@ -329,8 +328,7 @@ def make_outputs_dir(root_dir: str, datasets: list, model_id: str, model_revisio
     if not model_revision:
         model_revision = 'default'
 
-    outputs_dir = os.path.join(root_dir,
-                               f"eval_{'-'.join(datasets)}_{model_id}_{model_revision}")
+    outputs_dir = os.path.join(root_dir, f"eval_{'-'.join(datasets)}_{model_id}_{model_revision}")
 
     return outputs_dir
 
@@ -419,22 +417,15 @@ def split_str_parts_by(text: str, delimiters: List[str]):
 
     while len(text) > 0:
         for char_idx, char in enumerate(text):
-            match_index = [
-                idx for idx, start_char in enumerate(all_start_chars)
-                if start_char == char
-            ]
+            match_index = [idx for idx, start_char in enumerate(all_start_chars) if start_char == char]
             is_delimiter = False
             for index in match_index:
-                if text[char_idx:char_idx
-                        + all_length[index]] == delimiters[index]:
+                if text[char_idx:char_idx + all_length[index]] == delimiters[index]:
                     if last_words:
                         if text_list:
                             text_list[-1]['content'] = last_words
                         else:
-                            text_list.append({
-                                'key': '',
-                                'content': last_words
-                            })
+                            text_list.append({'key': '', 'content': last_words})
                     last_words = ''
                     text_list.append({'key': delimiters[index]})
                     text = text[char_idx + all_length[index]:]
@@ -451,9 +442,7 @@ def split_str_parts_by(text: str, delimiters: List[str]):
     return text_list
 
 
-def calculate_loss_scale(response: str,
-                         use_loss_scale=False
-                         ) -> Tuple[List[str], List[float]]:
+def calculate_loss_scale(response: str, use_loss_scale=False) -> Tuple[List[str], List[float]]:
     """Calculate the loss scale by splitting the agent response.
     This algorithm comes from paper: https://arxiv.org/pdf/2309.00986.pdf
     Agent response format:
@@ -474,10 +463,7 @@ def calculate_loss_scale(response: str,
         A tuple of agent response parts and their weights.
     """
     if 'Action:' in response and 'Observation:' in response and use_loss_scale:
-        agent_keyword = [
-            'Action:', 'Action Input:', 'Thought:', 'Final Answer:',
-            'Observation:'
-        ]
+        agent_keyword = ['Action:', 'Action Input:', 'Thought:', 'Final Answer:', 'Observation:']
         agent_parts = split_str_parts_by(response, agent_keyword)
         weights = []
         agent_content = []
@@ -518,17 +504,15 @@ def _get_closet_bucket(bucket_sizes, data_length):
     return cloest_length
 
 
-def pad_and_split_batch(padding_to, input_ids, attention_mask, labels,
-                        loss_scale, max_length, tokenizer, rank, world_size):
+def pad_and_split_batch(padding_to, input_ids, attention_mask, labels, loss_scale, max_length, tokenizer, rank,
+                        world_size):
     if padding_to is None:
         longest_len = input_ids.shape[-1]
         bucket_sizes = get_bucket_sizes(max_length)
         bucket_data_length = _get_closet_bucket(bucket_sizes, longest_len)
         padding_length = bucket_data_length - input_ids.shape[1]
-        input_ids = F.pad(input_ids, (0, padding_length), 'constant',
-                          tokenizer.pad_token_id)
-        attention_mask = F.pad(attention_mask, (0, padding_length), 'constant',
-                               0)
+        input_ids = F.pad(input_ids, (0, padding_length), 'constant', tokenizer.pad_token_id)
+        attention_mask = F.pad(attention_mask, (0, padding_length), 'constant', 0)
         if loss_scale:
             loss_scale = F.pad(loss_scale, (0, padding_length), 'constant', 0.)
         labels = F.pad(labels, (0, padding_length), 'constant', -100)
@@ -607,7 +591,7 @@ def get_latest_folder_path(work_dir):
 
     # timestamp parser
     def parse_timestamp(folder_name):
-        return datetime.strptime(folder_name, "%Y%m%d_%H%M%S")
+        return datetime.strptime(folder_name, '%Y%m%d_%H%M%S')
 
     # Find the latest folder
     latest_folder = max(timestamped_folders, key=parse_timestamp)
@@ -623,3 +607,18 @@ def csv_to_list(file_path: str) -> List[dict]:
         result = [row for row in csv_reader]
 
     return result
+
+
+def seed_everything(seed: int):
+    """Set all random seeds to a fixed value for reproducibility.
+
+    Args:
+        seed (int): The seed value.
+    """
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(seed)
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False
diff --git a/examples/example_eval_perf.py b/examples/example_eval_perf.py
new file mode 100644
index 00000000..c0c05bf5
--- /dev/null
+++ b/examples/example_eval_perf.py
@@ -0,0 +1,97 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+from evalscope.perf.main import run_perf_benchmark
+
+
+def run_perf():
+    task_cfg = {
+        'url': 'http://127.0.0.1:8000/v1/chat/completions',
+        'parallel': 1,
+        'model': 'qwen2.5',
+        'number': 15,
+        'api': 'openai',
+        'dataset': 'openqa',
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_stream():
+    task_cfg = {
+        'url': 'http://127.0.0.1:8000/v1/chat/completions',
+        'parallel': 1,
+        'model': 'qwen2.5',
+        'number': 15,
+        'api': 'openai',
+        'dataset': 'openqa',
+        'stream': True,
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_speed_benchmark():
+    task_cfg = {
+        'url': 'http://127.0.0.1:8000/v1/completions',
+        'parallel': 1,
+        'model': 'qwen2.5',
+        'api': 'openai',
+        'dataset': 'speed_benchmark',
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_local():
+    task_cfg = {
+        'parallel': 1,
+        'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+        'number': 5,
+        'api': 'local',
+        'dataset': 'openqa',
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_local_stream():
+    task_cfg = {
+        'parallel': 1,
+        'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+        'number': 5,
+        'api': 'local',
+        'dataset': 'openqa',
+        'stream': True,
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_local_speed_benchmark():
+    task_cfg = {
+        'parallel': 1,
+        'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+        'api': 'local_vllm',
+        'dataset': 'speed_benchmark',
+        'min_tokens': 2048,
+        'max_tokens': 2048,
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+def run_perf_local_custom_prompt():
+    task_cfg = {
+        'parallel': 1,
+        'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+        'api': 'local',
+        'number': 10,
+        'prompt': '写一个诗歌',
+        'min_tokens': 100,
+        'max_tokens': 1024,
+        'debug': True,
+    }
+    run_perf_benchmark(task_cfg)
+
+
+if __name__ == '__main__':
+    run_perf_local_custom_prompt()
diff --git a/requirements/framework.txt b/requirements/framework.txt
index 956a45c4..7b285996 100644
--- a/requirements/framework.txt
+++ b/requirements/framework.txt
@@ -1,11 +1,10 @@
-# pip3 install --no-dependencies -r requirements/framework.txt
-torch
 absl-py
 #auto-gptq
 accelerate
 cachetools
 datasets>=3.0.0,<=3.0.1
 editdistance
+jieba
 jsonlines
 matplotlib
 modelscope[framework]
@@ -19,6 +18,7 @@ pyyaml
 regex
 requests
 requests-toolbelt
+rouge-chinese
 rouge-score>=0.1.0
 sacrebleu
 scikit-learn
@@ -28,9 +28,9 @@ simple-ddl-parser
 # streamlit
 tabulate
 tiktoken
+# pip3 install --no-dependencies -r requirements/framework.txt
+torch
 tqdm
 transformers>=4.33
 transformers_stream_generator
-jieba
-rouge-chinese
-# wandb
\ No newline at end of file
+# wandb
diff --git a/evalscope/perf/requirements.txt b/requirements/perf.txt
similarity index 66%
rename from evalscope/perf/requirements.txt
rename to requirements/perf.txt
index 19aec29d..ee4abff3 100644
--- a/evalscope/perf/requirements.txt
+++ b/requirements/perf.txt
@@ -1,6 +1,9 @@
 # http requests
 aiohttp
+fastapi
+# for calc result.
+numpy
+sse_starlette
 # for tokenizer
 transformers
-# for calc result.
-numpy
\ No newline at end of file
+unicorn
diff --git a/requirements/vlmeval.txt b/requirements/vlmeval.txt
index 1f811d37..2f617dee 100644
--- a/requirements/vlmeval.txt
+++ b/requirements/vlmeval.txt
@@ -1,1 +1,1 @@
-ms-vlmeval>=0.0.5
+ms-vlmeval>=0.0.9
diff --git a/setup.py b/setup.py
index 9eac254b..e613d5b8 100644
--- a/setup.py
+++ b/setup.py
@@ -66,8 +66,7 @@ def parse_line(line):
                     if ';' in rest:
                         # Handle platform specific dependencies
                         # http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies
-                        version, platform_deps = map(str.strip,
-                                                     rest.split(';'))
+                        version, platform_deps = map(str.strip, rest.split(';'))
                         info['platform_deps'] = platform_deps
                     else:
                         version = rest  # NOQA
@@ -81,8 +80,7 @@ def parse_require_file(fpath):
                 if line.startswith('http'):
                     print('skip http requirements %s' % line)
                     continue
-                if line and not line.startswith('#') and not line.startswith(
-                        '--'):
+                if line and not line.startswith('#') and not line.startswith('--'):
                     for info in parse_line(line):
                         yield info
                 elif line and line.startswith('--find-links'):
@@ -116,27 +114,9 @@ def gen_packages_items():
     return gen_packages_items()
 
 
-def pack_resource():
-    import shutil
-    # pack resource such as configs and tools
-    root_dir = 'package/'
-    if os.path.isdir(root_dir):
-        shutil.rmtree(root_dir)
-    os.makedirs(root_dir)
-
-    proj_dir = root_dir + 'evalscope/'
-    shutil.copytree('evalscope', proj_dir)
-    shutil.copytree('requirements', root_dir + 'requirements')
-    shutil.copy('requirements.txt', root_dir + 'requirements.txt')
-    # shutil.copy('./MANIFEST.in', 'package/MANIFEST.in')
-    shutil.copy('./README.md', root_dir + 'README.md')
-
-
 if __name__ == '__main__':
     print('Usage: python3 setup.py bdist_wheel or pip3 install .[opencompass] for test')
 
-    pack_resource()
-    os.chdir('package')
     install_requires, deps_link = parse_requirements('requirements/framework.txt')
 
     extra_requires = {}
@@ -144,12 +124,14 @@ def pack_resource():
     extra_requires['opencompass'], _ = parse_requirements('requirements/opencompass.txt')
     extra_requires['vlmeval'], _ = parse_requirements('requirements/vlmeval.txt')
     extra_requires['rag'], _ = parse_requirements('requirements/rag.txt')
+    extra_requires['perf'], _ = parse_requirements('requirements/perf.txt')
     extra_requires['inner'], _ = parse_requirements('requirements/inner.txt')
 
     all_requires.extend(install_requires)
     all_requires.extend(extra_requires['opencompass'])
     all_requires.extend(extra_requires['vlmeval'])
     all_requires.extend(extra_requires['rag'])
+    all_requires.extend(extra_requires['perf'])
     extra_requires['all'] = all_requires
 
     setup(
@@ -164,9 +146,13 @@ def pack_resource():
         # github url to be added
         url='https://github.com/modelscope/evalscope',
         include_package_data=True,
-        package_data={'evalscope': ['registry/tasks/*.yaml',
-                                    'benchmarks/bbh/cot_prompts/*.txt',
-                                    'third_party/longbench_write/resources/*',]},
+        package_data={
+            'evalscope': [
+                'registry/tasks/*.yaml',
+                'benchmarks/bbh/cot_prompts/*.txt',
+                'third_party/longbench_write/resources/*',
+            ]
+        },
         packages=find_packages(exclude=('configs', 'demo')),
         classifiers=[
             'Development Status :: 4 - Beta',
@@ -180,9 +166,7 @@ def pack_resource():
         python_requires='>=3.8',
         zip_safe=False,
         install_requires=install_requires,
-        entry_points={  
-            'console_scripts': ['evalscope=evalscope.cli.cli:run_cmd']
-        },
+        entry_points={'console_scripts': ['evalscope=evalscope.cli.cli:run_cmd']},
         dependency_links=deps_link,
         extras_require=extra_requires,
     )


Test patch:
diff --git a/docs/en/user_guides/stress_test.md b/docs/en/user_guides/stress_test.md
index 39fd1db4..02c2f403 100644
--- a/docs/en/user_guides/stress_test.md
+++ b/docs/en/user_guides/stress_test.md
@@ -1,174 +1,361 @@
-# Stress Testing
-A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats.
-## Usage
+# Model Inference Stress Test
 
-### Command line  
-```bash
-evalscope perf --help
-usage: evalscope <command> [<args>] perf [-h] --model MODEL [--url URL] [--connect-timeout CONNECT_TIMEOUT] [--read-timeout READ_TIMEOUT] [-n NUMBER] [--parallel PARALLEL] [--rate RATE]
-                                       [--log-every-n-query LOG_EVERY_N_QUERY] [--headers KEY1=VALUE1 [KEY1=VALUE1 ...]] [--wandb-api-key WANDB_API_KEY] [--name NAME] [--debug] [--tokenizer-path TOKENIZER_PATH]
-                                       [--api API] [--max-prompt-length MAX_PROMPT_LENGTH] [--min-prompt-length MIN_PROMPT_LENGTH] [--prompt PROMPT] [--query-template QUERY_TEMPLATE] [--dataset DATASET]
-                                       [--dataset-path DATASET_PATH] [--frequency-penalty FREQUENCY_PENALTY] [--logprobs] [--max-tokens MAX_TOKENS] [--n-choices N_CHOICES] [--seed SEED] [--stop STOP] [--stream]
-                                       [--temperature TEMPERATURE] [--top-p TOP_P]
-
-options:
-  -h, --help            show this help message and exit
-  --model MODEL         The test model name.
-  --url URL
-  --connect-timeout CONNECT_TIMEOUT
-                        The network connection timeout
-  --read-timeout READ_TIMEOUT
-                        The network read timeout
-  -n NUMBER, --number NUMBER
-                        How many requests to be made, if None, will will send request base dataset or prompt.
-  --parallel PARALLEL   Set number of concurrency request, default 1
-  --rate RATE           Number of requests per second. default None, if it set to -1,then all the requests are sent at time 0. Otherwise, we use Poisson process to synthesize the request arrival times. Mutual exclusion
-                        with parallel
-  --log-every-n-query LOG_EVERY_N_QUERY
-                        Logging every n query.
-  --headers KEY1=VALUE1 [KEY1=VALUE1 ...]
-                        Extra http headers accepts by key1=value1 key2=value2. The headers will be use for each query.You can use this parameter to specify http authorization and other header.
-  --wandb-api-key WANDB_API_KEY
-                        The wandb api key, if set the metric will be saved to wandb.
-  --name NAME           The wandb db result name and result db name, default: {model_name}_{current_time}
-  --debug               Debug request send.
-  --tokenizer-path TOKENIZER_PATH
-                        Specify the tokenizer weight path, used to calculate the number of input and output tokens,usually in the same directory as the model weight.
-  --api API             Specify the service api, current support [openai|dashscope]you can define your custom parser with python, and specify the python file path, reference api_plugin_base.py,
-  --max-prompt-length MAX_PROMPT_LENGTH
-                        Maximum input prompt length
-  --min-prompt-length MIN_PROMPT_LENGTH
-                        Minimum input prompt length.
-  --prompt PROMPT       Specified the request prompt, all the query will use this prompt, You can specify local file via @file_path, the prompt will be the file content.
-  --query-template QUERY_TEMPLATE
-                        Specify the query template, should be a json string, or local file,with local file, specified with @local_file_path,will will replace model and prompt in the template.
-  --dataset DATASET     Specify the dataset [openqa|longalpaca|line_by_line]you can define your custom dataset parser with python, and specify the python file path, reference dataset_plugin_base.py,
-  --dataset-path DATASET_PATH
-                        Path to the dataset file, Used in conjunction with dataset. If dataset is None, each line defaults to a prompt.
-  --frequency-penalty FREQUENCY_PENALTY
-                        The frequency_penalty value.
-  --logprobs            The logprobs.
-  --max-tokens MAX_TOKENS
-                        The maximum number of tokens can be generated.
-  --n-choices N_CHOICES
-                        How may chmpletion choices to generate.
-  --seed SEED           The random seed.
-  --stop STOP           The stop generating tokens.
-  --stop-token-ids      Set the stop token ids.
-  --stream              Stream output with SSE.
-  --temperature TEMPERATURE
-                        The sample temperature.
-  --top-p TOP_P         Sampling top p.
+A stress testing tool for large language models that can be customized to support various dataset formats and different API protocol formats, with default support for OpenAI API format.
 
+## Installation
+```shell
+# Install additional dependencies
+pip install evalscope[perf] -U
 ```
-### Result
+
+## Quick Start
+
+The model inference performance stress test tool can be launched using the following two methods:
+
+::::{tab-set}
+:::{tab-item} Command Line
 ```bash
- Total requests: 10
- Succeed requests: 10
- Failed requests: 0
- Average QPS: 0.256
- Average latency: 3.859
- Throughput(average output tokens per second): 23.317
- Average time to first token: 0.007
- Average input tokens per request: 21.800
- Average output tokens per request: 91.100
- Average time per output token: 0.04289
- Average package per request: 93.100
- Average package latency: 0.042
- Percentile of time to first token: 
-     p50: 0.0021
-     p66: 0.0023
-     p75: 0.0025
-     p80: 0.0030
-     p90: 0.0526
-     p95: 0.0526
-     p98: 0.0526
-     p99: 0.0526
- Percentile of request latency: 
-     p50: 3.9317
-     p66: 3.9828
-     p75: 4.0153
-     p80: 7.2801
-     p90: 7.7003
-     p95: 7.7003
-     p98: 7.7003
-     p99: 7.7003
+evalscope perf \
+    --url "http://127.0.0.1:8000/v1/chat/completions" \
+    --parallel 1 \
+    --model qwen2.5 \
+    --number 15 \
+    --api openai \
+    --dataset openqa \
+    --stream
 ```
+:::
+
+:::{tab-item} Python Script
+```python
+from evalscope.perf.main import run_perf_benchmark
+
+task_cfg = {"url": "http://127.0.0.1:8000/v1/chat/completions",
+            "parallel": 1,
+            "model": "qwen2.5",
+            "number": 15,
+            "api": "openai",
+            "dataset": "openqa",
+            "stream": True}
+run_perf_benchmark(task_cfg)
+```
+:::
+::::
+
+Parameter Description:
+
+- `url`: URL address for requests
+- `parallel`: Number of parallel request tasks
+- `model`: Name of the model used
+- `number`: Number of requests
+- `api`: API service used
+- `dataset`: Name of the dataset
+- `stream`: Whether to enable streaming
+
+### Output Results
+```text
+Benchmarking summary: 
++----------------------------------------------+------------------------------------------------+
+| key                                          | Value                                          |
++==============================================+================================================+
+| Time taken for tests (seconds)               | 7.539                                          |
++----------------------------------------------+------------------------------------------------+
+| Number of concurrency                        | 1                                              |
++----------------------------------------------+------------------------------------------------+
+| Total requests                               | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Succeeded requests                           | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Failed requests                              | 0                                              |
++----------------------------------------------+------------------------------------------------+
+| Average QPS                                  | 1.99                                           |
++----------------------------------------------+------------------------------------------------+
+| Average latency                              | 0.492                                          |
++----------------------------------------------+------------------------------------------------+
+| Average time to first token                  | 0.026                                          |
++----------------------------------------------+------------------------------------------------+
+| Throughput (average output tokens per second)| 334.006                                        |
++----------------------------------------------+------------------------------------------------+
+| Average time per output token                | 0.00299                                        |
++----------------------------------------------+------------------------------------------------+
+| Average package per request                  | 167.867                                        |
++----------------------------------------------+------------------------------------------------+
+| Average package latency                      | 0.003                                          |
++----------------------------------------------+------------------------------------------------+
+| Average input tokens per request             | 40.133                                         |
++----------------------------------------------+------------------------------------------------+
+| Average output tokens per request            | 167.867                                        |
++----------------------------------------------+------------------------------------------------+
+| Expected number of requests                  | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Result DB path                               | ./outputs/qwen2.5_benchmark_20241107_201413.db |
++----------------------------------------------+------------------------------------------------+
+
+Percentile results: 
++------------+---------------------+---------+
+| Percentile | First Chunk Latency | Latency |
++------------+---------------------+---------+
+|    10%     |       0.0178        | 0.1577  |
+|    25%     |       0.0183        | 0.2358  |
+|    50%     |       0.0199        | 0.4311  |
+|    66%     |       0.0218        | 0.6317  |
+|    75%     |       0.0429        | 0.7121  |
+|    80%     |       0.0432        | 0.7957  |
+|    90%     |       0.0432        | 0.9153  |
+|    95%     |       0.0433        | 0.9897  |
+|    98%     |       0.0433        | 0.9897  |
+|    99%     |       0.0433        | 0.9897  |
++------------+---------------------+---------+
+```
+
+### Metrics Explanation
+
+| **Metric**                                  | **Description**                | **Value**       |
+|---------------------------------------------|--------------------------------|-----------------|
+| Total requests                              | Total number of requests       | 15              |
+| Succeeded requests                          | Number of successful requests  | 15              |
+| Failed requests                             | Number of failed requests      | 0               |
+| Average QPS                                 | Average requests per second    | 1.99            |
+| Average latency                             | Average latency of all requests| 0.492           |
+| Throughput (average output tokens per second)| Number of output tokens per second| 334.006      |
+| Average time to first token                 | Average time to first token    | 0.026           |
+| Average input tokens per request            | Average input tokens per request| 40.133         |
+| Average output tokens per request           | Average output tokens per request| 167.867       |
+| Average time per output token               | Average time per output token  | 0.00299         |
+| Average package per request                 | Average packages per request   | 167.867         |
+| Average package latency                     | Average package latency        | 0.003           |
+| Percentile of time to first token (p10, ..., p99)| Percentile for first token delay|               |
+| Percentile of request latency (p10, ..., p99)| Percentile for request latency |                 |
+
+
+## Full Parameter Description
+
+Execute `evalscope perf --help` to get a full parameter description:
+
+### Basic Settings
+- `--model`: Name of the test model.
+- `--url`: Specify the API address.
+- `--name`: Name for the wandb database result and result database, default is `{model_name}_{current_time}`, optional.
+- `--api`: Specify the service API, currently supports [openai|dashscope|local|local_vllm].
+  - Select `openai` to use the API supporting OpenAI, requiring the `--url` parameter.
+  - Select `dashscope` to use the API supporting DashScope, requiring the `--url` parameter.
+  - Select `local` to use local files as models and perform inference using transformers. `--model` should be the model file path or model_id, which will be automatically downloaded from modelscope, e.g., `Qwen/Qwen2.5-0.5B-Instruct`.
+  - Select `local_vllm` to use local files as models and start the vllm inference service. `--model` should be the model file path or model_id, which will be automatically downloaded from modelscope, e.g., `Qwen/Qwen2.5-0.5B-Instruct`.
+- `--attn-implementation`: Attention implementation method, default is None, optional [flash_attention_2|eager|sdpa], only effective when `api` is `local`.
+- `--api-key`: API key, optional.
+- `--debug`: Output debug information.
+
+### Network Configuration
+- `--connect-timeout`: Network connection timeout, default is 120 seconds.
+- `--read-timeout`: Network read timeout, default is 120 seconds.
+- `--headers`: Additional HTTP headers, formatted as `key1=value1 key2=value2`. This header will be used for each query.
+
+### Request Control
+- `--number`: Number of requests sent, default is None, meaning requests are sent based on the dataset size.
+- `--parallel`: Set the number of concurrent requests, default is 1.
+- `--rate`: Number of requests per second, default is None. If set to -1, all requests are sent at time 0. Otherwise, requests are synthesized using a Poisson process. Mutually exclusive with parallel.
+- `--log-every-n-query`: Log every n queries, default is 10.
+- `--stream`: Use SSE stream output, default is False.
+
+### Prompt Settings
+- `--max-prompt-length`: Maximum input prompt length, default is `sys.maxsize`. Prompts exceeding this length will be discarded.
+- `--min-prompt-length`: Minimum input prompt length, default is 0. Prompts shorter than this will be discarded.
+- `--prompt`: Specify request prompt, a string or local file, taking precedence over `dataset`. When using a local file, specify the file path with `@/path/to/file`, e.g., `@./prompt.txt`.
+- `--query-template`: Specify query template, a `JSON` string or local file. When using a local file, specify the file path with `@/path/to/file`, e.g., `@./query_template.json`.
+
+### Dataset Configuration
+- `--dataset` specifies the dataset [openqa|longalpaca|line_by_line]. You can also use a custom dataset parser in Python; refer to the [Custom Dataset Guide](#custom-datasets).
+  - `line_by_line` treats each line as a separate prompt and requires a `dataset_path`.
+  - `longalpaca` will use `item['instruction']` as the prompt. If `dataset_path` is not specified, it will be automatically downloaded from modelscope.
+  - `openqa` will use `item['question']` as the prompt. If `dataset_path` is not specified, it will be automatically downloaded from modelscope.
+- `--dataset-path`: Path to the dataset file, used in combination with the dataset. `openqa` and `longalpaca` do not need a specified dataset path and will be downloaded automatically; `line_by_line` requires a local dataset file, which will be loaded line by line.
+
+### Model Settings
+- `--tokenizer-path`: Optional, specify the path to tokenizer weights for calculating the number of input and output tokens, usually in the same directory as the model weights.
+- `--frequency-penalty`: Frequency penalty value.
+- `--logprobs`: Log probabilities.
+- `--max-tokens`: Maximum number of tokens that can be generated.
+- `--min-tokens`: Minimum number of tokens to be generated.
+- `--n-choices`: Number of completion choices generated.
+- `--seed`: Random seed, default is 42.
+- `--stop`: Tokens that stop generation.
+- `--stop-token-ids`: Set stop token IDs.
+- `--temperature`: Sampling temperature.
+- `--top-p`: Top_p sampling.
+
+### Data Storage
+- `--wandb-api-key`: wandb API key, if set, metrics will be saved to wandb.
+
+## Example
+
+### Using Local Model Inference
+
+This project supports inference using local transformers and vllm (vllm needs to be installed first). The `--model` can be filled with a modelscope model name, such as `Qwen/Qwen2.5-0.5B-Instruct`; or you can directly specify the model weight path, such as `/path/to/model_weights`, without needing to specify the `--url` parameter.
+
+**Inference using transformers**
 
-### Metrics Description
-| **Metrics**                                     | **Description**                       | **Values**        |
-|------------------------------------------|-----------------------------|-----------------|
-| Total requests                          | Total requests                     | 10              |
-| Succeed requests                        | Succeed requests                 | 10              |
-| Failed requests                         | Failed requests                  | 0               |
-| Average QPS                            | Average queries per second             | 0.256           |
-| Average latency                         | Average latency for total requests                    | 3.859           |
-| Throughput                              | Output tokens per second        | 23.317          |
-| Average time to first token            | Average time to first token            | 0.007           |
-| Average input tokens per request        | Average input tokens per request    | 21.800          |
-| Average output tokens per request       | Average output tokens per request    | 91.100          |
-| Average time per output token           | Average time per output token      | 0.04289         |
-| Average package per request             | Average package per request         | 93.100          |
-| Average package latency                  | Average package latency                   | 0.042           |
-| Percentile of time to first token (p50, ..., p99)     | Percentile of time to first token      |   p50=0.0021, ..., p99=0.0526         |
-| Percentile of request latency (p50, ..., p99)          | Percentile of request latency          |  p50=3.9317, ..., p99=7.7003         |
-
-
-### Request parameter  
-You can set request parameter's in query-template and with (--stop,--stream,--temperature, etc), the argument parameter will replace or add to the request.
-#### request with parameters
-Sample request llama3 vllm openai compatible interface.
 ```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '
' --dataset openqa --debug
+evalscope perf \
+ --model 'Qwen/Qwen2.5-0.5B-Instruct' \
+ --attn-implementation flash_attention_2 \  # Optional, or choose from [flash_attention_2|eager|sdpa]
+ --number 20 \
+ --rate 2 \
+ --api local \
+ --dataset openqa
 ```
+
+**Inference using vllm**
 ```bash
-evalscope perf 'http://host:port/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 -n 10000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
+evalscope perf \
+ --model 'Qwen/Qwen2.5-0.5B-Instruct' \
+ --number 20 \
+ --rate 2 \
+ --api local_vllm \
+ --dataset openqa
 ```
 
-#### query-template usage.
-When you need to process more complex requests, you can use templates to simplify the command line.
-If both the template and the parameter are present, the value in the parameter will prevail.
-Query-template example:
+### Using `prompt`
+```bash
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --number 20 \
+ --api openai \
+ --temperature 0.9 \
+ --max-tokens 1024 \
+ --prompt 'Write a science fiction story, please begin your performance'
+```
+You can also use a local file as a prompt:
 ```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 12 --model 'llama3' --log-every-n-query 10 --read-timeout=120 -n 1 --max-prompt-length 128000 --api openai --query-template '{"model": "%m", "messages": [], "stream": true, "stream_options":{"include_usage": true},"n": 3, "stop_token_ids": [128001, 128009]}' --dataset openqa --dataset-path './datasets/open_qa.jsonl'
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --number 20 \
+ --api openai \
+ --temperature 0.9 \
+ --max-tokens 1024 \
+ --prompt @prompt.txt
 ```
-For messages, the dataset processor message will replace messages in the query-template.
 
-#### Start the client
+### Complex Requests
+Using `stop`, `stream`, `temperature`, etc.:
+
 ```bash
-# test openai service
-evalscope perf --url 'https://api.openai.com/v1/chat/completions' --parallel 1 --headers 'Authorization=Bearer YOUR_OPENAI_API_KEY' --model 'gpt-4o' --dataset-path 'THE_DATA_TO/open_qa.jsonl'  --log-every-n-query 10 --read-timeout=120  -n 100 --max-prompt-length 128000 --api openai --stream --dataset openqa
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --temperature 0.7 \
+ --max-tokens 1024 \
+ --stop '
' \
+ --dataset openqa \
+ --stream
 ```
 
+### Using `query-template`
+
+You can set request parameters in the `query-template`:
+
+```bash
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true, "skip_special_tokens": false, "stop": ["
"], "temperature": 0.7, "max_tokens": 1024}' \
+ --dataset openqa 
+```
+Where `%m` and `%p` will be replaced by the model name and the prompt.
+
+You can set request parameters in the query-template:
+
+```{code-block} json
+:caption: template.json
+
+{
+   "model":"%m",
+   "messages":[
+      {
+         "role":"user",
+         "content":"%p"
+      }
+   ],
+   "stream":true,
+   "skip_special_tokens":false,
+   "stop":[
+      "
"
+   ],
+   "temperature":0.7,
+   "max_tokens":1024
+}
+```
 ```bash
-##### open qa dataset and 
-#### dataset address: https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl
-evalscope perf --url 'http://IP:PORT/v1/chat/completions' --parallel 1 --model 'qwen' --log-every-n-query 1 --read-timeout=120 -n 1000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --query-template @template.json \
+ --dataset openqa 
 ```
 
-## FAQ
+Here’s the translation of the provided text into English:
 
-### How to log metrics to wandb
---wandb-api-key 'your_wandb_api_key'  --name 'name_of_wandb_and_result_db'  
+### Using wandb to Record Test Results
+
+Please install wandb:
+```bash
+pip install wandb
+```
+
+When starting, add the following parameters:
+```bash
+--wandb-api-key 'wandb_api_key'
+--name 'name_of_wandb_log'
+```  
 
 ![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
 
-### How to debug
---debug 
-with --debug option, we will output the request and response.
+### Debugging Requests
+Use the `--debug` option to output the requests and responses.
+
+### Analyzing Results
+During testing, this tool saves all data into an SQLite3 database, including requests and responses. You can analyze the test data after testing.
 
-### How to analysis result.
-The tool will save all data during the test to the sqlite3 database, including requests and responses. You can analyze the test data after the test.
 ```python
 import sqlite3
 import base64
 import pickle
 import json
+
 result_db_path = 'db_name.db'
 con = sqlite3.connect(result_db_path)
 query_sql = "SELECT request, response_messages, prompt_tokens, completion_tokens \
-                FROM result WHERE success='True'"
-# how to save base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"), 
+                FROM result WHERE success='1'"
+
 with con:
     rows = con.execute(query_sql).fetchall()
     if len(rows) > 0:
@@ -182,39 +369,111 @@ with con:
             response_content = ''
             for response in responses:
                 response = json.loads(response)
+                if not response['choices']:
+                   continue
                 response_content += response['choices'][0]['delta']['content']
-            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' % (request['messages'][0]['content'], row[2], response_content, row[3]))
+            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' %
+                  (request['messages'][0]['content'], row[2], response_content,
+                   row[3]))
 ```
 
-### Api supported
-Currently supports openai, dashscope, zhipu API request. You can specify api with --api.
-You can custom your request with --query-template, you can specify a json string as:
-'{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}'
-or a local file with @to_query_template_path. We will replace %m with model and %p with prompt.
-
-### How to extend API
-To extend api you can create sub class of `ApiPluginBase`, annotation with @register_api("name_of_api")
-with build_request build request via model, prompt, and
-query_template. you can reference opanai_api.py
-parse_responses return number_of_prompt_tokens and number_of_completion_tokens.
+## Speed Benchmark
+To conduct speed tests and obtain the speed benchmarks reported by [Qwen Official](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html), please use `--dataset [speed_benchmark|speed_benchmark_long]`.
+
+- `speed_benchmark`: Tests prompts of lengths [1, 6144, 14336, 30720] with a fixed output of 2048 tokens.
+- `speed_benchmark_long`: Tests prompts of lengths [63488, 129024] with a fixed output of 2048 tokens.
+
+```{note}
+For speed testing, the `--url` option should use the `/v1/completions` endpoint instead of the `/v1/chat/completions` endpoint to avoid the additional processing of chat templates affecting the input length.
+```
+
+### Inference Based on Transformer
+```bash
+CUDA_VISIBLE_DEVICES=0 evalscope perf \
+ --rate 1 \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
+ --attn-implementation flash_attention_2 \
+ --log-every-n-query 5 \
+ --connect-timeout 6000 \
+ --read-timeout 6000 \
+ --max-tokens 2048 \
+ --min-tokens 2048 \
+ --api local \
+ --dataset speed_benchmark \
+ --debug
+```
+
+Example Output:
+```
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |      50.69      |      0.97      |
+|     6144      |      51.36      |      1.23      |
+|     14336     |      49.93      |      1.59      |
+|     30720     |      49.56      |      2.34      |
++---------------+-----------------+----------------+
+```
+
+### Inference Based on vLLM
+```bash
+CUDA_VISIBLE_DEVICES=0 evalscope perf \
+ --rate 1 \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
+ --log-every-n-query 5 \
+ --connect-timeout 6000 \
+ --read-timeout 6000 \
+ --max-tokens 2048 \
+ --min-tokens 2048 \
+ --api local_vllm \
+ --dataset speed_benchmark 
+```
+Example Output (Note that vllm reserves GPU memory, so GPU usage is not shown here):
+```
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |     343.08      |      0.0       |
+|     6144      |     334.71      |      0.0       |
+|     14336     |     318.88      |      0.0       |
+|     30720     |     292.86      |      0.0       |
++---------------+-----------------+----------------+
+```
+
+
+## Custom API
+Currently, the supported API request formats are openai and dashscope. To extend the API, you can inherit from the `ApiPluginBase` class and use the `@register_api("api_name")` decorator. You will need to implement the following two methods:
+
+- The `build_request()` method constructs the request using `messages` and the `model` and `query_template` from `param`. This request will then be sent to the target API.
+
+- The `parse_responses()` method returns the counts of `prompt_tokens` and `completion_tokens`, which are used to calculate inference speed.
+
+Refer to the following code:
+
 ```python
-class ApiPluginBase:
+from typing import Any, Dict, List, Tuple
+from evalscope.perf.arguments import Arguments
+
+@register_api('custom')
+class CustomPlugin(ApiPluginBase):
     def __init__(self, model_path: str) -> None:
         self.model_path = model_path
         
     @abstractmethod
-    def build_request(self, messages: List[Dict], param: QueryParameters)->Dict:
-        """Build a api request body.
+    def build_request(self, messages: List[Dict], param: Arguments) -> Dict:
+        """Build an API request body.
 
         Args:
-            messages (List[Dict]): The messages generated by dataset.
+            messages (List[Dict]): The messages generated by the dataset.
             param (QueryParameters): The query parameters.
 
         Raises:
             NotImplementedError: Not implemented.
 
         Returns:
-            Dict: The api request body.
+            Dict: The API request body.
         """
         raise NotImplementedError
     
@@ -222,12 +481,12 @@ class ApiPluginBase:
     def parse_responses(self, 
                         responses: List, 
                         request: Any=None,
-                        **kwargs:Any) -> Tuple[int, int]:
-        """Parser responses and return number of request and response tokens.
+                        **kwargs: Any) -> Tuple[int, int]:
+        """Parse responses and return the number of request and response tokens.
 
         Args:
-            responses (List[bytes]): List of http response body, for stream output,
-                there are multiple responses, each is bytes, for general only one. 
+            responses (List[bytes]): List of HTTP response bodies. For stream output,
+                there may be multiple responses, each represented as bytes; otherwise, there is generally only one. 
             request (Any): The request body.
 
         Returns:
@@ -236,71 +495,28 @@ class ApiPluginBase:
         raise NotImplementedError  
 ```
 
-### Dataset supported
-Currently supports line by line, longalpaca and openqa data set.
-line by line with each line as a prompt.
-longalpaca will get item['instruction'] as prompt.
-openqa will get item['question'] as prompt.
+## Custom Datasets
+To customize a dataset, you can inherit from the `DatasetPluginBase` class and use the `@register_dataset('dataset_name')` decorator. Then, implement the `build_messages` method to return a message in the format specified in the [OpenAI API documentation](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages).
 
-### How to extension dataset.
-To extend api you can create sub class of `DatasetPluginBase`, annotation with @register_dataset('name_of_dataset')
-implement build_prompt api return a prompt.
 ```python
-class DatasetPluginBase:
-    def __init__(self, query_parameters: QueryParameters):
-        """Build data set plugin
-
-        Args:
-            dataset_path (str, optional): The input dataset path. Defaults to None.
-        """
-        self.query_parameters = query_parameters
-
-    def __next__(self):
-        for item in self.build_messages():
-            yield item
-        raise StopIteration
-
-    def __iter__(self):
-        return self.build_messages()
-    
-    @abstractmethod
-    def build_messages(self)->Iterator[List[Dict]]:
-        """Build the request.
-
-        Raises:
-            NotImplementedError: The request is not impletion.
-
-        Yields:
-            Iterator[List[Dict]]: Yield request messages.
-        """
-        raise NotImplementedError
-    
-    def dataset_line_by_line(self, dataset: str)->Iterator[str]:
-        """Get content line by line of dataset.
-
-        Args:
-            dataset (str): The dataset path.
-
-        Yields:
-            Iterator[str]: Each line of file.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            for line in f:
-                yield line
-    
-    def dataset_json_list(self, dataset: str)->Iterator[Dict]:
-        """Read data from file which is list of requests.
-           Sample: https://huggingface.co/datasets/Yukang/LongAlpaca-12k
-
-        Args:
-            dataset (str): The dataset path.
-
-        Yields:
-            Iterator[Dict]: The each request object.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            content = f.read()
-        data = json.loads(content)
-        for item in data:
-            yield item      
+from typing import Dict, Iterator, List
+
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
+
+@register_dataset('custom')
+class CustomDatasetPlugin(DatasetPluginBase):
+    """Read dataset and return a prompt.
+    """
+
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
+
+    def build_messages(self) -> Iterator[List[Dict]]:
+        for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
+            prompt = item.strip()
+            if len(prompt) > self.query_parameters.min_prompt_length and len(
+                    prompt) < self.query_parameters.max_prompt_length:
+                yield [{'role': 'user', 'content': prompt}]
 ```
\ No newline at end of file
diff --git a/docs/zh/user_guides/stress_test.md b/docs/zh/user_guides/stress_test.md
index 0db998bb..1e98ba50 100644
--- a/docs/zh/user_guides/stress_test.md
+++ b/docs/zh/user_guides/stress_test.md
@@ -1,162 +1,351 @@
 # 模型推理性能压测
-一个专注于大型语言模型的压力测试工具，可以自定义以支持各种数据集格式和不同的API协议格式。
+一个大语言模型的压力测试工具，可以自定义以支持各种数据集格式和不同的API协议格式，默认支持OpenAI API格式。
 
-## 使用方法
+## 环境准备
+```shell
+# 安装额外依赖
+pip install evalscope[perf] -U
+```
+
+## 快速使用
+可以使用以下两种方式启动模型推理性能压测工具：
 
-### 命令行
+::::{tab-set}
+:::{tab-item} 命令行启动
 ```bash
-evalscope perf --help
-usage: evalscope <command> [<args>] perf [-h] --model MODEL [--url URL] [--connect-timeout CONNECT_TIMEOUT] [--read-timeout READ_TIMEOUT] [-n NUMBER] [--parallel PARALLEL] [--rate RATE]
-                                       [--log-every-n-query LOG_EVERY_N_QUERY] [--headers KEY1=VALUE1 [KEY1=VALUE1 ...]] [--wandb-api-key WANDB_API_KEY] [--name NAME] [--debug] [--tokenizer-path TOKENIZER_PATH]
-                                       [--api API] [--max-prompt-length MAX_PROMPT_LENGTH] [--min-prompt-length MIN_PROMPT_LENGTH] [--prompt PROMPT] [--query-template QUERY_TEMPLATE] [--dataset DATASET]
-                                       [--dataset-path DATASET_PATH] [--frequency-penalty FREQUENCY_PENALTY] [--logprobs] [--max-tokens MAX_TOKENS] [--n-choices N_CHOICES] [--seed SEED] [--stop STOP] [--stream]
-                                       [--temperature TEMPERATURE] [--top-p TOP_P]
-options:
-  -h, --help            显示此帮助信息并退出
-  --model MODEL         测试模型名称。
-  --url URL
-  --connect-timeout CONNECT_TIMEOUT
-                        网络连接超时
-  --read-timeout READ_TIMEOUT
-                        网络读取超时
-  -n NUMBER, --number NUMBER
-                        发出的请求数量，如果为None，则将基于数据集或提示发送请求。
-  --parallel PARALLEL   设置并发请求的数量，默认为1
-  --rate RATE           每秒请求的数量，默认为None。如果设置为-1，则所有请求将在时间0发送。否则，我们使用泊松过程合成请求到达时间。与parallel互斥
-  --log-every-n-query LOG_EVERY_N_QUERY
-                        每n个查询记录日志。
-  --headers KEY1=VALUE1 [KEY1=VALUE1 ...]
-                        额外的HTTP头，格式为key1=value1 key2=value2。该头将用于每个查询。您可以使用此参数指定HTTP授权和其他头信息。
-  --wandb-api-key WANDB_API_KEY
-                        wandb API密钥，如果设置，则度量将保存到wandb。
-  --name NAME           wandb数据库结果名称和结果数据库名称，默认为: {model_name}_{current_time}
-  --debug               调试请求发送。
-  --tokenizer-path TOKENIZER_PATH
-                        指定分词器权重路径，用于计算输入和输出的token数量，通常与模型权重在同一目录下。
-  --api API             指定服务API，目前支持[openai|dashscope]，您可以使用python定义自定义解析器，并指定python文件路径，参考api_plugin_base.py。
-  --max-prompt-length MAX_PROMPT_LENGTH
-                        最大输入prompt长度
-  --min-prompt-length MIN_PROMPT_LENGTH
-                        最小输入prompt长度。
-  --prompt PROMPT       指定请求prompt，所有查询将使用此prompt。您可以通过@file_path指定本地文件，prompt将为文件内容。
-  --query-template QUERY_TEMPLATE
-                        指定查询模板，应该是一个JSON字符串或本地文件，使用本地文件时，通过@local_file_path指定，将在模板中替换模型和prompt。
-  --dataset DATASET     指定数据集[openqa|longalpaca|line_by_line]，您可以使用python定义自定义数据集解析器，并指定python文件路径，参考dataset_plugin_base.py。
-  --dataset-path DATASET_PATH
-                        数据集文件的路径，与数据集结合使用。如果数据集为None，则每一行默认为一个prompt。
-  --frequency-penalty FREQUENCY_PENALTY
-                        frequency_penalty值。
-  --logprobs            对数概率。
-  --max-tokens MAX_TOKENS
-                        可以生成的最大token数量。
-  --n-choices N_CHOICES
-                        生成的补全选择数量。
-  --seed SEED           随机种子。
-  --stop STOP           停止生成的tokens。
-  --stop-token-ids      设置停止生成的token的ID。
-  --stream              使用SSE流输出。
-  --temperature TEMPERATURE
-                        采样温度。
-  --top-p TOP_P         top_p采样。
+evalscope perf \
+    --url "http://127.0.0.1:8000/v1/chat/completions" \
+    --parallel 1 \
+    --model qwen2.5 \
+    --number 15 \
+    --api openai \
+    --dataset openqa \
+    --stream
 ```
-### 结果
-```bash
- Total requests: 10
- Succeed requests: 10
- Failed requests: 0
- Average QPS: 0.256
- Average latency: 3.859
- Throughput(average output tokens per second): 23.317
- Average time to first token: 0.007
- Average input tokens per request: 21.800
- Average output tokens per request: 91.100
- Average time per output token: 0.04289
- Average package per request: 93.100
- Average package latency: 0.042
- Percentile of time to first token: 
-     p50: 0.0021
-     p66: 0.0023
-     p75: 0.0025
-     p80: 0.0030
-     p90: 0.0526
-     p95: 0.0526
-     p98: 0.0526
-     p99: 0.0526
- Percentile of request latency: 
-     p50: 3.9317
-     p66: 3.9828
-     p75: 4.0153
-     p80: 7.2801
-     p90: 7.7003
-     p95: 7.7003
-     p98: 7.7003
-     p99: 7.7003
+:::
+
+:::{tab-item} Python脚本启动
+```python
+from evalscope.perf.main import run_perf_benchmark
+
+task_cfg = {"url": "http://127.0.0.1:8000/v1/chat/completions",
+            "parallel": 1,
+            "model": "qwen2.5",
+            "number": 15,
+            "api": "openai",
+            "dataset": "openqa",
+            "stream": True}
+run_perf_benchmark(task_cfg)
+```
+:::
+::::
+参数说明：
+
+- `url`: 请求的URL地址
+- `parallel`: 并行请求的任务数量
+- `model`: 使用的模型名称
+- `number`: 请求数量
+- `api`: 使用的API服务
+- `dataset`: 数据集名称
+- `stream`: 是否启用流式处理
+
+
+### 输出结果
+```text
+Benchmarking summary: 
++----------------------------------------------+------------------------------------------------+
+| key                                          | Value                                          |
++==============================================+================================================+
+| Time taken for tests (senconds)              | 7.539                                          |
++----------------------------------------------+------------------------------------------------+
+| Number of concurrency                        | 1                                              |
++----------------------------------------------+------------------------------------------------+
+| Total requests                               | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Succeed requests                             | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Failed requests                              | 0                                              |
++----------------------------------------------+------------------------------------------------+
+| Average QPS                                  | 1.99                                           |
++----------------------------------------------+------------------------------------------------+
+| Average latency                              | 0.492                                          |
++----------------------------------------------+------------------------------------------------+
+| Average time to first token                  | 0.026                                          |
++----------------------------------------------+------------------------------------------------+
+| Throughput(average output tokens per second) | 334.006                                        |
++----------------------------------------------+------------------------------------------------+
+| Average time per output token                | 0.00299                                        |
++----------------------------------------------+------------------------------------------------+
+| Average package per request                  | 167.867                                        |
++----------------------------------------------+------------------------------------------------+
+| Average package latency                      | 0.003                                          |
++----------------------------------------------+------------------------------------------------+
+| Average input tokens per request             | 40.133                                         |
++----------------------------------------------+------------------------------------------------+
+| Average output tokens per request            | 167.867                                        |
++----------------------------------------------+------------------------------------------------+
+| Expected number of requests                  | 15                                             |
++----------------------------------------------+------------------------------------------------+
+| Result DB path                               | ./outputs/qwen2.5_benchmark_20241107_201413.db |
++----------------------------------------------+------------------------------------------------+
+
+Percentile results: 
++------------+---------------------+---------+
+| Percentile | First Chunk Latency | Latency |
++------------+---------------------+---------+
+|    10%     |       0.0178        | 0.1577  |
+|    25%     |       0.0183        | 0.2358  |
+|    50%     |       0.0199        | 0.4311  |
+|    66%     |       0.0218        | 0.6317  |
+|    75%     |       0.0429        | 0.7121  |
+|    80%     |       0.0432        | 0.7957  |
+|    90%     |       0.0432        | 0.9153  |
+|    95%     |       0.0433        | 0.9897  |
+|    98%     |       0.0433        | 0.9897  |
+|    99%     |       0.0433        | 0.9897  |
++------------+---------------------+---------+
 ```
 
-#### 指标说明
+### 指标说明
 
 | **指标**                                     | **说明**                       | **数值**        |
 |------------------------------------------|-----------------------------|-----------------|
-| Total requests                          | 总请求数                     | 10              |
-| Succeeded requests                      | 成功请求                     | 10              |
-| Failed requests                         | 失败请求                      | 0               |
-| Average QPS                            | 每秒平均查询                 | 0.256           |
-| Average latency                         | 所有请求的平均延迟          | 3.859           |
-| Throughput                              | 每秒输出tokens              | 23.317          |
-| Average time to first token            | 第一次token的平均时间       | 0.007           |
-| Average input tokens per request        | 每个请求的平均输入tokens    | 21.800          |
-| Average output tokens per request       | 每个请求的平均输出tokens    | 91.100          |
-| Average time per output token           | 每个输出token的平均时间     | 0.04289         |
-| Average package per request             | 每个请求的平均包数          | 93.100          |
-| Average package latency                  | 每个包的平均延迟            | 0.042           |
-| Percentile of time to first token (p50, ..., p99) | 第一次token的时间百分位     |   p50=0.0021, ..., p99=0.0526         |
-| Percentile of request latency (p50, ..., p99)          | 请求延迟的百分位          |  p50=3.9317, ..., p99=7.7003         |
-
-
-### 请求参数  
-您可以在`query-template`中设置请求参数，也可以使用（`--stop`，`--stream`，`--temperature`等），参数将替换或添加到请求中。
-#### 带参数的请求
-示例请求 llama3 
+| Total requests                          | 总请求数                     | 15              |
+| Succeeded requests                      | 成功请求数                     | 15              |
+| Failed requests                         | 失败请求数                      | 0               |
+| Average QPS                            | 每秒平均请求数                 | 1.99          |
+| Average latency                         | 所有请求的平均延迟          | 0.492          |
+| Throughput(average output tokens per second)  | 每秒输出token数量              | 334.006          |
+| Average time to first token            | 首token的平均延时       |    0.026         |
+| Average input tokens per request        | 每个请求的平均输入token数量    | 40.133         |
+| Average output tokens per request       | 每个请求的平均输出token数量    | 167.867            |
+| Average time per output token           | 输出每个token的平均时间     | 0.00299         |
+| Average package per request             | 每个请求的平均包数          | 167.867           |
+| Average package latency                  | 每个包的平均延迟            | 0.003            |
+| Percentile of time to first token (p10, ..., p99) | 首token延时百分位     |            |
+| Percentile of request latency (p10, ..., p99)          | 请求延迟的百分位          |          |
+
+
+## 全部参数说明
+执行 `evalscope perf --help` 可获取全部参数说明：
+
+
+### 基本设置
+- `--model` 测试模型名称。
+- `--url` 指定API地址。
+- `--name` wandb数据库结果名称和结果数据库名称，默认为: `{model_name}_{current_time}`，可选。
+- `--api` 指定服务API，目前支持[openai|dashscope|local|local_vllm]。
+  - 指定为`openai`，则使用支持OpenAI的API，需要提供`--url`参数。
+  - 指定为`dashscope`，则使用支持DashScope的API，需要提供`--url`参数。
+  - 指定为`local`，则使用本地文件作为模型，并使用transformers进行推理。`--model`为模型文件路径，也可为model_id，将自动从modelscope下载模型，例如`Qwen/Qwen2.5-0.5B-Instruct`。
+  - 指定为`local_vllm`，则使用本地文件作为模型，并启动vllm推理服务。`--model`为模型文件路径，也可为model_id，将自动从modelscope下载模型，例如`Qwen/Qwen2.5-0.5B-Instruct`。
+- `--attn-implementation` Attention实现方式，默认为None，可选[flash_attention_2|eager|sdpa]，仅在`api`为`local`时有效。
+- `--api-key` API密钥，可选。
+- `--debug` 输出调试信息。
+
+### 网络配置
+- `--connect-timeout` 网络连接超时，默认为120s。
+- `--read-timeout` 网络读取超时，默认为120s。
+- `--headers` 额外的HTTP头，格式为`key1=value1 key2=value2`。该头将用于每个查询。
+
+### 请求控制
+- `--number` 发出的请求数量，默认为None，表示基于数据集数量发送请求。
+- `--parallel` 设置并发请求的数量，默认为1。
+- `--rate` 每秒请求的数量，默认为None。如果设置为-1，则所有请求将在时间0发送。否则，我们使用泊松过程合成请求到达时间。与parallel互斥。
+- `--log-every-n-query` 每n个查询记录日志，默认为10。
+- `--stream` 使用SSE流输出，默认为False。
+
+### Prompt设置
+- `--max-prompt-length` 最大输入prompt长度，默认为`sys.maxsize`，大于该值时，将丢弃prompt。
+- `--min-prompt-length` 最小输入prompt长度，默认为0，小于该值时，将丢弃prompt。
+- `--prompt` 指定请求prompt，一个字符串或本地文件，使用优先级高于`dataset`。使用本地文件时，通过`@/path/to/file`指定文件路径，例如`@./prompt.txt`。
+- `--query-template` 指定查询模板，一个`JSON`字符串或本地文件，使用本地文件时，通过`@/path/to/file`指定文件路径，例如`@./query_template.json`。
+
+### 数据集配置
+- `--dataset` 指定数据集[openqa|longalpaca|line_by_line]，您也可以使用python自定义数据集解析器，参考[自定义数据集指南](#自定义数据集)。
+  - `line_by_line`逐行将每一行作为一个提示，需提供`dataset_path`。
+  - `longalpaca` 将获取 `item['instruction']` 作为提示，不指定`dataset_path`将从modelscope自动下载。
+  - `openqa` 将获取 `item['question']` 作为提示，不指定`dataset_path`将从modelscope自动下载。
+- `--dataset-path` 数据集文件的路径，与数据集结合使用。openqa与longalpaca可不指定数据集路径，将自动下载；line_by_line必须指定本地数据集文件，将一行一行加载。
+
+### 模型设置
+- `--tokenizer-path` 可选，指定分词器权重路径，用于计算输入和输出的token数量，通常与模型权重在同一目录下。
+- `--frequency-penalty` frequency_penalty值。
+- `--logprobs` 对数概率。
+- `--max-tokens` 可以生成的最大token数量。
+- `--min-tokens` 生成的最少token数量。
+- `--n-choices` 生成的补全选择数量。
+- `--seed` 随机种子，默认为42。
+- `--stop` 停止生成的tokens。
+- `--stop-token-ids` 设置停止生成的token的ID。
+- `--temperature` 采样温度。
+- `--top-p` top_p采样。
+
+### 数据存储
+- `--wandb-api-key` wandb API密钥，如果设置，则度量将保存到wandb。
+
+
+## 使用示例
+
+### 使用本地模型推理
+
+本项目支持本地transformers进行推理和vllm推理（需先安装vllm）， `--model`可以填入modelscope模型名称，例如`Qwen/Qwen2.5-0.5B-Instruct`；也可以直接指定模型权重路径，例如`/path/to/model_weights`，无需指定`--url`参数。
+
+**1. 使用transformers进行推理**
+
+指定`--api local`：
 ```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/open_qa.jsonl' -n 1 --max-prompt-length 128000 --api openai --stream --stop '
' --dataset openqa --debug
+evalscope perf \
+ --model 'Qwen/Qwen2.5-0.5B-Instruct' \
+ --attn-implementation flash_attention_2 \  # 可不填，或选[flash_attention_2|eager|sdpa]
+ --number 20 \
+ --rate 2 \
+ --api local \
+ --dataset openqa
 ```
 
+**2. 使用vllm进行推理**
+
+指定`--api local_vllm`：
 ```bash
-evalscope perf 'http://host:port/v1/chat/completions' --parallel 128 --model 'qwen' --log-every-n-query 10 --read-timeout=120 -n 10000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
+evalscope perf \
+ --model 'Qwen/Qwen2.5-0.5B-Instruct' \
+ --number 20 \
+ --rate 2 \
+ --api local_vllm \
+ --dataset openqa
 ```
 
-#### query-template 使用说明
-当你需要处理更复杂的请求时，可以使用模板来简化命令行。如果同时存在模板和参数，则参数中的值将优先使用。
-query-template 示例：
+### 使用`prompt`
 ```bash
-evalscope perf --url 'http://127.0.0.1:8000/v1/chat/completions' --parallel 12 --model 'llama3' --log-every-n-query 10 --read-timeout=120 -n 1 --max-prompt-length 128000 --api openai --query-template '{"model": "%m", "messages": [], "stream": true, "stream_options":{"include_usage": true},"n": 3, "stop_token_ids": [128001, 128009]}' --dataset openqa --dataset-path './datasets/open_qa.jsonl'
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --number 20 \
+ --api openai \
+ --temperature 0.9 \
+ --max-tokens 1024 \
+ --prompt '写一个科幻小说，请开始你的表演'
+```
+也可以使用本地文件作为prompt：
+```bash
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --number 20 \
+ --api openai \
+ --temperature 0.9 \
+ --max-tokens 1024 \
+ --prompt @prompt.txt
 ```
-对于 `messages` 字段, 数据集处理器的 message 将替换 query-template 中的 messages.
 
-#### 启动客户端
+### 复杂请求
+使用`stop`，`stream`，`temperature`等：
+
 ```bash
-# 测试 openai 服务
-evalscope perf --url 'https://api.openai.com/v1/chat/completions' --parallel 1 --headers 'Authorization=Bearer YOUR_OPENAI_API_KEY' --model 'gpt-4o' --dataset-path 'THE_DATA_TO/open_qa.jsonl'  --log-every-n-query 10 --read-timeout=120  -n 100 --max-prompt-length 128000 --api openai --stream --dataset openqa
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --temperature 0.7 \
+ --max-tokens 1024 \
+ --stop '
' \
+ --dataset openqa \
+ --stream
 ```
 
+### 使用`query-template`
+
+您可以在`query-template`中设置请求参数：
+
 ```bash
-##### open qa dataset and 
-#### dataset address: https://huggingface.co/datasets/Hello-SimpleAI/HC3-Chinese/blob/main/open_qa.jsonl
-evalscope perf --url 'http://IP:PORT/v1/chat/completions' --parallel 1 --model 'qwen' --log-every-n-query 1 --read-timeout=120 -n 1000 --max-prompt-length 128000 --tokenizer-path "THE_PATH_TO_TOKENIZER/Qwen1.5-32B/" --api openai --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}' --dataset openqa --dataset-path 'THE_PATH_TO_DATASETS/open_qa.jsonl'
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --query-template '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true, "skip_special_tokens": false, "stop": ["
"], "temperature": 0.7, "max_tokens": 1024}' \
+ --dataset openqa 
+```
+其中`%m`和`%p`会被替换为模型名称和prompt。
+
+您也可以使用本地`query-template.json`文件：
+
+```{code-block} json
+:caption: template.json
+
+{
+   "model":"%m",
+   "messages":[
+      {
+         "role":"user",
+         "content":"%p"
+      }
+   ],
+   "stream":true,
+   "skip_special_tokens":false,
+   "stop":[
+      "
"
+   ],
+   "temperature":0.7,
+   "max_tokens":1024
+}
 ```
+```bash
+evalscope perf \
+ --url 'http://127.0.0.1:8000/v1/chat/completions' \
+ --rate 2 \
+ --model 'qwen2.5' \
+ --log-every-n-query 10 \
+ --read-timeout 120 \
+ --connect-timeout 120 \
+ --number 20 \
+ --max-prompt-length 128000 \
+ --min-prompt-length 128 \
+ --api openai \
+ --query-template @template.json \
+ --dataset openqa 
+```
+
 
-### FAQ
+### 使用wandb记录测试结果
 
-#### 如何使用wandb记录测试结果
---wandb-api-key 'your_wandb_api_key'  --name 'name_of_wandb_and_result_db'  
+请安装wandb：
+```bash
+pip install wandb
+```
+
+在启动时，添加以下参数，即可：
+```bash
+--wandb-api-key 'wandb_api_key'
+--name 'name_of_wandb_log'
+```  
 
 ![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
 
-#### 如何调试
-使用 --debug 选项，我们将输出请求和响应。
+
+### 调试请求
+使用 `--debug` 选项，我们将输出请求和响应。
 
 
-#### 如何分析结果
+### 分析结果
 该工具在测试期间会将所有数据保存到 sqlite3 数据库中，包括请求和响应。您可以在测试后分析测试数据。
 
 ```python
@@ -167,7 +356,7 @@ import json
 result_db_path = 'db_name.db'
 con = sqlite3.connect(result_db_path)
 query_sql = "SELECT request, response_messages, prompt_tokens, completion_tokens \
-                FROM result WHERE success='True'"
+                FROM result WHERE success='1'"
 # how to save base64.b64encode(pickle.dumps(benchmark_data["request"])).decode("ascii"), 
 with con:
     rows = con.execute(query_sql).fetchall()
@@ -182,35 +371,99 @@ with con:
             response_content = ''
             for response in responses:
                 response = json.loads(response)
+                if not response['choices']:
+                   continue
                 response_content += response['choices'][0]['delta']['content']
-            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' % (request['messages'][0]['content'], row[2], response_content, row[3]))
+            print('prompt: %s, tokens: %s, completion: %s, tokens: %s' %
+                  (request['messages'][0]['content'], row[2], response_content,
+                   row[3]))
 ```
 
-#### 支持的 API 有哪些
-目前支持的 API 请求有 openai、dashscope 和 zhipu。您可以通过 --api 指定 API。
+## Speed Benchmark
+若想进行速度测试，得到[Qwen官方](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html)报告的速度基准，请使用 `--dataset [speed_benchmark|speed_benchmark_long]`。
+
+- `speed_benchmark`: 测试[1, 6144, 14336, 30720]长度的prompt，固定输出2048个token。
+- `speed_benchmark_long`: 测试[63488, 129024]长度的prompt，固定输出2048个token。
+
+```{note}
+速度测试`--url`需要使用`/v1/completions`端点，而不是`/v1/chat/completions`，避免chat template的额外处理对输入长度有影响。
+```
+
+### 基于Transformer推理
+```bash
+CUDA_VISIBLE_DEVICES=0 evalscope perf \
+ --rate 1 \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
+ --attn-implementation flash_attention_2 \
+ --log-every-n-query 5 \
+ --connect-timeout 6000 \
+ --read-timeout 6000 \
+ --max-tokens 2048 \
+ --min-tokens 2048 \
+ --api local \
+ --dataset speed_benchmark \
+ --debug
+```
 
-您可以使用 --query-template 自定义您的请求，您可以指定一个 JSON 字符串如下：
-```json
-{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}
+输出示例：
+```
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |      50.69      |      0.97      |
+|     6144      |      51.36      |      1.23      |
+|     14336     |      49.93      |      1.59      |
+|     30720     |      49.56      |      2.34      |
++---------------+-----------------+----------------+
 ```
 
-或使用本地文件 `@to_query_template_path`。我们将用 `model` 替换 `%m`，用 `prompt` 替换 `%p`。
+### 基于vLLM推理
+```bash
+CUDA_VISIBLE_DEVICES=0 evalscope perf \
+ --rate 1 \
+ --model Qwen/Qwen2.5-0.5B-Instruct \
+ --log-every-n-query 5 \
+ --connect-timeout 6000 \
+ --read-timeout 6000 \
+ --max-tokens 2048 \
+ --min-tokens 2048 \
+ --api local_vllm \
+ --dataset speed_benchmark 
+```
+输出示例（注意vllm预留显存，此处不显示GPU使用情况）：
+```
+Speed Benchmark Results:
++---------------+-----------------+----------------+
+| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
++---------------+-----------------+----------------+
+|       1       |     343.08      |      0.0       |
+|     6144      |     334.71      |      0.0       |
+|     14336     |     318.88      |      0.0       |
+|     30720     |     292.86      |      0.0       |
++---------------+-----------------+----------------+
+```
 
 
-#### 如何扩展 API
-要扩展 API，您可以创建 `ApiPluginBase` 的子类，并使用 `@register_api("api 名称")` 进行注解。
+## 自定义请求 API
+目前支持的 API 请求格式有 openai、dashscope。要扩展 API，您可以继承 `ApiPluginBase` 类，并使用 `@register_api("api名称")` 进行注解，需实现如下两个方法：
 
-通过 model、prompt 和 query_template 来构建请求的 build_request。您可以参考 opanai_api.py。
+- `build_request()`方法通过 `messages` 和 `param`中的`model`和`query_template` 来构建请求，改请求后续发送到目标API。
 
-`parse_responses` 方法将返回 `prompt_tokens` 和 `completion_tokens` 的数量。
+- `parse_responses()` 方法将返回 `prompt_tokens` 和 `completion_tokens` 的数量，用于计算推理速度。
 
+参考如下代码：
 ```python
-class ApiPluginBase:
+from typing import Any, Dict, List, Tuple
+from evalscope.perf.arguments import Arguments
+
+@register_api('custom')
+class CustomPlugin(ApiPluginBase):
     def __init__(self, model_path: str) -> None:
         self.model_path = model_path
         
     @abstractmethod
-    def build_request(self, messages: List[Dict], param: QueryParameters)->Dict:
+    def build_request(self, messages: List[Dict], param: Arguments) -> Dict:
         """Build a api request body.
 
         Args:
@@ -243,76 +496,31 @@ class ApiPluginBase:
         raise NotImplementedError  
 ```
 
-#### 支持的数据集
-目前支持逐行（line by line）、longalpaca 和 openqa 数据集。
-
-- 逐行方法将每一行作为一个提示。
+## 自定义数据集
 
-- longalpaca 将获取 item['instruction'] 作为提示。
-
-- openqa 将获取 item['question'] 作为提示。
-
-#### 如何扩展数据集
-要扩展数据集，您可以创建 `DatasetPluginBase` 的子类，并使用 `@register_dataset('数据集名称')` 进行注解，
-
-然后实现 `build_prompt` 方法以返回一个prompt。
+要自定义数据集，您可以继承 `DatasetPluginBase` 类，并使用 `@register_dataset('数据集名称')` 进行注解，然后实现 `build_messages` 方法以返回一个message，格式参考[OpenAI API](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages)。
 
 ```python
-class DatasetPluginBase:
-    def __init__(self, query_parameters: QueryParameters):
-        """Build data set plugin
+from typing import Dict, Iterator, List
 
-        Args:
-            dataset_path (str, optional): The input dataset path. Defaults to None.
-        """
-        self.query_parameters = query_parameters
+from evalscope.perf.arguments import Arguments
+from evalscope.perf.plugin.datasets.base import DatasetPluginBase
+from evalscope.perf.plugin.registry import register_dataset
 
-    def __next__(self):
-        for item in self.build_messages():
-            yield item
-        raise StopIteration
 
-    def __iter__(self):
-        return self.build_messages()
-    
-    @abstractmethod
-    def build_messages(self)->Iterator[List[Dict]]:
-        """Build the request.
+@register_dataset('custom')
+class CustomDatasetPlugin(DatasetPluginBase):
+    """Read dataset and return prompt.
+    """
 
-        Raises:
-            NotImplementedError: The request is not impletion.
-
-        Yields:
-            Iterator[List[Dict]]: Yield request messages.
-        """
-        raise NotImplementedError
-    
-    def dataset_line_by_line(self, dataset: str)->Iterator[str]:
-        """Get content line by line of dataset.
-
-        Args:
-            dataset (str): The dataset path.
+    def __init__(self, query_parameters: Arguments):
+        super().__init__(query_parameters)
 
-        Yields:
-            Iterator[str]: Each line of file.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            for line in f:
-                yield line
-    
-    def dataset_json_list(self, dataset: str)->Iterator[Dict]:
-        """Read data from file which is list of requests.
-           Sample: https://huggingface.co/datasets/Yukang/LongAlpaca-12k
-
-        Args:
-            dataset (str): The dataset path.
-
-        Yields:
-            Iterator[Dict]: The each request object.
-        """
-        with open(dataset, 'r', encoding='utf-8') as f:
-            content = f.read()
-        data = json.loads(content)
-        for item in data:
-            yield item      
+    def build_messages(self) -> Iterator[List[Dict]]:
+        for item in self.dataset_line_by_line(self.query_parameters.dataset_path):
+            prompt = item.strip()
+            if len(prompt) > self.query_parameters.min_prompt_length and len(
+                    prompt) < self.query_parameters.max_prompt_length:
+                yield [{'role': 'user', 'content': prompt}]
+   
 ```
\ No newline at end of file
diff --git a/evalscope/preprocess/__init__.py b/tests/perf/__init__.py
similarity index 100%
rename from evalscope/preprocess/__init__.py
rename to tests/perf/__init__.py
diff --git a/tests/perf/test_perf.py b/tests/perf/test_perf.py
new file mode 100644
index 00000000..a55b4d15
--- /dev/null
+++ b/tests/perf/test_perf.py
@@ -0,0 +1,96 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+import os
+import unittest
+
+from evalscope.perf.main import run_perf_benchmark
+from evalscope.utils import test_level_list
+
+
+class TestPerf(unittest.TestCase):
+
+    def setUp(self) -> None:
+        pass
+
+    def tearDown(self) -> None:
+        pass
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf(self):
+        task_cfg = {
+            'url': 'http://127.0.0.1:8000/v1/chat/completions',
+            'parallel': 1,
+            'model': 'qwen2.5',
+            'number': 15,
+            'api': 'openai',
+            'dataset': 'openqa',
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf_stream(self):
+        task_cfg = {
+            'url': 'http://127.0.0.1:8000/v1/chat/completions',
+            'parallel': 1,
+            'model': 'qwen2.5',
+            'number': 15,
+            'api': 'openai',
+            'dataset': 'openqa',
+            'stream': True,
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf_speed_benchmark(self):
+        task_cfg = {
+            'url': 'http://127.0.0.1:8000/v1/completions',
+            'parallel': 1,
+            'model': 'qwen2.5',
+            'api': 'openai',
+            'dataset': 'speed_benchmark',
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf_local(self):
+        task_cfg = {
+            'parallel': 1,
+            'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+            'number': 5,
+            'api': 'local',
+            'dataset': 'openqa',
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf_local_stream(self):
+        task_cfg = {
+            'parallel': 1,
+            'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+            'number': 5,
+            'api': 'local',
+            'dataset': 'openqa',
+            'stream': True,
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), 'skip test in current test level')
+    def test_run_perf_local_speed_benchmark(self):
+        task_cfg = {
+            'parallel': 1,
+            'model': 'Qwen/Qwen2.5-0.5B-Instruct',
+            'api': 'local_vllm',
+            'dataset': 'speed_benchmark',
+            'min_tokens': 2048,
+            'max_tokens': 2048,
+            'debug': True,
+        }
+        run_perf_benchmark(task_cfg)
+
+
+if __name__ == '__main__':
+    unittest.main(buffer=False)