Repository: modelscope/evalscope (language: python)

Issue:
AttributeError: can't set attribute 'split'
**Running this script results in an error. How can I fix it?**
**- script**
```
python -m evalscope.run \
 --model qwen/Qwen2-0.5B-Instruct \
 --template-type qwen \
 --datasets arc 
```

**- error**
```
Generating test split: 169 examples [00:00, 2188.70 examples/s]
Generating dev split: 5 examples [00:00, 998.88 examples/s]
Traceback (most recent call last):
  File "/root/miniconda3/envs/evalscope_py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/evalscope_py310/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/run.py", line 404, in <module>
    main()
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/run.py", line 398, in main
    run_task(task_cfg)
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/run.py", line 345, in run_task
    evaluator = Evaluator(
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/evaluator/evaluator.py", line 101, in __init__
    self.dataset = self.data_adapter.load(dataset_name_or_path=dataset_name_or_path,
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/benchmarks/data_adapter.py", line 79, in load
    dataset = Benchmark.load(dataset_name=dataset_name_or_path,
  File "/workspace/nfs/10_performance/evalscope/package/evalscope/benchmarks/benchmark.py", line 49, in load
    dataset.split = split
```


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 2b02831c..c36b136a 100644
--- a/README.md
+++ b/README.md
@@ -29,30 +29,11 @@ English | [简体中文](README_zh.md)
 
 ## 📝 Introduction
 
-Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework.
-
-### Framework Features
-- **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc.
-- **Evaluation Metrics**: Implements various commonly used evaluation metrics.
-- **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families.
-- **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models.
-- **Evaluation Reports**: Automatically generates evaluation reports.
-- **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including:
-  - **Single mode**: Scoring a single model.
-  - **Pairwise-baseline mode**: Comparing against a baseline model.
-  - **Pairwise (all) mode**: Pairwise comparison among all models.
-- **Visualization Tools**: Provides intuitive displays of evaluation results.
-- **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html).
-- **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation.
-- **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
-- **Full-Link Support**: Through seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency.
-
-
-<details><summary>Overall Architecture</summary>
+EvalScope is the official model evaluation and performance benchmarking framework launched by the [ModelScope](https://modelscope.cn/) community. It comes with built-in common benchmarks and evaluation metrics, such as MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope supports various types of model evaluations, including LLMs, multimodal LLMs, embedding models, and reranker models. It is also applicable to multiple evaluation scenarios, such as end-to-end RAG evaluation, arena mode, and model inference performance stress testing. Moreover, with the seamless integration of the ms-swift training framework, evaluations can be initiated with a single click, providing full end-to-end support from model training to evaluation 🚀
 
 <p align="center">
   <img src="docs/en/_static/images/evalscope_framework.png" width="70%">
-  <br>Fig 1. EvalScope Framework.
+  <br>EvalScope Framework.
 </p>
 
 The architecture includes the following modules:
@@ -62,14 +43,15 @@ The architecture includes the following modules:
     - **Native**: EvalScope’s own **default evaluation framework**, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.
     - **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
     - **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
+    - **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
     - **ThirdParty**: Other third-party evaluation tasks, such as ToolBench.
 4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
 5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
 6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.
-</details>
 
 
 ## 🎉 News
+- 🔥 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
 - 🔥 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [📖 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
 - 🔥 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
 - 🔥 **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
@@ -240,9 +222,10 @@ run_task(task_cfg=your_task_cfg)
 ## Evaluation Backend
 EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:
 - **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.
-- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/opencompass_backend.html)
-- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/vlmevalkit_backend.html)
-- **ThirdParty**: The third-party task, e.g. [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html), you can contribute your own evaluation task to EvalScope as third-party backend.
+- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
+- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
+- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
+- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
 
 ## Custom Dataset Evaluation
 EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [📖User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html)
@@ -271,6 +254,8 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 
 
 ## TO-DO List
+- [x] RAG evaluation
+- [x] VLM evaluation
 - [x] Agents evaluation
 - [x] vLLM
 - [ ] Distributed evaluating
@@ -282,3 +267,7 @@ Refer to : [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free)
 - [ ] Auto-reviewer
   - [ ] Qwen-max
 
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/evalscope&type=Date)](https://star-history.com/#modelscope/evalscope&Date)
\ No newline at end of file
diff --git a/README_zh.md b/README_zh.md
index 358d9eb8..54705cd9 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -30,32 +30,15 @@
 
 
 ## 📝 简介
-大模型（包括大语言模型和多模态模型）评估，已成为评价和改进大模型的重要流程和手段，为了更好地支持大模型的评测，我们提出了EvalScope框架。
-
-### 框架特点
-- **基准数据集**：预置了多个常用测试基准，包括：MMLU、CMMLU、C-Eval、GSM8K、ARC、HellaSwag、TruthfulQA、MATH、HumanEval等。
-- **评估指标**：实现了多种常用评估指标。
-- **模型接入**：统一的模型接入机制，兼容多个系列模型的Generate、Chat接口。
-- **自动评估**：包括客观题自动评估和使用专家模型进行的复杂任务评估。
-- **评估报告**：自动生成评估报告。
-- **竞技场(Arena)模式**：用于模型间的比较以及模型的客观评估，支持多种评估模式，包括：
-  - **Single mode**：对单个模型进行评分。
-  - **Pairwise-baseline mode**：与基线模型进行对比。
-  - **Pairwise (all) mode**：所有模型间的两两对比。
-- **可视化工具**：提供直观的评估结果展示。
-- **模型性能评估**：提供模型推理服务压测工具和详细统计，详见[模型性能评估文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test.html)。
-- **OpenCompass集成**：支持OpenCompass作为评测后端，对其进行了高级封装和任务简化，您可以更轻松地提交任务进行评估。
-- **VLMEvalKit集成**：支持VLMEvalKit作为评测后端，轻松发起多模态评测任务，支持多种多模态模型和数据集。
-- **全链路支持**：通过与[ms-swift](https://github.com/modelscope/ms-swift)训练框架的无缝集成，实现模型训练、模型部署、模型评测、评测报告查看的一站式开发流程，提升用户的开发效率。
-
-<details><summary>框架架构</summary>
+
+EvalScope是[魔搭社区](https://modelscope.cn/)官方推出的模型评估与性能基准测试框架，内置多个常用测试基准和评估指标，如MMLU、CMMLU、C-Eval、GSM8K、ARC、HellaSwag、TruthfulQA、MATH和HumanEval等；支持多种类型的模型评测，包括LLM、多模态LLM、embedding模型和reranker模型。EvalScope还适用于多种评测场景，如端到端RAG评测、竞技场模式和模型推理性能压测等。此外，通过ms-swift训练框架的无缝集成，可一键发起评测，实现了模型训练到评测的全链路支持🚀
 
 <p align="center">
     <img src="docs/en/_static/images/evalscope_framework.png" style="width: 70%;">
-    <br>图 1. EvalScope 整体架构图.
+    <br>EvalScope 整体架构图.
 </p>
 
-包括以下模块：
+EvalScope包括以下模块：
 
 1. **Model Adapter**: 模型适配器，用于将特定模型的输出转换为框架所需的格式，支持API调用的模型和本地运行的模型。
 
@@ -65,6 +48,7 @@
     - **Native**：EvalScope自身的**默认评测框架**，支持多种评估模式，包括单模型评估、竞技场模式、Baseline模型对比模式等。
     - **OpenCompass**：支持[OpenCompass](https://github.com/open-compass/opencompass)作为评测后端，对其进行了高级封装和任务简化，您可以更轻松地提交任务进行评估。
     - **VLMEvalKit**：支持[VLMEvalKit](https://github.com/open-compass/VLMEvalKit)作为评测后端，轻松发起多模态评测任务，支持多种多模态模型和数据集。
+    - **RAGEval**：支持RAG评估，支持使用[MTEB/CMTEB](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html)进行embedding模型和reranker的独立评测，以及使用[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)进行端到端评测。
     - **ThirdParty**：其他第三方评估任务，如ToolBench。
 
 4. **Performance Evaluator**: 模型性能评测，负责具体衡量模型推理服务性能，包括性能评测、压力测试、性能评测报告生成、可视化。
@@ -73,9 +57,9 @@
 
 6. **Visualization**: 可视化结果，帮助用户更直观地理解评估结果，便于分析和比较不同模型的表现。
 
-</details>
 
 ## 🎉 新闻
+- 🔥 **[2024.10.8]** 支持RAG评测，包括使用[MTEB/CMTEB](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html)进行embedding模型和reranker的独立评测，以及使用[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)进行端到端评测。
 - 🔥 **[2024.09.18]** 我们的文档增加了博客模块，包含一些评测相关的技术调研和分享，欢迎[📖阅读](https://evalscope.readthedocs.io/zh-cn/latest/blog/index.html)
 - 🔥 **[2024.09.12]** 支持 LongWriter 评估，您可以使用基准测试 [LongBench-Write](evalscope/third_party/longbench_write/README.md) 来评测长输出的质量以及输出长度。
 - 🔥 **[2024.08.30]** 支持自定义数据集评测，包括文本数据集和多模态图文数据集。
@@ -254,9 +238,10 @@ run_task(task_cfg=your_task_cfg)
 ## 使用其他评测后端
 EvalScope支持使用第三方评测框架发起评测任务，我们称之为评测后端 (Evaluation Backend)。目前支持的Evaluation Backend有：
 - **Native**：EvalScope自身的**默认评测框架**，支持多种评估模式，包括单模型评估、竞技场模式、Baseline模型对比模式等。
-- [OpenCompass](https://github.com/open-compass/opencompass)：通过EvalScope作为入口，发起OpenCompass的评测任务，轻量级、易于定制、支持与LLM微调框架[ms-wift](https://github.com/modelscope/swift)的无缝集成，[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/opencompass_backend.html)
-- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)：通过EvalScope作为入口，发起VLMEvalKit的多模态评测任务，支持多种多模态模型和数据集，支持与LLM微调框架[ms-wift](https://github.com/modelscope/swift)的无缝集成，[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/vlmevalkit_backend.html)
-- **ThirdParty**: 第三方评估任务，如[ToolBench](https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolbench.html)。
+- [OpenCompass](https://github.com/open-compass/opencompass)：通过EvalScope作为入口，发起OpenCompass的评测任务，轻量级、易于定制、支持与LLM微调框架[ms-wift](https://github.com/modelscope/swift)的无缝集成：[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/opencompass_backend.html)
+- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)：通过EvalScope作为入口，发起VLMEvalKit的多模态评测任务，支持多种多模态模型和数据集，支持与LLM微调框架[ms-wift](https://github.com/modelscope/swift)的无缝集成：[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/vlmevalkit_backend.html)
+- **RAGEval**：通过EvalScope作为入口，发起RAG评测任务，支持使用[MTEB/CMTEB](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html)进行embedding模型和reranker的独立评测，以及使用[RAGAS](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)进行端到端评测：[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/index.html)
+- **ThirdParty**: 第三方评估任务，如[ToolBench](https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolbench.html)、[LongBench-Write](https://evalscope.readthedocs.io/zh-cn/latest/third_party/longwriter.html)。
 
 ## 自定义数据集评测
 EvalScope支持自定义数据集评测，具体请参考：自定义数据集评测[📖使用指南](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset.html)
@@ -280,6 +265,8 @@ ModelScope LLM Leaderboard大模型评测榜单旨在提供一个客观、全面
 
 
 ## TO-DO List
+- [x] RAG evaluation
+- [x] VLM evaluation
 - [x] Agents evaluation
 - [x] vLLM
 - [ ] Distributed evaluating
@@ -292,3 +279,6 @@ ModelScope LLM Leaderboard大模型评测榜单旨在提供一个客观、全面
   - [ ] Qwen-max
 
 
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/evalscope&type=Date)](https://star-history.com/#modelscope/evalscope&Date)
\ No newline at end of file
diff --git a/docs/en/_static/images/evalscope_framework.png b/docs/en/_static/images/evalscope_framework.png
index 5f0541a7..2ba90b5a 100644
Binary files a/docs/en/_static/images/evalscope_framework.png and b/docs/en/_static/images/evalscope_framework.png differ
diff --git a/docs/en/advanced_guides/custom_dataset.md b/docs/en/advanced_guides/custom_dataset.md
index 86b8ffe1..04d6ba5c 100644
--- a/docs/en/advanced_guides/custom_dataset.md
+++ b/docs/en/advanced_guides/custom_dataset.md
@@ -145,7 +145,7 @@ Custom dataset evaluation requires the use of `VLMEvalKit` and requires the inst
 ```shell
 pip install evalscope[vlmeval]
 ```
-Reference: [Using VLMEvalKit for Evaluation Backend](../user_guides/vlmevalkit_backend.md)
+Reference: [Using VLMEvalKit for Evaluation Backend](../user_guides/backend/vlmevalkit_backend.md)
 ````
 
 ### Multiple-Choice Question Format (MCQ)
diff --git a/docs/en/advanced_guides/custom_model.md b/docs/en/advanced_guides/custom_model.md
index c551e7bf..5fc32ec4 100644
--- a/docs/en/advanced_guides/custom_model.md
+++ b/docs/en/advanced_guides/custom_model.md
@@ -10,7 +10,7 @@ See [Basic Usage Guide: Submitting Evaluation Tasks with run_task](../get_starte
 
 ### Method 2: Using OpenCompass Evaluation
 ```{seealso}
-See [OpenCompass Usage Guide](../user_guides/opencompass_backend.md)
+See [OpenCompass Usage Guide](../user_guides/backend/opencompass_backend.md)
 ```
 -----
 ## VLM Model Evaluation
@@ -18,5 +18,5 @@ Currently, evaluation of multimodal models only supports the VLMEvalKit backend.
 
 ### Using VLMEvalKit Backend
 ```{seealso}
-See [VLMEvalKit Usage Guide](../user_guides/vlmevalkit_backend.md)
+See [VLMEvalKit Usage Guide](../user_guides/backend/vlmevalkit_backend.md)
 ```
\ No newline at end of file
diff --git a/docs/en/get_started/introduction.md b/docs/en/get_started/introduction.md
index 14cb05b8..73a6fc19 100644
--- a/docs/en/get_started/introduction.md
+++ b/docs/en/get_started/introduction.md
@@ -1,5 +1,5 @@
 # Introduction
-Large Language Model Evaluation (LLMs Evaluation) is a key process for assessing and improving large models. To enhance the efficiency and effectiveness of this evaluation process, we have designed the **EvalScope** framework.
+EvalScope is the official model evaluation and performance benchmarking framework launched by the [ModelScope](https://modelscope.cn/) community. It comes with built-in common benchmarks and evaluation metrics, such as MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope supports various types of model evaluations, including LLMs, multimodal LLMs, embedding models, and reranker models. It is also applicable to multiple evaluation scenarios, such as end-to-end RAG evaluation, arena mode, and model inference performance stress testing. Moreover, with the seamless integration of the ms-swift training framework, evaluations can be initiated with a single click, providing full end-to-end support from model training to evaluation.
 
 ## Overall Architecture
 ![EvalScope Architecture Diagram](../_static/images/evalscope_framework.png)
@@ -13,6 +13,7 @@ The architecture includes the following modules:
     - **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
     - **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
     - **ThirdParty**: Other third-party evaluation tasks, such as [ToolBench](../third_party/toolbench.md).
+    - **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](../user_guides/backend/rageval_backend/mteb.md), as well as end-to-end evaluation using [RAGAS](../user_guides/backend/rageval_backend/ragas.md).
 4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
 5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
 6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.
diff --git a/docs/en/get_started/supported_dataset.md b/docs/en/get_started/supported_dataset.md
index a0ed5d7c..c0983c98 100644
--- a/docs/en/get_started/supported_dataset.md
+++ b/docs/en/get_started/supported_dataset.md
@@ -2,7 +2,7 @@
 
 ## 1. Native Supported Datasets
 ```{tip}
-The framework currently supports the following datasets. If the dataset you need is not in the list, please submit an issue, or use the [OpenCompass backend](../user_guides/opencompass_backend.md) for evaluation, or use the [VLMEvalKit backend](../user_guides/vlmevalkit_backend.md) for multi-modal model evaluation.
+The framework currently supports the following datasets. If the dataset you need is not in the list, please submit an issue, or use the [OpenCompass backend](../user_guides/backend/opencompass_backend.md) for evaluation, or use the [VLMEvalKit backend](../user_guides/backend/vlmevalkit_backend.md) for multi-modal model evaluation.
 ```
 
 | Dataset Name       | Link                                                                                   | Status | Note |
diff --git a/docs/en/index.md b/docs/en/index.md
new file mode 100644
index 00000000..b9cf9c56
--- /dev/null
+++ b/docs/en/index.md
@@ -0,0 +1,53 @@
+![EvalScope Logo](_static/images/evalscope_logo.png)
+
+<br>
+
+# Welcome to the EvalScope Tutorial!
+
+## Getting Started
+To help users get up to speed quickly, we recommend the following process:
+- For users who want to use EvalScope, we recommend first reading the [Quick Start](get_started/introduction.md) section to set up the environment and start a mini experiment to familiarize themselves with the process.
+- For some basic usage, we suggest users read the [User Guides](user_guides/offline_evaluation.md), which include [How to Use EvalScope for Offline Evaluation](user_guides/offline_evaluation.md), [How to Use Arena Mode for Evaluation](user_guides/arena.md), [How to Use Other Evaluation Backends](user_guides/backend/index.md), and [How to Use the Model Service Stress Testing Tool](user_guides/stress_test.md).
+- If you want to customize more modules, such as adding datasets and models, we provide an [Advanced Tutorial](advanced_guides/custom_dataset.md).
+- Additionally, we offer [Third-Party Tools](third_party/toolbench.md) to help users quickly evaluate models, such as using ToolBench for evaluation.
+- Finally, we provide [Best Practices](best_practice/swift_integration.md) to assist users in evaluation, such as how to use Swift for evaluation.
+We always welcome PRs and issues from users to improve EvalScope.
+
+:::{toctree}
+:hidden:
+:caption: 🚀 Quick Start
+get_started/introduction.md
+get_started/installation.md
+get_started/basic_usage.md
+get_started/supported_dataset.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 🔧 User Guides
+user_guides/offline_evaluation.md
+user_guides/arena.md
+user_guides/stress_test.md
+user_guides/backend/index.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 🛠️ Advanced Tutorials
+advanced_guides/custom_dataset.md
+advanced_guides/custom_model.md
+:::
+:::{toctree}
+:hidden:
+:caption: 🧰 Third-Party Tools
+third_party/toolbench.md
+third_party/longwriter.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 📖 Best Practices
+best_practice/swift_integration.md
+best_practice/llm_full_stack.md
+best_practice/experiments.md
+:::
\ No newline at end of file
diff --git a/docs/en/index.rst b/docs/en/index.rst
deleted file mode 100644
index 7a6af65c..00000000
--- a/docs/en/index.rst
+++ /dev/null
@@ -1,77 +0,0 @@
-.. image:: _static/images/evalscope_logo.png
-
-.. raw:: html
-
-    <br>
-
-
-Welcome to the EvalScope Tutorial!
-==========================================
-Getting Started
--------------------------------
-To help users quickly get started, we recommend the following flow:
-
-- For those who want to use EvalScope, we recommend first reading the QuickStart_ section to set up the environment and initiate a mini-experiment to familiarize yourself with the process.
-
-- For some basic usages, we suggest users read the Tutorials_, which include how to perform offline evaluations with EvalScope, how to use Arena models for evaluations, how to utilize other evaluation backends, and how to use the model service stress testing tool.
-
-- If you wish to customize more modules, such as adding datasets and models, we provide the AdvancedTutorials_.
-
-- Additionally, we offer Third-PartyTools_ to help users quickly evaluate models, such as using ToolBench for evaluations.
-
-- Finally, we provide BestPractices_ to assist users with evaluations, such as how to use Swift for evaluations.
-
-We always welcome users' PRs and Issues to improve EvalScope.
-
-Contents
----------------
-.. _QuickStart:
-.. toctree::
-   :maxdepth: 1
-   :caption: Quick Start
-   
-   get_started/introduction.md
-   get_started/installation.md
-   get_started/basic_usage.md
-   get_started/supported_dataset.md
-
-.. _Tutorials:
-.. toctree::
-   :maxdepth: 1
-   :caption: Tutorials
-   
-   user_guides/offline_evaluation.md
-   user_guides/arena.md
-   user_guides/opencompass_backend.md
-   user_guides/vlmevalkit_backend.md
-   user_guides/stress_test.md
-
-.. _AdvancedTutorials:
-.. toctree::
-   :maxdepth: 1
-   :caption: Advanced Tutorials
-   
-   advanced_guides/custom_dataset.md
-   advanced_guides/custom_model.md
-
-.. _Third-PartyTools:
-.. toctree::
-   :maxdepth: 1
-   :caption: Third-Party Tools
-   
-   third_party/toolbench.md
-   third_party/longwriter.md
-
-.. _BestPractices:
-.. toctree::
-   :maxdepth: 1
-   :caption: Best Practices
-   
-   best_practice/swift_integration.md
-   best_practice/llm_full_stack.md
-   best_practice/experiments.md
-
-.. Index and Tables
-.. ==================
-.. * :ref:`genindex`
-.. * :ref:`search`
\ No newline at end of file
diff --git a/docs/en/user_guides/backend/index.md b/docs/en/user_guides/backend/index.md
new file mode 100644
index 00000000..de4f56ce
--- /dev/null
+++ b/docs/en/user_guides/backend/index.md
@@ -0,0 +1,35 @@
+# Evaluation Backends
+:::{toctree}
+:hidden:
+opencompass_backend.md
+vlmevalkit_backend.md
+rageval_backend/index.md
+:::
+
+EvalScope supports multiple evaluation backends to integrate various tools for different evaluation tasks, detailed as follows:
+
+::::{grid} 2
+:::{grid-item-card} OpenCompass
+:link: opencompass
+:link-type: ref
+
++++
+Initiate lightweight OpenCompass evaluation tasks through EvalScope, supporting various LLM evaluation tasks.
+:::
+
+:::{grid-item-card} VLMEvalKit
+:link: vlmeval
+:link-type: ref
+
++++
+Initiate lightweight VLMEvalKit evaluation tasks through EvalScope, supporting various VLM evaluation tasks.
+:::
+
+:::{grid-item-card} RAGEval
+:link: rageval
+:link-type: ref
+
++++
+Supports independent and end-to-end evaluation for RAG: allows independent evaluation of the retrieval module using MTEB/CMTEB, as well as end-to-end evaluation using RAGAS.
+:::
+::::
\ No newline at end of file
diff --git a/docs/en/user_guides/opencompass_backend.md b/docs/en/user_guides/backend/opencompass_backend.md
similarity index 97%
rename from docs/en/user_guides/opencompass_backend.md
rename to docs/en/user_guides/backend/opencompass_backend.md
index e5b7f6b9..d3a9fb8a 100644
--- a/docs/en/user_guides/opencompass_backend.md
+++ b/docs/en/user_guides/backend/opencompass_backend.md
@@ -1,4 +1,5 @@
-# OpenCompass Backend
+(opencompass)=
+# OpenCompass
 
 To facilitate the use of the OpenCompass evaluation backend, we have customized the OpenCompass source code and named it `ms-opencompass`. This version optimizes task configuration and execution and supports installation via PyPI, enabling users to initiate lightweight OpenCompass evaluation tasks through EvalScope. Additionally, we initially provide an interface for evaluation tasks based on the OpenAI API format. You can deploy model services using [ms-swift](https://github.com/modelscope/swift) , [vLLM](https://github.com/vllm-project/vllm) , [LMDeploy](https://github.com/InternLM/lmdeploy) , [Ollama](https://ollama.ai/), etc., to launch model inference services.
 
@@ -10,7 +11,7 @@ pip install evalscope[opencompass] -U
 
 ## 2. Data Preparation
 ````{note}
-There are three ways to download datasets. The automatic download method supports fewer datasets than the manual download method. Please use as needed. For detailed dataset information, refer to the [OpenCompass Dataset List](https://hub.opencompass.org.cn/home).
+There are two ways to download datasets. The automatic download method supports fewer datasets than the manual download method. Please use as needed. For detailed dataset information, refer to the [OpenCompass Dataset List](https://hub.opencompass.org.cn/home).
 You can view the dataset name list using the following code:
 ```python
 from evalscope.backend.opencompass import OpenCompassBackendManager
diff --git a/docs/en/user_guides/backend/rageval_backend/images/eval_result.png b/docs/en/user_guides/backend/rageval_backend/images/eval_result.png
new file mode 100644
index 00000000..aea097c4
Binary files /dev/null and b/docs/en/user_guides/backend/rageval_backend/images/eval_result.png differ
diff --git a/docs/en/user_guides/backend/rageval_backend/images/generation_process.png b/docs/en/user_guides/backend/rageval_backend/images/generation_process.png
new file mode 100644
index 00000000..1120d927
Binary files /dev/null and b/docs/en/user_guides/backend/rageval_backend/images/generation_process.png differ
diff --git a/docs/en/user_guides/backend/rageval_backend/index.md b/docs/en/user_guides/backend/rageval_backend/index.md
new file mode 100644
index 00000000..589a5d4a
--- /dev/null
+++ b/docs/en/user_guides/backend/rageval_backend/index.md
@@ -0,0 +1,32 @@
+(rageval)=
+# RAGEval
+:::{toctree}
+:hidden:
+mteb.md
+ragas.md
+:::
+
+After the research on RAG evaluation-related topics [investigated](../../../blog/RAG/RAG_Evaluation.md), this project now supports two methods for evaluating RAG: Independent Evaluation and End-to-End Evaluation:
+
+- **Independent Evaluation Method**: Evaluates the Retrieval Module and Generation Module separately. The evaluation metrics for the Retrieval Module include **Hit Rate, Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), Precision**, etc. These metrics are used to measure the effectiveness of the system in ranking items based on queries or tasks. The evaluation for the Generation Module is usually conducted during the End-to-End Evaluation. The metrics focus primarily on contextual relevance, measuring the relevance of retrieved documents to the query.
+
+- **End-to-End Evaluation Method**: Evaluates the final response generated by the RAG model for a given input, including the relevance and alignment of the model-generated answer with the input query. From the perspective of content generation objectives, evaluation can be divided into unannotated and annotated content. Metrics for unannotated content include **Answer Fidelity, Answer Relevance, Harmlessness**, while metrics for annotated content include Accuracy and Exact Match (EM).
+
+This framework supports independent evaluation of the retrieval module using [MTEB/CMTEB](mteb.md) and end-to-end evaluation using [RAGAS](ragas.md).
+
+
+::::{grid} 2
+:::{grid-item-card}  MTEB
+:link: mteb
+:link-type: ref
+
+Independent evaluation of the retrieval module, supporting embedding models and reranker models.
+:::
+
+:::{grid-item-card}  RAGAS
+:link: ragas
+:link-type: ref
+
+End-to-end evaluation, supporting the automatic generation of evaluation sets.
+:::
+::::
\ No newline at end of file
diff --git a/docs/en/user_guides/backend/rageval_backend/mteb.md b/docs/en/user_guides/backend/rageval_backend/mteb.md
new file mode 100644
index 00000000..63a9eb70
--- /dev/null
+++ b/docs/en/user_guides/backend/rageval_backend/mteb.md
@@ -0,0 +1,601 @@
+(mteb)=
+
+# MTEB
+This framework supports [MTEB](https://github.com/embeddings-benchmark/mteb) and [CMTEB](https://github.com/embeddings-benchmark/mteb), with the following details:
+
+- MTEB (Massive Text Embedding Benchmark) is a large-scale benchmark designed to measure the performance of text embedding models across diverse embedding tasks. MTEB includes 56 datasets covering 8 tasks and supports over 112 different languages. The goal of this benchmark is to assist developers in finding the best text embedding models suitable for various tasks.
+
+- C-MTEB (Chinese Massive Text Embedding Benchmark) is a dedicated evaluation benchmark for Chinese text vectors, built on MTEB, aimed at assessing the performance of Chinese text vector models. C-MTEB collects 35 public datasets and is divided into 6 categories of evaluation tasks, including retrieval, re-ranking, semantic text similarity (STS), classification, pair classification, and clustering.
+
+## Supported Datasets
+Here is an overview of the available tasks and datasets in C-MTEB:
+
+| Name | Hub Link | Description | Type | Category | Number of Test Samples |
+|-----|-----|---------------------------|-----|-----|-----|
+| [T2Retrieval](https://arxiv.org/abs/2304.03679) | [C-MTEB/T2Retrieval](https://modelscope.cn/datasets/C-MTEB/T2Retrieval) | T2Ranking: A large-scale Chinese paragraph ranking benchmark | Retrieval | s2p | 24,832 |
+| [MMarcoRetrieval](https://github.com/unicamp-dl/mMARCO) | [C-MTEB/MMarcoRetrieval](https://modelscope.cn/datasets/C-MTEB/MMarcoRetrieval) | mMARCO is the multilingual version of the MS MARCO paragraph ranking dataset | Retrieval | s2p | 7,437 |
+| [DuRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/DuRetrieval](https://modelscope.cn/datasets/C-MTEB/DuRetrieval) | A large-scale Chinese web search engine paragraph retrieval benchmark | Retrieval | s2p | 4,000 |
+| [CovidRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/CovidRetrieval](https://modelscope.cn/datasets/C-MTEB/CovidRetrieval) | COVID-19 news articles | Retrieval | s2p | 949 |
+| [CmedqaRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/CmedqaRetrieval](https://modelscope.cn/datasets/C-MTEB/CmedqaRetrieval) | Online medical consultation texts | Retrieval | s2p | 3,999 |
+| [EcomRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/EcomRetrieval](https://modelscope.cn/datasets/C-MTEB/EcomRetrieval) | Paragraph retrieval dataset collected from Alibaba e-commerce search engine systems | Retrieval | s2p | 1,000 |
+| [MedicalRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/MedicalRetrieval](https://modelscope.cn/datasets/C-MTEB/MedicalRetrieval) | Paragraph retrieval dataset collected from Alibaba medical search engine systems | Retrieval | s2p | 1,000 |
+| [VideoRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/VideoRetrieval](https://modelscope.cn/datasets/C-MTEB/VideoRetrieval) | Paragraph retrieval dataset collected from Alibaba video search engine systems | Retrieval | s2p | 1,000 |
+| [T2Reranking](https://arxiv.org/abs/2304.03679) | [C-MTEB/T2Reranking](https://modelscope.cn/datasets/C-MTEB/T2Reranking) | T2Ranking: A large-scale Chinese paragraph ranking benchmark | Re-ranking | s2p | 24,382 |
+| [MMarcoReranking](https://github.com/unicamp-dl/mMARCO) | [C-MTEB/MMarco-reranking](https://modelscope.cn/datasets/C-MTEB/Mmarco-reranking) | mMARCO is the multilingual version of the MS MARCO paragraph ranking dataset | Re-ranking | s2p | 7,437 |
+| [CMedQAv1](https://github.com/zhangsheng93/cMedQA) | [C-MTEB/CMedQAv1-reranking](https://modelscope.cn/datasets/C-MTEB/CMedQAv1-reranking) | Chinese community medical Q&A | Re-ranking | s2p | 2,000 |
+| [CMedQAv2](https://github.com/zhangsheng93/cMedQA2) | [C-MTEB/CMedQAv2-reranking](https://modelscope.cn/datasets/C-MTEB/C-MTEB/CMedQAv2-reranking) | Chinese community medical Q&A | Re-ranking | s2p | 4,000 |
+| [Ocnli](https://arxiv.org/abs/2010.05444) | [C-MTEB/OCNLI](https://modelscope.cn/datasets/C-MTEB/OCNLI) | Original Chinese natural language inference dataset | Pair Classification | s2s | 3,000 |
+| [Cmnli](https://modelscope.cn/datasets/clue/viewer/cmnli) | [C-MTEB/CMNLI](https://modelscope.cn/datasets/C-MTEB/CMNLI) | Chinese multi-class natural language inference | Pair Classification | s2s | 139,000 |
+| [CLSClusteringS2S](https://arxiv.org/abs/2209.05034) | [C-MTEB/CLSClusteringS2S](https://modelscope.cn/datasets/C-MTEB/C-MTEB/CLSClusteringS2S) | Clustering titles from the CLS dataset. Clustering based on 13 sets of main categories. | Clustering | s2s | 10,000 |
+| [CLSClusteringP2P](https://arxiv.org/abs/2209.05034) | [C-MTEB/CLSClusteringP2P](https://modelscope.cn/datasets/C-MTEB/CLSClusteringP2P) | Clustering titles + abstracts from the CLS dataset. Clustering based on 13 sets of main categories. | Clustering | p2p | 10,000 |
+| [ThuNewsClusteringS2S](http://thuctc.thunlp.org/) | [C-MTEB/ThuNewsClusteringS2S](https://modelscope.cn/datasets/C-MTEB/ThuNewsClusteringS2S) | Clustering titles from the THUCNews dataset | Clustering | s2s | 10,000 |
+| [ThuNewsClusteringP2P](http://thuctc.thunlp.org/) | [C-MTEB/ThuNewsClusteringP2P](https://modelscope.cn/datasets/C-MTEB/ThuNewsClusteringP2P) | Clustering titles + abstracts from the THUCNews dataset | Clustering | p2p | 10,000 |
+| [ATEC](https://github.com/IceFlameWorm/NLP_Datasets/tree/master/ATEC) | [C-MTEB/ATEC](https://modelscope.cn/datasets/C-MTEB/ATEC) | ATEC NLP Sentence Pair Similarity Competition | STS | s2s | 20,000 |
+| [BQ](https://huggingface.co/datasets/shibing624/nli_zh) | [C-MTEB/BQ](https://modelscope.cn/datasets/C-MTEB/BQ) | Banking Question Semantic Similarity | STS | s2s | 10,000 |
+| [LCQMC](https://huggingface.co/datasets/shibing624/nli_zh) | [C-MTEB/LCQMC](https://modelscope.cn/datasets/C-MTEB/LCQMC) | Large-scale Chinese Question Matching Corpus | STS | s2s | 12,500 |
+| [PAWSX](https://arxiv.org/pdf/1908.11828.pdf) | [C-MTEB/PAWSX](https://modelscope.cn/datasets/C-MTEB/PAWSX) | Translated PAWS evaluation pairs | STS | s2s | 2,000 |
+| [STSB](https://github.com/pluto-junzeng/CNSD) | [C-MTEB/STSB](https://modelscope.cn/datasets/C-MTEB/STSB) | Translated STS-B into Chinese | STS | s2s | 1,360 |
+| [AFQMC](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/AFQMC](https://modelscope.cn/datasets/C-MTEB/AFQMC) | Ant Financial Question Matching Corpus | STS | s2s | 3,861 |
+| [QBQTC](https://github.com/CLUEbenchmark/QBQTC) | [C-MTEB/QBQTC](https://modelscope.cn/datasets/C-MTEB/QBQTC) | QQ Browser Query Title Corpus | STS | s2s | 5,000 |
+| [TNews](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/TNews-classification](https://modelscope.cn/datasets/C-MTEB/TNews-classification) | News Short Text Classification | Classification | s2s | 10,000 |
+| [IFlyTek](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/IFlyTek-classification](https://modelscope.cn/datasets/C-MTEB/IFlyTek-classification) | Long Text Classification of Application Descriptions | Classification | s2s | 2,600 |
+| [Waimai](https://github.com/SophonPlus/ChineseNlpCorpus/blob/master/datasets/waimai_10k/intro.ipynb) | [C-MTEB/waimai-classification](https://modelscope.cn/datasets/C-MTEB/waimai-classification) | Sentiment Analysis of User Reviews on Food Delivery Platforms | Classification | s2s | 1,000 |
+| [OnlineShopping](https://github.com/SophonPlus/ChineseNlpCorpus/blob/master/datasets/online_shopping_10_cats/intro.ipynb) | [C-MTEB/OnlineShopping-classification](https://modelscope.cn/datasets/C-MTEB/OnlineShopping-classification) | Sentiment Analysis of User Reviews on Online Shopping Websites | Classification | s2s | 1,000 |
+| [MultilingualSentiment](https://github.com/tyqiangz/multilingual-sentiment-datasets) | [C-MTEB/MultilingualSentiment-classification](https://modelscope.cn/datasets/C-MTEB/MultilingualSentiment-classification) | A set of multilingual sentiment datasets grouped into three categories: positive, neutral, negative | Classification | s2s | 3,000 |
+| [JDReview](https://huggingface.co/datasets/kuroneko5943/jd21) | [C-MTEB/JDReview-classification](https://modelscope.cn/datasets/C-MTEB/JDReview-classification) | Reviews of iPhone | Classification | s2s | 533 |
+
+For retrieval tasks, a sample of 100,000 candidates (including the ground truth) is drawn from the entire corpus to reduce inference costs.
+
+## Environment Setup
+Install dependencies
+```bash
+pip install mteb
+```
+
+## Configure Evaluation Parameters
+The framework supports two evaluation modes: single-stage evaluation and two-stage evaluation:
+- **Single-stage evaluation**: Directly use the model for prediction and calculate metrics.
+- **Two-stage evaluation**: Use the model for prediction, then apply a retrieval model for reranking, and calculate metrics.
+
+### Single-stage Evaluation
+Example configuration file:
+```python
+one_stage_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "MTEB",
+        "model": [
+            {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+                "pooling_mode": None,
+                "max_seq_length": 512,
+                "prompt": "",
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 128,
+                },
+            }
+        ],
+        "eval": {
+            "tasks": [
+                "TNews",
+                "CLSClusteringS2S",
+                "T2Reranking",
+                "T2Retrieval",
+                "ATEC",
+            ],
+            "verbosity": 2,
+            "output_folder": "outputs",
+            "overwrite_results": True,
+            "limits": 500,
+        },
+    },
+}
+```
+#### Parameter Explanation
+- `eval_backend`: Default value is `RAGEval`, indicating the use of the RAGEval evaluation backend.
+- `eval_config`: A dictionary containing the following fields:
+    - `tool`: Evaluation tool, using `MTEB`.
+    - `model`: List of model configurations, **only one model can be placed here for single-stage evaluation**, containing the following fields:
+        - `model_name_or_path`: `str` Model name or path
+        - `is_cross_encoder`: `bool` Indicates whether the model is a cross-encoder, default is False.  
+        - `pooling_mode`: `Optional[str]` Pooling mode, possible values are: “cls”, “lasttoken”, “max”, “mean”, “mean_sqrt_len_tokens” or “weightedmean”.  
+        - `max_seq_length`: `int` Maximum sequence length, default is 512.  
+        - `prompt`: `str` Prompt for large language model-based assistance.  
+        - `model_kwargs`: `dict` Model keyword arguments, default value is `{"torch_dtype": "auto"}`.  
+        - `config_kwargs`: `Dict[str, Any]` Configuration keyword arguments, default is an empty dictionary.  
+        - `encode_kwargs`: `dict` Encoding keyword arguments, default is:  
+            ```python  
+            {  
+                "show_progress_bar": True,  
+                "batch_size": 32
+            }  
+            ```  
+        - `hub`: `str` Model source, can be "modelscope" or "huggingface".  
+    - `eval`: Dictionary containing the following fields:
+        - `tasks`: `List[str]` Task names  
+        - `verbosity`: `int` Level of detail, ranging from 0-3  
+        - `output_folder`: `str` Output folder, default is "outputs"  
+        - `overwrite_results`: `bool` Indicates whether to overwrite results, default is True  
+        - `limits`: `Optional[int]` Limiting the number of samples, default is None  
+        - `hub`: `str` Model source, can be "modelscope" or "huggingface"  
+        - `top_k`: `int` Select the top K results, used in retrieval tasks  
+
+
+### Two-stage Evaluation
+Example configuration file: first perform retrieval, then reranking:
+```python
+two_stage_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "MTEB",
+        "model": [
+            {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+                "is_cross_encoder": False,
+                "max_seq_length": 512,
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 64,
+                },
+            },
+            {
+                "model_name_or_path": "OpenBMB/MiniCPM-Reranker",
+                "is_cross_encoder": True,
+                "max_seq_length": 512,
+                "prompt": "Generate a retrieval representation for this question",
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 32,
+                },
+            },
+        ],
+        "eval": {
+            "tasks": ["T2Retrieval"],
+            "verbosity": 2,
+            "output_folder": "outputs",
+            "overwrite_results": True,
+            "limits": 100,
+        },
+    },
+}
+```
+#### Parameter Explanation
+Basic parameters are the same as in single-stage evaluation, differing in the `model` field, where two models are required: the first model for retrieval and the second model for reranking, with the reranking model needing to be a cross encoder.
+
+## Model Evaluation
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+logger = get_logger()
+
+one_stage_task_cfg = one_stage_task_cfg
+# or
+# two_stage_task_cfg = two_stage_task_cfg
+
+# Run task
+run_task(task_cfg=one_stage_task_cfg) # or run_task(task_cfg=two_stage_task_cfg)
+```
+
+The following is an example of the output:
+
+**One-Stage Evaluation**
+
+<details><summary>Outputs</summary>
+
+```{code-block} json
+:caption: outputs/m3e-base/master/TNews.json
+
+{
+  "dataset_revision": "317f262bf1e6126357bbe89e875451e4b0938fe4",
+  "evaluation_time": 16.50650382041931,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "validation": [
+      {
+        "accuracy": 0.4744,
+        "f1": 0.44562489526640825,
+        "f1_weighted": 0.47540307398330806,
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.4744,
+        "scores_per_experiment": [
+          {
+            "accuracy": 0.48,
+            "f1": 0.4536376605217497,
+            "f1_weighted": 0.47800277926811163
+          },
+          {
+            "accuracy": 0.48,
+            "f1": 0.44713633954639176,
+            "f1_weighted": 0.4826984434763292
+          },
+          {
+            "accuracy": 0.462,
+            "f1": 0.433365706955334,
+            "f1_weighted": 0.4640970055245127
+          },
+          {
+            "accuracy": 0.484,
+            "f1": 0.4586732839614161,
+            "f1_weighted": 0.4857359110392786
+          },
+          {
+            "accuracy": 0.462,
+            "f1": 0.4293797541165097,
+            "f1_weighted": 0.4632657330831137
+          },
+          {
+            "accuracy": 0.474,
+            "f1": 0.44775120246296396,
+            "f1_weighted": 0.4737182842092953
+          },
+          {
+            "accuracy": 0.47,
+            "f1": 0.4431197566080463,
+            "f1_weighted": 0.4714830140231783
+          },
+          {
+            "accuracy": 0.472,
+            "f1": 0.44322381694059326,
+            "f1_weighted": 0.47100005556357255
+          },
+          {
+            "accuracy": 0.484,
+            "f1": 0.45454749692062835,
+            "f1_weighted": 0.4856239367465818
+          },
+          {
+            "accuracy": 0.476,
+            "f1": 0.44541393463044954,
+            "f1_weighted": 0.47840557689910646
+          }
+        ]
+      }
+    ]
+  },
+  "task_name": "TNews"
+}
+```
+</details>
+
+<br>
+
+**Two-stage Evaluation**
+
+<details><summary>first stage</summary>
+
+```{code-block} json
+:caption: outputs/stage1/m3e-base/v1/T2Retrieval.json
+
+{
+  "dataset_revision": "8731a845f1bf500a4f111cf1070785c793d10e64",
+  "evaluation_time": 599.5170171260834,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "dev": [
+      {
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.73143,
+        "map_at_1": 0.22347,
+        "map_at_10": 0.63237,
+        "map_at_100": 0.67533,
+        "map_at_1000": 0.67651,
+        "map_at_20": 0.66282,
+        "map_at_3": 0.43874,
+        "map_at_5": 0.54049,
+        "mrr_at_1": 0.7898912852884447,
+        "mrr_at_10": 0.8402654617870331,
+        "mrr_at_100": 0.8421827758769684,
+        "mrr_at_1000": 0.8422583001072272,
+        "mrr_at_20": 0.8415411456315557,
+        "mrr_at_3": 0.8307469752761716,
+        "mrr_at_5": 0.8368029984218875,
+        "nauc_map_at_1000_diff1": 0.17749400860890877,
+        "nauc_map_at_1000_max": 0.42844516520725967,
+        "nauc_map_at_1000_std": 0.18789871694419072,
+        "nauc_map_at_100_diff1": 0.17747467084779375,
+        "nauc_map_at_100_max": 0.42732291785494575,
+        "nauc_map_at_100_std": 0.18694287087286737,
+        "nauc_map_at_10_diff1": 0.19976199493034202,
+        "nauc_map_at_10_max": 0.3374436217668296,
+        "nauc_map_at_10_std": 0.07951451707732717,
+        "nauc_map_at_1_diff1": 0.41727578149080663,
+        "nauc_map_at_1_max": -0.1402656422184478,
+        "nauc_map_at_1_std": -0.26168722519030313,
+        "nauc_map_at_20_diff1": 0.1811898211371171,
+        "nauc_map_at_20_max": 0.40563441466210043,
+        "nauc_map_at_20_std": 0.15927727170010608,
+        "nauc_map_at_3_diff1": 0.31255422845809033,
+        "nauc_map_at_3_max": 0.007523677231905161,
+        "nauc_map_at_3_std": -0.19578481884353466,
+        "nauc_map_at_5_diff1": 0.26073699217160473,
+        "nauc_map_at_5_max": 0.14665611579604088,
+        "nauc_map_at_5_std": -0.09600383298672226,
+        "nauc_mrr_at_1000_diff1": 0.3819666309367981,
+        "nauc_mrr_at_1000_max": 0.6285393024619401,
+        "nauc_mrr_at_1000_std": 0.3294970299417527,
+        "nauc_mrr_at_100_diff1": 0.3819436006743644,
+        "nauc_mrr_at_100_max": 0.6286346262471935,
+        "nauc_mrr_at_100_std": 0.32963045935037844,
+        "nauc_mrr_at_10_diff1": 0.3819124721154632,
+        "nauc_mrr_at_10_max": 0.6292778905762176,
+        "nauc_mrr_at_10_std": 0.3298187966196067,
+        "nauc_mrr_at_1_diff1": 0.3862589251033909,
+        "nauc_mrr_at_1_max": 0.589976680174432,
+        "nauc_mrr_at_1_std": 0.2780515387897469,
+        "nauc_mrr_at_20_diff1": 0.38198959771391816,
+        "nauc_mrr_at_20_max": 0.6290569436652999,
+        "nauc_mrr_at_20_std": 0.3301570340189363,
+        "nauc_mrr_at_3_diff1": 0.3825046940733129,
+        "nauc_mrr_at_3_max": 0.6282507269128365,
+        "nauc_mrr_at_3_std": 0.3260807934869131,
+        "nauc_mrr_at_5_diff1": 0.3816317396711923,
+        "nauc_mrr_at_5_max": 0.6288655177904692,
+        "nauc_mrr_at_5_std": 0.3298854062538469,
+        "nauc_ndcg_at_1000_diff1": 0.21319598381916555,
+        "nauc_ndcg_at_1000_max": 0.5328295949130256,
+        "nauc_ndcg_at_1000_std": 0.2946773445135694,
+        "nauc_ndcg_at_100_diff1": 0.2089807772703975,
+        "nauc_ndcg_at_100_max": 0.5239397690321543,
+        "nauc_ndcg_at_100_std": 0.29123456982125717,
+        "nauc_ndcg_at_10_diff1": 0.20555333230027603,
+        "nauc_ndcg_at_10_max": 0.44316027023003046,
+        "nauc_ndcg_at_10_std": 0.1921835220940756,
+        "nauc_ndcg_at_1_diff1": 0.3862589251033909,
+        "nauc_ndcg_at_1_max": 0.589976680174432,
+        "nauc_ndcg_at_1_std": 0.2780515387897469,
+        "nauc_ndcg_at_20_diff1": 0.20754208582741446,
+        "nauc_ndcg_at_20_max": 0.4786092392092643,
+        "nauc_ndcg_at_20_std": 0.23536973680564616,
+        "nauc_ndcg_at_3_diff1": 0.1902823773882388,
+        "nauc_ndcg_at_3_max": 0.5400466380622567,
+        "nauc_ndcg_at_3_std": 0.2713874990424778,
+        "nauc_ndcg_at_5_diff1": 0.18279298790691637,
+        "nauc_ndcg_at_5_max": 0.4916119327522918,
+        "nauc_ndcg_at_5_std": 0.2375397192963552,
+        "nauc_precision_at_1000_diff1": -0.20510380600112582,
+        "nauc_precision_at_1000_max": 0.4958820760698651,
+        "nauc_precision_at_1000_std": 0.5402465580496146,
+        "nauc_precision_at_100_diff1": -0.1994322347949809,
+        "nauc_precision_at_100_max": 0.5206762748551254,
+        "nauc_precision_at_100_std": 0.5568154081333078,
+        "nauc_precision_at_10_diff1": -0.16707155441197413,
+        "nauc_precision_at_10_max": 0.5600612846655972,
+        "nauc_precision_at_10_std": 0.49419688804691536,
+        "nauc_precision_at_1_diff1": 0.3862589251033909,
+        "nauc_precision_at_1_max": 0.589976680174432,
+        "nauc_precision_at_1_std": 0.2780515387897469,
+        "nauc_precision_at_20_diff1": -0.18471041949530417,
+        "nauc_precision_at_20_max": 0.5458950955439645,
+        "nauc_precision_at_20_std": 0.5355982267058214,
+        "nauc_precision_at_3_diff1": -0.03826790088047189,
+        "nauc_precision_at_3_max": 0.5833083970750171,
+        "nauc_precision_at_3_std": 0.380196662597275,
+        "nauc_precision_at_5_diff1": -0.11789367842600275,
+        "nauc_precision_at_5_max": 0.5708494593335263,
+        "nauc_precision_at_5_std": 0.42860609671688105,
+        "nauc_recall_at_1000_diff1": 0.1341309660059583,
+        "nauc_recall_at_1000_max": 0.5923755841077135,
+        "nauc_recall_at_1000_std": 0.5980459502693942,
+        "nauc_recall_at_100_diff1": 0.12181394285840096,
+        "nauc_recall_at_100_max": 0.47090136790318127,
+        "nauc_recall_at_100_std": 0.3959369184297595,
+        "nauc_recall_at_10_diff1": 0.17356300971546512,
+        "nauc_recall_at_10_max": 0.25475707245853674,
+        "nauc_recall_at_10_std": 0.041819982320384745,
+        "nauc_recall_at_1_diff1": 0.41727578149080663,
+        "nauc_recall_at_1_max": -0.1402656422184478,
+        "nauc_recall_at_1_std": -0.26168722519030313,
+        "nauc_recall_at_20_diff1": 0.14273713155999543,
+        "nauc_recall_at_20_max": 0.36251116771924663,
+        "nauc_recall_at_20_std": 0.1912123941692314,
+        "nauc_recall_at_3_diff1": 0.2873719855400218,
+        "nauc_recall_at_3_max": -0.041198403561830285,
+        "nauc_recall_at_3_std": -0.21921947922872737,
+        "nauc_recall_at_5_diff1": 0.23680082643694844,
+        "nauc_recall_at_5_max": 0.06580524171324151,
+        "nauc_recall_at_5_std": -0.14104561361502632,
+        "ndcg_at_1": 0.78989,
+        "ndcg_at_10": 0.73143,
+        "ndcg_at_100": 0.78829,
+        "ndcg_at_1000": 0.80026,
+        "ndcg_at_20": 0.75787,
+        "ndcg_at_3": 0.7417,
+        "ndcg_at_5": 0.72641,
+        "precision_at_1": 0.78989,
+        "precision_at_10": 0.37304,
+        "precision_at_100": 0.04828,
+        "precision_at_1000": 0.00511,
+        "precision_at_20": 0.21403,
+        "precision_at_3": 0.65461,
+        "precision_at_5": 0.54942,
+        "recall_at_1": 0.22347,
+        "recall_at_10": 0.73318,
+        "recall_at_100": 0.91093,
+        "recall_at_1000": 0.97197,
+        "recall_at_20": 0.81286,
+        "recall_at_3": 0.46573,
+        "recall_at_5": 0.59383
+      }
+    ]
+  },
+  "task_name": "T2Retrieval"
+}
+
+```
+</details>
+
+<details><summary>second stage</summary>
+
+```{code-block} json
+:caption: outputs/stage2/jina-reranker-v2-base-multilingual/master/T2Retrieval.json
+
+{
+  "dataset_revision": "8731a845f1bf500a4f111cf1070785c793d10e64",
+  "evaluation_time": 332.15709686279297,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "dev": [
+      {
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.661,
+        "map_at_1": 0.24264,
+        "map_at_10": 0.56291,
+        "map_at_100": 0.56291,
+        "map_at_1000": 0.56291,
+        "map_at_20": 0.56291,
+        "map_at_3": 0.4714,
+        "map_at_5": 0.56291,
+        "mrr_at_1": 0.841969139049623,
+        "mrr_at_10": 0.8689147524694633,
+        "mrr_at_100": 0.8689147524694633,
+        "mrr_at_1000": 0.8689147524694633,
+        "mrr_at_20": 0.8689147524694633,
+        "mrr_at_3": 0.8664883979192248,
+        "mrr_at_5": 0.8689147524694633,
+        "nauc_map_at_1000_diff1": 0.12071580301051653,
+        "nauc_map_at_1000_max": 0.2536691069727338,
+        "nauc_map_at_1000_std": 0.343624832364704,
+        "nauc_map_at_100_diff1": 0.12071580301051653,
+        "nauc_map_at_100_max": 0.2536691069727338,
+        "nauc_map_at_100_std": 0.343624832364704,
+        "nauc_map_at_10_diff1": 0.12071580301051653,
+        "nauc_map_at_10_max": 0.2536691069727338,
+        "nauc_map_at_10_std": 0.343624832364704,
+        "nauc_map_at_1_diff1": 0.47964980727810325,
+        "nauc_map_at_1_max": -0.08015044571696166,
+        "nauc_map_at_1_std": 0.3507257834956417,
+        "nauc_map_at_20_diff1": 0.12071580301051653,
+        "nauc_map_at_20_max": 0.2536691069727338,
+        "nauc_map_at_20_std": 0.343624832364704,
+        "nauc_map_at_3_diff1": 0.23481937699306626,
+        "nauc_map_at_3_max": 0.10372745264123306,
+        "nauc_map_at_3_std": 0.45345158923063256,
+        "nauc_map_at_5_diff1": 0.12071580301051653,
+        "nauc_map_at_5_max": 0.2536691069727338,
+        "nauc_map_at_5_std": 0.343624832364704,
+        "nauc_mrr_at_1000_diff1": 0.23393918304502795,
+        "nauc_mrr_at_1000_max": 0.8703379129725659,
+        "nauc_mrr_at_1000_std": 0.5785333616122065,
+        "nauc_mrr_at_100_diff1": 0.23393918304502795,
+        "nauc_mrr_at_100_max": 0.8703379129725659,
+        "nauc_mrr_at_100_std": 0.5785333616122065,
+        "nauc_mrr_at_10_diff1": 0.23393918304502795,
+        "nauc_mrr_at_10_max": 0.8703379129725659,
+        "nauc_mrr_at_10_std": 0.5785333616122065,
+        "nauc_mrr_at_1_diff1": 0.2520016067648708,
+        "nauc_mrr_at_1_max": 0.8560897633767299,
+        "nauc_mrr_at_1_std": 0.5642467684745208,
+        "nauc_mrr_at_20_diff1": 0.23393918304502795,
+        "nauc_mrr_at_20_max": 0.8703379129725659,
+        "nauc_mrr_at_20_std": 0.5785333616122065,
+        "nauc_mrr_at_3_diff1": 0.2343988881957151,
+        "nauc_mrr_at_3_max": 0.8695482778251757,
+        "nauc_mrr_at_3_std": 0.5799167198804328,
+        "nauc_mrr_at_5_diff1": 0.23393918304502795,
+        "nauc_mrr_at_5_max": 0.8703379129725659,
+        "nauc_mrr_at_5_std": 0.5785333616122065,
+        "nauc_ndcg_at_1000_diff1": 0.11252208055013257,
+        "nauc_ndcg_at_1000_max": 0.3417865079349515,
+        "nauc_ndcg_at_1000_std": 0.3623961771041499,
+        "nauc_ndcg_at_100_diff1": 0.11252208055013257,
+        "nauc_ndcg_at_100_max": 0.3417865079349515,
+        "nauc_ndcg_at_100_std": 0.3623961771041499,
+        "nauc_ndcg_at_10_diff1": 0.10015448775533999,
+        "nauc_ndcg_at_10_max": 0.3761759074862075,
+        "nauc_ndcg_at_10_std": 0.35152523471339914,
+        "nauc_ndcg_at_1_diff1": 0.2524564785684737,
+        "nauc_ndcg_at_1_max": 0.8566368743831702,
+        "nauc_ndcg_at_1_std": 0.5635391925059349,
+        "nauc_ndcg_at_20_diff1": 0.11228113618796766,
+        "nauc_ndcg_at_20_max": 0.34274993051851965,
+        "nauc_ndcg_at_20_std": 0.36216437469674284,
+        "nauc_ndcg_at_3_diff1": -0.062134030685870506,
+        "nauc_ndcg_at_3_max": 0.7183183844837573,
+        "nauc_ndcg_at_3_std": 0.3352626268658533,
+        "nauc_ndcg_at_5_diff1": -0.04476981761624879,
+        "nauc_ndcg_at_5_max": 0.6272060974309411,
+        "nauc_ndcg_at_5_std": 0.21341258393783158,
+        "nauc_precision_at_1000_diff1": -0.3554940965683014,
+        "nauc_precision_at_1000_max": 0.605443274008298,
+        "nauc_precision_at_1000_std": -0.13073611213585504,
+        "nauc_precision_at_100_diff1": -0.35549409656830133,
+        "nauc_precision_at_100_max": 0.6054432740082977,
+        "nauc_precision_at_100_std": -0.13073611213585531,
+        "nauc_precision_at_10_diff1": -0.3554940965683011,
+        "nauc_precision_at_10_max": 0.6054432740082981,
+        "nauc_precision_at_10_std": -0.1307361121358551,
+        "nauc_precision_at_1_diff1": 0.2524564785684737,
+        "nauc_precision_at_1_max": 0.8566368743831702,
+        "nauc_precision_at_1_std": 0.5635391925059349,
+        "nauc_precision_at_20_diff1": -0.3554940965683011,
+        "nauc_precision_at_20_max": 0.6054432740082981,
+        "nauc_precision_at_20_std": -0.1307361121358551,
+        "nauc_precision_at_3_diff1": -0.3377658698816377,
+        "nauc_precision_at_3_max": 0.6780151277792397,
+        "nauc_precision_at_3_std": 0.12291559606586676,
+        "nauc_precision_at_5_diff1": -0.3554940965683011,
+        "nauc_precision_at_5_max": 0.6054432740082981,
+        "nauc_precision_at_5_std": -0.1307361121358551,
+        "nauc_recall_at_1000_diff1": 0.1091970342988605,
+        "nauc_recall_at_1000_max": 0.18339955163544436,
+        "nauc_recall_at_1000_std": 0.30756376767627086,
+        "nauc_recall_at_100_diff1": 0.1091970342988605,
+        "nauc_recall_at_100_max": 0.18339955163544436,
+        "nauc_recall_at_100_std": 0.30756376767627086,
+        "nauc_recall_at_10_diff1": 0.1091970342988605,
+        "nauc_recall_at_10_max": 0.18339955163544436,
+        "nauc_recall_at_10_std": 0.30756376767627086,
+        "nauc_recall_at_1_diff1": 0.47964980727810325,
+        "nauc_recall_at_1_max": -0.08015044571696166,
+        "nauc_recall_at_1_std": 0.3507257834956417,
+        "nauc_recall_at_20_diff1": 0.1091970342988605,
+        "nauc_recall_at_20_max": 0.18339955163544436,
+        "nauc_recall_at_20_std": 0.30756376767627086,
+        "nauc_recall_at_3_diff1": 0.22013063499116758,
+        "nauc_recall_at_3_max": 0.054749114965246065,
+        "nauc_recall_at_3_std": 0.4258163949018153,
+        "nauc_recall_at_5_diff1": 0.1091970342988605,
+        "nauc_recall_at_5_max": 0.18339955163544436,
+        "nauc_recall_at_5_std": 0.30756376767627086,
+        "ndcg_at_1": 0.84188,
+        "ndcg_at_10": 0.661,
+        "ndcg_at_100": 0.6534,
+        "ndcg_at_1000": 0.6534,
+        "ndcg_at_20": 0.65358,
+        "ndcg_at_3": 0.7826,
+        "ndcg_at_5": 0.74517,
+        "precision_at_1": 0.84188,
+        "precision_at_10": 0.27471,
+        "precision_at_100": 0.02747,
+        "precision_at_1000": 0.00275,
+        "precision_at_20": 0.13736,
+        "precision_at_3": 0.68633,
+        "precision_at_5": 0.54942,
+        "recall_at_1": 0.24264,
+        "recall_at_10": 0.59383,
+        "recall_at_100": 0.59383,
+        "recall_at_1000": 0.59383,
+        "recall_at_20": 0.59383,
+        "recall_at_3": 0.48934,
+        "recall_at_5": 0.59383
+      }
+    ]
+  },
+  "task_name": "T2Retrieval"
+}
+```
+
+</details>
\ No newline at end of file
diff --git a/docs/en/user_guides/backend/rageval_backend/ragas.md b/docs/en/user_guides/backend/rageval_backend/ragas.md
new file mode 100644
index 00000000..8b5d08fc
--- /dev/null
+++ b/docs/en/user_guides/backend/rageval_backend/ragas.md
@@ -0,0 +1,221 @@
+(ragas)=
+# RAGAS
+
+This framework supports [RAGAS](https://github.com/explodinggradients/ragas) (Retrieval Augmented Generation Assessment), a dedicated framework for evaluating the performance of Retrieval Augmented Generation (RAG) processes. Some of its core evaluation metrics include:
+
+- Faithfulness: Measures the factual accuracy of generated answers, ensuring the truthfulness and reliability of the generated content.
+- Answer Relevance: Assesses the relevance of answers to the given questions, verifying whether the response directly addresses the user's query.
+- Context Precision: Evaluates the precision of the context used to generate answers, ensuring that relevant information is selected from the context.
+- Context Relevancy: Measures the relevance of the chosen context to the question, helping to improve context selection to enhance answer accuracy.
+- Context Recall: Evaluates the completeness of retrieved relevant contextual information, ensuring that no key context is omitted.
+
+In addition, RAGAS also provides tools for automatic test data generation for user convenience.
+
+## Environment Setup
+Install dependencies
+
+```shell
+pip install ragas
+```
+
+## Dataset Preparation
+
+Example of evaluation dataset is as follows:
+```json
+{
+    "question": [
+        "When was the first super bowl?",
+        "Who won the most super bowls?"
+    ],
+    "answer": [
+        "The first superbowl was held on Jan 15, 1967",
+        "The most super bowls have been won by The New England Patriots"
+    ],
+    "contexts": [
+        [
+            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,"
+        ],
+        [
+            "The Green Bay Packers...Green Bay, Wisconsin.",
+            "The Packers compete...Football Conference"
+        ]
+    ],
+    "ground_truth": [
+        "The first superbowl was held on January 15, 1967",
+        "The New England Patriots have won the Super Bowl a record six times"
+    ]
+}
+```
+Required fields include:
+- question: Question
+- answer: Answer
+- contexts: List of contexts
+- ground_truth: Standard answer
+### Automatic Data Generation
+
+RAGAS offers the capability to automatically generate test data, allowing users to specify parameters such as test set size, data distribution, generator, and LLM, to automatically produce test datasets. The specific steps are as follows:
+
+```{figure} images/generation_process.png
+
+Ragas adopts a novel approach to generating evaluation data. An ideal evaluation dataset should encompass various types of questions encountered in real-world applications, including questions of different difficulty levels. By default, large language models (LLMs) struggle to create diverse samples since they often follow common paths. Inspired by works like [Evol-Instruct](https://arxiv.org/abs/2304.12244), Ragas achieves this goal by adopting an evolutionary generation paradigm, where questions with different characteristics (such as reasoning, conditions, multiple contexts, etc.) are systematically constructed based on the provided document set. This method ensures comprehensive coverage of the various components' performance in your pipeline, making the evaluation process more robust.
+```
+
+#### Configuration Task
+```python
+generate_testset_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "RAGAS",
+        "testset_generation": {
+            "docs": ["README.md"],
+            "test_size": 5,
+            "output_file": "outputs/testset.json",
+            "distribution": {"simple": 0.5, "multi_context": 0.4, "reasoning": 0.1},
+            "generator_llm": {
+                "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                "template_type": "qwen",
+            },
+            "critic_llm": {
+                "model_name_or_path": "QwenCollection/Ragas-critic-llm-Qwen1.5-GPTQ",
+                "template_type": "qwen",
+            },
+            "embeddings": {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+            },
+        }
+    },
+}
+```
+Configuration file description:
+- `eval_backend`: string, description: The name of the evaluation backend, e.g., "RAGEval".
+- `eval_config`: dictionary, description: Contains detailed information about the evaluation configuration.
+  - `tool`: string, description: The name of the evaluation tool, e.g., "RAGAS".
+  - `testset_generation`: dictionary, description: Configuration for test set generation:
+    - `docs`: list, description: A list of documents needed for test set generation, e.g., ["README.md"].
+    - `test_size`: integer, description: The size of the generated test set, e.g., 5.
+    - `output_file`: string, description: The path for the generated output file, e.g., "outputs/testset.json".
+    - `distribution`: dictionary, description: Configuration for the distribution of test set content.
+      - `simple`: float, description: The distribution proportion of simple content, e.g., 0.5.
+      - `multi_context`: float, description: The distribution proportion of multi-context content, e.g., 0.4.
+      - `reasoning`: float, description: The distribution proportion of reasoning content, e.g., 0.1.
+
+    - `generator_llm`: dictionary, description: Configuration for the generator LLM (Large Language Model):
+        - If using a local model, the following parameters are supported:
+            - `model_name_or_path`: string, description: The name or path of the generator model. Enter a name, e.g., "qwen/Qwen2-7B-Instruct" to automatically download the model from ModelScope; enter a path to load the model locally.
+            - `template_type`: string, description: Template type, e.g., "qwen".
+            - `generation_config`: dictionary, description: Generation configuration, e.g., `{"temperature": 0.7}`.
+        - If using an API model, the following parameters are supported:
+            - `model_name`: string, custom name for the model.
+            - `api_base`: string, custom base URL.
+            - `api_key`: optional, your API key.
+    - `critic_llm`: dictionary, description: Configuration for the evaluator LLM, same as above.
+    - `embeddings`: dictionary, description: Configuration for the embedding model.
+      - `model_name_or_path`: string, description: The name or path of the embedding model, e.g., "AI-ModelScope/m3e-base".
+
+#### Execute Task
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+logger = get_logger()
+
+# Run task
+run_task(task_cfg=generate_testset_task_cfg) 
+```
+
+Using examples generated from the project's README documentation as follows:
+
+<details><summary>Click to view the automatically generated dataset</summary>
+
+``` json
+[
+    {
+        "question": "What is the purpose of specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command?",
+        "contexts": [
+            " to run the custom code? [y\/N]`, please type `y`. #### Basic Parameter Descriptions - `--model`: Specifies the `model_id` of the model on [ModelScope](https:\/\/modelscope.cn\/), allowing automatic download. For example, see the [Qwen2-0.5B-Instruct model link](https:\/\/modelscope.cn\/models\/qwen\/Qwen2-0.5B-Instruct\/summary); you can also use a local path, such as `\/path\/to\/model`. - `--template-type`: Specifies the template type corresponding to the model. Refer to the `Default Template` field in the [template table](https:\/\/swift.readthedocs.io\/en\/latest\/Instruction\/Supported-models-datasets.html#llm) for filling in this field. - `--datasets`: The dataset name, allowing multiple datasets to be specified, separated by spaces; these datasets will be automatically downloaded. Refer to the [supported datasets list](https:\/\/evalscope.readthedocs.io\/en\/latest\/get_started\/supported_dataset.html) for available options. ### 2. Parameterized Evaluation If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands: **Example 1:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --model-args revision=master,precision=torch.float16,device_map=auto \\ --datasets gsm8k ceval \\ --use-cache true \\ --limit 10 ``` **Example 2:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --generation-config do_sample=false,temperature=0.0 \\ --datasets ceval \\ --dataset-args '{\"ceval\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\ --limit 10 ``` #### Parameter Descriptions In addition to the three [basic parameters](#basic-parameter-descriptions), the other parameters are as follows: - `--model-args`: Model loading parameters, separated by commas, in `key=value` format. - `--generation-config`: Generation parameters, separated by commas, in `key=value` format. - `do_sample`: Whether to use sampling, default is `false`. - `max_new_tokens`: Maximum generation length, default is 1024. - `temperature`: Sampling temperature. - `top_p`: Sampling threshold. - `top_k`: Sampling threshold. - `--use-cache`: Whether to use local cache, default is `false`. If set to `true`, previously evaluated model and dataset combinations will not be evaluated again, and will be read directly from the local cache. - `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format, where the key is the dataset name and the value is the parameter; note that these must correspond one-to-one with the values in `--datasets`. - `--few_shot_num`: Number of few-shot examples. - `--few_shot_random`: Whether to randomly sample few-shot data; if not specified, defaults to `true`. - `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation. ### 3. Use the run_task Function to Submit an Evaluation Task Using the `run_task` function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields: #### 1. Configuration Task Dictionary Parameters ```python import torch from evalscope.constants import DEFAULT_ROOT_CACHE_DIR # Example your_task_cfg = { 'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'}, 'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512}, 'dataset_args': {}, 'dry_run': False, 'model': 'qwen\/Qwen2-0.5B-Instruct', 'template_type': 'qwen', 'datasets': ['arc', 'hellasw"
+        ],
+        "ground_truth": "The purpose of specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command is to associate the correct evaluation template with the chosen model. This ensures that the evaluation process uses the appropriate instructions, prompts, and settings tailored for that specific model, enhancing the relevance and effectiveness of the evaluation.",
+        "answer": "Specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command is crucial because it corresponds to the specific structure and requirements of the model being used. This ensures that the evaluation process aligns correctly with the model's capabilities and expected input format, leading to accurate and meaningful results."
+    },
+    {
+        "question": "What automated outputs does the EvalScope framework produce following a model evaluation?",
+        "contexts": [
+            "\ud83d\udcd6 Documents &nbsp | &nbsp \ud83d\udcd6 \u4e2d\u6587\u6587\u6863",
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "The EvalScope framework automatically generates evaluation reports following a model evaluation.",
+        "answer": "The EvalScope framework produces automated evaluation reports following a model evaluation. These reports include results from automatic evaluations of objective questions and complex task evaluations using expert models."
+    },
+    {
+        "question": "How does EvalScope framework create detailed reports for big models using benchmark datasets, metrics, and backends like OpenCompass or VLMEvalKit?",
+        "contexts": [
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "EvalScope framework creates detailed reports for big models by utilizing benchmark datasets, implementing evaluation metrics, and integrating with backends such as OpenCompass or VLMEvalKit. It supports automated evaluation of objective questions and complex tasks, offers a unified model access mechanism compatible with various model families, and includes features like Arena Mode for model comparisons. Additionally, it generates evaluation reports and provides visualization tools for intuitive result displays. Seamless integration with frameworks like ms-swift enhances the development process, offering a comprehensive solution for model training, deployment, evaluation, and reporting.",
+        "answer": "EvalScope framework creates detailed reports for big models by utilizing benchmark datasets, metrics, and backends such as OpenCompass or VLMEvalKit. It automates the evaluation process, generating comprehensive reports that reflect the model's performance across various tasks and metrics. This is achieved through its integrated backend support, which simplifies task submission and enables advanced encapsulation, making it easier to assess model capabilities in both single and comparative evaluations."
+    },
+    {
+        "question": "How can one tailor model assessments in `evalscope`, and which parameters related to model and dataset configurations can be modified for customization, given the tool's capabilities for model-specific loading, generation, and evaluation dataset setup?",
+        "contexts": [
+            " to run the custom code? [y\/N]`, please type `y`. #### Basic Parameter Descriptions - `--model`: Specifies the `model_id` of the model on [ModelScope](https:\/\/modelscope.cn\/), allowing automatic download. For example, see the [Qwen2-0.5B-Instruct model link](https:\/\/modelscope.cn\/models\/qwen\/Qwen2-0.5B-Instruct\/summary); you can also use a local path, such as `\/path\/to\/model`. - `--template-type`: Specifies the template type corresponding to the model. Refer to the `Default Template` field in the [template table](https:\/\/swift.readthedocs.io\/en\/latest\/Instruction\/Supported-models-datasets.html#llm) for filling in this field. - `--datasets`: The dataset name, allowing multiple datasets to be specified, separated by spaces; these datasets will be automatically downloaded. Refer to the [supported datasets list](https:\/\/evalscope.readthedocs.io\/en\/latest\/get_started\/supported_dataset.html) for available options. ### 2. Parameterized Evaluation If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands: **Example 1:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --model-args revision=master,precision=torch.float16,device_map=auto \\ --datasets gsm8k ceval \\ --use-cache true \\ --limit 10 ``` **Example 2:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --generation-config do_sample=false,temperature=0.0 \\ --datasets ceval \\ --dataset-args '{\"ceval\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\ --limit 10 ``` #### Parameter Descriptions In addition to the three [basic parameters](#basic-parameter-descriptions), the other parameters are as follows: - `--model-args`: Model loading parameters, separated by commas, in `key=value` format. - `--generation-config`: Generation parameters, separated by commas, in `key=value` format. - `do_sample`: Whether to use sampling, default is `false`. - `max_new_tokens`: Maximum generation length, default is 1024. - `temperature`: Sampling temperature. - `top_p`: Sampling threshold. - `top_k`: Sampling threshold. - `--use-cache`: Whether to use local cache, default is `false`. If set to `true`, previously evaluated model and dataset combinations will not be evaluated again, and will be read directly from the local cache. - `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format, where the key is the dataset name and the value is the parameter; note that these must correspond one-to-one with the values in `--datasets`. - `--few_shot_num`: Number of few-shot examples. - `--few_shot_random`: Whether to randomly sample few-shot data; if not specified, defaults to `true`. - `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation. ### 3. Use the run_task Function to Submit an Evaluation Task Using the `run_task` function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields: #### 1. Configuration Task Dictionary Parameters ```python import torch from evalscope.constants import DEFAULT_ROOT_CACHE_DIR # Example your_task_cfg = { 'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'}, 'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512}, 'dataset_args': {}, 'dry_run': False, 'model': 'qwen\/Qwen2-0.5B-Instruct', 'template_type': 'qwen', 'datasets': ['arc', 'hellasw",
+            "ag'], 'work_dir': DEFAULT_ROOT_CACHE_DIR, 'outputs': DEFAULT_ROOT_CACHE_DIR, 'mem_cache': False, 'dataset_hub': 'ModelScope', 'dataset_dir': DEFAULT_ROOT_CACHE_DIR, 'limit': 10, 'debug': False } ``` Here, `DEFAULT_ROOT_CACHE_DIR` is set to `'~\/.cache\/evalscope'`. #### 2. Execute Task with run_task ```python from evalscope.run import run_task run_task(task_cfg=your_task_cfg) ``` ## Evaluation Backend EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes: - **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode. - [OpenCompass](https:\/\/github.com\/open-compass\/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/opencompass_backend.html) - [VLMEvalKit](https:\/\/github.com\/open-compass\/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/vlmevalkit_backend.html) - **ThirdParty**: The third-party task, e.g. [ToolBench](https:\/\/evalscope.readthedocs.io\/en\/latest\/third_party\/toolbench.html), you can contribute your own evaluation task to EvalScope as third-party backend. ## Custom Dataset Evaluation EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [\ud83d\udcd6User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html) ## Offline Evaluation You can use local dataset to evaluate the model without internet connection. Refer to: Offline Evaluation [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/offline_evaluation.html) ## Arena Mode The Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report. Refer to: Arena Mode [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html) ## Model Serving Performance Evaluation A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats. Refer to: Model Serving Performance Evaluation [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html) ## Leaderboard The LLM Leaderboard aims to provide an objective and comprehensive evaluation standard and platform to help researchers and developers understand and compare the performance of models on various tasks on ModelScope. Refer to: [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free) ## TO-DO List - [x] Agents evaluation - [x] vLLM - [ ] Distributed evaluating - [x] Multi-modal evaluation - [ ] Benchmarks - [ ] GAIA - [ ] GPQA - [x] MBPP - [ ] Auto-reviewer - [ ] Qwen-max"
+        ],
+        "ground_truth": "To tailor model assessments in `evalscope`, one can modify several parameters related to model and dataset configurations. For model-specific loading, `model-args` allows setting parameters like `revision`, `precision`, and `device_map`. For generation, `generation-config` lets users control aspects such as `do_sample`, `temperature`, `max_new_tokens`, and `top_p`. To customize evaluation datasets, `dataset-args` provides configuration options for each specified dataset. Additionally, parameters like `few_shot_num`, `few_shot_random`, and `limit` enable adjustments for few-shot learning and limiting the number of evaluation samples per dataset.",
+        "answer": "To tailor model assessments in `evalscope`, you can modify parameters related to model and dataset configurations. Key parameters include:\n\n- `--model`: Specifies the model ID or local path.\n- `--template-type`: Corresponds to the model's template type.\n- `--datasets`: Allows specifying multiple datasets to evaluate, which are automatically downloaded.\n- `--model-args`: Model loading parameters, like `revision`, `precision`, and `device_map`.\n- `--generation-config`: Generation parameters, including `do_sample`, `temperature`, and `max_new_tokens`.\n- `--use-cache`: Determines if local cache should be used.\n- `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format.\n- `--few_shot_num`: Number of few-shot examples.\n- `--limit`: Maximum number of evaluation samples per dataset.\n\nThese parameters enable customization for model-specific loading, generation, and evaluation dataset setup."
+    },
+    {
+        "question": "How does the OpenCompass Integration streamline task submission for evaluation compared to EvalScope and VLMEvalKit?",
+        "contexts": [
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "The OpenCompass Integration streamlines task submission for evaluation by providing advanced encapsulation and task simplification. This allows for easier task submission for evaluation, making the process more efficient and user-friendly.",
+        "answer": "The OpenCompass Integration in EvalScope streamlines task submission for evaluation by offering advanced encapsulation and simplifying tasks, making it easier to submit tasks for assessment compared to EvalScope's direct use of the Evaluation Backend or VLMEvalKit."
+    }
+]
+```
+</details>
+
+## Configuration for Evaluation Task
+```python
+eval_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "RAGAS",
+        "eval": {
+            "testset_file": "outputs/testset.json",
+            "critic_llm": {
+                "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                "template_type": "qwen",
+            },
+            "embeddings": {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+            },
+            "metrics": [
+                "faithfulness",
+                "answer_relevancy",
+                "context_precision",
+                "answer_correctness",
+            ],
+        },
+    },
+}
+```
+The basic parameters are consistent with those of the automatic evaluation task, with the following differences:
+- `testset_file`: Specifies the path to the evaluation dataset file, defaulting to `outputs/testset.json`.
+- `metrics`: Specifies the evaluation metrics, refer to the [metrics list](https://docs.ragas.io/en/latest/concepts/metrics/index.html).
+
+## Execute Evaluation
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+logger = get_logger()
+
+# Run task
+run_task(task_cfg=eval_task_cfg)
+```
+
+The output evaluation results are as follows:
+
+```{figure} images/eval_result.png
+
+Evaluation Results
+```
\ No newline at end of file
diff --git a/docs/en/user_guides/vlmevalkit_backend.md b/docs/en/user_guides/backend/vlmevalkit_backend.md
similarity index 99%
rename from docs/en/user_guides/vlmevalkit_backend.md
rename to docs/en/user_guides/backend/vlmevalkit_backend.md
index 4da6d937..6690786d 100644
--- a/docs/en/user_guides/vlmevalkit_backend.md
+++ b/docs/en/user_guides/backend/vlmevalkit_backend.md
@@ -1,4 +1,5 @@
-# VLMEvalKit Backend
+(vlmeval)=
+# VLMEvalKit
 
 To facilitate the use of the VLMEvalKit evaluation backend, we have customized the VLMEvalKit source code, naming it `ms-vlmeval`. This version encapsulates the configuration and execution of evaluation tasks and supports installation via PyPI, allowing users to initiate lightweight VLMEvalKit evaluation tasks through EvalScope. Additionally, we support interface evaluation tasks based on the OpenAI API format, and you can deploy multi-modal model services using [ms-swift](https://github.com/modelscope/swift) , [vLLM](https://github.com/vllm-project/vllm) , [LMDeploy](https://github.com/InternLM/lmdeploy) , [Ollama](https://ollama.ai/), etc.
 
diff --git a/docs/zh/_static/images/evalscope_framework.png b/docs/zh/_static/images/evalscope_framework.png
index 5f0541a7..2ba90b5a 100644
Binary files a/docs/zh/_static/images/evalscope_framework.png and b/docs/zh/_static/images/evalscope_framework.png differ
diff --git a/docs/zh/advanced_guides/custom_dataset.md b/docs/zh/advanced_guides/custom_dataset.md
index 4ecb1143..14446ce4 100644
--- a/docs/zh/advanced_guides/custom_dataset.md
+++ b/docs/zh/advanced_guides/custom_dataset.md
@@ -162,7 +162,7 @@ run_task(task_cfg=task_cfg)
 ```shell
 pip install evalscope[vlmeval]
 ```
-参考：[使用VLMEvalKit评测后端](../user_guides/vlmevalkit_backend.md)
+参考：[使用VLMEvalKit评测后端](../user_guides/backend/vlmevalkit_backend.md)
 ````
 
 ### 选择题格式（MCQ）
diff --git a/docs/zh/advanced_guides/custom_model.md b/docs/zh/advanced_guides/custom_model.md
index 547588a1..41d29a72 100644
--- a/docs/zh/advanced_guides/custom_model.md
+++ b/docs/zh/advanced_guides/custom_model.md
@@ -12,7 +12,7 @@
 
 ### 方式2. 使用 OpenCompass 评估
 ```{seealso}
-参考 [OpenCompass使用指南](../user_guides/opencompass_backend.md)
+参考 [OpenCompass使用指南](../user_guides/backend/opencompass_backend.md)
 ```
 
 -----
@@ -24,5 +24,5 @@
 ### 使用 VLMEvalKit 评估后端
 
 ```{seealso}
-参考 [VLMEvalKit使用指南](../user_guides/vlmevalkit_backend.md)
+参考 [VLMEvalKit使用指南](../user_guides/backend/vlmevalkit_backend.md)
 ```
diff --git a/docs/zh/get_started/introduction.md b/docs/zh/get_started/introduction.md
index fe657a14..781eac57 100644
--- a/docs/zh/get_started/introduction.md
+++ b/docs/zh/get_started/introduction.md
@@ -1,6 +1,6 @@
 # 简介
 
-大型语言模型评估（LLMs Evaluation）是评价和改进大型模型的关键流程。为了提升这一评估过程的效率和效果，我们设计了**EvalScope**框架。
+EvalScope是[魔搭社区](https://modelscope.cn/)官方推出的模型评估与性能基准测试框架，内置多个常用测试基准和评估指标，如MMLU、CMMLU、C-Eval、GSM8K、ARC、HellaSwag、TruthfulQA、MATH和HumanEval等；支持多种类型的模型评测，包括LLM、多模态LLM、embedding模型和reranker模型。EvalScope还适用于多种评测场景，如端到端RAG评测、竞技场模式和模型推理性能压测等。此外，通过ms-swift训练框架的无缝集成，可一键发起评测，实现了模型训练到评测的全链路支持。
 
 ## 整体架构
 ![EvalScope 架构图](../_static/images/evalscope_framework.png)
@@ -17,7 +17,7 @@
     - **OpenCompass**：支持[OpenCompass](https://github.com/open-compass/opencompass)作为评测后端，对其进行了高级封装和任务简化，您可以更轻松地提交任务进行评估。
     - **VLMEvalKit**：支持[VLMEvalKit](https://github.com/open-compass/VLMEvalKit)作为评测后端，轻松发起多模态评测任务，支持多种多模态模型和数据集。
     - **ThirdParty**：其他第三方评估任务，如[ToolBench](../third_party/toolbench.md)。
-
+    - **RAGEval**：支持RAG评估，支持使用[MTEB/CMTEB](../user_guides/backend/rageval_backend/mteb.md)进行embedding模型和reranker的独立评测，以及使用[RAGAS](../user_guides/backend/rageval_backend/ragas.md)进行端到端评测。
 4. **Performance Evaluator**: 模型性能评测，负责具体衡量模型推理服务性能，包括性能评测、压力测试、性能评测报告生成、可视化。
 
 5. **Evaluation Report**: 最终生成的评估报告，总结模型的性能表现，报告可以用于决策和进一步的模型优化。
diff --git a/docs/zh/get_started/supported_dataset.md b/docs/zh/get_started/supported_dataset.md
index afaa55bc..cf85e14e 100644
--- a/docs/zh/get_started/supported_dataset.md
+++ b/docs/zh/get_started/supported_dataset.md
@@ -3,7 +3,7 @@
 ## 1. 原生支持的数据集
 
 ```{tip}
-目前框架原生支持如下数据集，若您需要的数据集不在列表中，请提交issue；或者使用[OpenCompass backend](../user_guides/opencompass_backend.md)进行语言模型评估；或使用[VLMEvalKit backend](../user_guides/vlmevalkit_backend.md)进行多模态模型评估。
+目前框架原生支持如下数据集，若您需要的数据集不在列表中，请提交issue；或者使用[OpenCompass backend](../user_guides/backend/opencompass_backend.md)进行语言模型评估；或使用[VLMEvalKit backend](../user_guides/backend/vlmevalkit_backend.md)进行多模态模型评估。
 ```
 
 | 名称        | 链接                                                                                   | 状态 | 备注 |
diff --git a/docs/zh/index.md b/docs/zh/index.md
new file mode 100644
index 00000000..a3bcc08f
--- /dev/null
+++ b/docs/zh/index.md
@@ -0,0 +1,59 @@
+![EvalScope Logo](_static/images/evalscope_logo.png)
+
+<br>
+
+# 欢迎来到 EvalScope 中文教程！
+
+## 上手路线
+
+为了用户能够快速上手，我们推荐以下流程：
+
+- 对于想要使用 EvalScope 的用户，我们推荐先阅读 [快速开始](get_started/introduction.md) 部分来设置环境，并启动一个迷你实验熟悉流程。
+- 对于一些基础使用，我们建议用户阅读 [教程](user_guides/offline_evaluation.md)，包括 [如何使用 EvalScope 进行离线评测](user_guides/offline_evaluation.md)、[如何使用竞技场模式进行评测](user_guides/arena.md)、[如何使用其他评测后端](user_guides/backend/index.md)、[如何使用模型服务压测工具](user_guides/stress_test.md)。
+- 若您想进行更多模块的自定义，例如增加数据集和模型，我们提供了 [进阶教程](advanced_guides/custom_dataset.md)。
+- 此外，我们提供了 [第三方工具](third_party/toolbench.md) 来帮助用户快速评测模型，例如使用 ToolBench 进行评测。
+- 最后，我们提供了 [最佳实践](best_practice/swift_integration.md) 来帮助用户进行评测，例如如何使用 Swift 进行评测。
+
+我们始终非常欢迎用户的 PRs 和 Issues 来完善 EvalScope。
+
+
+:::{toctree}
+:hidden:
+:caption: 🚀 快速开始
+
+get_started/introduction.md
+get_started/installation.md
+get_started/basic_usage.md
+get_started/supported_dataset.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 🔧 教程
+user_guides/offline_evaluation.md
+user_guides/arena.md
+user_guides/stress_test.md
+user_guides/backend/index.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 🛠️ 进阶教程
+advanced_guides/custom_dataset.md
+advanced_guides/custom_model.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 🧰 第三方工具
+third_party/toolbench.md
+third_party/longwriter.md
+:::
+
+:::{toctree}
+:hidden:
+:caption: 📖 最佳实践
+best_practice/swift_integration.md
+best_practice/llm_full_stack.md
+best_practice/experiments.md
+:::
diff --git a/docs/zh/index.rst b/docs/zh/index.rst
deleted file mode 100644
index 0ce429e2..00000000
--- a/docs/zh/index.rst
+++ /dev/null
@@ -1,73 +0,0 @@
-.. image:: _static/images/evalscope_logo.png
-
-.. raw:: html
-
-    <br>
-
-欢迎来到 EvalScope 中文教程！
-==========================================
-
-上手路线
--------------------------------
-
-为了用户能够快速上手，我们推荐以下流程：
-
-- 对于想要使用 EvalScope 的用户，我们推荐先阅读 快速开始_ 部分来设置环境，并启动一个迷你实验熟悉流程。
-
-- 对于一些基础使用，我们建议用户阅读 教程_ ，包括 如何使用 EvalScope 进行离线评测、如何使用 竞技场模式 进行评测、如何使用其他评测后端、如何使用模型服务压测工具。
-
-- 若您想进行更多模块的自定义，例如增加数据集和模型，我们提供了 进阶教程_ 。
-
-- 此外，我们提供了 第三方工具_ 来帮助用户快速评测模型，例如使用 ToolBench 进行评测。
-
-- 最后，我们提供了 最佳实践_ 来帮助用户进行评测，例如如何使用 Swift 进行评测。
-
-我们始终非常欢迎用户的 PRs 和 Issues 来完善 EvalScope
-
-目录
--------------
-.. _快速开始:
-.. toctree::
-   :maxdepth: 1
-   :caption: 快速开始
-   
-   get_started/introduction.md
-   get_started/installation.md
-   get_started/basic_usage.md
-   get_started/supported_dataset.md
-
-.. _教程:
-.. toctree::
-   :maxdepth: 1
-   :caption: 教程
-   
-   user_guides/offline_evaluation.md
-   user_guides/arena.md
-   user_guides/opencompass_backend.md
-   user_guides/vlmevalkit_backend.md
-   user_guides/stress_test.md
-
-.. _进阶教程:
-.. toctree::
-   :maxdepth: 1
-   :caption: 进阶教程
-   
-   advanced_guides/custom_dataset.md
-   advanced_guides/custom_model.md
-
-.. _第三方工具:
-.. toctree::
-   :maxdepth: 1
-   :caption: 第三方工具
-   
-   third_party/toolbench.md
-   third_party/longwriter.md
-
-.. _最佳实践:
-.. toctree::
-   :maxdepth: 1
-   :caption: 最佳实践
-   
-   best_practice/swift_integration.md
-   best_practice/llm_full_stack.md
-   best_practice/experiments.md
diff --git a/docs/zh/user_guides/backend/index.md b/docs/zh/user_guides/backend/index.md
new file mode 100644
index 00000000..3cf741d8
--- /dev/null
+++ b/docs/zh/user_guides/backend/index.md
@@ -0,0 +1,39 @@
+# 评测后端
+
+:::{toctree}
+:hidden:
+
+opencompass_backend.md
+vlmevalkit_backend.md
+rageval_backend/index.md
+:::
+
+EvalScope支持多个评测后端，以集成多个工具实现不同的评测任务，具体介绍如下：
+
+::::{grid} 2
+
+:::{grid-item-card} OpenCompass
+:link: opencompass
+:link-type: ref
+
++++
+通过EvalScope发起轻量化的OpenCompass评估任务，支持多种LLM评测任务。
+:::
+
+:::{grid-item-card} VLMEvalKit
+:link: vlmeval
+:link-type: ref
+
++++
+通过EvalScope发起轻量化的VLMEvalKit评估任务，支持多种VLM评测任务。
+:::
+
+:::{grid-item-card} RAGEval
+:link: rageval
+:link-type: ref
+
++++
+支持RAG独立评估和端到端评估：支持使用MTEB/CMTEB进行检索模块的独立评测，以及使用RAGAS进行端到端评测。
+:::
+
+::::
diff --git a/docs/zh/user_guides/opencompass_backend.md b/docs/zh/user_guides/backend/opencompass_backend.md
similarity index 99%
rename from docs/zh/user_guides/opencompass_backend.md
rename to docs/zh/user_guides/backend/opencompass_backend.md
index dfc68aeb..a80e10eb 100644
--- a/docs/zh/user_guides/opencompass_backend.md
+++ b/docs/zh/user_guides/backend/opencompass_backend.md
@@ -1,4 +1,6 @@
-# OpenCompass 评测后端
+(opencompass)=
+
+# OpenCompass
 
 为便于使用OpenCompass 评测后端，我们基于OpenCompass源码做了定制，命名为`ms-opencompass`，该版本在原版基础上对评估任务的配置和执行做了一些优化，并支持pypi安装方式，使得用户可以通过EvalScope发起轻量化的OpenCompass评估任务。同时，我们先期开放了基于OpenAI API格式的接口评估任务，您可以使用[ms-swift](https://github.com/modelscope/swift)、[vLLM](https://github.com/vllm-project/vllm)、[LMDeploy](https://github.com/InternLM/lmdeploy)、[Ollama](https://ollama.ai/)等模型服务，来拉起模型推理服务。
 
@@ -14,7 +16,7 @@ pip install evalscope[opencompass] -U
 ````{note}
 有以下两种方式下载数据集
 
-数据集的详细信息可以参考[OpenCompass数据集列表](../get_started/supported_dataset.md#2-opencompass支持的数据集)
+数据集的详细信息可以参考[OpenCompass数据集列表](../../get_started/supported_dataset.md#2-opencompass支持的数据集)
 
 您可以使用以下方式，来查看数据集的名称列表：
 ```python
diff --git a/docs/zh/user_guides/backend/rageval_backend/images/eval_result.png b/docs/zh/user_guides/backend/rageval_backend/images/eval_result.png
new file mode 100644
index 00000000..aea097c4
Binary files /dev/null and b/docs/zh/user_guides/backend/rageval_backend/images/eval_result.png differ
diff --git a/docs/zh/user_guides/backend/rageval_backend/images/generation_process.png b/docs/zh/user_guides/backend/rageval_backend/images/generation_process.png
new file mode 100644
index 00000000..1120d927
Binary files /dev/null and b/docs/zh/user_guides/backend/rageval_backend/images/generation_process.png differ
diff --git a/docs/zh/user_guides/backend/rageval_backend/index.md b/docs/zh/user_guides/backend/rageval_backend/index.md
new file mode 100644
index 00000000..7c22920b
--- /dev/null
+++ b/docs/zh/user_guides/backend/rageval_backend/index.md
@@ -0,0 +1,33 @@
+(rageval)=
+
+# RAGEval
+:::{toctree}
+:hidden:
+:maxdepth: 1
+
+mteb.md
+ragas.md
+:::
+
+经过RAG评测相关的[调研](../../../blog/RAG/RAG_Evaluation.md)，本项目现支持两种评估RAG的方法：独立评估(Independent Evaluation)和端到端评估(End-to-End Evaluation)：
+
+- 独立评估方法：单独评估检索模块(Retrieval Module)和生成模块(Generation Module)，其中检索模块评估指标包括指标包括 **命中率(Hit Rate)、平均排名倒数(Mean Reciprocal Rank, MRR)、归一化折扣累积增益(Normalized Discounted Cumulative Gain, NDCG)、准确率(Precision)** 等，这些指标用于测量系统在根据查询或任务排名项目方面的有效性。而生成模块评估通常在端到端评估中进行。评估指标主要关注上下文相关性，测量检索到的文档与查询问题的相关性​​。
+
+- 端到端评估方法：评估RAG模型对给定输入生成的最终响应，包括模型生成答案与输入查询的相关性和对齐程度。从内容生成目标视角来评估可以将评估划分为未标记内容和已标记内容。未标记内容评估指标包括 **答案忠实度(Answer Fidelity)、答案相关性(Answer Relevance)、无害性(Harmlessness)** 等，而已标记内容评估指标包括准确率(Accuracy)和精确匹配(Exact Match, EM)。
+
+本框架支持使用[MTEB/CMTEB](mteb.md)进行检索模块的独立评测，以及使用[RAGAS](ragas.md)进行端到端评测。
+
+::::{grid} 2
+:::{grid-item-card}  MTEB
+:link: mteb
+:link-type: ref
+
+进行检索模块的独立评测，支持embedding模型和reranker模型。
+:::
+:::{grid-item-card}  RAGAS
+:link: ragas
+:link-type: ref
+
+进行端到端评测，支持自动生成评测集。
+:::
+::::
\ No newline at end of file
diff --git a/docs/zh/user_guides/backend/rageval_backend/mteb.md b/docs/zh/user_guides/backend/rageval_backend/mteb.md
new file mode 100644
index 00000000..1b8ee21d
--- /dev/null
+++ b/docs/zh/user_guides/backend/rageval_backend/mteb.md
@@ -0,0 +1,605 @@
+(mteb)=
+
+# MTEB
+
+本框架支持 [MTEB](https://github.com/embeddings-benchmark/mteb) 和 [CMTEB](https://github.com/embeddings-benchmark/mteb)，具体介绍如下：
+
+- MTEB（Massive Text Embedding Benchmark）是一个大规模的基准测试，旨在衡量文本嵌入模型在多样化嵌入任务上的性能。MTEB 包括56个数据集，涵盖8个任务，并且支持超过112种不同的语言。这个基准测试的目标是帮助开发者找到适用于多种任务的最佳文本嵌入模型。
+
+- C-MTEB（Chinese Massive Text Embedding Benchmark）是一个专门针对中文文本向量的评测基准，它基于MTEB构建，旨在评估中文文本向量模型的性能。C-MTEB收集了35个公共数据集，并分为6类评估任务，包括检索（retrieval）、重排序（re-ranking）、语义文本相似度（STS）、分类（classification）、对分类（pair classification）和聚类（clustering）。
+
+## 支持的数据集
+
+以下是C-MTEB中可用任务和数据集的概述：
+
+| 名称 | Hub链接 | 描述 | 类型 | 类别 | 测试样本数量 |
+|-----|-----|---------------------------|-----|-----|-----|
+| [T2Retrieval](https://arxiv.org/abs/2304.03679) | [C-MTEB/T2Retrieval](https://modelscope.cn/datasets/C-MTEB/T2Retrieval) | T2Ranking：一个大规模的中文段落排序基准 | 检索 | s2p | 24,832 |
+| [MMarcoRetrieval](https://github.com/unicamp-dl/mMARCO) | [C-MTEB/MMarcoRetrieval](https://modelscope.cn/datasets/C-MTEB/MMarcoRetrieval) | mMARCO是MS MARCO段落排序数据集的多语言版本 | 检索 | s2p | 7,437 |
+| [DuRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/DuRetrieval](https://modelscope.cn/datasets/C-MTEB/DuRetrieval) | 一个大规模的中文网页搜索引擎段落检索基准 | 检索 | s2p | 4,000 |
+| [CovidRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/CovidRetrieval](https://modelscope.cn/datasets/C-MTEB/CovidRetrieval) | COVID-19新闻文章 | 检索 | s2p | 949 |
+| [CmedqaRetrieval](https://aclanthology.org/2022.emnlp-main.357.pdf) | [C-MTEB/CmedqaRetrieval](https://modelscope.cn/datasets/C-MTEB/CmedqaRetrieval) | 在线医疗咨询文本 | 检索 | s2p | 3,999 |
+| [EcomRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/EcomRetrieval](https://modelscope.cn/datasets/C-MTEB/EcomRetrieval) | 从阿里巴巴电商领域搜索引擎系统收集的段落检索数据集 | 检索 | s2p | 1,000 |
+| [MedicalRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/MedicalRetrieval](https://modelscope.cn/datasets/C-MTEB/MedicalRetrieval) | 从阿里巴巴医疗领域搜索引擎系统收集的段落检索数据集 | 检索 | s2p | 1,000 |
+| [VideoRetrieval](https://arxiv.org/abs/2203.03367) | [C-MTEB/VideoRetrieval](https://modelscope.cn/datasets/C-MTEB/VideoRetrieval) | 从阿里巴巴视频领域搜索引擎系统收集的段落检索数据集 | 检索 | s2p | 1,000 |
+| [T2Reranking](https://arxiv.org/abs/2304.03679) | [C-MTEB/T2Reranking](https://modelscope.cn/datasets/C-MTEB/T2Reranking) | T2Ranking：一个大规模的中文段落排序基准 | 重新排序 | s2p | 24,382 |
+| [MMarcoReranking](https://github.com/unicamp-dl/mMARCO) | [C-MTEB/MMarco-reranking](https://modelscope.cn/datasets/C-MTEB/Mmarco-reranking) | mMARCO是MS MARCO段落排序数据集的多语言版本 | 重新排序 | s2p | 7,437 |
+| [CMedQAv1](https://github.com/zhangsheng93/cMedQA) | [C-MTEB/CMedQAv1-reranking](https://modelscope.cn/datasets/C-MTEB/CMedQAv1-reranking) | 中文社区医疗问答 | 重新排序 | s2p | 2,000 |
+| [CMedQAv2](https://github.com/zhangsheng93/cMedQA2) | [C-MTEB/CMedQAv2-reranking](https://modelscope.cn/datasets/C-MTEB/C-MTEB/CMedQAv2-reranking) | 中文社区医疗问答 | 重新排序 | s2p | 4,000 |
+| [Ocnli](https://arxiv.org/abs/2010.05444) | [C-MTEB/OCNLI](https://modelscope.cn/datasets/C-MTEB/OCNLI) | 原始中文自然语言推理数据集 | 配对分类 | s2s | 3,000 |
+| [Cmnli](https://modelscope.cn/datasets/clue/viewer/cmnli) | [C-MTEB/CMNLI](https://modelscope.cn/datasets/C-MTEB/CMNLI) | 中文多类别自然语言推理 | 配对分类 | s2s | 139,000 |
+| [CLSClusteringS2S](https://arxiv.org/abs/2209.05034) | [C-MTEB/CLSClusteringS2S](https://modelscope.cn/datasets/C-MTEB/C-MTEB/CLSClusteringS2S) | 从CLS数据集中聚类标题。基于主要类别的13个集合的聚类。 | 聚类 | s2s | 10,000 |
+| [CLSClusteringP2P](https://arxiv.org/abs/2209.05034) | [C-MTEB/CLSClusteringP2P](https://modelscope.cn/datasets/C-MTEB/CLSClusteringP2P) | 从CLS数据集中聚类标题+摘要。基于主要类别的13个集合的聚类。 | 聚类 | p2p | 10,000 |
+| [ThuNewsClusteringS2S](http://thuctc.thunlp.org/) | [C-MTEB/ThuNewsClusteringS2S](https://modelscope.cn/datasets/C-MTEB/ThuNewsClusteringS2S) | 从THUCNews数据集中聚类标题 | 聚类 | s2s | 10,000 |
+| [ThuNewsClusteringP2P](http://thuctc.thunlp.org/) | [C-MTEB/ThuNewsClusteringP2P](https://modelscope.cn/datasets/C-MTEB/ThuNewsClusteringP2P) | 从THUCNews数据集中聚类标题+摘要 | 聚类 | p2p | 10,000 |
+| [ATEC](https://github.com/IceFlameWorm/NLP_Datasets/tree/master/ATEC) | [C-MTEB/ATEC](https://modelscope.cn/datasets/C-MTEB/ATEC) | ATEC NLP句子对相似性竞赛 | STS | s2s | 20,000 |
+| [BQ](https://huggingface.co/datasets/shibing624/nli_zh) | [C-MTEB/BQ](https://modelscope.cn/datasets/C-MTEB/BQ) | 银行问题语义相似性 | STS | s2s | 10,000 |
+| [LCQMC](https://huggingface.co/datasets/shibing624/nli_zh) | [C-MTEB/LCQMC](https://modelscope.cn/datasets/C-MTEB/LCQMC) | 大规模中文问题匹配语料库 | STS | s2s | 12,500 |
+| [PAWSX](https://arxiv.org/pdf/1908.11828.pdf) | [C-MTEB/PAWSX](https://modelscope.cn/datasets/C-MTEB/PAWSX) | 翻译的PAWS评估对 | STS | s2s | 2,000 |
+| [STSB](https://github.com/pluto-junzeng/CNSD) | [C-MTEB/STSB](https://modelscope.cn/datasets/C-MTEB/STSB) | 将STS-B翻译成中文 | STS | s2s | 1,360 |
+| [AFQMC](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/AFQMC](https://modelscope.cn/datasets/C-MTEB/AFQMC) | 蚂蚁金服问答匹配语料库 | STS | s2s | 3,861 |
+| [QBQTC](https://github.com/CLUEbenchmark/QBQTC) | [C-MTEB/QBQTC](https://modelscope.cn/datasets/C-MTEB/QBQTC) | QQ浏览器查询标题语料库 | STS | s2s | 5,000 |
+| [TNews](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/TNews-classification](https://modelscope.cn/datasets/C-MTEB/TNews-classification) | 新闻短文本分类 | 分类 | s2s | 10,000 |
+| [IFlyTek](https://github.com/CLUEbenchmark/CLUE) | [C-MTEB/IFlyTek-classification](https://modelscope.cn/datasets/C-MTEB/IFlyTek-classification) | 应用描述的长文本分类 | 分类 | s2s | 2,600 |
+| [Waimai](https://github.com/SophonPlus/ChineseNlpCorpus/blob/master/datasets/waimai_10k/intro.ipynb) | [C-MTEB/waimai-classification](https://modelscope.cn/datasets/C-MTEB/waimai-classification) | 外卖平台用户评论的情感分析 | 分类 | s2s | 1,000 |
+| [OnlineShopping](https://github.com/SophonPlus/ChineseNlpCorpus/blob/master/datasets/online_shopping_10_cats/intro.ipynb) | [C-MTEB/OnlineShopping-classification](https://modelscope.cn/datasets/C-MTEB/OnlineShopping-classification) | 在线购物网站用户评论的情感分析 | 分类 | s2s | 1,000 |
+| [MultilingualSentiment](https://github.com/tyqiangz/multilingual-sentiment-datasets) | [C-MTEB/MultilingualSentiment-classification](https://modelscope.cn/datasets/C-MTEB/MultilingualSentiment-classification) | 一组按三类分组的多语言情感数据集--正面、中立、负面 | 分类 | s2s | 3,000 |
+| [JDReview](https://huggingface.co/datasets/kuroneko5943/jd21) | [C-MTEB/JDReview-classification](https://modelscope.cn/datasets/C-MTEB/JDReview-classification) | iPhone的评论 | 分类 | s2s | 533 |
+
+对于检索任务，从整个语料库中抽样100,000个候选项（包括真实值），以降低推理成本。
+
+## 环境准备
+安装依赖包
+```bash
+pip install evalscope[rag] -U
+```
+
+## 配置评测参数
+框架支持两种评测方式：单阶段评测 和 两阶段评测：
+- 单阶段评测：直接使用模型预测，并计算指标。
+- 两阶段评测：使用模型预测，再使用检索模型进行reranking，并计算指标。
+
+### 单阶段评测
+配置文件示例如下：
+```python
+one_stage_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "MTEB",
+        "model": [
+            {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+                "pooling_mode": None,
+                "max_seq_length": 512,
+                "prompt": "",
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 128,
+                },
+            }
+        ],
+        "eval": {
+            "tasks": [
+                "TNews",
+                "CLSClusteringS2S",
+                "T2Reranking",
+                "T2Retrieval",
+                "ATEC",
+            ],
+            "verbosity": 2,
+            "output_folder": "outputs",
+            "overwrite_results": True,
+            "limits": 500,
+        },
+    },
+}
+```
+#### 参数说明
+
+- `eval_backend`：默认值为 `RAGEval`，表示使用 RAGEval 评测后端。
+- `eval_config`：字典，包含以下字段：
+    - `tool`：评测工具，使用 `MTEB`。
+    - `model`： 模型配置列表，**单阶段评测时只能放置一个模型**，包含以下字段：
+        - `model_name_or_path`: `str` 模型名称或路径
+        - `is_cross_encoder`: `bool` 模型是否为交叉编码器，默认为 False。  
+        - `pooling_mode`: `Optional[str]` 池化模式，可选值为：“cls”、“lasttoken”、“max”、“mean”、“mean_sqrt_len_tokens”或“weightedmean”。  
+        - `max_seq_length`: `int` 最大序列长度，默认为 512。  
+        - `prompt`: `str` 用于基于大语言模型的提示。  
+        - `model_kwargs`: `dict` 模型的关键字参数，默认值为 `{"torch_dtype": "auto"}`。  
+        - `config_kwargs`: `Dict[str, Any]` 配置的关键字参数，默认为空字典。  
+        - `encode_kwargs`: `dict` 编码的关键字参数，默认值为：  
+            ```python  
+            {  
+                "show_progress_bar": True,  
+                "batch_size": 32
+            }  
+            ```  
+        - `hub`: `str` 模型来源，可以是 "modelscope" 或 "huggingface"。  
+    - `eval`：字典，包含以下字段：
+        - `tasks`: `List[str]` 任务名称  
+        - `verbosity`: `int` 详细程度，范围为 0-3  
+        - `output_folder`: `str` 输出文件夹，默认为 "outputs"  
+        - `overwrite_results`: `bool` 是否覆盖结果，默认为 True  
+        - `limits`: `Optional[int]` 限制样本数量，默认为 None  
+        - `hub`: `str` 模型来源，可以是 "modelscope" 或 "huggingface"  
+        - `top_k`: `int` 选取前 K 个结果，检索任务使用  
+
+### 两阶段评测
+配置文件示例如下，先进行检索，再进行reranking：
+```python
+two_stage_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "MTEB",
+        "model": [
+            {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+                "is_cross_encoder": False,
+                "max_seq_length": 512,
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 64,
+                },
+            },
+            {
+                "model_name_or_path": "OpenBMB/MiniCPM-Reranker",
+                "is_cross_encoder": True,
+                "max_seq_length": 512,
+                "prompt": "为这个问题生成一个检索用的表示",
+                "model_kwargs": {"torch_dtype": "auto"},
+                "encode_kwargs": {
+                    "batch_size": 32,
+                },
+            },
+        ],
+        "eval": {
+            "tasks": ["T2Retrieval"],
+            "verbosity": 2,
+            "output_folder": "outputs",
+            "overwrite_results": True,
+            "limits": 100,
+        },
+    },
+}
+```
+#### 参数说明
+
+基本参数与单阶段相同，不同的地方在于model字段，这里需要传入两个模型，第一个模型用于检索，第二个模型用于reranking，reranking的模型需要使用cross encoder。
+
+## 模型评测
+
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+logger = get_logger()
+
+one_stage_task_cfg = one_stage_task_cfg
+# or
+# two_stage_task_cfg = two_stage_task_cfg
+
+# Run task
+run_task(task_cfg=one_stage_task_cfg) # or run_task(task_cfg=two_stage_task_cfg)
+```
+
+输出结果如下：
+
+**单阶段评测**
+
+<details><summary>输出：</summary>
+
+```{code-block} json
+:caption: outputs/m3e-base/master/TNews.json
+
+{
+  "dataset_revision": "317f262bf1e6126357bbe89e875451e4b0938fe4",
+  "evaluation_time": 16.50650382041931,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "validation": [
+      {
+        "accuracy": 0.4744,
+        "f1": 0.44562489526640825,
+        "f1_weighted": 0.47540307398330806,
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.4744,
+        "scores_per_experiment": [
+          {
+            "accuracy": 0.48,
+            "f1": 0.4536376605217497,
+            "f1_weighted": 0.47800277926811163
+          },
+          {
+            "accuracy": 0.48,
+            "f1": 0.44713633954639176,
+            "f1_weighted": 0.4826984434763292
+          },
+          {
+            "accuracy": 0.462,
+            "f1": 0.433365706955334,
+            "f1_weighted": 0.4640970055245127
+          },
+          {
+            "accuracy": 0.484,
+            "f1": 0.4586732839614161,
+            "f1_weighted": 0.4857359110392786
+          },
+          {
+            "accuracy": 0.462,
+            "f1": 0.4293797541165097,
+            "f1_weighted": 0.4632657330831137
+          },
+          {
+            "accuracy": 0.474,
+            "f1": 0.44775120246296396,
+            "f1_weighted": 0.4737182842092953
+          },
+          {
+            "accuracy": 0.47,
+            "f1": 0.4431197566080463,
+            "f1_weighted": 0.4714830140231783
+          },
+          {
+            "accuracy": 0.472,
+            "f1": 0.44322381694059326,
+            "f1_weighted": 0.47100005556357255
+          },
+          {
+            "accuracy": 0.484,
+            "f1": 0.45454749692062835,
+            "f1_weighted": 0.4856239367465818
+          },
+          {
+            "accuracy": 0.476,
+            "f1": 0.44541393463044954,
+            "f1_weighted": 0.47840557689910646
+          }
+        ]
+      }
+    ]
+  },
+  "task_name": "TNews"
+}
+```
+</details>
+
+<br>
+
+**两阶段评测**
+
+<details><summary>阶段一</summary>
+
+```{code-block} json
+:caption: outputs/stage1/m3e-base/v1/T2Retrieval.json
+
+{
+  "dataset_revision": "8731a845f1bf500a4f111cf1070785c793d10e64",
+  "evaluation_time": 599.5170171260834,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "dev": [
+      {
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.73143,
+        "map_at_1": 0.22347,
+        "map_at_10": 0.63237,
+        "map_at_100": 0.67533,
+        "map_at_1000": 0.67651,
+        "map_at_20": 0.66282,
+        "map_at_3": 0.43874,
+        "map_at_5": 0.54049,
+        "mrr_at_1": 0.7898912852884447,
+        "mrr_at_10": 0.8402654617870331,
+        "mrr_at_100": 0.8421827758769684,
+        "mrr_at_1000": 0.8422583001072272,
+        "mrr_at_20": 0.8415411456315557,
+        "mrr_at_3": 0.8307469752761716,
+        "mrr_at_5": 0.8368029984218875,
+        "nauc_map_at_1000_diff1": 0.17749400860890877,
+        "nauc_map_at_1000_max": 0.42844516520725967,
+        "nauc_map_at_1000_std": 0.18789871694419072,
+        "nauc_map_at_100_diff1": 0.17747467084779375,
+        "nauc_map_at_100_max": 0.42732291785494575,
+        "nauc_map_at_100_std": 0.18694287087286737,
+        "nauc_map_at_10_diff1": 0.19976199493034202,
+        "nauc_map_at_10_max": 0.3374436217668296,
+        "nauc_map_at_10_std": 0.07951451707732717,
+        "nauc_map_at_1_diff1": 0.41727578149080663,
+        "nauc_map_at_1_max": -0.1402656422184478,
+        "nauc_map_at_1_std": -0.26168722519030313,
+        "nauc_map_at_20_diff1": 0.1811898211371171,
+        "nauc_map_at_20_max": 0.40563441466210043,
+        "nauc_map_at_20_std": 0.15927727170010608,
+        "nauc_map_at_3_diff1": 0.31255422845809033,
+        "nauc_map_at_3_max": 0.007523677231905161,
+        "nauc_map_at_3_std": -0.19578481884353466,
+        "nauc_map_at_5_diff1": 0.26073699217160473,
+        "nauc_map_at_5_max": 0.14665611579604088,
+        "nauc_map_at_5_std": -0.09600383298672226,
+        "nauc_mrr_at_1000_diff1": 0.3819666309367981,
+        "nauc_mrr_at_1000_max": 0.6285393024619401,
+        "nauc_mrr_at_1000_std": 0.3294970299417527,
+        "nauc_mrr_at_100_diff1": 0.3819436006743644,
+        "nauc_mrr_at_100_max": 0.6286346262471935,
+        "nauc_mrr_at_100_std": 0.32963045935037844,
+        "nauc_mrr_at_10_diff1": 0.3819124721154632,
+        "nauc_mrr_at_10_max": 0.6292778905762176,
+        "nauc_mrr_at_10_std": 0.3298187966196067,
+        "nauc_mrr_at_1_diff1": 0.3862589251033909,
+        "nauc_mrr_at_1_max": 0.589976680174432,
+        "nauc_mrr_at_1_std": 0.2780515387897469,
+        "nauc_mrr_at_20_diff1": 0.38198959771391816,
+        "nauc_mrr_at_20_max": 0.6290569436652999,
+        "nauc_mrr_at_20_std": 0.3301570340189363,
+        "nauc_mrr_at_3_diff1": 0.3825046940733129,
+        "nauc_mrr_at_3_max": 0.6282507269128365,
+        "nauc_mrr_at_3_std": 0.3260807934869131,
+        "nauc_mrr_at_5_diff1": 0.3816317396711923,
+        "nauc_mrr_at_5_max": 0.6288655177904692,
+        "nauc_mrr_at_5_std": 0.3298854062538469,
+        "nauc_ndcg_at_1000_diff1": 0.21319598381916555,
+        "nauc_ndcg_at_1000_max": 0.5328295949130256,
+        "nauc_ndcg_at_1000_std": 0.2946773445135694,
+        "nauc_ndcg_at_100_diff1": 0.2089807772703975,
+        "nauc_ndcg_at_100_max": 0.5239397690321543,
+        "nauc_ndcg_at_100_std": 0.29123456982125717,
+        "nauc_ndcg_at_10_diff1": 0.20555333230027603,
+        "nauc_ndcg_at_10_max": 0.44316027023003046,
+        "nauc_ndcg_at_10_std": 0.1921835220940756,
+        "nauc_ndcg_at_1_diff1": 0.3862589251033909,
+        "nauc_ndcg_at_1_max": 0.589976680174432,
+        "nauc_ndcg_at_1_std": 0.2780515387897469,
+        "nauc_ndcg_at_20_diff1": 0.20754208582741446,
+        "nauc_ndcg_at_20_max": 0.4786092392092643,
+        "nauc_ndcg_at_20_std": 0.23536973680564616,
+        "nauc_ndcg_at_3_diff1": 0.1902823773882388,
+        "nauc_ndcg_at_3_max": 0.5400466380622567,
+        "nauc_ndcg_at_3_std": 0.2713874990424778,
+        "nauc_ndcg_at_5_diff1": 0.18279298790691637,
+        "nauc_ndcg_at_5_max": 0.4916119327522918,
+        "nauc_ndcg_at_5_std": 0.2375397192963552,
+        "nauc_precision_at_1000_diff1": -0.20510380600112582,
+        "nauc_precision_at_1000_max": 0.4958820760698651,
+        "nauc_precision_at_1000_std": 0.5402465580496146,
+        "nauc_precision_at_100_diff1": -0.1994322347949809,
+        "nauc_precision_at_100_max": 0.5206762748551254,
+        "nauc_precision_at_100_std": 0.5568154081333078,
+        "nauc_precision_at_10_diff1": -0.16707155441197413,
+        "nauc_precision_at_10_max": 0.5600612846655972,
+        "nauc_precision_at_10_std": 0.49419688804691536,
+        "nauc_precision_at_1_diff1": 0.3862589251033909,
+        "nauc_precision_at_1_max": 0.589976680174432,
+        "nauc_precision_at_1_std": 0.2780515387897469,
+        "nauc_precision_at_20_diff1": -0.18471041949530417,
+        "nauc_precision_at_20_max": 0.5458950955439645,
+        "nauc_precision_at_20_std": 0.5355982267058214,
+        "nauc_precision_at_3_diff1": -0.03826790088047189,
+        "nauc_precision_at_3_max": 0.5833083970750171,
+        "nauc_precision_at_3_std": 0.380196662597275,
+        "nauc_precision_at_5_diff1": -0.11789367842600275,
+        "nauc_precision_at_5_max": 0.5708494593335263,
+        "nauc_precision_at_5_std": 0.42860609671688105,
+        "nauc_recall_at_1000_diff1": 0.1341309660059583,
+        "nauc_recall_at_1000_max": 0.5923755841077135,
+        "nauc_recall_at_1000_std": 0.5980459502693942,
+        "nauc_recall_at_100_diff1": 0.12181394285840096,
+        "nauc_recall_at_100_max": 0.47090136790318127,
+        "nauc_recall_at_100_std": 0.3959369184297595,
+        "nauc_recall_at_10_diff1": 0.17356300971546512,
+        "nauc_recall_at_10_max": 0.25475707245853674,
+        "nauc_recall_at_10_std": 0.041819982320384745,
+        "nauc_recall_at_1_diff1": 0.41727578149080663,
+        "nauc_recall_at_1_max": -0.1402656422184478,
+        "nauc_recall_at_1_std": -0.26168722519030313,
+        "nauc_recall_at_20_diff1": 0.14273713155999543,
+        "nauc_recall_at_20_max": 0.36251116771924663,
+        "nauc_recall_at_20_std": 0.1912123941692314,
+        "nauc_recall_at_3_diff1": 0.2873719855400218,
+        "nauc_recall_at_3_max": -0.041198403561830285,
+        "nauc_recall_at_3_std": -0.21921947922872737,
+        "nauc_recall_at_5_diff1": 0.23680082643694844,
+        "nauc_recall_at_5_max": 0.06580524171324151,
+        "nauc_recall_at_5_std": -0.14104561361502632,
+        "ndcg_at_1": 0.78989,
+        "ndcg_at_10": 0.73143,
+        "ndcg_at_100": 0.78829,
+        "ndcg_at_1000": 0.80026,
+        "ndcg_at_20": 0.75787,
+        "ndcg_at_3": 0.7417,
+        "ndcg_at_5": 0.72641,
+        "precision_at_1": 0.78989,
+        "precision_at_10": 0.37304,
+        "precision_at_100": 0.04828,
+        "precision_at_1000": 0.00511,
+        "precision_at_20": 0.21403,
+        "precision_at_3": 0.65461,
+        "precision_at_5": 0.54942,
+        "recall_at_1": 0.22347,
+        "recall_at_10": 0.73318,
+        "recall_at_100": 0.91093,
+        "recall_at_1000": 0.97197,
+        "recall_at_20": 0.81286,
+        "recall_at_3": 0.46573,
+        "recall_at_5": 0.59383
+      }
+    ]
+  },
+  "task_name": "T2Retrieval"
+}
+
+```
+</details>
+
+<details><summary>阶段二</summary>
+
+```{code-block} json
+:caption: outputs/stage2/jina-reranker-v2-base-multilingual/master/T2Retrieval.json
+
+{
+  "dataset_revision": "8731a845f1bf500a4f111cf1070785c793d10e64",
+  "evaluation_time": 332.15709686279297,
+  "kg_co2_emissions": null,
+  "mteb_version": "1.14.15",
+  "scores": {
+    "dev": [
+      {
+        "hf_subset": "default",
+        "languages": [
+          "cmn-Hans"
+        ],
+        "main_score": 0.661,
+        "map_at_1": 0.24264,
+        "map_at_10": 0.56291,
+        "map_at_100": 0.56291,
+        "map_at_1000": 0.56291,
+        "map_at_20": 0.56291,
+        "map_at_3": 0.4714,
+        "map_at_5": 0.56291,
+        "mrr_at_1": 0.841969139049623,
+        "mrr_at_10": 0.8689147524694633,
+        "mrr_at_100": 0.8689147524694633,
+        "mrr_at_1000": 0.8689147524694633,
+        "mrr_at_20": 0.8689147524694633,
+        "mrr_at_3": 0.8664883979192248,
+        "mrr_at_5": 0.8689147524694633,
+        "nauc_map_at_1000_diff1": 0.12071580301051653,
+        "nauc_map_at_1000_max": 0.2536691069727338,
+        "nauc_map_at_1000_std": 0.343624832364704,
+        "nauc_map_at_100_diff1": 0.12071580301051653,
+        "nauc_map_at_100_max": 0.2536691069727338,
+        "nauc_map_at_100_std": 0.343624832364704,
+        "nauc_map_at_10_diff1": 0.12071580301051653,
+        "nauc_map_at_10_max": 0.2536691069727338,
+        "nauc_map_at_10_std": 0.343624832364704,
+        "nauc_map_at_1_diff1": 0.47964980727810325,
+        "nauc_map_at_1_max": -0.08015044571696166,
+        "nauc_map_at_1_std": 0.3507257834956417,
+        "nauc_map_at_20_diff1": 0.12071580301051653,
+        "nauc_map_at_20_max": 0.2536691069727338,
+        "nauc_map_at_20_std": 0.343624832364704,
+        "nauc_map_at_3_diff1": 0.23481937699306626,
+        "nauc_map_at_3_max": 0.10372745264123306,
+        "nauc_map_at_3_std": 0.45345158923063256,
+        "nauc_map_at_5_diff1": 0.12071580301051653,
+        "nauc_map_at_5_max": 0.2536691069727338,
+        "nauc_map_at_5_std": 0.343624832364704,
+        "nauc_mrr_at_1000_diff1": 0.23393918304502795,
+        "nauc_mrr_at_1000_max": 0.8703379129725659,
+        "nauc_mrr_at_1000_std": 0.5785333616122065,
+        "nauc_mrr_at_100_diff1": 0.23393918304502795,
+        "nauc_mrr_at_100_max": 0.8703379129725659,
+        "nauc_mrr_at_100_std": 0.5785333616122065,
+        "nauc_mrr_at_10_diff1": 0.23393918304502795,
+        "nauc_mrr_at_10_max": 0.8703379129725659,
+        "nauc_mrr_at_10_std": 0.5785333616122065,
+        "nauc_mrr_at_1_diff1": 0.2520016067648708,
+        "nauc_mrr_at_1_max": 0.8560897633767299,
+        "nauc_mrr_at_1_std": 0.5642467684745208,
+        "nauc_mrr_at_20_diff1": 0.23393918304502795,
+        "nauc_mrr_at_20_max": 0.8703379129725659,
+        "nauc_mrr_at_20_std": 0.5785333616122065,
+        "nauc_mrr_at_3_diff1": 0.2343988881957151,
+        "nauc_mrr_at_3_max": 0.8695482778251757,
+        "nauc_mrr_at_3_std": 0.5799167198804328,
+        "nauc_mrr_at_5_diff1": 0.23393918304502795,
+        "nauc_mrr_at_5_max": 0.8703379129725659,
+        "nauc_mrr_at_5_std": 0.5785333616122065,
+        "nauc_ndcg_at_1000_diff1": 0.11252208055013257,
+        "nauc_ndcg_at_1000_max": 0.3417865079349515,
+        "nauc_ndcg_at_1000_std": 0.3623961771041499,
+        "nauc_ndcg_at_100_diff1": 0.11252208055013257,
+        "nauc_ndcg_at_100_max": 0.3417865079349515,
+        "nauc_ndcg_at_100_std": 0.3623961771041499,
+        "nauc_ndcg_at_10_diff1": 0.10015448775533999,
+        "nauc_ndcg_at_10_max": 0.3761759074862075,
+        "nauc_ndcg_at_10_std": 0.35152523471339914,
+        "nauc_ndcg_at_1_diff1": 0.2524564785684737,
+        "nauc_ndcg_at_1_max": 0.8566368743831702,
+        "nauc_ndcg_at_1_std": 0.5635391925059349,
+        "nauc_ndcg_at_20_diff1": 0.11228113618796766,
+        "nauc_ndcg_at_20_max": 0.34274993051851965,
+        "nauc_ndcg_at_20_std": 0.36216437469674284,
+        "nauc_ndcg_at_3_diff1": -0.062134030685870506,
+        "nauc_ndcg_at_3_max": 0.7183183844837573,
+        "nauc_ndcg_at_3_std": 0.3352626268658533,
+        "nauc_ndcg_at_5_diff1": -0.04476981761624879,
+        "nauc_ndcg_at_5_max": 0.6272060974309411,
+        "nauc_ndcg_at_5_std": 0.21341258393783158,
+        "nauc_precision_at_1000_diff1": -0.3554940965683014,
+        "nauc_precision_at_1000_max": 0.605443274008298,
+        "nauc_precision_at_1000_std": -0.13073611213585504,
+        "nauc_precision_at_100_diff1": -0.35549409656830133,
+        "nauc_precision_at_100_max": 0.6054432740082977,
+        "nauc_precision_at_100_std": -0.13073611213585531,
+        "nauc_precision_at_10_diff1": -0.3554940965683011,
+        "nauc_precision_at_10_max": 0.6054432740082981,
+        "nauc_precision_at_10_std": -0.1307361121358551,
+        "nauc_precision_at_1_diff1": 0.2524564785684737,
+        "nauc_precision_at_1_max": 0.8566368743831702,
+        "nauc_precision_at_1_std": 0.5635391925059349,
+        "nauc_precision_at_20_diff1": -0.3554940965683011,
+        "nauc_precision_at_20_max": 0.6054432740082981,
+        "nauc_precision_at_20_std": -0.1307361121358551,
+        "nauc_precision_at_3_diff1": -0.3377658698816377,
+        "nauc_precision_at_3_max": 0.6780151277792397,
+        "nauc_precision_at_3_std": 0.12291559606586676,
+        "nauc_precision_at_5_diff1": -0.3554940965683011,
+        "nauc_precision_at_5_max": 0.6054432740082981,
+        "nauc_precision_at_5_std": -0.1307361121358551,
+        "nauc_recall_at_1000_diff1": 0.1091970342988605,
+        "nauc_recall_at_1000_max": 0.18339955163544436,
+        "nauc_recall_at_1000_std": 0.30756376767627086,
+        "nauc_recall_at_100_diff1": 0.1091970342988605,
+        "nauc_recall_at_100_max": 0.18339955163544436,
+        "nauc_recall_at_100_std": 0.30756376767627086,
+        "nauc_recall_at_10_diff1": 0.1091970342988605,
+        "nauc_recall_at_10_max": 0.18339955163544436,
+        "nauc_recall_at_10_std": 0.30756376767627086,
+        "nauc_recall_at_1_diff1": 0.47964980727810325,
+        "nauc_recall_at_1_max": -0.08015044571696166,
+        "nauc_recall_at_1_std": 0.3507257834956417,
+        "nauc_recall_at_20_diff1": 0.1091970342988605,
+        "nauc_recall_at_20_max": 0.18339955163544436,
+        "nauc_recall_at_20_std": 0.30756376767627086,
+        "nauc_recall_at_3_diff1": 0.22013063499116758,
+        "nauc_recall_at_3_max": 0.054749114965246065,
+        "nauc_recall_at_3_std": 0.4258163949018153,
+        "nauc_recall_at_5_diff1": 0.1091970342988605,
+        "nauc_recall_at_5_max": 0.18339955163544436,
+        "nauc_recall_at_5_std": 0.30756376767627086,
+        "ndcg_at_1": 0.84188,
+        "ndcg_at_10": 0.661,
+        "ndcg_at_100": 0.6534,
+        "ndcg_at_1000": 0.6534,
+        "ndcg_at_20": 0.65358,
+        "ndcg_at_3": 0.7826,
+        "ndcg_at_5": 0.74517,
+        "precision_at_1": 0.84188,
+        "precision_at_10": 0.27471,
+        "precision_at_100": 0.02747,
+        "precision_at_1000": 0.00275,
+        "precision_at_20": 0.13736,
+        "precision_at_3": 0.68633,
+        "precision_at_5": 0.54942,
+        "recall_at_1": 0.24264,
+        "recall_at_10": 0.59383,
+        "recall_at_100": 0.59383,
+        "recall_at_1000": 0.59383,
+        "recall_at_20": 0.59383,
+        "recall_at_3": 0.48934,
+        "recall_at_5": 0.59383
+      }
+    ]
+  },
+  "task_name": "T2Retrieval"
+}
+```
+
+</details>
\ No newline at end of file
diff --git a/docs/zh/user_guides/backend/rageval_backend/ragas.md b/docs/zh/user_guides/backend/rageval_backend/ragas.md
new file mode 100644
index 00000000..4c373871
--- /dev/null
+++ b/docs/zh/user_guides/backend/rageval_backend/ragas.md
@@ -0,0 +1,227 @@
+(ragas)=
+
+# RAGAS
+
+本框架支持[RAGAS](https://github.com/explodinggradients/ragas)（Retrieval Augmented Generation Assessment），一个专门用于评估检索增强生成（RAG）流程性能的框架。其具有的一些核心评估指标包括：
+
+- 忠实性（Faithfulness）：衡量生成答案的事实准确性，确保生成内容的真实性和可靠性。
+- 答案相关性（Answer Relevance）：评估答案与给定问题的相关性，验证响应是否直接解决了用户的查询。
+- 上下文精准度（Context Precision）：评估用于生成答案的上下文的精确度，确保从上下文中选择了相关信息。
+- 上下文相关性（Context Relevancy）：衡量所选上下文与问题的相关性，帮助改进上下文选择以提高答案的准确性。
+- 上下文召回率（Context Recall）：评估检索到的相关上下文信息的完整性，确保没有遗漏关键上下文。
+
+此外，RAGAS还提供了自动测试数据生成的工具，方便用户使用。
+
+## 环境准备
+安装依赖
+```shell
+pip install evalscope[rag] -U
+```
+
+## 数据集准备
+评测数据集示例如下：
+```json
+{
+    "question": [
+        "When was the first super bowl?",
+        "Who won the most super bowls?"
+    ],
+    "answer": [
+        "The first superbowl was held on Jan 15, 1967",
+        "The most super bowls have been won by The New England Patriots"
+    ],
+    "contexts": [
+        [
+            "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,"
+        ],
+        [
+            "The Green Bay Packers...Green Bay, Wisconsin.",
+            "The Packers compete...Football Conference"
+        ]
+    ],
+    "ground_truth": [
+        "The first superbowl was held on January 15, 1967",
+        "The New England Patriots have won the Super Bowl a record six times"
+    ]
+}
+```
+需要的字段包括：
+- question：问题
+- answer：答案
+- contexts：上下文列表
+- ground_truth：标准答案
+
+### 自动生成数据集
+RAGAS提供了自动生成测试数据的功能，用户可以指定测试集大小、数据分布、生成器和 LLM 等参数，自动生成测试数据集。具体步骤如下：
+
+```{figure} images/generation_process.png
+
+Ragas采用了一种新颖的评估数据生成方法。理想的评估数据集应该涵盖在实际应用中遇到的各种类型的问题，包括不同难度等级的问题。默认情况下，大语言模型（LLMs）不擅长创建多样化的样本，因为它们往往遵循常见的路径。受到[Evol-Instruct](https://arxiv.org/abs/2304.12244)等工作的启发，Ragas通过采用进化生成范式来实现这一目标，在这一过程中，具有不同特征的问题（如推理、条件、多个上下文等）会根据提供的文档集被系统地构建。这种方法确保了对您管道中各个组件性能的全面覆盖，从而使评估过程更加稳健。
+```
+
+#### 配置任务
+```python
+generate_testset_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "RAGAS",
+        "testset_generation": {
+            "docs": ["README.md"],
+            "test_size": 5,
+            "output_file": "outputs/testset.json",
+            "distribution": {"simple": 0.5, "multi_context": 0.4, "reasoning": 0.1},
+            "generator_llm": {
+                "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                "template_type": "qwen",
+            },
+            "critic_llm": {
+                "model_name_or_path": "QwenCollection/Ragas-critic-llm-Qwen1.5-GPTQ",
+                "template_type": "qwen",
+            },
+            "embeddings": {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+            },
+        }
+    },
+}
+
+```
+配置文件说明：
+- `eval_backend`: 字符串, 描述: 评估后端的名称，例如 "RAGEval"。
+- `eval_config`: 字典, 描述: 包含评估配置的详细信息。
+  - `tool`: 字符串, 描述: 评估工具的名称，例如 "RAGAS"。
+  - `testset_generation`: 字典, 描述: 测试集生成的配置。
+    - `docs`: 列表, 描述: 测试集生成所需的文档列表，例如 ["README.md"]。
+    - `test_size`: 整数, 描述: 生成测试集的大小，例如 5。
+    - `output_file`: 字符串, 描述: 生成的输出文件路径，例如 "outputs/testset.json"。
+    - `distribution`: 字典, 描述: 测试集内容的分布配置。
+      - `simple`: 浮点数, 描述: 简单内容的分布比例，例如 0.5。
+      - `multi_context`: 浮点数, 描述: 多上下文内容的分布比例，例如 0.4。
+      - `reasoning`: 浮点数, 描述: 推理内容的分布比例，例如 0.1。
+    - `generator_llm`: 字典, 描述: 生成器 LLM（大语言模型）的配置：
+      - 若使用本地模型，支持如下参数：
+        - `model_name_or_path`: 字符串, 描述: 生成器模型的名称或路径，填入名称，例如 "qwen/Qwen2-7B-Instruct"可以从modelscope自动下载模型；填入路径则从本地加载模型。
+        - `template_type`: 字符串, 描述: 模板类型，例如 "qwen"。
+        - `generation_config`: 字典, 描述: 生成配置，例如 `{"temperature": 0.7}`。
+      - 若使用API模型，支持如下参数：
+        - `model_name`: 字符串，自定义模型的名称。
+        - `api_base`: 字符串， 自定义基础URL。
+        - `api_key`: 可选，你的API密钥。
+    - `critic_llm`: 字典, 描述: 评价器 LLM 的配置，支持的配置项同上。
+    - `embeddings`: 字典, 描述: 嵌入模型的配置。
+      - `model_name_or_path`: 字符串, 描述: 嵌入模型的名称或路径，例如 "AI-ModelScope/m3e-base"。
+
+
+#### 执行任务
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+# Run task
+run_task(task_cfg=generate_testset_task_cfg) 
+```
+
+
+使用本项目README文档生成的示例如下：
+
+<details><summary>点击查看自动生成的数据集</summary>
+
+``` json
+[
+    {
+        "question": "What is the purpose of specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command?",
+        "contexts": [
+            " to run the custom code? [y\/N]`, please type `y`. #### Basic Parameter Descriptions - `--model`: Specifies the `model_id` of the model on [ModelScope](https:\/\/modelscope.cn\/), allowing automatic download. For example, see the [Qwen2-0.5B-Instruct model link](https:\/\/modelscope.cn\/models\/qwen\/Qwen2-0.5B-Instruct\/summary); you can also use a local path, such as `\/path\/to\/model`. - `--template-type`: Specifies the template type corresponding to the model. Refer to the `Default Template` field in the [template table](https:\/\/swift.readthedocs.io\/en\/latest\/Instruction\/Supported-models-datasets.html#llm) for filling in this field. - `--datasets`: The dataset name, allowing multiple datasets to be specified, separated by spaces; these datasets will be automatically downloaded. Refer to the [supported datasets list](https:\/\/evalscope.readthedocs.io\/en\/latest\/get_started\/supported_dataset.html) for available options. ### 2. Parameterized Evaluation If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands: **Example 1:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --model-args revision=master,precision=torch.float16,device_map=auto \\ --datasets gsm8k ceval \\ --use-cache true \\ --limit 10 ``` **Example 2:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --generation-config do_sample=false,temperature=0.0 \\ --datasets ceval \\ --dataset-args '{\"ceval\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\ --limit 10 ``` #### Parameter Descriptions In addition to the three [basic parameters](#basic-parameter-descriptions), the other parameters are as follows: - `--model-args`: Model loading parameters, separated by commas, in `key=value` format. - `--generation-config`: Generation parameters, separated by commas, in `key=value` format. - `do_sample`: Whether to use sampling, default is `false`. - `max_new_tokens`: Maximum generation length, default is 1024. - `temperature`: Sampling temperature. - `top_p`: Sampling threshold. - `top_k`: Sampling threshold. - `--use-cache`: Whether to use local cache, default is `false`. If set to `true`, previously evaluated model and dataset combinations will not be evaluated again, and will be read directly from the local cache. - `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format, where the key is the dataset name and the value is the parameter; note that these must correspond one-to-one with the values in `--datasets`. - `--few_shot_num`: Number of few-shot examples. - `--few_shot_random`: Whether to randomly sample few-shot data; if not specified, defaults to `true`. - `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation. ### 3. Use the run_task Function to Submit an Evaluation Task Using the `run_task` function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields: #### 1. Configuration Task Dictionary Parameters ```python import torch from evalscope.constants import DEFAULT_ROOT_CACHE_DIR # Example your_task_cfg = { 'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'}, 'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512}, 'dataset_args': {}, 'dry_run': False, 'model': 'qwen\/Qwen2-0.5B-Instruct', 'template_type': 'qwen', 'datasets': ['arc', 'hellasw"
+        ],
+        "ground_truth": "The purpose of specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command is to associate the correct evaluation template with the chosen model. This ensures that the evaluation process uses the appropriate instructions, prompts, and settings tailored for that specific model, enhancing the relevance and effectiveness of the evaluation.",
+        "answer": "Specifying the 'template type' when conducting a more customized evaluation using the evalscope/run.py command is crucial because it corresponds to the specific structure and requirements of the model being used. This ensures that the evaluation process aligns correctly with the model's capabilities and expected input format, leading to accurate and meaningful results."
+    },
+    {
+        "question": "What automated outputs does the EvalScope framework produce following a model evaluation?",
+        "contexts": [
+            "\ud83d\udcd6 Documents &nbsp | &nbsp \ud83d\udcd6 \u4e2d\u6587\u6587\u6863",
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "The EvalScope framework automatically generates evaluation reports following a model evaluation.",
+        "answer": "The EvalScope framework produces automated evaluation reports following a model evaluation. These reports include results from automatic evaluations of objective questions and complex task evaluations using expert models."
+    },
+    {
+        "question": "How does EvalScope framework create detailed reports for big models using benchmark datasets, metrics, and backends like OpenCompass or VLMEvalKit?",
+        "contexts": [
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "EvalScope framework creates detailed reports for big models by utilizing benchmark datasets, implementing evaluation metrics, and integrating with backends such as OpenCompass or VLMEvalKit. It supports automated evaluation of objective questions and complex tasks, offers a unified model access mechanism compatible with various model families, and includes features like Arena Mode for model comparisons. Additionally, it generates evaluation reports and provides visualization tools for intuitive result displays. Seamless integration with frameworks like ms-swift enhances the development process, offering a comprehensive solution for model training, deployment, evaluation, and reporting.",
+        "answer": "EvalScope framework creates detailed reports for big models by utilizing benchmark datasets, metrics, and backends such as OpenCompass or VLMEvalKit. It automates the evaluation process, generating comprehensive reports that reflect the model's performance across various tasks and metrics. This is achieved through its integrated backend support, which simplifies task submission and enables advanced encapsulation, making it easier to assess model capabilities in both single and comparative evaluations."
+    },
+    {
+        "question": "How can one tailor model assessments in `evalscope`, and which parameters related to model and dataset configurations can be modified for customization, given the tool's capabilities for model-specific loading, generation, and evaluation dataset setup?",
+        "contexts": [
+            " to run the custom code? [y\/N]`, please type `y`. #### Basic Parameter Descriptions - `--model`: Specifies the `model_id` of the model on [ModelScope](https:\/\/modelscope.cn\/), allowing automatic download. For example, see the [Qwen2-0.5B-Instruct model link](https:\/\/modelscope.cn\/models\/qwen\/Qwen2-0.5B-Instruct\/summary); you can also use a local path, such as `\/path\/to\/model`. - `--template-type`: Specifies the template type corresponding to the model. Refer to the `Default Template` field in the [template table](https:\/\/swift.readthedocs.io\/en\/latest\/Instruction\/Supported-models-datasets.html#llm) for filling in this field. - `--datasets`: The dataset name, allowing multiple datasets to be specified, separated by spaces; these datasets will be automatically downloaded. Refer to the [supported datasets list](https:\/\/evalscope.readthedocs.io\/en\/latest\/get_started\/supported_dataset.html) for available options. ### 2. Parameterized Evaluation If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands: **Example 1:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --model-args revision=master,precision=torch.float16,device_map=auto \\ --datasets gsm8k ceval \\ --use-cache true \\ --limit 10 ``` **Example 2:** ```shell python evalscope\/run.py \\ --model qwen\/Qwen2-0.5B-Instruct \\ --template-type qwen \\ --generation-config do_sample=false,temperature=0.0 \\ --datasets ceval \\ --dataset-args '{\"ceval\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\ --limit 10 ``` #### Parameter Descriptions In addition to the three [basic parameters](#basic-parameter-descriptions), the other parameters are as follows: - `--model-args`: Model loading parameters, separated by commas, in `key=value` format. - `--generation-config`: Generation parameters, separated by commas, in `key=value` format. - `do_sample`: Whether to use sampling, default is `false`. - `max_new_tokens`: Maximum generation length, default is 1024. - `temperature`: Sampling temperature. - `top_p`: Sampling threshold. - `top_k`: Sampling threshold. - `--use-cache`: Whether to use local cache, default is `false`. If set to `true`, previously evaluated model and dataset combinations will not be evaluated again, and will be read directly from the local cache. - `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format, where the key is the dataset name and the value is the parameter; note that these must correspond one-to-one with the values in `--datasets`. - `--few_shot_num`: Number of few-shot examples. - `--few_shot_random`: Whether to randomly sample few-shot data; if not specified, defaults to `true`. - `--limit`: Maximum number of evaluation samples per dataset; if not specified, all will be evaluated, which is useful for quick validation. ### 3. Use the run_task Function to Submit an Evaluation Task Using the `run_task` function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields: #### 1. Configuration Task Dictionary Parameters ```python import torch from evalscope.constants import DEFAULT_ROOT_CACHE_DIR # Example your_task_cfg = { 'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'}, 'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512}, 'dataset_args': {}, 'dry_run': False, 'model': 'qwen\/Qwen2-0.5B-Instruct', 'template_type': 'qwen', 'datasets': ['arc', 'hellasw",
+            "ag'], 'work_dir': DEFAULT_ROOT_CACHE_DIR, 'outputs': DEFAULT_ROOT_CACHE_DIR, 'mem_cache': False, 'dataset_hub': 'ModelScope', 'dataset_dir': DEFAULT_ROOT_CACHE_DIR, 'limit': 10, 'debug': False } ``` Here, `DEFAULT_ROOT_CACHE_DIR` is set to `'~\/.cache\/evalscope'`. #### 2. Execute Task with run_task ```python from evalscope.run import run_task run_task(task_cfg=your_task_cfg) ``` ## Evaluation Backend EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes: - **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode. - [OpenCompass](https:\/\/github.com\/open-compass\/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/opencompass_backend.html) - [VLMEvalKit](https:\/\/github.com\/open-compass\/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/vlmevalkit_backend.html) - **ThirdParty**: The third-party task, e.g. [ToolBench](https:\/\/evalscope.readthedocs.io\/en\/latest\/third_party\/toolbench.html), you can contribute your own evaluation task to EvalScope as third-party backend. ## Custom Dataset Evaluation EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [\ud83d\udcd6User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset.html) ## Offline Evaluation You can use local dataset to evaluate the model without internet connection. Refer to: Offline Evaluation [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/offline_evaluation.html) ## Arena Mode The Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report. Refer to: Arena Mode [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html) ## Model Serving Performance Evaluation A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats. Refer to: Model Serving Performance Evaluation [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test.html) ## Leaderboard The LLM Leaderboard aims to provide an objective and comprehensive evaluation standard and platform to help researchers and developers understand and compare the performance of models on various tasks on ModelScope. Refer to: [Leaderboard](https://modelscope.cn/leaderboard/58/ranking?type=free) ## TO-DO List - [x] Agents evaluation - [x] vLLM - [ ] Distributed evaluating - [x] Multi-modal evaluation - [ ] Benchmarks - [ ] GAIA - [ ] GPQA - [x] MBPP - [ ] Auto-reviewer - [ ] Qwen-max"
+        ],
+        "ground_truth": "To tailor model assessments in `evalscope`, one can modify several parameters related to model and dataset configurations. For model-specific loading, `model-args` allows setting parameters like `revision`, `precision`, and `device_map`. For generation, `generation-config` lets users control aspects such as `do_sample`, `temperature`, `max_new_tokens`, and `top_p`. To customize evaluation datasets, `dataset-args` provides configuration options for each specified dataset. Additionally, parameters like `few_shot_num`, `few_shot_random`, and `limit` enable adjustments for few-shot learning and limiting the number of evaluation samples per dataset.",
+        "answer": "To tailor model assessments in `evalscope`, you can modify parameters related to model and dataset configurations. Key parameters include:\n\n- `--model`: Specifies the model ID or local path.\n- `--template-type`: Corresponds to the model's template type.\n- `--datasets`: Allows specifying multiple datasets to evaluate, which are automatically downloaded.\n- `--model-args`: Model loading parameters, like `revision`, `precision`, and `device_map`.\n- `--generation-config`: Generation parameters, including `do_sample`, `temperature`, and `max_new_tokens`.\n- `--use-cache`: Determines if local cache should be used.\n- `--dataset-args`: Evaluation dataset configuration parameters, provided in JSON format.\n- `--few_shot_num`: Number of few-shot examples.\n- `--limit`: Maximum number of evaluation samples per dataset.\n\nThese parameters enable customization for model-specific loading, generation, and evaluation dataset setup."
+    },
+    {
+        "question": "How does the OpenCompass Integration streamline task submission for evaluation compared to EvalScope and VLMEvalKit?",
+        "contexts": [
+            "## \ud83d\udccb Table of Contents - [Introduction](#introduction) - [News](#News) - [Installation](#installation) - [Quick Start](#quick-start) - [Evaluation Backend](#evaluation-backend) - [Custom Dataset Evaluation](#custom-dataset-evaluation) - [Offline Evaluation](#offline-evaluation) - [Arena Mode](#arena-mode) - [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation) - [Leaderboard](#leaderboard) ## \ud83d\udcdd Introduction Large Model (including Large Language Models, Multi-modal Large Language Models) evaluation has become a critical process for assessing and improving LLMs. To better support the evaluation of large models, we propose the EvalScope framework. ### Framework Features - **Benchmark Datasets**: Preloaded with several commonly used test benchmarks, including MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, HumanEval, etc. - **Evaluation Metrics**: Implements various commonly used evaluation metrics. - **Model Access**: A unified model access mechanism that is compatible with the Generate and Chat interfaces of multiple model families. - **Automated Evaluation**: Includes automatic evaluation of objective questions and complex task evaluation using expert models. - **Evaluation Reports**: Automatically generates evaluation reports. - **Arena Mode**: Used for comparisons between models and objective evaluation of models, supporting various evaluation modes, including: - **Single mode**: Scoring a single model. - **Pairwise-baseline mode**: Comparing against a baseline model. - **Pairwise (all) mode**: Pairwise comparison among all models. - **Visualization Tools**: Provides intuitive displays of evaluation results. - **Model Performance Evaluation**: Offers a performance testing tool for model inference services and detailed statistics, see [Model Performance Evaluation Documentation](https:\/\/evalscope.readthedocs.io\/en\/latest\/user_guides\/stress_test.html). - **OpenCompass Integration**: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing for easier task submission for evaluation. - **VLMEvalKit Integration**: Supports VLMEvalKit as the evaluation backend, facilitating the initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets. - **Full-Link Support**: Through seamless integration with the [ms-swift](https:\/\/github.com\/modelscope\/ms-swift) training framework, provides a one-stop development process for model training, model deployment, model evaluation, and report viewing, enhancing user development efficiency."
+        ],
+        "ground_truth": "The OpenCompass Integration streamlines task submission for evaluation by providing advanced encapsulation and task simplification. This allows for easier task submission for evaluation, making the process more efficient and user-friendly.",
+        "answer": "The OpenCompass Integration in EvalScope streamlines task submission for evaluation by offering advanced encapsulation and simplifying tasks, making it easier to submit tasks for assessment compared to EvalScope's direct use of the Evaluation Backend or VLMEvalKit."
+    }
+]
+```
+
+</details>
+
+## 配置评测任务
+
+```python
+eval_task_cfg = {
+    "eval_backend": "RAGEval",
+    "eval_config": {
+        "tool": "RAGAS",
+        "eval": {
+            "testset_file": "outputs/testset.json",
+            "critic_llm": {
+                "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                "template_type": "qwen",
+            },
+            "embeddings": {
+                "model_name_or_path": "AI-ModelScope/m3e-base",
+            },
+            "metrics": [
+                "faithfulness",
+                "answer_relevancy",
+                "context_precision",
+                "answer_correctness",
+            ],
+        },
+    },
+}
+```
+基本参数与自动评测任务配置一致，不同点在于：
+- `testset_file` ：指定评测数据集文件路径，默认为 `outputs/testset.json`
+- `metrics`: 指定评测指标，参考[metrics 列表](https://docs.ragas.io/en/latest/concepts/metrics/index.html)
+
+
+## 执行评测
+
+```python
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+# Run task
+run_task(task_cfg=eval_task_cfg)
+```
+输出评测结果如下：
+
+```{figure} images/eval_result.png
+
+评测结果
+```
diff --git a/docs/zh/user_guides/vlmevalkit_backend.md b/docs/zh/user_guides/backend/vlmevalkit_backend.md
similarity index 99%
rename from docs/zh/user_guides/vlmevalkit_backend.md
rename to docs/zh/user_guides/backend/vlmevalkit_backend.md
index bb4cf775..e3f22c75 100644
--- a/docs/zh/user_guides/vlmevalkit_backend.md
+++ b/docs/zh/user_guides/backend/vlmevalkit_backend.md
@@ -1,11 +1,13 @@
-# VLMEvalKit 评测后端
+(vlmeval)=
+
+# VLMEvalKit
 
 为便于使用VLMEvalKit 评测后端，我们基于VLMEvalKit源码做了定制，命名为`ms-vlmeval`，该版本在原版基础上对评估任务的配置和执行进行了封装，并支持pypi安装方式，使得用户可以通过EvalScope发起轻量化的VLMEvalKit评估任务。同时，我们支持基于OpenAI API格式的接口评估任务，您可以使用[ms-swift](https://github.com/modelscope/swift)、[vLLM](https://github.com/vllm-project/vllm)、[LMDeploy](https://github.com/InternLM/lmdeploy)、[Ollama](https://ollama.ai/)等模型服务，部署多模态模型服务。
 
 ## 1. 环境准备
 ```shell
 # 安装额外依赖
-pip install evalscope[vlmeval]
+pip install evalscope[vlmeval] -U
 ```
 
 ## 2. 数据准备
diff --git a/evalscope/backend/__init__.py b/evalscope/backend/__init__.py
index 8458bda6..e69de29b 100644
--- a/evalscope/backend/__init__.py
+++ b/evalscope/backend/__init__.py
@@ -1,3 +0,0 @@
-# Copyright (c) Alibaba, Inc. and its affiliates.
-
-from evalscope.backend.opencompass.backend_manager import OpenCompassBackendManager
diff --git a/evalscope/backend/rag_eval/__init__.py b/evalscope/backend/rag_eval/__init__.py
new file mode 100644
index 00000000..19a34891
--- /dev/null
+++ b/evalscope/backend/rag_eval/__init__.py
@@ -0,0 +1,3 @@
+from evalscope.backend.rag_eval.utils.embedding import EmbeddingModel
+from evalscope.backend.rag_eval.utils.llm import LLM
+from evalscope.backend.rag_eval.backend_manager import RAGEvalBackendManager
\ No newline at end of file
diff --git a/evalscope/backend/rag_eval/backend_manager.py b/evalscope/backend/rag_eval/backend_manager.py
new file mode 100644
index 00000000..02cfcc2f
--- /dev/null
+++ b/evalscope/backend/rag_eval/backend_manager.py
@@ -0,0 +1,68 @@
+import os
+from typing import Optional, Union
+from evalscope.utils import is_module_installed, get_valid_list
+from evalscope.backend.base import BackendManager
+from evalscope.utils.logger import get_logger
+
+
+logger = get_logger()
+
+
+class RAGEvalBackendManager(BackendManager):
+    def __init__(self, config: Union[str, dict], **kwargs):
+        """BackendManager for VLM Evaluation Kit
+
+        Args:
+            config (Union[str, dict]): the configuration yaml-file or the configuration dictionary
+        """
+        super().__init__(config, **kwargs)
+
+    @staticmethod
+    def _check_env(module_name: str):
+        if is_module_installed(module_name):
+            logger.info(f"Check `{module_name}` Installed")
+        else:
+            logger.error(f"Please install `{module_name}` first")
+
+    def run_mteb(self):
+        from evalscope.backend.rag_eval.cmteb import ModelArguments, EvalArguments
+        from evalscope.backend.rag_eval.cmteb import one_stage_eval, two_stage_eval
+
+        if len(self.model_args) > 2:
+            raise ValueError("Not support multiple models yet")
+
+        # Convert arguments to dictionary
+        model_args_list = [ModelArguments(**args).to_dict() for args in self.model_args]
+        eval_args = EvalArguments(**self.eval_args).to_dict()
+
+        if len(model_args_list) == 1:
+            one_stage_eval(model_args_list[0], eval_args)
+        else:  # len(model_args_list) == 2
+            two_stage_eval(model_args_list[0], model_args_list[1], eval_args)
+
+    def run_ragas(self):
+        from evalscope.backend.rag_eval.ragas import rag_eval, testset_generation
+        from evalscope.backend.rag_eval.ragas import (
+            TestsetGenerationArguments,
+            EvaluationArguments,
+        )
+
+        if self.testset_args is not None:
+            testset_generation(TestsetGenerationArguments(**self.testset_args))
+        if self.eval_args is not None:
+            rag_eval(EvaluationArguments(**self.eval_args))
+
+    def run(self, *args, **kwargs):
+        tool = self.config_d.pop("tool")
+        if tool.lower() == "mteb":
+            self._check_env("mteb")
+            self.model_args = self.config_d["model"]
+            self.eval_args = self.config_d["eval"]
+            self.run_mteb()
+        elif tool.lower() == "ragas":
+            self._check_env("ragas")
+            self.testset_args = self.config_d.get("testset_generation", None)
+            self.eval_args = self.config_d.get("eval", None)
+            self.run_ragas()
+        else:
+            raise ValueError(f"Unknown tool: {tool}")
diff --git a/evalscope/backend/rag_eval/cmteb/__init__.py b/evalscope/backend/rag_eval/cmteb/__init__.py
new file mode 100644
index 00000000..b77705e1
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/__init__.py
@@ -0,0 +1,4 @@
+from evalscope.backend.rag_eval.cmteb.tasks import *
+from evalscope.backend.rag_eval.cmteb.base import *
+from evalscope.backend.rag_eval.cmteb.arguments import ModelArguments, EvalArguments
+from evalscope.backend.rag_eval.cmteb.task_template import one_stage_eval, two_stage_eval
\ No newline at end of file
diff --git a/evalscope/backend/rag_eval/cmteb/arguments.py b/evalscope/backend/rag_eval/cmteb/arguments.py
new file mode 100644
index 00000000..895d4435
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/arguments.py
@@ -0,0 +1,59 @@
+from dataclasses import dataclass, field
+from typing import List, Optional, Union, Dict, Any
+
+
+@dataclass
+class ModelArguments:
+    # Arguments for embeding model: sentence transformer or cross encoder
+    model_name_or_path: str = ""  # model name or path
+    is_cross_encoder: bool = False  # whether the model is a cross encoder
+    # pooling mode: Either “cls”, “lasttoken”, “max”, “mean”, “mean_sqrt_len_tokens”, or “weightedmean”.
+    pooling_mode: Optional[str] = None
+    max_seq_length: int = 512  # max sequence length
+    # prompt for llm based model
+    prompt: str = ""
+    # model kwargs
+    model_kwargs: dict = field(default_factory=lambda: {"torch_dtype": "auto"})
+    # config kwargs
+    config_kwargs: Dict[str, Any] = field(default_factory=dict)
+    # encode kwargs
+    encode_kwargs: dict = field(
+        default_factory=lambda: {"show_progress_bar": True, "batch_size": 32}
+    )
+    hub: str = "modelscope"  # modelscope or huggingface
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "model_name_or_path": self.model_name_or_path,
+            "is_cross_encoder": self.is_cross_encoder,
+            "pooling_mode": self.pooling_mode,
+            "max_seq_length": self.max_seq_length,
+            "prompt": self.prompt,
+            "model_kwargs": self.model_kwargs,
+            "config_kwargs": self.config_kwargs,
+            "encode_kwargs": self.encode_kwargs,
+            "hub": self.hub,
+        }
+
+
+@dataclass
+class EvalArguments:
+    # Evaluation
+    tasks: List[str] = field(default_factory=list)  # task names
+    verbosity: int = 2  # verbosity level 0-3
+    output_folder: str = "outputs"  # output folder
+    overwrite_results: bool = True  # overwrite results
+    limits: Optional[int] = None  # limit number of samples
+    hub: str = "modelscope"  # modelscope or huggingface
+    top_k: int = 5
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "tasks": self.tasks,
+            "verbosity": self.verbosity,
+            "output_folder": self.output_folder,
+            "overwrite_results": self.overwrite_results,
+            "limits": self.limits,
+            "hub": self.hub,
+            "top_k": 5,
+        }
diff --git a/evalscope/backend/rag_eval/cmteb/base.py b/evalscope/backend/rag_eval/cmteb/base.py
new file mode 100644
index 00000000..29dce5da
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/base.py
@@ -0,0 +1,89 @@
+from collections import defaultdict
+from typing import List
+from mteb import AbsTask
+from datasets import DatasetDict
+from modelscope import MsDataset
+import datasets
+from evalscope.backend.rag_eval.cmteb.tasks import CLS_DICT, CLS_RETRIEVAL
+
+__all__ = ["TaskBase"]
+
+
+
+class TaskBase:
+
+    @staticmethod
+    def get_tasks(task_names, **kwargs) -> List[AbsTask]:
+
+        return [TaskBase.get_task(task_name, **kwargs) for task_name in task_names]
+
+    @staticmethod
+    def get_task(task_name, **kwargs) -> AbsTask:
+
+        if task_name not in CLS_DICT:
+            from mteb.overview import TASKS_REGISTRY
+
+            task_cls = TASKS_REGISTRY[task_name]
+            if task_cls.metadata.type != "Retrieval":
+                task_cls.load_data = load_data
+        else:
+            task_cls = CLS_DICT[task_name]
+            task_cls.load_data = load_data
+        # init task instance
+        task_instance = task_cls()
+        return task_instance
+
+
+def load_data(self, **kwargs):
+    """Load dataset from the hub, compatible with ModelScope and Hugging Face."""
+    if self.data_loaded:
+        return
+
+    limits = kwargs.get("limits", None)
+    hub = kwargs.get("hub", "modelscope")
+    name = self.metadata_dict.get("name")
+    path = self.metadata_dict["dataset"].get("path")
+
+    assert path is not None, "Path must be specified in dataset"
+
+    # Loading the dataset based on the source hub
+    if hub == "modelscope":
+        import re
+
+        path = re.sub(r"^mteb/", "MTEB/", path)
+        dataset = MsDataset.load(path)
+    else:
+        dataset = datasets.load_dataset(**self.metadata_dict["dataset"])  # type: ignore
+
+    if limits is not None:
+        dataset = {
+            split: dataset[split].select(range(min(limits, len(dataset[split]))))
+            for split in dataset.keys()
+        }
+
+    if name in CLS_RETRIEVAL:
+        self.corpus, self.queries, self.relevant_docs = load_retrieval_data(
+            dataset,
+            path,
+            self.metadata_dict["eval_splits"],
+        )
+
+    self.dataset = dataset
+    self.dataset_transform()
+    self.data_loaded = True
+
+
+def load_retrieval_data(dataset, dataset_name: str, eval_splits: list) -> tuple:
+    eval_split = eval_splits[0]
+    qrels = MsDataset.load(dataset_name + "-qrels")[eval_split]
+
+    corpus = {e["id"]: {"text": e["text"]} for e in dataset["corpus"]}
+    queries = {e["id"]: e["text"] for e in dataset["queries"]}
+    relevant_docs = defaultdict(dict)
+    for e in qrels:
+        relevant_docs[e["qid"]][e["pid"]] = e["score"]
+
+    corpus = DatasetDict({eval_split: corpus})
+    queries = DatasetDict({eval_split: queries})
+    relevant_docs = DatasetDict({eval_split: relevant_docs})
+    return corpus, queries, relevant_docs
diff --git a/evalscope/backend/rag_eval/cmteb/task_template.py b/evalscope/backend/rag_eval/cmteb/task_template.py
new file mode 100644
index 00000000..de3daa36
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/task_template.py
@@ -0,0 +1,83 @@
+import os
+import mteb
+from evalscope.backend.rag_eval import EmbeddingModel
+from evalscope.backend.rag_eval import cmteb
+from mteb.task_selection import results_to_dataframe
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def show_results(output_folder, model, results):
+    model_name = model.mteb_model_meta.model_name_as_path()
+    revision = model.mteb_model_meta.revision
+
+    results_df = results_to_dataframe({model_name: {revision: results}})
+
+    save_path = os.path.join(
+        output_folder,
+        model_name,
+        revision,
+    )
+    logger.info(f"Evaluation results:\n{results_df.to_markdown()}")
+    logger.info(f"Evaluation results saved in {os.path.abspath(save_path)}")
+
+
+def one_stage_eval(
+    model_args,
+    eval_args,
+) -> None:
+    # load model
+    model = EmbeddingModel.load(**model_args)
+
+    # load task first to update instructions
+    tasks = cmteb.TaskBase.get_tasks(task_names=eval_args["tasks"])
+    evaluation = mteb.MTEB(tasks=tasks)
+
+    # run evaluation
+    results = evaluation.run(model, **eval_args)
+
+    # save and log results
+    show_results(eval_args["output_folder"], model, results)
+
+
+def two_stage_eval(
+    model1_args,
+    model2_args,
+    eval_args,
+) -> None:
+    """a two-stage run with the second stage reading results saved from the first stage."""
+    # load model
+    dual_encoder = EmbeddingModel.load(**model1_args)
+    cross_encoder = EmbeddingModel.load(**model2_args)
+
+    first_stage_path = f"{eval_args['output_folder']}/stage1"
+    second_stage_path = f"{eval_args['output_folder']}/stage2"
+
+    tasks = cmteb.TaskBase.get_tasks(task_names=eval_args["tasks"])
+    for task in tasks:
+        evaluation = mteb.MTEB(tasks=[task])
+
+        # stage 1: run dual encoder
+        evaluation.run(
+            dual_encoder,
+            save_predictions=True,
+            output_folder=first_stage_path,
+            overwrite_results=True,
+            hub=eval_args["hub"],
+            limits=eval_args["limits"],
+        )
+        # stage 2: run cross encoder
+        results = evaluation.run(
+            cross_encoder,
+            top_k=eval_args["top_k"],
+            save_predictions=True,
+            output_folder=second_stage_path,
+            previous_results=f"{first_stage_path}/{task.metadata.name}_default_predictions.json",
+            overwrite_results=True,
+            hub=eval_args["hub"],
+            limits=eval_args["limits"],
+        )
+
+        # save and log results
+        show_results(second_stage_path, cross_encoder, results)
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/Classification.py b/evalscope/backend/rag_eval/cmteb/tasks/Classification.py
new file mode 100644
index 00000000..85e99412
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/Classification.py
@@ -0,0 +1,302 @@
+from mteb.abstasks.AbsTaskClassification import AbsTaskClassification
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class TNews(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="TNews",
+        description="Short Text Classification for News",
+        reference="https://www.cluebenchmarks.com/introduce.html",
+        dataset={
+            "path": "C-MTEB/TNews-classification",
+            "revision": "317f262bf1e6126357bbe89e875451e4b0938fe4",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@inproceedings {xu-etal-2020-clue,
+ title = "{CLUE}: A {C}hinese Language Understanding Evaluation Benchmark",
+ author = "Xu, Liang  and
+    Hu, Hai and
+    Zhang, Xuanwei and
+    Li, Lu and
+    Cao, Chenjie and
+    Li, Yudong and
+    Xu, Yechen and
+    Sun, Kai and
+    Yu, Dian and
+    Yu, Cong and
+    Tian, Yin and
+    Dong, Qianqian and
+    Liu, Weitang and
+    Shi, Bo and
+    Cui, Yiming and
+    Li, Junyi and
+    Zeng, Jun and
+    Wang, Rongzhao and
+    Xie, Weijian and
+    Li, Yanting and
+    Patterson, Yina and
+    Tian, Zuoyu and
+    Zhang, Yiwen and
+    Zhou, He and
+    Liu, Shaoweihua and
+    Zhao, Zhe and
+    Zhao, Qipeng and
+    Yue, Cong and
+    Zhang, Xinrui and
+    Yang, Zhengliang and
+    Richardson, Kyle and
+    Lan, Zhenzhong ",
+ booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
+ month = dec,
+ year = "2020",
+ address = "Barcelona, Spain (Online)",
+ publisher = "International Committee on Computational Linguistics",
+ url = "https://aclanthology.org/2020.coling-main.419",
+ doi = "10.18653/v1/2020.coling-main.419",
+ pages = "4762--4772",
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+        return metadata_dict
+
+
+class IFlyTek(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="IFlyTek",
+        description="Long Text classification for the description of Apps",
+        reference="https://www.cluebenchmarks.com/introduce.html",
+        dataset={
+            "path": "C-MTEB/IFlyTek-classification",
+            "revision": "421605374b29664c5fc098418fe20ada9bd55f8a",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@inproceedings {xu-etal-2020-clue,
+ title = "{CLUE}: A {C}hinese Language Understanding Evaluation Benchmark",
+ author = "Xu, Liang  and
+    Hu, Hai and
+    Zhang, Xuanwei and
+    Li, Lu and
+    Cao, Chenjie and
+    Li, Yudong and
+    Xu, Yechen and
+    Sun, Kai and
+    Yu, Dian and
+    Yu, Cong and
+    Tian, Yin and
+    Dong, Qianqian and
+    Liu, Weitang and
+    Shi, Bo and
+    Cui, Yiming and
+    Li, Junyi and
+    Zeng, Jun and
+    Wang, Rongzhao and
+    Xie, Weijian and
+    Li, Yanting and
+    Patterson, Yina and
+    Tian, Zuoyu and
+    Zhang, Yiwen and
+    Zhou, He and
+    Liu, Shaoweihua and
+    Zhao, Zhe and
+    Zhao, Qipeng and
+    Yue, Cong and
+    Zhang, Xinrui and
+    Yang, Zhengliang and
+    Richardson, Kyle and
+    Lan, Zhenzhong ",
+ booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
+ month = dec,
+ year = "2020",
+ address = "Barcelona, Spain (Online)",
+ publisher = "International Committee on Computational Linguistics",
+ url = "https://aclanthology.org/2020.coling-main.419",
+ doi = "10.18653/v1/2020.coling-main.419",
+ pages = "4762--4772",
+ abstract = "The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.cluebenchmarks.com",
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+        metadata_dict["n_experiments"] = 5
+        return metadata_dict
+
+
+class MultilingualSentiment(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="MultilingualSentiment",
+        description="A collection of multilingual sentiments datasets grouped into 3 classes -- positive, neutral, negative",
+        reference="https://github.com/tyqiangz/multilingual-sentiment-datasets",
+        dataset={
+            "path": "C-MTEB/MultilingualSentiment-classification",
+            "revision": "46958b007a63fdbf239b7672c25d0bea67b5ea1a",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation", "test"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+        return metadata_dict
+
+
+class JDReview(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="JDReview",
+        description="review for iphone",
+        reference="https://aclanthology.org/2023.nodalida-1.20/",
+        dataset={
+            "path": "C-MTEB/JDReview-classification",
+            "revision": "b7c64bd89eb87f8ded463478346f76731f07bf8b",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@article{xiao2023c,
+  title={C-pack: Packaged resources to advance general chinese embedding},
+  author={Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighof, Niklas},
+  journal={arXiv preprint arXiv:2309.07597},
+  year={2023}
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+        return metadata_dict
+
+
+class OnlineShopping(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="OnlineShopping",
+        description="Sentiment Analysis of User Reviews on Online Shopping Websites",
+        reference="https://aclanthology.org/2023.nodalida-1.20/",
+        dataset={
+            "path": "C-MTEB/OnlineShopping-classification",
+            "revision": "e610f2ebd179a8fda30ae534c3878750a96db120",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@article{xiao2023c,
+  title={C-pack: Packaged resources to advance general chinese embedding},
+  author={Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighof, Niklas},
+  journal={arXiv preprint arXiv:2309.07597},
+  year={2023}
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+        return metadata_dict
+
+
+class Waimai(AbsTaskClassification):
+    metadata = TaskMetadata(
+        name="Waimai",
+        description="Sentiment Analysis of user reviews on takeaway platforms",
+        reference="https://aclanthology.org/2023.nodalida-1.20/",
+        dataset={
+            "path": "C-MTEB/waimai-classification",
+            "revision": "339287def212450dcaa9df8c22bf93e9980c7023",
+        },
+        type="Classification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@article{xiao2023c,
+  title={C-pack: Packaged resources to advance general chinese embedding},
+  author={Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighof, Niklas},
+  journal={arXiv preprint arXiv:2309.07597},
+  year={2023}
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["samples_per_label"] = 32
+
+        return metadata_dict
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/Clustering.py b/evalscope/backend/rag_eval/cmteb/tasks/Clustering.py
new file mode 100644
index 00000000..0442a961
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/Clustering.py
@@ -0,0 +1,252 @@
+import itertools
+
+from datasets import Dataset, DatasetDict
+
+from mteb.abstasks.AbsTaskClustering import AbsTaskClustering
+from mteb.abstasks.AbsTaskClusteringFast import (
+    AbsTaskClusteringFast,
+    check_label_distribution,
+)
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+NUM_SAMPLES = 2048
+
+
+class CLSClusteringFastS2S(AbsTaskClusteringFast):
+    max_document_to_embed = NUM_SAMPLES
+    max_fraction_of_documents_to_embed = None
+
+    metadata = TaskMetadata(
+        name="CLSClusteringS2S",
+        description="Clustering of titles from CLS dataset. Clustering of 13 sets on the main category.",
+        reference="https://arxiv.org/abs/2209.05034",
+        dataset={
+            "path": "C-MTEB/CLSClusteringS2S",
+            "revision": "e458b3f5414b62b7f9f83499ac1f5497ae2e869f",
+        },
+        type="Clustering",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="v_measure",
+        date=("2022-01-01", "2022-09-12"),
+        domains=["Academic", "Written"],
+        task_subtypes=["Thematic clustering", "Topic classification"],
+        license="Apache-2.0",
+        annotations_creators="derived",
+        dialect=[],
+        sample_creation="found",
+        bibtex_citation="""@misc{li2022csl,
+            title={CSL: A Large-scale Chinese Scientific Literature Dataset}, 
+            author={Yudong Li and Yuqing Zhang and Zhe Zhao and Linlin Shen and Weijie Liu and Weiquan Mao and Hui Zhang},
+            year={2022},
+            eprint={2209.05034},
+            archivePrefix={arXiv},
+            primaryClass={cs.CL}
+        }""",
+        descriptive_stats={
+            "n_samples": {"test": NUM_SAMPLES},
+            "avg_character_length": {},
+        },
+    )
+
+    def dataset_transform(self):
+        ds = {}
+        for split in self.metadata.eval_splits:
+            labels = list(itertools.chain.from_iterable(self.dataset[split]["labels"]))
+            sentences = list(
+                itertools.chain.from_iterable(self.dataset[split]["sentences"])
+            )
+
+            check_label_distribution(self.dataset[split])
+
+            ds[split] = Dataset.from_dict({"labels": labels, "sentences": sentences})
+        self.dataset = DatasetDict(ds)
+        self.dataset = self.stratified_subsampling(
+            self.dataset,
+            self.seed,
+            self.metadata.eval_splits,
+            label="labels",
+            n_samples=NUM_SAMPLES,
+        )
+
+
+class CLSClusteringFastP2P(AbsTaskClusteringFast):
+    max_document_to_embed = NUM_SAMPLES
+    max_fraction_of_documents_to_embed = None
+
+    metadata = TaskMetadata(
+        name="CLSClusteringP2P",
+        description="Clustering of titles + abstract from CLS dataset. Clustering of 13 sets on the main category.",
+        reference="https://arxiv.org/abs/2209.05034",
+        dataset={
+            "path": "C-MTEB/CLSClusteringP2P",
+            "revision": "4b6227591c6c1a73bc76b1055f3b7f3588e72476",
+        },
+        type="Clustering",
+        category="p2p",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="v_measure",
+        date=("2022-01-01", "2022-09-12"),
+        domains=["Academic", "Written"],
+        task_subtypes=["Thematic clustering", "Topic classification"],
+        license="Apache-2.0",
+        annotations_creators="derived",
+        dialect=[],
+        sample_creation="found",
+        bibtex_citation="""@misc{li2022csl,
+            title={CSL: A Large-scale Chinese Scientific Literature Dataset}, 
+            author={Yudong Li and Yuqing Zhang and Zhe Zhao and Linlin Shen and Weijie Liu and Weiquan Mao and Hui Zhang},
+            year={2022},
+            eprint={2209.05034},
+            archivePrefix={arXiv},
+            primaryClass={cs.CL}
+        }""",
+        descriptive_stats={
+            "n_samples": {"test": NUM_SAMPLES},
+            "avg_character_length": {},
+        },
+    )
+
+    def dataset_transform(self):
+        ds = {}
+        for split in self.metadata.eval_splits:
+            labels = list(itertools.chain.from_iterable(self.dataset[split]["labels"]))
+            sentences = list(
+                itertools.chain.from_iterable(self.dataset[split]["sentences"])
+            )
+
+            check_label_distribution(self.dataset[split])
+
+            ds[split] = Dataset.from_dict({"labels": labels, "sentences": sentences})
+        self.dataset = DatasetDict(ds)
+        self.dataset = self.stratified_subsampling(
+            self.dataset,
+            self.seed,
+            self.metadata.eval_splits,
+            label="labels",
+            n_samples=NUM_SAMPLES,
+        )
+
+
+class ThuNewsClusteringFastS2S(AbsTaskClusteringFast):
+    max_document_to_embed = NUM_SAMPLES
+    max_fraction_of_documents_to_embed = None
+
+    metadata = TaskMetadata(
+        name="ThuNewsClusteringS2S",
+        dataset={
+            "path": "C-MTEB/ThuNewsClusteringS2S",
+            "revision": "8a8b2caeda43f39e13c4bc5bea0f8a667896e10d",
+        },
+        description="Clustering of titles from the THUCNews dataset",
+        reference="http://thuctc.thunlp.org/",
+        type="Clustering",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="v_measure",
+        date=("2006-01-01", "2007-01-01"),
+        domains=["News", "Written"],
+        task_subtypes=["Thematic clustering", "Topic classification"],
+        license="Not specified",
+        annotations_creators="derived",
+        dialect=[],
+        sample_creation="found",
+        bibtex_citation="""@software{THUCTC,
+  author = {Sun, M. and Li, J. and Guo, Z. and Yu, Z. and Zheng, Y. and Si, X. and Liu, Z.},
+  title = {THUCTC: An Efficient Chinese Text Classifier},
+  year = {2016},
+  note = {THU Chinese Text Classification Toolkit},
+  publisher = {THU Natural Language Processing Lab},
+  url = {https://github.com/thunlp/THUCTC}
+}""",
+        descriptive_stats={
+            "n_samples": {"test": NUM_SAMPLES},
+            "avg_character_length": {},
+        },
+    )
+
+    def dataset_transform(self):
+        ds = {}
+        for split in self.metadata.eval_splits:
+            labels = list(itertools.chain.from_iterable(self.dataset[split]["labels"]))
+            sentences = list(
+                itertools.chain.from_iterable(self.dataset[split]["sentences"])
+            )
+
+            check_label_distribution(self.dataset[split])
+
+            ds[split] = Dataset.from_dict({"labels": labels, "sentences": sentences})
+        self.dataset = DatasetDict(ds)
+        self.dataset = self.stratified_subsampling(
+            self.dataset,
+            self.seed,
+            self.metadata.eval_splits,
+            label="labels",
+            n_samples=NUM_SAMPLES,
+        )
+
+
+class ThuNewsClusteringFastP2P(AbsTaskClusteringFast):
+    max_document_to_embed = NUM_SAMPLES
+    max_fraction_of_documents_to_embed = None
+
+    metadata = TaskMetadata(
+        name="ThuNewsClusteringP2P",
+        dataset={
+            "path": "C-MTEB/ThuNewsClusteringP2P",
+            "revision": "5798586b105c0434e4f0fe5e767abe619442cf93",
+        },
+        description="Clustering of titles + abstracts from the THUCNews dataset",
+        reference="http://thuctc.thunlp.org/",
+        type="Clustering",
+        category="p2p",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="v_measure",
+        date=("2006-01-01", "2007-01-01"),
+        domains=["News", "Written"],
+        task_subtypes=["Thematic clustering", "Topic classification"],
+        license="Not specified",
+        annotations_creators="derived",
+        dialect=[],
+        sample_creation="found",
+        bibtex_citation="""@software{THUCTC,
+  author = {Sun, M. and Li, J. and Guo, Z. and Yu, Z. and Zheng, Y. and Si, X. and Liu, Z.},
+  title = {THUCTC: An Efficient Chinese Text Classifier},
+  year = {2016},
+  note = {THU Chinese Text Classification Toolkit},
+  publisher = {THU Natural Language Processing Lab},
+  url = {https://github.com/thunlp/THUCTC}
+}""",
+        descriptive_stats={
+            "n_samples": {"test": NUM_SAMPLES},
+            "avg_character_length": {},
+        },
+    )
+
+    def dataset_transform(self):
+        ds = {}
+        for split in self.metadata.eval_splits:
+            labels = list(itertools.chain.from_iterable(self.dataset[split]["labels"]))
+            sentences = list(
+                itertools.chain.from_iterable(self.dataset[split]["sentences"])
+            )
+
+            check_label_distribution(self.dataset[split])
+
+            ds[split] = Dataset.from_dict({"labels": labels, "sentences": sentences})
+        self.dataset = DatasetDict(ds)
+        self.dataset = self.stratified_subsampling(
+            self.dataset,
+            self.seed,
+            self.metadata.eval_splits,
+            label="labels",
+            n_samples=NUM_SAMPLES,
+        )
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/PairClassification.py b/evalscope/backend/rag_eval/cmteb/tasks/PairClassification.py
new file mode 100644
index 00000000..81045613
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/PairClassification.py
@@ -0,0 +1,113 @@
+from mteb.abstasks.AbsTaskPairClassification import AbsTaskPairClassification
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class Ocnli(AbsTaskPairClassification):
+    metadata = TaskMetadata(
+        name="Ocnli",
+        description="Original Chinese Natural Language Inference dataset",
+        reference="https://arxiv.org/abs/2010.05444",
+        dataset={
+            "path": "C-MTEB/OCNLI",
+            "revision": "66e76a618a34d6d565d5538088562851e6daa7ec",
+        },
+        type="PairClassification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation"],
+        eval_langs=["cmn-Hans"],
+        main_score="max_accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{hu2020ocnli,
+            title={OCNLI: Original Chinese Natural Language Inference}, 
+            author={Hai Hu and Kyle Richardson and Liang Xu and Lu Li and Sandra Kuebler and Lawrence S. Moss},
+            year={2020},
+            eprint={2010.05444},
+            archivePrefix={arXiv},
+            primaryClass={cs.CL}
+        }""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    def dataset_transform(self):
+        self.dataset = self.dataset.rename_column("sent1", "sentence1")
+        self.dataset = self.dataset.rename_column("sent2", "sentence2")
+
+
+class Cmnli(AbsTaskPairClassification):
+    metadata = TaskMetadata(
+        name="Cmnli",
+        description="Chinese Multi-Genre NLI",
+        reference="https://huggingface.co/datasets/clue/viewer/cmnli",
+        dataset={
+            "path": "C-MTEB/CMNLI",
+            "revision": "41bc36f332156f7adc9e38f53777c959b2ae9766",
+        },
+        type="PairClassification",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation", "test"],
+        eval_langs=["cmn-Hans"],
+        main_score="max_accuracy",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@inproceedings{xu-etal-2020-clue,
+            title = "{CLUE}: A {C}hinese Language Understanding Evaluation Benchmark",
+            author = "Xu, Liang  and
+            Hu, Hai  and
+            Zhang, Xuanwei  and
+            Li, Lu  and
+            Cao, Chenjie  and
+            Li, Yudong  and
+            Xu, Yechen  and
+            Sun, Kai  and
+            Yu, Dian  and
+            Yu, Cong  and
+            Tian, Yin  and
+            Dong, Qianqian  and
+            Liu, Weitang  and
+            Shi, Bo  and
+            Cui, Yiming  and
+            Li, Junyi  and
+            Zeng, Jun  and
+            Wang, Rongzhao  and
+            Xie, Weijian  and
+            Li, Yanting  and
+            Patterson, Yina  and
+            Tian, Zuoyu  and
+            Zhang, Yiwen  and
+            Zhou, He  and
+            Liu, Shaoweihua  and
+            Zhao, Zhe  and
+            Zhao, Qipeng  and
+            Yue, Cong  and
+            Zhang, Xinrui  and
+            Yang, Zhengliang  and
+            Richardson, Kyle  and
+            Lan, Zhenzhong",
+            booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
+            month = dec,
+            year = "2020",
+            address = "Barcelona, Spain (Online)",
+            publisher = "International Committee on Computational Linguistics",
+            url = "https://aclanthology.org/2020.coling-main.419",
+            doi = "10.18653/v1/2020.coling-main.419",
+            pages = "4762--4772",
+        }""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    def dataset_transform(self):
+        self.dataset = self.dataset.rename_column("sent1", "sentence1")
+        self.dataset = self.dataset.rename_column("sent2", "sentence2")
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/Reranking.py b/evalscope/backend/rag_eval/cmteb/tasks/Reranking.py
new file mode 100644
index 00000000..0b61d83a
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/Reranking.py
@@ -0,0 +1,153 @@
+from typing import Any
+from mteb.abstasks.AbsTask import ScoresDict
+from mteb.abstasks.AbsTaskReranking import AbsTaskReranking
+from mteb.abstasks.TaskMetadata import HFSubset, TaskMetadata
+from mteb.encoder_interface import Encoder, EncoderWithQueryCorpusEncode
+
+
+class T2Reranking(AbsTaskReranking):
+    metadata = TaskMetadata(
+        name="T2Reranking",
+        description="T2Ranking: A large-scale Chinese Benchmark for Passage Ranking",
+        reference="https://arxiv.org/abs/2304.03679",
+        dataset={
+            "path": "C-MTEB/T2Reranking",
+            "revision": "76631901a18387f85eaa53e5450019b87ad58ef9",
+        },
+        type="Reranking",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="map",
+        date=None,
+        form=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xie2023t2ranking,
+      title={T2Ranking: A large-scale Chinese Benchmark for Passage Ranking}, 
+      author={Xiaohui Xie and Qian Dong and Bingning Wang and Feiyang Lv and Ting Yao and Weinan Gan and Zhijing Wu and Xiangsheng Li and Haitao Li and Yiqun Liu and Jin Ma},
+      year={2023},
+      eprint={2304.03679},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR}
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+
+class MMarcoReranking(AbsTaskReranking):
+    metadata = TaskMetadata(
+        name="MMarcoReranking",
+        description="mMARCO is a multilingual version of the MS MARCO passage ranking dataset",
+        reference="https://github.com/unicamp-dl/mMARCO",
+        dataset={
+            "path": "C-MTEB/Mmarco-reranking",
+            "revision": "8e0c766dbe9e16e1d221116a3f36795fbade07f6",
+        },
+        type="Reranking",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="map",
+        date=None,
+        form=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{bonifacio2021mmarco,
+      title={mMARCO: A Multilingual Version of MS MARCO Passage Ranking Dataset}, 
+      author={Luiz Henrique Bonifacio and Vitor Jeronymo and Hugo Queiroz Abonizio and Israel Campiotti and Marzieh Fadaee and  and Roberto Lotufo and Rodrigo Nogueira},
+      year={2021},
+      eprint={2108.13897},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+
+class CMedQAv1(AbsTaskReranking):
+    metadata = TaskMetadata(
+        name="CMedQAv1",
+        description="Chinese community medical question answering",
+        reference="https://github.com/zhangsheng93/cMedQA",
+        dataset={
+            "path": "C-MTEB/CMedQAv1-reranking",
+            "revision": "8d7f1e942507dac42dc58017c1a001c3717da7df",
+        },
+        type="Reranking",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="map",
+        date=("2017-01-01", "2017-07-26"),
+        domains=["Medical", "Written"],
+        task_subtypes=[],
+        license="Not specified",
+        annotations_creators="expert-annotated",
+        dialect=[],
+        sample_creation="found",
+        bibtex_citation="""@article{zhang2017chinese,
+  title={Chinese Medical Question Answer Matching Using End-to-End Character-Level Multi-Scale CNNs},
+  author={Zhang, Sheng and Zhang, Xin and Wang, Hui and Cheng, Jiajun and Li, Pei and Ding, Zhaoyun},
+  journal={Applied Sciences},
+  volume={7},
+  number={8},
+  pages={767},
+  year={2017},
+  publisher={Multidisciplinary Digital Publishing Institute}
+}""",
+        descriptive_stats={
+            "n_samples": {"test": 2000},
+            "avg_character_length": {"test": 165},
+        },
+    )
+
+
+class CMedQAv2(AbsTaskReranking):
+    metadata = TaskMetadata(
+        name="CMedQAv2",
+        description="Chinese community medical question answering",
+        reference="https://github.com/zhangsheng93/cMedQA2",
+        dataset={
+            "path": "C-MTEB/CMedQAv2-reranking",
+            "revision": "23d186750531a14a0357ca22cd92d712fd512ea0",
+        },
+        type="Reranking",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="map",
+        date=None,
+        form=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@ARTICLE{8548603, 
+author={S. Zhang and X. Zhang and H. Wang and L. Guo and S. Liu}, 
+journal={IEEE Access}, 
+title={Multi-Scale Attentive Interaction Networks for Chinese Medical Question Answer Selection}, 
+year={2018}, 
+volume={6}, 
+number={}, 
+pages={74061-74071}, 
+keywords={Biomedical imaging;Data mining;Semantics;Medical services;Feature extraction;Knowledge discovery;Medical question answering;interactive attention;deep learning;deep neural networks}, 
+doi={10.1109/ACCESS.2018.2883637}, 
+ISSN={2169-3536}, 
+month={},}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/Retrieval.py b/evalscope/backend/rag_eval/cmteb/tasks/Retrieval.py
new file mode 100644
index 00000000..0fb3e221
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/Retrieval.py
@@ -0,0 +1,345 @@
+from mteb import AbsTaskRetrieval
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+
+class T2Retrieval(AbsTaskRetrieval):
+    ignore_identical_ids = True
+
+    metadata = TaskMetadata(
+        name="T2Retrieval",
+        description="T2Ranking: A large-scale Chinese Benchmark for Passage Ranking",
+        reference="https://arxiv.org/abs/2304.03679",
+        dataset={
+            "path": "C-MTEB/T2Retrieval",
+            "revision": "8731a845f1bf500a4f111cf1070785c793d10e64",
+            "qrel_revision": "1c83b8d1544e529875e3f6930f3a1fcf749a8e97",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xie2023t2ranking,
+      title={T2Ranking: A large-scale Chinese Benchmark for Passage Ranking}, 
+      author={Xiaohui Xie and Qian Dong and Bingning Wang and Feiyang Lv and Ting Yao and Weinan Gan and Zhijing Wu and Xiangsheng Li and Haitao Li and Yiqun Liu and Jin Ma},
+      year={2023},
+      eprint={2304.03679},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR}
+}""",
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 874.1184182791619,
+                    "average_query_length": 10.938847974750132,
+                    "num_documents": 118605,
+                    "num_queries": 22812,
+                    "average_relevant_docs_per_query": 5.213571804313519,
+                }
+            },
+        },
+    )
+
+
+class MMarcoRetrieval(AbsTaskRetrieval):
+    ignore_identical_ids = True
+
+    metadata = TaskMetadata(
+        name="MMarcoRetrieval",
+        description="MMarcoRetrieval",
+        reference="https://arxiv.org/abs/2309.07597",
+        dataset={
+            "path": "C-MTEB/MMarcoRetrieval",
+            "revision": "539bbde593d947e2a124ba72651aafc09eb33fc2",
+            "qrel_revision": "bae08bb7bddbedb96c7e7db52018a55167b67f89",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xiao2024cpack,
+      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
+      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff and Defu Lian and Jian-Yun Nie},
+      year={2024},
+      eprint={2309.07597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}""",
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 114.41787048392986,
+                    "average_query_length": 10.51131805157593,
+                    "num_documents": 106813,
+                    "num_queries": 6980,
+                    "average_relevant_docs_per_query": 1.0654727793696275,
+                }
+            },
+        },
+    )
+
+
+class DuRetrieval(AbsTaskRetrieval):
+    metadata = TaskMetadata(
+        name="DuRetrieval",
+        description="A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine",
+        reference="https://aclanthology.org/2022.emnlp-main.357.pdf",
+        dataset={
+            "path": "C-MTEB/DuRetrieval",
+            "revision": "a1a333e290fe30b10f3f56498e3a0d911a693ced",
+            "qrel_revision": "497b7bd1bbb25cb3757ff34d95a8be50a3de2279",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{qiu2022dureaderretrieval,
+      title={DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine}, 
+      author={Yifu Qiu and Hongyu Li and Yingqi Qu and Ying Chen and Qiaoqiao She and Jing Liu and Hua Wu and Haifeng Wang},
+      year={2022},
+      eprint={2203.10232},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}""",
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 331.3219967800322,
+                    "average_query_length": 9.289,
+                    "num_documents": 100001,
+                    "num_queries": 2000,
+                    "average_relevant_docs_per_query": 4.9195,
+                }
+            },
+        },
+    )
+
+
+class CovidRetrieval(AbsTaskRetrieval):
+    metadata = TaskMetadata(
+        name="CovidRetrieval",
+        description="COVID-19 news articles",
+        reference="https://arxiv.org/abs/2203.03367",
+        dataset={
+            "path": "C-MTEB/CovidRetrieval",
+            "revision": "1271c7809071a13532e05f25fb53511ffce77117",
+            "qrel_revision": "a9f41b7cdf24785531d12417ce0d1157ed4b39ca",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 332.4152658473415,
+                    "average_query_length": 25.9304531085353,
+                    "num_documents": 100001,
+                    "num_queries": 949,
+                    "average_relevant_docs_per_query": 1.0105374077976819,
+                }
+            },
+        },
+    )
+
+
+class CmedqaRetrieval(AbsTaskRetrieval):
+    metadata = TaskMetadata(
+        name="CmedqaRetrieval",
+        description="Online medical consultation text. Used the CMedQAv2 as its underlying dataset.",
+        reference="https://aclanthology.org/2022.emnlp-main.357.pdf",
+        dataset={
+            "path": "C-MTEB/CmedqaRetrieval",
+            "revision": "cd540c506dae1cf9e9a59c3e06f42030d54e7301",
+            "qrel_revision": "279d737f36c731c8ff6e2b055f31fe02216fa23d",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 307.7710222897771,
+                    "average_query_length": 48.470367591897976,
+                    "num_documents": 100001,
+                    "num_queries": 3999,
+                    "average_relevant_docs_per_query": 1.86271567891973,
+                }
+            },
+        },
+    )
+
+
+class EcomRetrieval(AbsTaskRetrieval):
+    ignore_identical_ids = True
+
+    metadata = TaskMetadata(
+        name="EcomRetrieval",
+        description="EcomRetrieval",
+        reference="https://arxiv.org/abs/2203.03367",
+        dataset={
+            "path": "C-MTEB/EcomRetrieval",
+            "revision": "687de13dc7294d6fd9be10c6945f9e8fec8166b9",
+            "qrel_revision": "39c90699b034ec22ac45b3abf5b0bbb5ffd421f9",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 32.98041664189015,
+                    "average_query_length": 6.798,
+                    "num_documents": 100902,
+                    "num_queries": 1000,
+                    "average_relevant_docs_per_query": 1.0,
+                }
+            },
+        },
+    )
+
+
+class MedicalRetrieval(AbsTaskRetrieval):
+    ignore_identical_ids = True
+
+    metadata = TaskMetadata(
+        name="MedicalRetrieval",
+        description="MedicalRetrieval",
+        reference="https://arxiv.org/abs/2203.03367",
+        dataset={
+            "path": "C-MTEB/MedicalRetrieval",
+            "revision": "2039188fb5800a9803ba5048df7b76e6fb151fc6",
+            "qrel_revision": "37b8efec53c54c3d9c6af212f6710b62ccdf895c",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 122.04231725066585,
+                    "average_query_length": 17.938,
+                    "num_documents": 100999,
+                    "num_queries": 1000,
+                    "average_relevant_docs_per_query": 1.0,
+                }
+            },
+        },
+    )
+
+
+class VideoRetrieval(AbsTaskRetrieval):
+    ignore_identical_ids = True
+
+    metadata = TaskMetadata(
+        name="VideoRetrieval",
+        description="VideoRetrieval",
+        reference="https://arxiv.org/abs/2203.03367",
+        dataset={
+            "path": "C-MTEB/VideoRetrieval",
+            "revision": "58c2597a5943a2ba48f4668c3b90d796283c5639",
+            "qrel_revision": "faa71382b6a29cf1778d1f436b963e75cb5b927c",
+        },
+        type="Retrieval",
+        category="s2p",
+        modalities=["text"],
+        eval_splits=["dev"],
+        eval_langs=["cmn-Hans"],
+        main_score="ndcg_at_10",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={
+            "n_samples": None,
+            "avg_character_length": {
+                "dev": {
+                    "average_document_length": 31.048855642524522,
+                    "average_query_length": 7.365,
+                    "num_documents": 100930,
+                    "num_queries": 1000,
+                    "average_relevant_docs_per_query": 1.0,
+                }
+            },
+        },
+    )
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/STS.py b/evalscope/backend/rag_eval/cmteb/tasks/STS.py
new file mode 100644
index 00000000..f21aeb72
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/STS.py
@@ -0,0 +1,302 @@
+from mteb.abstasks.AbsTaskSTS import AbsTaskSTS
+from mteb.abstasks.TaskMetadata import TaskMetadata
+
+class ATEC(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="ATEC",
+        dataset={
+            "path": "C-MTEB/ATEC",
+            "revision": "0f319b1142f28d00e055a6770f3f726ae9b7d865",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation", "test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@inproceedings{raghu-etal-2021-end,
+    title = "End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs",
+    author = "Raghu, Dinesh  and
+      Agarwal, Shantanu  and
+      Joshi, Sachindra  and
+      {Mausam}",
+    editor = "Moens, Marie-Francine  and
+      Huang, Xuanjing  and
+      Specia, Lucia  and
+      Yih, Scott Wen-tau",
+    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
+    month = nov,
+    year = "2021",
+    address = "Online and Punta Cana, Dominican Republic",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.emnlp-main.357",
+    doi = "10.18653/v1/2021.emnlp-main.357",
+    pages = "4348--4366",
+    abstract = "We propose a novel problem within end-to-end learning of task oriented dialogs (TOD), in which the dialog system mimics a troubleshooting agent who helps a user by diagnosing their problem (e.g., car not starting). Such dialogs are grounded in domain-specific flowcharts, which the agent is supposed to follow during the conversation. Our task exposes novel technical challenges for neural TOD, such as grounding an utterance to the flowchart without explicit annotation, referring to additional manual pages when user asks a clarification question, and ability to follow unseen flowcharts at test time. We release a dataset (FLODIAL) consisting of 2,738 dialogs grounded on 12 different troubleshooting flowcharts. We also design a neural model, FLONET, which uses a retrieval-augmented generation architecture to train the dialog agent. Our experiments find that FLONET can do zero-shot transfer to unseen flowcharts, and sets a strong baseline for future research.",
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 1
+        return metadata_dict
+
+
+class BQ(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="BQ",
+        dataset={
+            "path": "C-MTEB/BQ",
+            "revision": "e3dda5e115e487b39ec7e618c0c6a29137052a55",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation", "test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xiao2024cpackpackagedresourcesadvance,
+      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
+      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff and Defu Lian and Jian-Yun Nie},
+      year={2024},
+      eprint={2309.07597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2309.07597}, 
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 1
+        return metadata_dict
+
+
+class LCQMC(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="LCQMC",
+        dataset={
+            "path": "C-MTEB/LCQMC",
+            "revision": "17f9b096f80380fce5ed12a9be8be7784b337daf",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xiao2024cpackpackagedresourcesadvance,
+      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
+      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff and Defu Lian and Jian-Yun Nie},
+      year={2024},
+      eprint={2309.07597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2309.07597}, 
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 1
+        return metadata_dict
+
+
+class PAWSX(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="PAWSX",
+        dataset={
+            "path": "C-MTEB/PAWSX",
+            "revision": "9c6a90e430ac22b5779fb019a23e820b11a8b5e1",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xiao2024cpackpackagedresourcesadvance,
+      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
+      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff and Defu Lian and Jian-Yun Nie},
+      year={2024},
+      eprint={2309.07597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2309.07597}, 
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 1
+        return metadata_dict
+
+
+class STSB(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="STSB",
+        dataset={
+            "path": "C-MTEB/STSB",
+            "revision": "0cde68302b3541bb8b3c340dc0644b0b745b3dc0",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation", "test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@misc{xiao2024cpackpackagedresourcesadvance,
+      title={C-Pack: Packaged Resources To Advance General Chinese Embedding}, 
+      author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff and Defu Lian and Jian-Yun Nie},
+      year={2024},
+      eprint={2309.07597},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2309.07597}, 
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 5
+        return metadata_dict
+
+
+class AFQMC(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="AFQMC",
+        dataset={
+            "path": "C-MTEB/AFQMC",
+            "revision": "b44c3b011063adb25877c13823db83bb193913c4",
+        },
+        description="A Chinese dataset for textual relatedness",
+        reference="https://aclanthology.org/2021.emnlp-main.357",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["validation"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation="""@inproceedings{raghu-etal-2021-end,
+    title = "End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs",
+    author = "Raghu, Dinesh  and
+      Agarwal, Shantanu  and
+      Joshi, Sachindra  and
+      {Mausam}",
+    editor = "Moens, Marie-Francine  and
+      Huang, Xuanjing  and
+      Specia, Lucia  and
+      Yih, Scott Wen-tau",
+    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
+    month = nov,
+    year = "2021",
+    address = "Online and Punta Cana, Dominican Republic",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.emnlp-main.357",
+    doi = "10.18653/v1/2021.emnlp-main.357",
+    pages = "4348--4366",
+    abstract = "We propose a novel problem within end-to-end learning of task oriented dialogs (TOD), in which the dialog system mimics a troubleshooting agent who helps a user by diagnosing their problem (e.g., car not starting). Such dialogs are grounded in domain-specific flowcharts, which the agent is supposed to follow during the conversation. Our task exposes novel technical challenges for neural TOD, such as grounding an utterance to the flowchart without explicit annotation, referring to additional manual pages when user asks a clarification question, and ability to follow unseen flowcharts at test time. We release a dataset (FLODIAL) consisting of 2,738 dialogs grounded on 12 different troubleshooting flowcharts. We also design a neural model, FLONET, which uses a retrieval-augmented generation architecture to train the dialog agent. Our experiments find that FLONET can do zero-shot transfer to unseen flowcharts, and sets a strong baseline for future research.",
+}""",
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
+
+    @property
+    def metadata_dict(self) -> dict[str, str]:
+        metadata_dict = super().metadata_dict
+        metadata_dict["min_score"] = 0
+        metadata_dict["max_score"] = 1
+        return metadata_dict
+
+
+class QBQTC(AbsTaskSTS):
+    metadata = TaskMetadata(
+        name="QBQTC",
+        dataset={
+            "path": "C-MTEB/QBQTC",
+            "revision": "790b0510dc52b1553e8c49f3d2afb48c0e5c48b7",
+        },
+        description="",
+        reference="https://github.com/CLUEbenchmark/QBQTC/tree/main/dataset",
+        type="STS",
+        category="s2s",
+        modalities=["text"],
+        eval_splits=["test"],
+        eval_langs=["cmn-Hans"],
+        main_score="cosine_spearman",
+        date=None,
+        domains=None,
+        task_subtypes=None,
+        license=None,
+        annotations_creators=None,
+        dialect=None,
+        sample_creation=None,
+        bibtex_citation=None,
+        descriptive_stats={"n_samples": None, "avg_character_length": None},
+    )
diff --git a/evalscope/backend/rag_eval/cmteb/tasks/__init__.py b/evalscope/backend/rag_eval/cmteb/tasks/__init__.py
new file mode 100644
index 00000000..5dc0623a
--- /dev/null
+++ b/evalscope/backend/rag_eval/cmteb/tasks/__init__.py
@@ -0,0 +1,64 @@
+from .Classification import *
+from .Clustering import *
+from .PairClassification import *
+from .Reranking import *
+from .Retrieval import *
+from .STS import *
+
+CLS_CLASSIFICATION = {
+    "TNews": TNews,
+    "IFlyTek": IFlyTek,
+    "MultilingualSentiment": MultilingualSentiment,
+    "JDReview": JDReview,
+    "OnlineShopping": OnlineShopping,
+    "Waimai": Waimai,
+}
+
+CLS_CLUSTERING = {
+    "CLSClusteringS2S": CLSClusteringFastS2S,
+    "CLSClusteringP2P": CLSClusteringFastP2P,
+    "ThuNewsClusteringS2S": ThuNewsClusteringFastS2S,
+    "ThuNewsClusteringP2P": ThuNewsClusteringFastP2P,
+}
+
+CLS_PAIR_CLASSIFICATION = {
+    "Ocnli": Ocnli,
+    "Cmnli": Cmnli,
+}
+
+CLS_RERANKING = {
+    "T2Reranking": T2Reranking,
+    "MMarcoReranking": MMarcoReranking,
+    "CMedQAv1": CMedQAv1,
+    "CMedQAv2": CMedQAv2,
+}
+
+CLS_RETRIEVAL = {
+    "T2Retrieval": T2Retrieval,
+    "MMarcoRetrieval": MMarcoRetrieval,
+    "DuRetrieval": DuRetrieval,
+    "CovidRetrieval": CovidRetrieval,
+    "CmedqaRetrieval": CmedqaRetrieval,
+    "EcomRetrieval": EcomRetrieval,
+    "MedicalRetrieval": MedicalRetrieval,
+    "VideoRetrieval": VideoRetrieval,
+}
+
+CLS_STS = {
+    "ATEC": ATEC,
+    "BQ": BQ,
+    "LCQMC": LCQMC,
+    "PAWSX": PAWSX,
+    "STSB": STSB,
+    "AFQMC": AFQMC,
+    "QBQTC": QBQTC,
+}
+
+CLS_DICT = {
+    **CLS_CLASSIFICATION,
+    **CLS_CLUSTERING,
+    **CLS_PAIR_CLASSIFICATION,
+    **CLS_RERANKING,
+    **CLS_RETRIEVAL,
+    **CLS_STS,
+}
diff --git a/evalscope/backend/rag_eval/ragas/__init__.py b/evalscope/backend/rag_eval/ragas/__init__.py
new file mode 100644
index 00000000..63874c62
--- /dev/null
+++ b/evalscope/backend/rag_eval/ragas/__init__.py
@@ -0,0 +1,2 @@
+from evalscope.backend.rag_eval.ragas.arguments import TestsetGenerationArguments, EvaluationArguments
+from evalscope.backend.rag_eval.ragas.task_template import testset_generation, rag_eval
\ No newline at end of file
diff --git a/evalscope/backend/rag_eval/ragas/arguments.py b/evalscope/backend/rag_eval/ragas/arguments.py
new file mode 100644
index 00000000..1eff8a94
--- /dev/null
+++ b/evalscope/backend/rag_eval/ragas/arguments.py
@@ -0,0 +1,37 @@
+from dataclasses import dataclass, field
+from typing import List, Optional, Union, Dict, Any
+
+
+@dataclass
+class TestsetGenerationArguments:
+    docs: List[str] = field(default_factory=list)
+    test_size: int = 10
+    output_file: str = "outputs/testset.json"
+    """
+    For local LLM support, you can use the following fields:
+        model_name_or_path: str
+        model_revision: str = "master"
+        template_type: str = "default"
+        generation_config: Optional[Dict]
+        
+    For API LLM support, you can use the following fields:
+        model_name="gpt-4o-mini"
+        api_base: str = "",
+        api_key: Optional[str] = None
+    """
+    generator_llm: Dict = field(default_factory=dict)
+    critic_llm: Dict = field(default_factory=dict)
+    embeddings: Dict = field(default_factory=dict)
+    distribution: str = field(
+        default_factory=lambda: {"simple": 0.5, "multi_context": 0.4, "reasoning": 0.1}
+    )
+
+
+@dataclass
+class EvaluationArguments:
+    testset_file: str
+    critic_llm: Dict = field(default_factory=dict)
+    embeddings: Dict = field(default_factory=dict)
+    metrics: List[str] = field(
+        default_factory=lambda: ["answer_relevancy", "faithfulness"]
+    )
diff --git a/evalscope/backend/rag_eval/ragas/task_template.py b/evalscope/backend/rag_eval/ragas/task_template.py
new file mode 100644
index 00000000..204f2620
--- /dev/null
+++ b/evalscope/backend/rag_eval/ragas/task_template.py
@@ -0,0 +1,117 @@
+import os
+from evalscope.backend.rag_eval import EmbeddingModel, LLM
+from evalscope.utils.logger import get_logger
+from .arguments import TestsetGenerationArguments, EvaluationArguments
+
+logger = get_logger()
+
+
+def testset_generation(args: TestsetGenerationArguments) -> None:
+    from langchain_community.document_loaders.unstructured import UnstructuredFileLoader
+    from ragas.testset.generator import TestsetGenerator
+    from ragas.testset.evolutions import simple, reasoning, multi_context
+    from ragas import RunConfig
+
+    # load data
+    file_path = args.docs
+    loader = UnstructuredFileLoader(file_path, mode="elements")
+    data = loader.load()
+
+    # generator with models
+    generator_llm = LLM.load(**args.generator_llm)
+    critic_llm = LLM.load(**args.critic_llm)
+    embeddings = EmbeddingModel.load(**args.embeddings)
+
+    # Change resulting question type distribution
+    distributions = {
+        simple: args.distribution["simple"],
+        multi_context: args.distribution["multi_context"],
+        reasoning: args.distribution["reasoning"],
+    }
+
+    generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
+
+    runconfig = RunConfig(timeout=30, max_retries=1, max_wait=30, max_workers=1)
+    testset = generator.generate_with_langchain_docs(
+        data,
+        args.test_size,
+        distributions,
+        with_debugging_logs=True,
+        is_async=False,
+        run_config=runconfig,
+    )
+
+    # save file
+    testset_df = testset.to_pandas()
+    output_path = os.path.dirname(args.output_file)
+    os.makedirs(output_path, exist_ok=True)
+    testset_df.to_json(args.output_file, indent=4, index=False, orient="records")
+
+    # get answer
+    testset_with_answer = get_answer(testset_df, generator_llm)
+    testset_with_answer.to_json(
+        args.output_file, indent=4, index=False, orient="records"
+    )
+
+
+def get_answer(testset_df, generator_llm):
+    template = """You are an assistant for question-answering tasks. 
+Use the following pieces of retrieved context to answer the question. 
+If you don't know the answer, just say that you don't know. 
+Use two sentences maximum and keep the answer concise.
+Question: {question} 
+Context: {contexts} 
+Answer:
+"""
+    answers = []
+    for index, row in testset_df.iterrows():
+        question = row["question"]
+        contexts = "\n".join(row["contexts"])
+
+        # Combine question and contexts as input for the LLM
+        input_text = template.format(question=question, contexts=contexts)
+
+        # Generate the answer using the generator LLM
+        answer = generator_llm.invoke(input_text)
+        answers.append(answer)
+
+    testset_df["answer"] = answers
+    return testset_df
+
+
+def rag_eval(
+    args: EvaluationArguments,
+) -> None:
+    from datasets import Dataset
+    from ragas import evaluate
+    from evalscope.backend.rag_eval import EmbeddingModel, LLM
+    from ragas import RunConfig
+    import importlib
+
+    def dynamic_import(module_name, *function_names):
+        # 动态导入指定模块
+        module = importlib.import_module(module_name)
+
+        functions = [getattr(module, name) for name in function_names]
+        return functions
+
+    llm = LLM.load(**args.critic_llm)
+    embedding = EmbeddingModel.load(**args.embeddings)
+
+    dataset = Dataset.from_json(args.testset_file)
+
+    runconfig = RunConfig(timeout=30, max_retries=1, max_wait=30, max_workers=1)
+    score = evaluate(
+        dataset,
+        metrics=dynamic_import("ragas.metrics", *args.metrics),
+        llm=llm,
+        embeddings=embedding,
+        run_config=runconfig,
+    )
+    score_df = score.to_pandas()
+    # logger.info(score_df.to_string())
+
+    output_path = args.testset_file.split(".")[0] + "_score.json"
+    score_df.to_json(output_path, indent=4, index=False, orient="records")
+
+    logger.info(f"Eval score saved to {output_path}")
diff --git a/evalscope/backend/rag_eval/utils/embedding.py b/evalscope/backend/rag_eval/utils/embedding.py
new file mode 100644
index 00000000..e2ddd3dc
--- /dev/null
+++ b/evalscope/backend/rag_eval/utils/embedding.py
@@ -0,0 +1,160 @@
+import os
+from typing import List, Optional, Union
+from sentence_transformers import models
+from sentence_transformers.SentenceTransformer import SentenceTransformer
+from sentence_transformers.cross_encoder import CrossEncoder
+from torch import Tensor
+from evalscope.backend.rag_eval.utils.tools import download_model
+from evalscope.utils.logger import get_logger
+from langchain_core.embeddings import Embeddings
+
+logger = get_logger()
+
+
+class BaseModel(Embeddings):
+    def __init__(
+        self,
+        model_name_or_path: str,
+        max_seq_length: int = 512,
+        prompt: str = "",
+        revision: Optional[str] = None,
+        **kwargs,
+    ):
+        self.model_name_or_path = model_name_or_path
+        self.max_seq_length = max_seq_length
+        self.model_kwargs = kwargs.pop("model_kwargs", {})
+        self.model_kwargs["trust_remote_code"] = True
+
+        self.config_kwargs = kwargs.pop("config_kwargs", {})
+        self.config_kwargs["trust_remote_code"] = True
+
+        self.encode_kwargs = kwargs.pop("encode_kwargs", {})
+        self.encode_kwargs["convert_to_tensor"] = True
+
+        self.prompt = prompt
+        self.revision = revision
+
+    @property
+    def mteb_model_meta(self):
+        """Model metadata for MTEB (Multilingual Task Embeddings Benchmark)"""
+        from mteb import ModelMeta
+
+        return ModelMeta(
+            name=os.path.basename(self.model_name_or_path),
+            revision=self.revision,
+            languages=None,
+            release_date=None,
+        )
+
+    def embed_documents(self, texts: List[str]) -> List[List[float]]:
+        """Embed search docs.
+
+        Args:
+            texts: List of text to embed.
+
+        Returns:
+            List of embeddings.
+        """
+        return self.encode(texts).tolist()
+
+    def embed_query(self, text: str) -> List[float]:
+        """Embed query text.
+
+        Args:
+            text: Text to embed.
+
+        Returns:
+            Embedding.
+        """
+        return self.encode(text).tolist()
+
+    def encode(self, texts: Union[str, List[str]], **kwargs) -> List[List[float]]:
+        """Embed query text."""
+        raise NotImplementedError
+
+
+class SentenceTransformerModel(BaseModel):
+    def __init__(
+        self, model_name_or_path: str, pooling_mode: Optional[str] = None, **kwargs
+    ):
+        super().__init__(model_name_or_path, **kwargs)
+
+        if not pooling_mode:
+            self.model = SentenceTransformer(
+                self.model_name_or_path,
+                config_kwargs=self.config_kwargs,
+                model_kwargs=self.model_kwargs,
+            )
+        else:
+            word_embedding_model = models.Transformer(
+                self.model_name_or_path,
+                config_args=self.config_kwargs,
+                model_args=self.model_kwargs,
+            )
+            pooling_model = models.Pooling(
+                word_embedding_model.get_word_embedding_dimension(),
+                pooling_mode=pooling_mode,
+            )
+            self.model = SentenceTransformer(
+                modules=[word_embedding_model, pooling_model],
+            )
+
+        self.model.max_seq_length = self.max_seq_length
+
+    def encode(self, texts: Union[str, List[str]], **kwargs) -> List[List[float]]:
+        kwargs.pop("prompt_name", "")  # remove prompt name, use prompt
+        self.encode_kwargs.update(kwargs)
+        embeddings = self.model.encode(texts, prompt=self.prompt, **self.encode_kwargs)
+        assert isinstance(embeddings, Tensor)
+        return embeddings
+
+
+class CrossEncoderModel(BaseModel):
+    def __init__(self, model_name_or_path: str, **kwargs):
+        super().__init__(model_name_or_path, **kwargs)
+        self.model = CrossEncoder(
+            self.model_name_or_path,
+            trust_remote_code=True,
+            max_length=self.max_seq_length,
+        )
+
+    def predict(self, sentences: List[List[str]], **kwargs) -> List[List[float]]:
+        self.encode_kwargs.update(kwargs)
+
+        if len(sentences[0]) == 3:  # Note: For mteb retrieval task
+            sentences = [
+                (self.prompt + query, docs) for query, docs, instruction in sentences
+            ]
+        embeddings = self.model.predict(sentences, **self.encode_kwargs)
+        assert isinstance(embeddings, Tensor)
+        return embeddings
+
+
+class EmbeddingModel:
+    """Custom embeddings"""
+
+    @staticmethod
+    def load(
+        model_name_or_path: str = "",
+        is_cross_encoder: bool = False,
+        hub: str = "modelscope",
+        revision: Optional[str] = "master",
+        **kwargs,
+    ):
+        # If model path does not exist and hub is 'modelscope', download the model
+        if not os.path.exists(model_name_or_path) and hub == "modelscope":
+            model_name_or_path = download_model(model_name_or_path, revision)
+
+        # Return different model instances based on whether it is a cross-encoder and pooling mode
+        if is_cross_encoder:
+            return CrossEncoderModel(
+                model_name_or_path,
+                revision=revision,
+                **kwargs,
+            )
+        else:
+            return SentenceTransformerModel(
+                model_name_or_path,
+                revision=revision,
+                **kwargs,
+            )
diff --git a/evalscope/backend/rag_eval/utils/llm.py b/evalscope/backend/rag_eval/utils/llm.py
new file mode 100644
index 00000000..d47432f6
--- /dev/null
+++ b/evalscope/backend/rag_eval/utils/llm.py
@@ -0,0 +1,73 @@
+import os
+from typing import Any, Dict, Iterator, List, Mapping, Optional
+
+from langchain_core.callbacks.manager import CallbackManagerForLLMRun
+from langchain_core.language_models.llms import LLM as BaseLLM
+from evalscope.models.model_adapter import ChatGenerationModelAdapter
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
+
+
+class LLM:
+    @staticmethod
+    def load(**kw):
+        api_base = kw.get("api_base", None)
+        if api_base:
+            return ChatOpenAI(
+                model_name=kw.get("model_name", ""),
+                openai_api_base=api_base,
+                openai_api_key=kw.get("api_key", ""),
+            )
+        else:
+            return LocalLLM(**kw)
+
+
+class LocalLLM(BaseLLM):
+    """A custom LLM that loads a model from a given path and performs inference."""
+
+    model_name_or_path: str
+    model_revision: str = "master"
+    template_type: str = "default"
+    model_name: Optional[str]
+    model: Optional[ChatGenerationModelAdapter]
+    generation_config: Optional[Dict]
+
+    def __init__(self, **kw):
+        super().__init__(**kw)
+        self.model_name = os.path.basename(self.model_name_or_path)
+        self.model = ChatGenerationModelAdapter(
+            model_id=self.model_name_or_path,
+            model_revision=self.model_revision,
+            template_type=self.template_type,
+            generation_config=self.generation_config,
+        )
+
+    def _call(
+        self,
+        prompt: str,
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[CallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> str:
+        """Run the LLM on the given input."""
+        infer_cfg = {"stop": stop}
+
+        response = self.model._model_generate(prompt, infer_cfg)
+        return response
+
+    @property
+    def _identifying_params(self) -> Dict[str, Any]:
+        """Return a dictionary of identifying parameters."""
+        return {
+            # The model name allows users to specify custom token counting
+            # rules in LLM monitoring applications (e.g., in LangSmith users
+            # can provide per token pricing for their model and monitor
+            # costs for the given LLM.)
+            "model_name": self.model_name,
+            "revision": self.model_revision,
+        }
+
+    @property
+    def _llm_type(self) -> str:
+        """Get the type of language model used by this chat model. Used for logging purposes only."""
+        return self.model_name
+
diff --git a/evalscope/backend/rag_eval/utils/tools.py b/evalscope/backend/rag_eval/utils/tools.py
new file mode 100644
index 00000000..c3afb03c
--- /dev/null
+++ b/evalscope/backend/rag_eval/utils/tools.py
@@ -0,0 +1,15 @@
+from modelscope import snapshot_download
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def download_model(model_id: str, revision: str):
+    """
+    default base dir: '~/.cache/modelscope/hub/model_id'
+    """
+    logger.info(f"Loading model {model_id} from modelscope")
+
+    model_path = snapshot_download(model_id=model_id, revision=revision)
+
+    return model_path
\ No newline at end of file
diff --git a/evalscope/backend/vlm_eval_kit/backend_manager.py b/evalscope/backend/vlm_eval_kit/backend_manager.py
index 3face706..63b18cc9 100644
--- a/evalscope/backend/vlm_eval_kit/backend_manager.py
+++ b/evalscope/backend/vlm_eval_kit/backend_manager.py
@@ -4,7 +4,6 @@
 from evalscope.utils.logger import get_logger
 from functools import partial
 import subprocess
-from dataclasses import dataclass
 import copy
 
 logger = get_logger()
diff --git a/evalscope/backend/vlm_eval_kit/custom_dataset.py b/evalscope/backend/vlm_eval_kit/custom_dataset.py
index 4a5b35a9..12916674 100644
--- a/evalscope/backend/vlm_eval_kit/custom_dataset.py
+++ b/evalscope/backend/vlm_eval_kit/custom_dataset.py
@@ -8,7 +8,7 @@ class CustomDataset:
 
     def load_data(self, dataset):
         # customize the loading of the dataset
-        data_path = os.path.join("~/LMUData", f'{dataset}.tsv')
+        data_path = os.path.join(os.path.expanduser("~/LMUData"), f'{dataset}.tsv')
         return load(data_path)
 
 
diff --git a/evalscope/run.py b/evalscope/run.py
index 90263ebc..3f84c3ca 100644
--- a/evalscope/run.py
+++ b/evalscope/run.py
@@ -207,6 +207,10 @@ def run_task(task_cfg: Union[str, dict, TaskConfig, List[TaskConfig]]) -> Union[
             from evalscope.backend.vlm_eval_kit import VLMEvalKitBackendManager
             vlm_eval_kit_backend_manager = VLMEvalKitBackendManager(config=eval_config)
             vlm_eval_kit_backend_manager.run()
+        elif eval_backend == EvalBackend.RAG_EVAL.value:
+            from evalscope.backend.rag_eval import RAGEvalBackendManager
+            rag_eval_backend_manager = RAGEvalBackendManager(config=eval_config)
+            rag_eval_backend_manager.run()
         # TODO: Add other evaluation backends
         elif eval_backend == EvalBackend.THIRD_PARTY.value:
             raise NotImplementedError(f'Not implemented for evaluation backend {eval_backend}')
diff --git a/evalscope/utils/logger.py b/evalscope/utils/logger.py
index d8a26b48..db87cc47 100644
--- a/evalscope/utils/logger.py
+++ b/evalscope/utils/logger.py
@@ -1,18 +1,20 @@
 # Copyright (c) Alibaba, Inc. and its affiliates.
 
+import importlib.util as iutil
 import logging
 from typing import Optional
 
 init_loggers = {}
+format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+formatter = logging.Formatter(format)
 
-formatter = logging.Formatter(
-    '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+logging.basicConfig(format=format, level=logging.INFO)
 
 
-def get_logger(log_file: Optional[str] = None,
-               log_level: int = logging.INFO,
-               file_mode: str = 'w'):
-    """ Get logging logger
+def get_logger(
+    log_file: Optional[str] = None, log_level: int = logging.INFO, file_mode: str = "w"
+):
+    """Get logging logger
 
     Args:
         log_file: Log filename, if specified, file handler will be added to
@@ -22,21 +24,39 @@ def get_logger(log_file: Optional[str] = None,
             specified (if filemode is unspecified, it defaults to 'w').
     """
 
-    logger_name = __name__.split('.')[0]
+    logger_name = __name__.split(".")[0]
     logger = logging.getLogger(logger_name)
-
+    logger.propagate = False
     if logger_name in init_loggers:
         add_file_handler_if_needed(logger, log_file, file_mode, log_level)
+        if logger.level != log_level:
+            logger.setLevel(log_level)
         return logger
 
-    for handler in logger.root.handlers:
-        if type(handler) is logging.StreamHandler:
-            handler.setLevel(logging.ERROR)
+    # handle duplicate logs to the console
+    # Starting in 1.8.0, PyTorch DDP attaches a StreamHandler <stderr> (NOTSET)
+    # to the root logger. As logger.propagate is True by default, this root
+    # level handler causes logging messages from rank>0 processes to
+    # unexpectedly show up on the console, creating much unwanted clutter.
+    # To fix this issue, we set the root logger's StreamHandler, if any, to log
+    # at the ERROR level.
+    torch_dist = False
+    is_worker0 = True
+    if iutil.find_spec("torch") is not None:
+        from modelscope.utils.torch_utils import is_dist, is_master
+
+        torch_dist = is_dist()
+        is_worker0 = is_master()
+
+    if torch_dist:
+        for handler in logger.root.handlers:
+            if type(handler) is logging.StreamHandler:
+                handler.setLevel(logging.ERROR)
 
     stream_handler = logging.StreamHandler()
     handlers = [stream_handler]
 
-    if log_file is not None:
+    if is_worker0 and log_file is not None:
         file_handler = logging.FileHandler(log_file, file_mode)
         handlers.append(file_handler)
 
@@ -45,7 +65,10 @@ def get_logger(log_file: Optional[str] = None,
         handler.setLevel(log_level)
         logger.addHandler(handler)
 
-    logger.setLevel(log_level)
+    if is_worker0:
+        logger.setLevel(log_level)
+    else:
+        logger.setLevel(logging.ERROR)
 
     init_loggers[logger_name] = True
 
@@ -57,7 +80,14 @@ def add_file_handler_if_needed(logger, log_file, file_mode, log_level):
         if isinstance(handler, logging.FileHandler):
             return
 
-    if log_file is not None:
+    if iutil.find_spec("torch") is not None:
+        from modelscope.utils.torch_utils import is_master
+
+        is_worker0 = is_master()
+    else:
+        is_worker0 = True
+
+    if is_worker0 and log_file is not None:
         file_handler = logging.FileHandler(log_file, file_mode)
         file_handler.setFormatter(formatter)
         file_handler.setLevel(log_level)
diff --git a/evalscope/utils/task_utils.py b/evalscope/utils/task_utils.py
index e66d81a0..289bd0e5 100644
--- a/evalscope/utils/task_utils.py
+++ b/evalscope/utils/task_utils.py
@@ -11,6 +11,9 @@ class EvalBackend(Enum):
     
     # Use VLM Eval Kit as the multi-modal model evaluation backend
     VLM_EVAL_KIT = 'VLMEvalKit'
+    
+    # Use RAGEval as the RAG evaluation backend
+    RAG_EVAL = 'RAGEval'
 
     # Use third-party evaluation backend/modules
     THIRD_PARTY = 'ThirdParty'
diff --git a/examples/example_eval_mteb.py b/examples/example_eval_mteb.py
new file mode 100644
index 00000000..0aefc499
--- /dev/null
+++ b/examples/example_eval_mteb.py
@@ -0,0 +1,93 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+
+"""
+1. Installation
+EvalScope: pip install mteb
+
+2. Run eval task
+"""
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+import torch
+
+logger = get_logger()
+
+
+def run_eval():
+
+    # Prepare the config
+
+    one_stage_task_cfg = {
+        "eval_backend": "RAGEval",
+        "eval_config": {
+            "tool": "MTEB",
+            "model": [
+                {
+                    "model_name_or_path": "AI-ModelScope/m3e-base",
+                    "pooling_mode": None,  # load from model config
+                    "max_seq_length": 512,
+                    "prompt": "",
+                    "model_kwargs": {"torch_dtype": "auto"},
+                    "encode_kwargs": {
+                        "batch_size": 128,
+                    },
+                }
+            ],
+            "eval": {
+                "tasks": [
+                    "TNews",
+                    "CLSClusteringS2S",
+                    "T2Reranking",
+                    "T2Retrieval",
+                    "ATEC",
+                ],
+                "verbosity": 2,
+                "output_folder": "outputs",
+                "overwrite_results": True,
+                "limits": 500,
+            },
+        },
+    }
+
+    two_stage_task_cfg = {
+        "eval_backend": "RAGEval",
+        "eval_config": {
+            "tool": "MTEB",
+            "model": [
+                {
+                    "model_name_or_path": "AI-ModelScope/m3e-base",
+                    "is_cross_encoder": False,
+                    "max_seq_length": 512,
+                    "prompt": "",
+                    "model_kwargs": {"torch_dtype": "auto"},
+                    "encode_kwargs": {
+                        "batch_size": 64,
+                    },
+                },
+                {
+                    "model_name_or_path": "OpenBMB/MiniCPM-Reranker",
+                    "is_cross_encoder": True,
+                    "max_seq_length": 512,
+                    "prompt": "为这个问题生成一个检索用的表示",
+                    "model_kwargs": {"torch_dtype": "auto"},
+                    "encode_kwargs": {
+                        "batch_size": 32,
+                    },
+                },
+            ],
+            "eval": {
+                "tasks": ["T2Retrieval"],
+                "verbosity": 2,
+                "output_folder": "outputs",
+                "overwrite_results": True,
+                "limits": 100,
+            },
+        },
+    }
+
+    # Run task
+    run_task(task_cfg=two_stage_task_cfg)
+
+
+if __name__ == "__main__":
+    run_eval()
diff --git a/examples/example_eval_rag.py b/examples/example_eval_rag.py
new file mode 100644
index 00000000..e04f85e2
--- /dev/null
+++ b/examples/example_eval_rag.py
@@ -0,0 +1,64 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+
+from evalscope.run import run_task
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+def run_eval():
+    generate_testset_task_cfg = {
+        "eval_backend": "RAGEval",
+        "eval_config": {
+            "tool": "RAGAS",
+            "testset_generation": {
+                "docs": ["README.md"],
+                "test_size": 5,
+                "output_file": "outputs/testset.json",
+                "distribution": {"simple": 0.5, "multi_context": 0.4, "reasoning": 0.1},
+                # The generator_llm is the component that generates the questions, and evolves the question to make it more relevant.
+                # The critic_llm is the component that filters the questions and nodes based on the question and node relevance.
+                "generator_llm": {
+                    "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                    "template_type": "qwen",
+                },
+                "critic_llm": {
+                    "model_name_or_path": "QwenCollection/Ragas-critic-llm-Qwen1.5-GPTQ",
+                    "template_type": "qwen",
+                },
+                "embeddings": {
+                    "model_name_or_path": "AI-ModelScope/m3e-base",
+                },
+            }
+        },
+    }
+    
+    eval_task_cfg = {
+        "eval_backend": "RAGEval",
+        "eval_config": {
+            "tool": "RAGAS",
+            "eval": {
+                "testset_file": "outputs/testset.json",
+                "critic_llm": {
+                    "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                    "template_type": "qwen",
+                },
+                "embeddings": {
+                    "model_name_or_path": "AI-ModelScope/m3e-base",
+                },
+                "metrics": [
+                    "faithfulness",
+                    "answer_relevancy",
+                    "context_precision",
+                    "answer_correctness",
+                ],
+            },
+        },
+    }
+
+    # Run task
+    run_task(task_cfg=eval_task_cfg) # or run_task(task_cfg=generate_testset_task_cfg)
+
+
+if __name__ == "__main__":
+    run_eval()
diff --git a/examples/tasks/eval_vlm_swift.yaml b/examples/tasks/eval_vlm_swift.yaml
index 9a6375b2..b27439cd 100644
--- a/examples/tasks/eval_vlm_swift.yaml
+++ b/examples/tasks/eval_vlm_swift.yaml
@@ -1,14 +1,18 @@
 eval_backend: VLMEvalKit
 eval_config:
   model: 
-    - type: qwen-vl-chat   # model id of the model
+    - type: internvl2-8b   # model id of the model
       name: CustomAPIModel # Don't change, must be CustomAPIModel for deploy evaluation
       api_base: http://localhost:8000/v1/chat/completions # deployed model api
       key: EMPTY
       temperature: 0.0
-      img_size: -1
+      img_size: 224
+      img_detail: low
   data:
-    - custom_vqa
+    - MVBench
+    # - Video-MME
+    # - MMBench-Video
+    # - custom_vqa
     # - MMStar
     # - COCO_VAL
     # - AI2D_TEST
@@ -21,6 +25,6 @@ eval_config:
   work_dir: outputs
   nproc: 1
   # judge model server config
-  # OPENAI_API_KEY: EMPTY
-  # OPENAI_API_BASE: http://localhost:8866/v1/chat/completions # judge model api
-  # LOCAL_LLM: qwen2-7b-instruct                               # judge model type
\ No newline at end of file
+  OPENAI_API_KEY: EMPTY
+  OPENAI_API_BASE: http://localhost:11434/v1/chat/completions # judge model api
+  LOCAL_LLM: llama3.1:latest                              # judge model type
\ No newline at end of file
diff --git a/requirements/rag.txt b/requirements/rag.txt
new file mode 100644
index 00000000..ea4283d2
--- /dev/null
+++ b/requirements/rag.txt
@@ -0,0 +1,2 @@
+ragas
+mteb>=0.14.16
\ No newline at end of file
diff --git a/setup.py b/setup.py
index a49e140e..9eac254b 100644
--- a/setup.py
+++ b/setup.py
@@ -143,11 +143,13 @@ def pack_resource():
     all_requires = []
     extra_requires['opencompass'], _ = parse_requirements('requirements/opencompass.txt')
     extra_requires['vlmeval'], _ = parse_requirements('requirements/vlmeval.txt')
+    extra_requires['rag'], _ = parse_requirements('requirements/rag.txt')
     extra_requires['inner'], _ = parse_requirements('requirements/inner.txt')
 
     all_requires.extend(install_requires)
     all_requires.extend(extra_requires['opencompass'])
     all_requires.extend(extra_requires['vlmeval'])
+    all_requires.extend(extra_requires['rag'])
     extra_requires['all'] = all_requires
 
     setup(


Test patch:
diff --git a/tests/rag/__init__.py b/tests/rag/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/rag/test_mteb.py b/tests/rag/test_mteb.py
new file mode 100644
index 00000000..d41def3e
--- /dev/null
+++ b/tests/rag/test_mteb.py
@@ -0,0 +1,105 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+
+import subprocess
+import unittest
+from evalscope.utils import test_level_list, is_module_installed
+from evalscope.utils.logger import get_logger
+from evalscope.run import run_task
+
+logger = get_logger()
+
+
+class TestMTEB(unittest.TestCase):
+
+    def setUp(self) -> None:
+        self._check_env("mteb")
+
+    def tearDown(self) -> None:
+        pass
+
+    @staticmethod
+    def _check_env(module_name: str):
+        if is_module_installed(module_name):
+            logger.info(f"{module_name} is installed.")
+        else:
+            raise ModuleNotFoundError(f"run: pip install {module_name}")
+
+    @unittest.skipUnless(0 in test_level_list(), "skip test in current test level")
+    def test_run_one_stage_mteb(self):
+        task_cfg = {
+            "eval_backend": "RAGEval",
+            "eval_config": {
+                "tool": "MTEB",
+                "model": [
+                    {
+                        "model_name_or_path": "AI-ModelScope/m3e-base",
+                        "pooling_mode": None,  # load from model config
+                        "max_seq_length": 512,
+                        "prompt": "",
+                        "model_kwargs": {"torch_dtype": "auto"},
+                        "encode_kwargs": {
+                            "batch_size": 128,
+                        },
+                    }
+                ],
+                "eval": {
+                    "tasks": [
+                        "TNews",
+                        "CLSClusteringS2S",
+                        "T2Reranking",
+                        "T2Retrieval",
+                        "ATEC",
+                    ],
+                    "verbosity": 2,
+                    "output_folder": "outputs",
+                    "overwrite_results": True,
+                    "limits": 500,
+                },
+            },
+        }
+
+        run_task(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), "skip test in current test level")
+    def test_run_two_stage_mteb(self):
+        task_cfg = {
+            "eval_backend": "RAGEval",
+            "eval_config": {
+                "tool": "MTEB",
+                "model": [
+                    {
+                        "model_name_or_path": "AI-ModelScope/m3e-base",
+                        "is_cross_encoder": False,
+                        "max_seq_length": 512,
+                        "prompt": "",
+                        "model_kwargs": {"torch_dtype": "auto"},
+                        "encode_kwargs": {
+                            "batch_size": 64,
+                        },
+                    },
+                    {
+                        "model_name_or_path": "OpenBMB/MiniCPM-Reranker",
+                        "is_cross_encoder": True,
+                        "max_seq_length": 512,
+                        "prompt": "为这个问题生成一个检索用的表示",
+                        "model_kwargs": {"torch_dtype": "auto"},
+                        "encode_kwargs": {
+                            "batch_size": 32,
+                        },
+                    },
+                ],
+                "eval": {
+                    "tasks": ["T2Retrieval"],
+                    "verbosity": 2,
+                    "output_folder": "outputs",
+                    "overwrite_results": True,
+                    "limits": 100,
+                },
+            },
+        }
+
+        run_task(task_cfg)
+
+
+if __name__ == "__main__":
+    unittest.main(buffer=False)
diff --git a/tests/rag/test_ragas.py b/tests/rag/test_ragas.py
new file mode 100644
index 00000000..7ff9d64c
--- /dev/null
+++ b/tests/rag/test_ragas.py
@@ -0,0 +1,123 @@
+# Copyright (c) Alibaba, Inc. and its affiliates.
+import os
+import unittest
+from evalscope.utils import test_level_list, is_module_installed
+from evalscope.utils.logger import get_logger
+from evalscope.run import run_task
+
+logger = get_logger()
+
+
+class TestRAGAS(unittest.TestCase):
+
+    def setUp(self) -> None:
+        self._check_env("ragas")
+
+    def tearDown(self) -> None:
+        pass
+
+    @staticmethod
+    def _check_env(module_name: str):
+        if is_module_installed(module_name):
+            logger.info(f"{module_name} is installed.")
+        else:
+            raise ModuleNotFoundError(f"run: pip install {module_name}")
+
+    @unittest.skipUnless(0 in test_level_list(), "skip test in current test level")
+    def test_run_generate_dataset(self):
+        task_cfg = {
+            "eval_backend": "RAGEval",
+            "eval_config": {
+                "tool": "RAGAS",
+                "testset_generation": {
+                    "docs": ["README.md"],
+                    "test_size": 5,
+                    "output_file": "outputs/testset.json",
+                    "distribution": {
+                        "simple": 0.5,
+                        "multi_context": 0.4,
+                        "reasoning": 0.1,
+                    },
+                    # The generator_llm is the component that generates the questions, and evolves the question to make it more relevant.
+                    # The critic_llm is the component that filters the questions and nodes based on the question and node relevance.
+                    "generator_llm": {
+                        "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                        "template_type": "qwen",
+                    },
+                    "critic_llm": {
+                        "model_name_or_path": "QwenCollection/Ragas-critic-llm-Qwen1.5-GPTQ",
+                        "template_type": "qwen",
+                    },
+                    "embeddings": {
+                        "model_name_or_path": "AI-ModelScope/m3e-base",
+                    },
+                },
+            },
+        }
+
+        logger.info(f">> Start to run task: {task_cfg}")
+
+        run_task(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), "skip test in current test level")
+    def test_run_rag_eval(self):
+        task_cfg = {
+            "eval_backend": "RAGEval",
+            "eval_config": {
+                "tool": "RAGAS",
+                "eval": {
+                    "testset_file": "outputs/testset.json",
+                    "critic_llm": {
+                        "model_name_or_path": "qwen/Qwen2-7B-Instruct",
+                        "template_type": "qwen",
+                    },
+                    "embeddings": {
+                        "model_name_or_path": "AI-ModelScope/m3e-base",
+                    },
+                    "metrics": [
+                        "faithfulness",
+                        "answer_relevancy",
+                        "context_precision",
+                        "answer_correctness",
+                    ],
+                },
+            },
+        }
+
+        logger.info(f">> Start to run task: {task_cfg}")
+
+        run_task(task_cfg)
+
+    @unittest.skipUnless(0 in test_level_list(), "skip test in current test level")
+    def test_run_rag_eval_api(self):
+        task_cfg = {
+            "eval_backend": "RAGEval",
+            "eval_config": {
+                "tool": "RAGAS",
+                "eval": {
+                    "testset_file": "outputs/testset.json",
+                    "critic_llm": {
+                        "model_name": "gpt-4o-mini",  # 自定义聊天模型名称
+                        "api_base": "http://127.0.0.1:8088/v1",  # 自定义基础URL
+                        "api_key": "xxxx",  # 你的API密钥
+                    },
+                    "embeddings": {
+                        "model_name_or_path": "AI-ModelScope/m3e-base",
+                    },
+                    "metrics": [
+                        "faithfulness",
+                        "answer_relevancy",
+                        "context_precision",
+                        "answer_correctness",
+                    ],
+                },
+            },
+        }
+
+        logger.info(f">> Start to run task: {task_cfg}")
+
+        run_task(task_cfg)
+
+
+if __name__ == "__main__":
+    unittest.main(buffer=False)