C²RBench: A Chinese Complex Reasoning Benchmark for Large Language Models
Authors: Junru Wu, Tianhao Shen, Linxi Su, Deyi Xiong
Accepted at: acl25

C²RBench: A Chinese Complex Reasoning Benchmark for Large Language Models

Junru Wu*, Tianhao Shen*, Linxi Su, Deyi Xiong†
College of Intelligence and Computing, Tianjin University, Tianjin, China
{wujunru, thshen, sulinxi, dyxiong}@tju.edu.cn

Abstract
Large language models (LLMs) have achieved remarkable progress in autonomous reasoning, evolving from basic text processing to sophisticated multimodal reasoning, a critical capability for general-purpose AI assistants. However, existing benchmarks usually fail to adequately capture the intricate multi-step reasoning demands inherent in real-world scenarios. To bridge this gap, we propose C²RBench: a Chinese Complex Reasoning Benchmark for evaluating multi-step, multimodal advanced reasoning capability of LLMs. C²RBench comprises 1,115 carefully curated Chinese tasks, which are organized into eight domain-specific subsets, each meticulously designed to mirror real-world challenges. This hierarchical benchmark features three difficulty tiers based on the number of reasoning steps required (average 8.44 steps per task), significantly exceeding existing benchmarks in cognitive complexity. Extensive evaluations of 20 LLMs (including DeepSeek-R1) and 24 multimodal large language models (MLLMs) on C²RBench reveal critical performance gaps: GPT-4.1¹ achieves only 52.11% accuracy, indicating substantial room for improvement. The dataset and evaluation code² are publicly available.

1 Introduction
Recent advances in reasoning capabilities of large language models (LLMs), such as o1-preview³ and DeepSeek-R1 (DeepSeek-AI et al., 2025) have significantly enhanced their ability to engage in more extended and deliberate reasoning processes, allowing them to solve increasingly complex problems compared to earlier LLMs (Yao et al., 2023). These reasoning capabilities allow models to perform multi-step reasoning, make better decisions, and adapt more effectively to diverse tasks (Chen et al., 2024; Wang et al., 2024).

Reasoning has become a critical factor in evaluating the intelligence and generalization of LLMs. However, existing reasoning benchmarks often fall short of capturing the requirements of modern reasoning language models (Liu et al., 2025; Huang and Chang, 2023). First, many current benchmarks often focus on a limited set of reasoning tasks with a small number of reasoning steps. For instance, GSM8K (Cobbe et al., 2021), a purely textual benchmark, tests basic mathematical operations requiring only 2 to 8 reasoning steps. Second, benchmarks like DRCD (Shao et al., 2019) and C-Eval (Huang et al., 2023) are not well aligned with real-world scenarios and fail to capture the complexity and contextual ambiguity of authentic reasoning problems. Third, most reasoning benchmarks are limited to English and unimodal tasks, lacking support for other languages like Chinese or multimodal formats that combine text, images, tables, or structured files (Li et al., 2024; Huang et al., 2024b).

To mitigate these challenges, we propose the Chinese Complex Reasoning Benchmark (C²RBench), a novel benchmark designed to evaluate LLM reasoning across multiple dimensions, including multi-task, multi-step, and multimodal reasoning in Chinese. As illustrated in Figure 1, it features broader and more reasoning steps than most existing benchmarks. Specifically, C²RBench

![Illustration of the features of C²RBench.](page_1016_172_388_246.png)

Figure 1: Illustration of the features of C²RBench.

*Equal contribution
†Corresponding author
¹https://openai.com/index/gpt-4-1/
²https://github.com/tjunlp-lab/C2Rbench
³https://openai.com/o1/
emphasizes multi-step reasoning, with an average of 8.44 reasoning steps per task. The tasks are organized into three levels of reasoning complexity, with level-3 tasks requiring an average of 13.03 reasoning steps. Additionally, C^2RBench supports multimodal reasoning, encompassing text, images, tables, and files. The tasks in C^2RBench are drawn from real-world scenarios, such as file processing, graph analysis, and scientific reasoning, in order to simulate the evaluation of LLM reasoning in practical applications. Although similar to the GAIA benchmark (Mialon et al., 2023), C^2RBench is larger (1,115 vs. 466 tasks) and more aligned with real-world scenarios, with a focus on Chinese rather than English.

The main contributions of C^2RBench are summarized as follows:

1. We present C^2RBench, the first large-scale Chinese benchmark that jointly evaluates multi-task, multi-step, and multimodal reasoning, addressing critical gaps in current LLM evaluation.

2. C^2RBench benefits from its high cognitive complexity and strong alignment with real-world reasoning tasks. It consists of 1,115 tasks across 8 subsets and 3 complexity levels, with an average of 4.63 reasoning steps per level-1 task, 8.56 per level-2 task and 13.03 per level-3 task.

3. We conduct extensive evaluations of 20 LLMs and 24 multimodal large language models (MLLMs) on C^2RBench. Even state-of-the-art models such as GPT-4.1 achieve only 52.11% accuracy, compared to 89.75% for humans, illustrating the formidable challenge posed by C^2RBench to advanced LLMs.

2 Related Work

As LLMs continue to advance, their ability to tackle complex reasoning tasks has become a key focus in both research and application (Guo et al., 2023; Wei et al., 2023; DeepSeek-AI et al., 2025). Existing benchmarks for evaluating reasoning capabilities can be broadly categorized into task-specific and comprehensive benchmarks.

Task Specific Reasoning Benchmarks. These benchmarks are tailored to assess a model’s reasoning ability in a particular domain. For instance, Hel-laSwag (Zellers et al., 2019) and SIQA (Sap et al., 2019) evaluate commonsense reasoning; MathVista (Lu et al., 2024), GSM8K (Cobbe et al., 2021) and BackMATH (Zhang and Xiong, 2025) focus on mathematical reasoning; ARC (Clark et al., 2018) targets scientific reasoning; CodeXGLUE (Lu et al., 2021) and ProBench (Yang et al., 2025) benchmark code reasoning and generation tasks; ToolBench (Qin et al., 2023) and CToolEval (Guo et al., 2024) evaluate the tool using capability of models; and RoleEval (Shen et al., 2023) aims to evaluate reasoning over knowledge required for role-playing. While these datasets provide valuable insights into specific reasoning capabilities, they fall short in evaluating general-purpose reasoning across diverse settings.

Comprehensive Reasoning Benchmarks. To systematically evaluate broader reasoning abilities, several comprehensive benchmarks have been introduced. MMMU (Yue et al., 2024a,b) spans multiple domains—ranging from science to humanities—using primarily multiple-choice questions. Similarly, C-Eval (Huang et al., 2023), M3KE (Liu et al., 2023a) and OpenEval (Liu et al., 2024) provide large-scale Chinese evaluation sets emphasizing advanced knowledge and reasoning. Other multimodal benchmarks such as MMBench (Liu et al., 2023b) and GaoKao-MM (Zong and Qiu, 2024) focus on visual reasoning in multiple-choice formats. GAIA (Mialon et al., 2023) offers open-ended tasks covering web search, code generation, multimodal understanding, and document processing.

Despite their contributions, existing benchmarks exhibit several limitations: (i) a lack of real-world contextual complexity, (ii) limited support for multi-step reasoning annotations, and (iii) insufficient coverage of the Chinese language and multimodal reasoning settings (Li et al., 2024; Huang et al., 2023). These constraints hinder their applicability for evaluating the capabilities of modern LLMs in practical, multilingual, and cognitively demanding scenarios.

To address these challenges, we propose C^2RBench, a Chinese benchmark designed for multi-task, multi-modal and multi-step reasoning scenarios in real-world applications. Detailed comparison of our dataset with previous datasets is presented in Table 1.

3 Dataset Curation

We follow the pipeline illustrated in Figure 2 to curate C^2RBench, aiming to evaluate the multi-step reasoning abilities of LLMs in real-world Chinese
<table>
  <tr>
    <th>Datasets</th>
    <th>MS Anno?</th>
    <th>Size</th>
    <th>Lang</th>
    <th>Domain</th>
    <th>Task Type</th>
  </tr>
  <tr>
    <td>MathVista</td>
    <td>no</td>
    <td>5.14K</td>
    <td>en</td>
    <td>Math</td>
    <td>Open&MC</td>
  </tr>
  <tr>
    <td>MMMU</td>
    <td>a few</td>
    <td>12K</td>
    <td>en</td>
    <td>Real-world</td>
    <td>Open&MC</td>
  </tr>
  <tr>
    <td>GAIA</td>
    <td>yes</td>
    <td>0.5K</td>
    <td>en</td>
    <td>Real-world</td>
    <td>Open</td>
  </tr>
  <tr>
    <td>GaoKao-MM</td>
    <td>no</td>
    <td>0.6K</td>
    <td>zh</td>
    <td>Exams</td>
    <td>MC</td>
  </tr>
  <tr>
    <td>C-Eval</td>
    <td>no</td>
    <td>14K</td>
    <td>zh</td>
    <td>Exams, Web</td>
    <td>MC</td>
  </tr>
  <tr>
    <td>DRCD</td>
    <td>no</td>
    <td>30K</td>
    <td>zh</td>
    <td>Reading</td>
    <td>Open</td>
  </tr>
  <tr>
    <td>C²RBench</td>
    <td><b>yes</b></td>
    <td>1.1K</td>
    <td><b>zh</b></td>
    <td><b>Real-world</b></td>
    <td><b>Open&MC</b></td>
  </tr>
</table>

Table 1: Comparison of C²RBench with existing reasoning benchmarks. The column “MS Anno?” indicates whether the dataset provides multi-step reasoning annotations. MC denotes Multi-choice QA, while open refers to open-ended QA.

![The dataset curation pipeline of C²RBench.](page_186_370_1077_180.png)

Figure 2: The dataset curation pipeline of C²RBench.

![Proportion of different subsets in C²RBench.](page_186_590_1077_180.png)

Figure 3: Proportion of different subsets in C²RBench.

scenarios.

3.1 Data Sources and Preprocessing

To ensure both authenticity and diversity, we collect samples from four main sources: (1) real-world materials such as civil service exams and classical literature; (2) authoritative databases including those from the China National Bureau of Statistics; (3) educational resources such as secondary and tertiary-level academic exam questions; and (4) structured datasets from open platforms like Kaggle. Additionally, we incorporate high-quality GPT-generated samples verified by human annotators (see §A.1), forming a hybrid corpus of authentic and synthetic data.

All collected data undergo rigorous preprocessing, adhering to three principles: format normalization with semantic preservation, consistent annotation practices via standardized workflows, and comprehensive annotator training. We further implement a multi-stage quality control mechanism involving progress audits, random sample reviews, and error correction cycles to maintain annotation consistency and data integrity.

3.2 Annotation

Each task is designed to meet the following criteria:

• Deterministic Answers: Questions must yield unique, unambiguous answers with concise solutions.

• Multi-Step Reasoning: All tasks require at least three reasoning steps, with clearly delineated intermediate inferences.

• Non-Retrievability: Problems must not be solvable via direct keyword search, ensuring genuine reasoning is required.

• Synthetic Data Validation: GPT-generated samples are subject to rigorous manual validation, including checks for semantic alignment and logical consistency.

• Contamination Prevention: We confirm that all questions cannot be answered via simple web queries to avoid data leakage.

To enforce these standards, we implement a dual-layer verification framework comprising automated rule-based checks and expert manual review. This ensures that all instances meet the necessary complexity thresholds and remain grounded in real-world reasoning scenarios.
3.3 Quality Control
Post-annotation, we apply a rigorous cross-verification protocol to ensure logical coherence, answer correctness, and supporting evidence relevance. Quality assurance is conducted in two iterative phases:

• Deduplication: Near-duplicate samples are identified using TF-IDF with cosine similarity and then manually validated before removal.

• Dynamic Difficulty Calibration: Question difficulty is dynamically adjusted based on solver feedback. Tasks exceeding 11 reasoning steps are automatically escalated to Level 3 difficulty, followed by human verification.

This comprehensive quality control framework guarantees both difficulty progression and content diversity while preserving pedagogical validity.

3.4 Task Construction Manual
For detailed information on how tasks are constructed, annotated, and verified, please refer to the Task Construction Manual in the Appendix A.2.

3.5 Statistics

Figure 4: Proportion of different difficulty levels in each subset.

C²RBench contains 1,115 manually curated instances across eight subsets: Math, Multimodal, Logical, General Knowledge, Graph Analysis, Reading, Science, and File Processing. Figure 3 illustrates their distribution.

To enable fine-grained analysis, we categorize tasks into three difficulty levels based on the number of annotated reasoning steps:

• Level 1 (3–6 steps): Basic reasoning, testing fundamental capabilities of LLMs.

• Level 2 (7–11 steps): Intermediate reasoning, probing multi-hop inference capacity.

• Level 3 (\( \geq 12 \) steps): Advanced reasoning, evaluating long-horizon inference and information synthesis.

The overall distribution of difficulty levels and reasoning steps is shown in Figure 4 and 5, providing a comprehensive overview of C²RBench structure and complexity. For more details, please refer to Appendix A.4.

![Distribution of annotated reasoning steps in C²RBench.](page_820_98_579_377.png)

Figure 5: Distribution of annotated reasoning steps in C²RBench.

4 Experiments
We conducted extensive experiments with 20 LLMs and 24 MLLMs on C²RBench to evaluate its effectiveness and utility for assessing the reasoning capabilities of LLMs and MLLMs. All models used for evaluation are summarized in Table 2.

4.1 Evaluation Settings
We evaluate a diverse set of models on C²RBench, encompassing both LLMs and MLLMs, including closed-source and open-source variants. All evaluations are conducted in a zero-shot setting to measure the models’ intrinsic reasoning capabilities without fine-tuning or few-shot demonstrations.

Accuracy (ACC) is adopted as the primary evaluation metric. Given the dataset’s mixture of open-ended and multiple-choice questions, rule-based matching proves insufficient for reliable evaluation. To address this, we use GPT-4o as an automatic evaluator. To verify its reliability, we perform human evaluation involving three trained annotators. In each of the 10 rounds, 100 model responses are randomly sampled and compared. GPT-4o achieves over 98% agreement with human judgments, validating its effectiveness as an evaluation proxy. The evaluation pipeline is illustrated in Figure 6.
<table>
  <tr>
    <th>Models</th>
    <th>Open Source?</th>
    <th>Model Size</th>
    <th>Multimodal?</th>
    <th>Reasoning?</th>
  </tr>
  <tr>
    <td>DeepSeek-R1 (DeepSeek-AI et al., 2023)</td>
    <td>Yes</td>
    <td>671B</td>
    <td>No</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>Doubaovision-V2.5-120G (DeepSeek-AI, 2024)</td>
    <td>Yes</td>
    <td>671B</td>
    <td>No</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Doubaovision-V3 (DeepSeek-AI et al., 2024)</td>
    <td>Yes</td>
    <td>671B</td>
    <td>No</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Doubaovision-pro-32k<sup>4</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Doubaovision-pro-2048<sup>5</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Doubaovision-pro-32k<sup>6</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Doubaovision-pro-32k<sup>7</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Emic-1.5 Flash<sup>8</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Emic-1.5 Turbo<sup>9</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash<sup>10</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Turbo<sup>11</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GLM-4-Long (GLM et al., 2024)</td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GLM-4v-plus<sup>12</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GLM-Zero-Alpha<sup>13</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GLM-Zero-Preview<sup>14</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GPT-3.5-Turbo<sup>15</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GPT-4<sup>16</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GPT-4-16B<sup>17</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GPT-4-Turbo<sup>18</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>GPT-4.5-Preview<sup>19</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>o1-mini<sup>20</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>o1-preview<sup>21</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Plato-EB (Agrawal et al., 2024)</td>
    <td>Yes</td>
    <td>32B</td>
    <td>No</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview<sup>22</sup></td>
    <td>Yes</td>
    <td>32B</td>
    <td>No</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>hunyuanturbo-lates<sup>23</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>hunyuanturbo-fast<sup>24</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Kimi-thinking<sup>25</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (Paterson et al., 2022)</td>
    <td>Yes</td>
    <td>11B, 72B</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (Paterson et al., 2023)</td>
    <td>Yes</td>
    <td>7B, 72B</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Moonshot-v1-32k-vision-viewer<sup>26</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Moonshot-v1-32k-vision-viewer<sup>27</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Qwen-3-235B-A22B<sup>28</sup></td>
    <td>Yes</td>
    <td>235B</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Qwen-VL-max (Bai et al., 2023)</td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Sensenova-1.2<sup>29</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>SparkX Ultra<sup>30</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>SparkX1<sup>31</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>Vi-Lighting (Wake et al., 2025)</td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
  <tr>
    <td>XVision-V2<sup>32</sup></td>
    <td>No</td>
    <td>undisclosed</td>
    <td>Yes</td>
    <td>No</td>
  </tr>
</table>

Table 2: Models evaluated on the C²RBench.

All experiments are conducted on machines equipped with eight NVIDIA A6000 GPUs (48GB memory each). Wherever possible, we use the official APIs released by the model developers for consistency and fairness.

To ensure reproducibility and control randomness, we standardize the decoding parameters as follows: we set temperature = 0 whenever supported. For models requiring a non-zero temperature (e.g., LLaMA), we use temperature = 0.001 and disable sampling by setting do_sample = False. Note that o1-preview only supports temperature = 1, and DeepSeek-R1 does not respond to temperature adjustments.

<sup>4</sup>https://www.volcengine.com/product/doubaovision-pro-32k
<sup>5</sup>https://www.volcengine.com/docs/82379/1536428
<sup>6</sup>https://ai.baidu.com/ai-doc/WENXINWORKSHOP/am3iH7x209a
<sup>7</sup>https://cloud.baidu.com/doc/WENXINWORKSHOP/s/hm9w09a
<sup>8</sup>https://deepmind.google/technologies/gemini/
<sup>9</sup>https://www.zhipuai.cn
<sup>10</sup>https://open.bigmodel.cn/dev/api/Reasoning-models/glm-z1
<sup>11</sup>https://openai.com
<sup>12</sup>https://huggingface.co/QwQ/QwQ-32B-Preview
<sup>13</sup>https://hunyuantencent.com
<sup>14</sup>https://platform.moonshot.cn
<sup>15</sup>https://qwenlm.github.io/blog/qwen2-v1/
<sup>16</sup>https://huggingface.co/Qwen/Qwen3-235B-A22B
<sup>17</sup>https://platform.sensenova.cn
<sup>18</sup>https://www.xfyun.cn
<sup>19</sup>https://platform.lingyiwanwu.com

<table>
  <tr>
    <th>Category</th>
    <th>Average Accuracy (%)</th>
  </tr>
  <tr>
    <td>Overall</td>
    <td>89.75</td>
  </tr>
  <tr>
    <td>File Processing</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>General Knowledge</td>
    <td>90.90</td>
  </tr>
  <tr>
    <td>Graph Analysis</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>Logical</td>
    <td>93.80</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>100.00</td>
  </tr>
  <tr>
    <td>Multimodal</td>
    <td>56.20</td>
  </tr>
  <tr>
    <td>Reading</td>
    <td>97.00</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>86.80</td>
  </tr>
</table>

Table 3: Human performance across different subsets..

4.2 Tested Models
The 44 tested models are summarized in Table 2.

MLLMs. We selected 24 MLLMs, including both open-source and closed-source models. Among these, two are specifically optimized for reasoning. For each model family, we selected the most recent and high-performing version available at the time of evaluation. Although the official documentation for Llama-3.2-Vision lists support for only eight languages (excluding Chinese), our experiments show that it can handle Chinese reasonably well, albeit with occasional code-switching.

Text-only LLMs. We included 20 text-only LLMs, 9 of which are explicitly designed for reasoning, such as DeepSeek-R1 and o1-preview.

4.3 Human Performance Evaluation
To benchmark human performance on C²RBench, we conducted a stratified sampling of 10% of the questions, ensuring balanced coverage across all subsets and difficulty levels. Eleven participants, all with at least undergraduate-level education, were recruited through targeted selection to ensure relevant domain expertise. Each question was independently answered by 2–3 individuals.

Human participants achieved an overall accuracy of 89.75%, significantly outperforming current LLMs. Table 3 presents accuracy by category. Additional details on the sampled questions and participant backgrounds are provided in Appendix A.7.

4.4 Results
Overall Performance: Human participants significantly outperformed all evaluated models, achieving an overall accuracy of **89.75%**. In contrast, the best-performing models, **Doubao-1.5-thinking-pro** and **GPT-4.1**, only reached **65.74%**
Prompt for generation
给你一个问题，请你首先给出思考过程，最后给出一个明确结论/选择填空。请务必保证分析与答案的格式一致。
(Given a question, first provide your reasoning process, then give a clear answer (choice/solution). Please respond in the format: [your analysis]\n[answer])
File
从所给四个选项中，选择最合适的—个填入问号处，使题意完整、逻辑正确。
(Select the most appropriate option from the given four choices to fill in the question mark, ensuring a consistent pattern.)
Question
Prompt for evaluation
给你一个参考答案和一个需要判断对错的回答，请你首先给出推理过程分析参考答案和判断理由，最后给出判断。参考答案完全正确，则认为参考答案与参考答案评分标准等同，判断结果为1；如果参考答案有部分错误或不完全正确，判断结果为0。如果参考答案为数值型数字结果，回答者需给出对数值或成对数值的解释。评分分析：[分析分数] = [分析分数] + [判断分数]
(Given a reference answer and a response that needs to be judged, first provide your reasoning process and judgment criteria, then determine whether the response is correct. If the reference answer is entirely correct, it is considered correct, and the score is 1. Otherwise, it is incorrect, and the score is 0. Note: Do not provide an explanation if the answer is a numerical result; the response must explicitly match the corresponding option or numerical result to receive a score. Please respond in the following format:
[Your analysis]\n[Your score]\n)
LLM to be tested
Generation from LLM
“分析：第一行花盆的颜色是黑色，白色，白色，白色。第二行花盆的颜色是黑色，白色，白色，白色。第三行花盆的颜色是黑色，白色，白色，白色。第一行花盆的形状和方向是圆形，正方形，正方形，正方形。第二行花盆的形状和方向是圆形，正方形，正方形，正方形。第三行花盆的形状和方向是圆形，正方形，正方形，正方形。第一行花盆的排列方式是水平排列。第二行花盆的排列方式是水平排列。第三行花盆的排列方式是水平排列。”
(Analysis: The flower pot colors in the first row are black, white, white, white. The flower pot colors in the second row are black, white, white, white. The flower pot colors in the third row are black, white, white, white. The shapes and orientations of the flower pots in the first row are circle, square, square, square. The shapes and orientations of the flower pots in the second row are circle, square, square, square. The shapes and orientations of the flower pots in the third row are circle, square, square, square. The arrangement of the flower pots in the first row is horizontal. The arrangement of the flower pots in the second row is horizontal. The arrangement of the flower pots in the third row is horizontal.)
Answer of LLM
A
Ground-Truth Answer
A
GPT-4o evaluation
“分析：题目与参考答案相同，都是选了A。因为参考答案完全正确，所以与参考答案评分标准等同一致，因此判断正确。”
(Analysis: The question and reference answer are the same, both selecting option A. Since the reference answer is entirely correct and the response is semantically equivalent with the same conclusion, the response is correct.)
Score
1.0

Figure 6: The pipeline for task evaluation in C^2RBench is illustrated using an example. LLMs represent all the models to be evaluated (see Table 2). The input to the LLM to be tested consists of the prompt for generation, file, and question. The answer of the LLM is derived by extracting its generation. GPT-4o is used as the evaluation model. The input to GPT-4o includes the prompt for evaluation, the answer of the LLM, and the ground truth answer. Finally, the score for the evaluated reasoning task is derived by extracting GPT-4o evaluation results.

and **52.11%**, respectively (see Figure 7 and 8). This substantial performance gap underscores the current limitations of LLMs in approximating general-purpose human reasoning capabilities.

Disparity between Open-source and Closed-source Models: The best-performing open-source MLLM (e.g., Qwen2-VL-72B-Instruct) achieves an overall accuracy of only **37.76%**, which lags far behind leading closed-source models such as **GPT-4.1**. Despite this gap, open-source models demonstrate strong potential on specific reasoning tasks. Notably, **Deepseek-R1** attains **76.10%** accuracy on the logical reasoning subset, outperforming closed-source models like o1-preview in this category.

Model Performance across Different Subsets: Figure 7 shows the performance of MLLMs across the eight subsets of C^2RBench. Current MLLMs exhibit notable weaknesses in Math reasoning and Multimodal reasoning. Additionally, as shown in Figure 10, text-only LLMs tend to perform poorly on tasks in the General Knowledge subset of C^2RBench, further indicating a limitation in handling certain types of reasoning tasks.

Model Performance across Different Levels: Figure 8 and 10 illustrate that as task complexity increases, the accuracy of all models tends to decrease. Taking GPT-4.1 as an example, its accuracy drops from 62.73% at Level 1 to 54.30% at Level 2, and further decreases to 34.35% at Level 3. This trend is consistent across all models, highlighting the difficulties large language models encounter when dealing with more complex reasoning tasks.

MLLMs Performance on Multimodal Reasoning: Figure 9 presents the performance of MLLMs on the multimodal subset. Open-source models, such as Qwen2-VL-7B-Instruct, attain an accuracy of 25.79%, slightly outperforming closed-source models like GPT-4o, which achieves an accuracy of 25.16%. Despite this, the overall performance on the multimodal subset remains low, revealing substantial room for improvement in both multimodal and multi-step reasoning tasks.

Insights of Logical Reasoning: Even the state-of-the-art models, such as o1-preview and DeepSeek-R1, achieve an accuracy of only 72.33% and 76.10% (see Figure 10 ). These results underscore the significant challenges posed by C^2RBench in advanced logical reasoning tasks.
Figure 7: MLLMs’ performance across different subsets.

Figure 8: MLLMs’ performance across different levels.

Further investigation of DeepSeek-R1’s performance via web interface testing resulted in a 74.48% accuracy on the logical reasoning subset. A detailed analysis of inference times (Table 4) reveals a positive correlation between reasoning time and task difficulty. Specifically, incorrect responses consistently require longer processing times (mean = 285.83s) compared to correct solutions (mean = 100.69s), in line with previous findings (Huang et al., 2024a). This pattern suggests that persistent reasoning loops contribute to inefficiencies in LLMs, emphasizing the importance of addressing unproductive computational cycles in future model development.

Additionally, an in-depth error analysis and case study (see Appendix A.5) were conducted to further explore the underlying factors affecting the model’s logical reasoning performance.

4.5 Error Analysis

To systematically assess the limitations in model performance, we conducted an extensive error typology analysis across four state-of-the-art LLMs.

The responses from each model were categorized into six distinct error classes through a multi-stage annotation process. (1) Perception & Input: Failures in processing multimodal inputs. (2)

<table>
  <tr>
    <th>Times (s)</th>
    <th>All</th>
    <th>Correct</th>
    <th>Wrong</th>
  </tr>
  <tr>
    <td>Overall</td>
    <td>147.26</td>
    <td>100.69</td>
    <td>285.83</td>
  </tr>
  <tr>
    <td>Level 1</td>
    <td>83.57</td>
    <td>63.88</td>
    <td>167.25</td>
  </tr>
  <tr>
    <td>Level 2</td>
    <td>132.49</td>
    <td>91.98</td>
    <td>281.00</td>
  </tr>
  <tr>
    <td>Level 3</td>
    <td>226.19</td>
    <td>158.37</td>
    <td>345.88</td>
  </tr>
</table>

Table 4: Average reasoning time per task for DeepSeek-R1 on the logical reasoning subset.

Knowledge & Comprehension: Gaps in domain-specific knowledge. (3) Logic & Reasoning: Breakdowns in multi-step reasoning chains. (4) Task Execution: Procedural errors in problem-solving. (5) Refusal & Inability: Unwarranted non-responses. (6) Generation & Output: Errors in formatting or linguistic generation. Note: Individual problems may exhibit multiple error types simultaneously. Full definitions and prompt for error analysis are provided in Appendix A.3.

Table 5 highlights two critical needs for LLMs in Chinese reasoning: improved cross-modal understanding to resolve perceptual-input mismatches, and more efforts dedicated to multi-step reasoning. The persistent prevalence of logical errors (exceeding 65% across all LLMs) particularly underscores the urgency of addressing these challenges.
Figure 9: Results of MLLMs on the multimodal reasoning subset.

Figure 10: Results of text-only LLMs on the Logical & Graph Analysis & Reading reasoning subset.

<table>
  <tr>
    <th>Error Type</th>
    <th>DeepSeek-R1 o1-preview</th>
    <th>GPT-4o Qwen2-VL-72B</th>
  </tr>
  <tr>
    <td>Perception & Input</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>4.42%</td>
    <td>3.17%</td>
  </tr>
  <tr>
    <td>Knowledge & Comprehension</td>
    <td>13.16%</td>
    <td>18.92%</td>
    <td>52.21%</td>
    <td>52.88%</td>
  </tr>
  <tr>
    <td>Logic & Reasoning</td>
    <td>97.37%</td>
    <td>97.73%</td>
    <td>69.39%</td>
    <td>65.56%</td>
  </tr>
  <tr>
    <td>Task Execution</td>
    <td>10.53%</td>
    <td>0.00%</td>
    <td>9.82%</td>
    <td>13.40%</td>
  </tr>
  <tr>
    <td>Refusal & Inability</td>
    <td>0.00%</td>
    <td>0.00%</td>
    <td>1.96%</td>
    <td>7.06%</td>
  </tr>
  <tr>
    <td>Generation & Output</td>
    <td>0.00%</td>
    <td>2.27%</td>
    <td>1.31%</td>
    <td>2.74%</td>
  </tr>
</table>

Table 5: Error type distribution across models. Please note that one output of LLM may involve multiple types of errors (e.g., both Knowledge & Comprehension and Logic & Reasoning), resulting in column totals exceeding 100%. It is important to note that DeepSeek-R1 and o1-preview were evaluated exclusively on the logical subset, while GPT-4o and Qwen2-VL-72B were assessed on the full C²RBench dataset.

4.6 Case Study

To further analyze the strengths and limitations of different MLLMs, we present two representative case studies.

Figure 21 illustrates a multimodal reasoning case that requires precise alignment between visual content and textual understanding. From the model outputs, it is evident that both Qwen2-VL-72B and LLaMA-3.2-90B-Vision struggle with fine-grained visual recognition and spatial reasoning, leading to incorrect conclusions. These errors suggest that despite their overall capabilities, current MLLMs still face challenges in extracting detailed visual cues and integrating them effectively into the reasoning process.

Figure 22 presents a logical reasoning scenario that highlights differences in reasoning chains among the evaluated models. These variations reflect the disparity in multi-step reasoning abilities and indicate that some models rely more heavily on surface-level patterns rather than deep inference.

More comprehensive discussions and additional examples can be found in Section A.5, where we further dissect model behaviors across a range of challenging scenarios.

4.7 Analysis of Reasoning Models

We analyzed token usage across tasks to evaluate multi-step reasoning efficiency, comparing Doubao-1.5-Thinking-Pro and Kimi-Thinking-Preview with direct-answer model GPT-4o, aiming to identify overthinking—excessive reasoning with little gain.

For Doubao and Kimi, we computed the following token categories: Reasoning Tokens, Answer Tokens, and Total Tokens, while for GPT-4o we
Figure 11: Overthinking analysis across task category, level, score, and final answer presence. Bars represent token counts for reasoning, answer, and total. In the “Final Response Presence” plots, 1 indicates absence of a final answer.

only counted the final response tokens.

Figure 11 illustrates a common phenomenon in reasoning models: while token usage increases in more complex tasks, higher reasoning token consumption does not necessarily improve accuracy and may even result in answer absence due to token exhaustion.

The phenomenon of overthinking becomes evident with increasing complexity in tasks. Both Doubao and Kimi show excessive reasoning steps without significant improvements in accuracy, often leading to token exhaustion before reaching a conclusion. In contrast, GPT-4o’s direct approach, despite potentially using fewer reasoning tokens, achieves more consistent results.

To address these issues, future advancements should focus on adaptive control mechanisms for reasoning depth, budget-aware planning for token usage, trace-aware evaluations to balance efficiency with accuracy, and dead-end detection to avoid unproductive reasoning loops. For a more detailed analysis of token consumption, see Appendix A.6.

5 Conclusion

In this paper, we have presented C^2RBench, a novel benchmark designed to evaluate the multi-step reasoning capabilities of LLMs in Chinese. C^2RBench consists of 1,115 tasks organized into eight distinct subsets. These tasks are categorized into three difficulty levels based on the number of steps involved. The key highlights of C^2RBench are its real-world scenarios, extensive task coverage, and its focus on Chinese. Our core contribution lies in providing a comprehensive and challenging benchmark for assessing LLMs’ performance on these complex, multi-step tasks in Chinese. Experimental evaluations on 24 MLLMs and 20 text-only LLMs indicate that C^2RBench remains challenging for state-of-the-art models. By systematically assessing complex reasoning across eight subsets, C^2RBench establishes a rigorous and comprehensive benchmark.
Limitation

Although C²RBench is comprehensive, like any benchmark, it has its limitations. The manual curation process may introduce biases, and its focus on high-difficulty, multi-step reasoning problems may not fully capture the capabilities required for LLMs. However, we believe strong benchmark performance is essential for evaluating advanced LLMs. The reasoning challenges posed by C²RBench are evident in the performance of 22 MLLMs and 20 text-only LLMs. To balance complexity and practicality, the dataset includes multiple-choice, single-choice, calculation-based tasks, and concise open-ended questions.

Acknowledgments

The present research was supported by the National Key Research and Development Program of China (Grant No. 2024YFE0203000). We would like to thank the anonymous reviewers for their insightful comments.

References

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lampe, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychinikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. 2024. Pixtral 12B. Preprint, arXiv:2410.07073.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Owen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966.

Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024. M²CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199–8221, Bangkok, Thailand. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. ArXiv, abs/1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. Preprint, arXiv:2110.14168.

DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. Preprint, arXiv:2405.04434.

DeepSeek-AI. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qiha Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqoing Wu, Shengfeng Ye, Tao Yun, Tian Pi, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song,
Zizheng Pan, Zhen Huang, Zhipei Xu, Zhongyu Zhang, and Zhen Zhang. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Preprint, arXiv:2501.12948.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuit Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuteng Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenjin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yao-hui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zhihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Ziheng Pan. 2024. DeepSeek-V3 Technical Report. Preprint, arXiv:2412.19437.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen-hui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Han-lin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaoahang Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. Preprint, arXiv:2406.12793.

Zishan Guo, Yufei Huang, and Deyi Xiong. 2024. CToolEval: A Chinese benchmark for LLM-powered agent evaluation in real-world API interactions. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15711–15724, Bangkok, Thailand. Association for Computational Linguistics.

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736.

Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards Reasoning in Large Language Models: A Survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaxiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024a. Large Language Models Cannot Self-Correct Reasoning Yet. In The Twelfth International Conference on Learning Representations.

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junting Liu, Chuncheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. In Advances in Neural Information Processing Systems.

Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. 2024b. OlympicArena: Benchmarking Multidiscipline Cognitive Reasoning for Superintelligent AI. arXiv preprint arXiv:2406.12753.

Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. 2024. A Survey on Benchmarks of Multimodal Large Language Models. Preprint, arXiv:2408.08632.
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. 2023a. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models. arXiv preprint arXiv:2305.10263.

Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinneng Ji, Tingting Cui, Liutao Liutao, et al. 2024. Openeval: Benchmarking chinese llms across capability, alignment and safety. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 190–210.

Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. 2025. Logical Reasoning in Large Language Models: A Survey. Preprint, arXiv:2502.09100.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023b. MMBench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun-yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In International Conference on Learning Representations (ICLR).

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664.

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. GAIA: a benchmark for General AI Assistants. Preprint, arXiv:2311.12983.

David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguía, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2022. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink. Preprint, arXiv:2204.05149.

Yuqia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolilm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQA: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473.

Chih Chieh Shao, Trois Liu, Yuting Lai, Yiyieng Tseng, and Sam Tsai. 2019. DRCD: a Chinese Machine Reading Comprehension Dataset. Preprint, arXiv:1806.00920.

Tianhao Shen, Sun Li, Quan Tu, and Deyi Xiong. 2023. Roleeval: A bilingual role evaluation benchmark for large language models. arXiv preprint arXiv:2312.16132.

Alan Wake, Bei Chen, C. X. Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Fan Zhou, Feng Hu, Ge Zhang, Guoyin Wang, Heng Ji, Howard Qiu, Jiangcheng Zhu, Jun Tian, Katherine Su, Lihuan Zhang, Liying Li, Ming Song, Mou Li, Peng Liu, Qicheng Hu, Shawn Wang, Shijun Zhou, Shiming Yang, Shiyong Li, Tianhang Zhu, Wen Xie, Wenhao Huang, Xiang He, Xiaobo Chen, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Yanpeng Li, Yongke Zhao, Yongzhen Luo, Yuchi Xu, Yuxuan Sha, Zhaodong Yan, Zhiyuan Liu, Zirui Zhang, and Zonghong Dai. 2025. Yi-Lightning Technical Report. Preprint, arXiv:2412.01253.

Yulong Wang, Tianhao Shen, Lifeng Liu, and Jian Xie. 2024. Sibyl: Simple yet effective agent framework for complex real-world reasoning. arXiv preprint arXiv:2407.10718.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Preprint, arXiv:2201.11903.

Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, and Deyi Xiong. 2025. Probench: Benchmarking large language models in competitive programming. arXiv preprint arXiv:2502.20868.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Preprint, arXiv:2305.10601.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqiqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Rubin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhao Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhui Chen. 2024a. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In Proceedings of CVPR.

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhui Chen, and Graham Neubig. 2024b. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. arXiv preprint arXiv:2409.02813.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a
Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

Shaowei Zhang and Deyi Xiong. 2025. BackMATH: Towards backward reasoning for solving math problems step by step. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 466–482, Abu Dhabi, UAE. Association for Computational Linguistics.

Yi Zong and Xipeng Qiu. 2024. GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation. Preprint, arXiv:2402.15745.

A Appendix

A.1 Prompts for Dataset Curation
In Figure 12, we present a complete prompt example required to construct a Graph Analysis task using ChatGPT. The generated content is manually reviewed each time to ensure its relevance and usability.

A.2 Task Construction Manual
Each instance in C^2RBench follows a standardized schema with the following fields:

• task_id: Unique identifier (category code + serial number).

• Question: Problem statement in Chinese.

• Label: Task category (Math, Multimodal, Logical, General Knowledge, Graph Analysis, Reading, Science, File Processing).

• Level: Difficulty level (1–3), determined by the number of reasoning steps.

• Final answer: A concise final answer.

• File_name: Optional multimodal inputs (e.g., images, PDFs, Excel files).

• Annotator Metadata: Includes the full reasoning chain (Steps) and number of steps.

A.2.1 Category Composition
The dataset includes 140 instances in each of the eight task categories, with each category designed to pose distinct domain-specific reasoning challenges. The categories are: Multimodal, Logical, Graph Analysis, Math, Science, File, General Knowledge, and Reading.

Each category is designed to address specific reasoning challenges:

• Multimodal: Includes graphical pattern completion and audio-based reasoning tasks.

• Logical: Abstract logic puzzles and reasoning tasks.

• Graph Analysis: Reasoning over related visual plots or statistical charts.

• Math: Symbolic and natural language math word problems (LaTeX encoded).

• Science: Physics, chemistry, and biology problems grounded in educational material.
Prompt for data source construction
根据[Technology and the Internet]这个主题，生成2个与数据源相关的csv格式数据。根据主题和数据生成标题，将其放在标记“[title]”之后。数据中包含的信息可以根据需要进行扩展和调整，以适应柱状图的类型。图表主题是[Technology and the Internet]。每个数据点的差异不应太大。数据的每一行代表一个类别或不同方面。（Generate two logically related CSV data points based on the theme of [Technology and the Internet]. Create a title based on the theme and the data, and place it after the tag "[title]". The information contained in the data can be expanded or adjusted as needed to fit the type of [bar chart]. The chart’s theme is [Technology and the Internet]. The differences between each data point should not be too large. Each row of the data represents different aspects of a particular category.)

Prompt for generating graph image files
假设您是专业的Python图表学家，请使用python模块以下数据绘制柱状图并保存为png。图像必须清晰直观。代码末尾有演示的缩进，请将数据转换为三个变量：data labels、数据值labels。Data labels表示第一列之外的列，例如：the labels for the x axis. Data labels应与之对应的每一行的标签。数据表示数据中的数值数组。使用柱状图类型绘制数据。[绘制所选图表类型的基本设置，绘制代码的具体设置，以增加多样性。图像必须另存为path to save.png。清除代码末尾的当前图像状态。在savefig(之前按shift+layout)自动调整图像大小。图的标题应为生成的图片标题。如果图片中的字符串太长，找到一种让所有字符都显示的方法，而不是被截断或堆叠在一起。生成的代码的开头和结尾不要有多余的前导引，例如“python code. python.”等。检查生成的代码以确保它不会报告错误，不包含未定义的函数。数据文件如下：（Assume you are a professional Python graphs expert. Please use Python to plot the data into a bar chart. Save the image as a png. The image must be clear and intuitive. The code at the end of the file has indentation. Plot the data into three variables: data labels, data values. Data labels represent the labels for each column except the first one. The data represents the array of values in the dataset. Plot the data using the [bar chart] type. [Plot the basic settings for the selected chart type.] [Specific settings for the plot code to add variety.] The image must be saved as [path to save].png. Clear the current image state at the end of the code. Automatically adjust the image size with tight layout() before calling savefig(). The chart title should be [Generated Chart Title]. If the strings in the image are too long, find a way to display all characters without them being covered or stacked together. The generated code should not contain unnecessary leading words at the beginning and end, such as “python code”, “python”, etc. Check the generated code to ensure it will not report errors or contain undefined functions. The data file is as follows.）

Prompt for generating QA
根据输入的具有给定标题和图表类型的csv数据，设计一个从两个图表中信息才能得到答案的hard难度的问答对，可以直接通过数据回答。每个答案不应包含任何提示、解释或注释等。答案应当是唯一的。并与推理到答案的过程链。（Design a hard-level question-answer pair based on the input CSV data with a given title and chart type, where the answer can only be obtained by combining information from both charts. The answer should be directly derived from the data. Each answer should not contain any hints, explanations, or comments. The answer must be unique. Also, provide the reasoning process chain that leads to the answer.）

Figure 12: Example prompts for constructing a Graph Analysis task using ChatGPT.

• File Processing: Multi-file document understanding (e.g., cross-referencing PDF and Excel).

• General Knowledge: Tasks for general knowledge reasoning.

• Reading: Reasoning tasks based on long Chinese passages.

A.2.2 Task Construction Guidelines
The construction of each task strictly follows the guidelines below:

• Reasoning Chain: Each question must include necessary reasoning steps. Each reasoning step must be logically required, concise, and context-dependent.

• Non-Redundancy: Each step of the task must be non-redundant, ensuring the reasoning process is minimal and effective.

• Non-Retrievability: Tasks are designed to prevent solving via search engine queries, avoiding shortcut solutions.

• Multi-Step Reasoning: All tasks must involve at least three reasoning steps (average reasoning steps: 8.44 steps, with up to 12.03 steps for Level 3 tasks).

A.2.3 Quality Assurance and Annotator Training
All annotators underwent detailed training using a handbook, and regular reviews were conducted. The training included:

• Ensuring task format consistency, annotation uniformity, and data quality.

• Regular inter-annotator agreement checks to ensure annotation quality.

• Secondary review for all Level 3 tasks.

• Verification of attachment integrity and accessibility.

A total of 373 annotation hours were logged, with an average time of 20 minutes per instance. Additionally, random sampling was used for quality checks to ensure data reliability.

A.3 Prompt for Error Analysis
The GPT-4o-based error analysis prompt is as follows:

You will be provided with: 1. A reference answer that follows standard reasoning steps. 2. A model-generated response containing errors.
Your task is to analyze the errors, attribute them to their root causes, and classify them into an appropriate error type and subtype based on the following categories:
1. **Perception & Input Errors** -
   Perceptual Error: Misinterprets visual or structured input (e.g., tables, charts). - Input Parsing Error: Mishandles input format, structure, or symbols (e.g., JSON, XML).
2. **Knowledge & Comprehension Errors** -
   - Lack of Knowledge: Lacks necessary domain knowledge. - Factual Error: Provides incorrect or hallucinated facts. - Misinterpretation of Question Intent: Misunderstands task requirements.
3. **Logical & Reasoning Errors** -
   Logical Reasoning Error: Produces inconsistent or invalid reasoning. - Inductive Reasoning Error: Incorrectly generalizes from specific cases. - Deductive Reasoning Error: Misapplies general rules to specific instances. - Long-range Dependency Error: Fails to maintain key context across reasoning steps. - Hypothesis Generation Error: Assumes unsupported premises. - Causal Reasoning Error: Misidentifies cause-effect relationships. - Analytical Reasoning Error: Struggles with multi-step analysis (e.g., mathematical proofs).
4. **Task Execution Errors** -
   Arithmetic Error: Makes numerical miscalculations. - Constraint Violation Error: Ignores explicit constraints (e.g., word limits, formatting). - Planning & Omission Error: Skips necessary reasoning steps or lacks structural coherence.
5. **Refusal & Inability Errors** -
   Overly Conservative Refusal: Unjustified refusal despite sufficient information. - Failure to Answer: States inability to respond when reasoning is possible. - Model Breakdown: Produces irrelevant, incoherent, or nonsensical output.
6. **Generation & Output Errors** -
   Grammar & Syntax Error: Contains grammatical or syntactical mistakes. - Disorganized Response: Presents information in a confusing manner. - Hallucination Error: Generates non-existent or fabricated content. - Redundancy & Repetition Error: Repeats content excessively. - Ambiguous Response: Provides vague or unclear answers.

**Response Format**:
- Analysis: {Detailed analysis of the errors, including their root causes}
- Error Type: {Main error category}
- Error Subtype: {Specific error subtype}

A.4 Dataset Details

We divide the dataset into eight subsets. Below, we introduce the data source and specific content of each subset and provide one example image for illustration.

A.4.1 Multimodal

This subset comes from graphical reasoning questions in civil service exams. It mainly tests the model’s ability to handle and reason over multimodal information. An example is shown in Figure 13.

![An example from the Multimodal subset.](page_1092_1347_388_180.png)

Figure 13: An example from the Multimodal subset.

A.4.2 Logical

This subset includes logical reasoning questions from major entrance exams and is used to evaluate the model’s reasoning and problem-solving abilities. See Figure 14.
Figure 14: An example from the Logical subset.

A.4.3 Graph Analysis
This subset is sourced from national statistics, government data, and academic databases. It includes a variety of chart types (e.g., bar charts, line charts, pie charts) and covers multiple domains such as business, education, environment, and transportation. See Figure 15.

Figure 15: An example from the Graph Analysis subset.

A.4.4 Math
This subset includes math problems from Chinese high school and university entrance exams, covering symbolic reasoning, natural language math problems, and spatial geometry. An example is shown in Figure 16.

A.4.5 Science
This subset includes tasks sourced from publicly available Chinese textbook materials. It covers a wide range of disciplines, including biology, physics, chemistry, engineering, and economics. See Figure 17.

A.4.6 File Processing
This subset comes from structured data provided by governments, international organizations, research institutions, and Kaggle competitions. It involves demographics, economics, health data, etc. An example is shown in Figure 18.

A.4.7 General Knowledge
This subset is designed to test multi-step reasoning ability beyond math and science. Some of the content was co-annotated with the help of ChatGPT. See Figure 19.

A.4.8 Reading Comprehension
This subset contains data from over 20 books, including detective novels (e.g., Sherlock Holmes), thematic novels, and academic papers, to evaluate reading comprehension and reasoning skills. See Figure 20.

A.5 Case Study
Figure 21 presents a multimodal case from C2Rbench, showcasing responses from four models: two high-performing closed-source models,

"task_id": "04012",
"Question": "已知函数 \( f(x)=\mathrm{e}^{-x}-a x^2+2 \)，\( g(x)=x^2+\cos x \)。当 \( a=15 \) 时，判断函数 \( f(x) \) 的单调性。", "(Given the functions \( f(x)=\mathrm{e}^{-x}-a x^2+2 \) and \( g(x)=x^2+\cos x \). Determine the monotonicity of the function \( f(x) \) when \( a=15 \).)"
"Label": "Math",
"Level": "1",
"Final answer": "单调递增",("Strictly increasing")
"File_name": "math_04012"
"Explanation": "1. 当 \( a=15 \) 时，\( f(x)=\mathrm{e}^{-x}-15 x^2+2 \)，以 \( f^{\prime}(x)=\mathrm{e}^{-x}-30 x \)。令 \( f^{\prime}(x)=0 \)，解得 \( x=\frac{1}{30}\ln 30 \)。
2. 当 \( x<\frac{1}{30}\ln 30 \) 时，\( f^{\prime}(x)>0 \)，\( f(x) \) 单调递增；
3. 当 \( x>\frac{1}{30}\ln 30 \) 时，\( f^{\prime}(x)<0 \)，\( f(x) \) 单调递减。",
"Number of steps": "6"

Figure 16: An example from the Math subset.

![An example from the Science subset.](page_1012_1342_388_246.png)

Figure 17: An example from the Science subset.
Doubao-1.5-Vision-Pro-32k and GPT-4o; and two high-performing open-source models, Qwen2-VL-72B and Llama-3.2-90B-Vision. This multimodal reasoning example effectively facilitates a comparative analysis of the performance differences among existing MLLMs.

Doubao-1.5-Vision-Pro-32k and GPT-4o provided both accurate analyses and correct answers. In contrast, Qwen2-VL-72B produced an accurate answer but with errors in its analysis process, while Llama-3.2-90B-Vision failed in both its analysis and the final answer.

Figure 22 illustrates a logical reasoning example from C^2RBench, showcasing responses from four models: one high-performing closed-source large reasoning model, o1-preview; one high-performing open-source large reasoning model, DeepSeek-R1; Hunyuan-turbo-latest, and GPT-4o.

For the logical (see Figure 21) and multimodal (see Figure 22) subsets, we show responses sampled from four models, with green text representing correct answers and red text representing incorrect ones.

A.6 Overthinking Analysis Setting

In this appendix, we present the detailed metrics for token consumption by the models under various evaluation conditions.

Token Breakdown:

• Reasoning Tokens: The number of tokens spent on intermediate reasoning steps.

• Answer Tokens: The tokens used to generate the final response.

• Total Tokens: The combined total of reasoning and answer tokens.

We track these values for each model (Doubao-1.5-Thinking-Pro, Kimi-Thinking-Preview, and GPT-4o) across different task types, highlighting inefficiencies in reasoning and areas for improvement.

A.7 Human Evaluation

A.7.1 Task Selection Strategy

We adopted a stratified random sampling strategy to ensure balanced and representative task coverage across the C^2RBench. Specifically, we sampled 10% of tasks from each combination of difficulty level (1–3) and category label, resulting in a total of 113 tasks. This approach guarantees proportional representation across task types and complexity levels.
"Question": "从所给四个选项中，选择最合适的填入问号处，使之呈现一定规律性。",
("Select the most appropriate option from the given four choices to fill in the question mark, ensuring a consistent pattern.")
"Label": "Multimodal", "Level": "2", "Final answer": "A",
"Steps": "1. 观察题目，本题考察图形推理规律。2. 观察每行的每个图形的元素（包括形状、花纹、黑白）在下一行中都重复出现。3. 分析：？处的图形特征：a4. 花盆应为黑色正方形，排除C、D项；a5. 花盆上应带有月牙形状；a6. 花朵应为圆形。4. 因此，本题正确答案为A选项。"
("1. Observe the problem. This question assesses the pattern of graphical rotation.2. Observe that the elements of each shape in a row (including shape, pattern and color) reappear in the next row.a3. Analyze the characteristics of the shape at the “?” position.a4. The flowerpot should be a black square, eliminating options C and D.a5. The stem should feature a crescent shape；a6. The flower should be circular.a7. Therefore, the correct answer is option A.")

<table>
  <tr>
    <th>Douhao-1.5-vision-pro-32k</th>
    <th>GPT-40</th>
    <th>Qwen2-VL-72B</th>
    <th>Llama-3.2-90B-Vision</th>
  </tr>
  <tr>
    <td>1. 首先观察图形特征：... 每行的花盆是每行要出现相同的元素。2. 然后具体分析每行元素：... a3. 最后看选项：... a4. 分析：通过每行图形的花盆形状、植物方向和植物顶端形状进行推理。... a5. 第三行第三列应该是黑色花盆，圆形花。a6. 答案：A</td>
    <td>分析：a. 观察图中的每一行和每一列，注意花盆的颜色、形状和花朵的形状。b. 由此，第三行第三列应该是黑色花盆，圆形花。a6. 答案：A<br><br>(Analysis: a. Observe each row and column in the image, focusing on the color of the flower pot, as well as the shape and orientation of the flower. ... a5. Thus, the missing element in the third row, third column should be a black flower pot with a circular flower. a6. Answer: A)</td>
    <td>分析：这是一道图形推理题。... 第一行的图形按顺序时针旋转90度，并且颜色交替出现。根据这个规律，第三行的最后一个图形应该是黑色的，并且是顺时针旋转90度后的形状。a6. 答案：A<br><br>(Analysis: This is a figure reasoning question. ... The first row rotates 90 degrees clockwise sequentially, with alternating colors. ... Therefore, the third row should also rotate 90 degrees. Based on this pattern, the last shape in the third row should be black and rotated 90 degrees clockwise. a6. Answer: A)</td>
    <td>分析：... 图形的颜色有黑色和白色两种。图形的形状有两种：一种是正方形在上，正方形在下。另一种是三角形在上，正方形在下。根据这些规律，可以推断出问号处的图形应该是黑色三角形在上，白色正方形在下。a6. 答案：D<br><br>(Analysis: The shapes come in two colors: black and white. There are two possible arrangements of the shapes: one with the square on top and the triangle on the bottom, and the other with the triangle on top and the square on the bottom. ... Based on these patterns, it can be deduced that the shape at the question mark should be a black triangle on top with a white square at the bottom. a6. Answer: D)</td>
  </tr>
</table>

Figure 21: An example from the Multimodal subset. Correct analyses are highlighted in green, while errors are marked in red.

A.7.2 Task Distribution

Table 6 summarizes the detailed distribution of the 113 sampled tasks, including task ID, category label, difficulty level (1–3), and average human score.

A.7.3 Participant Demographics

A total of ten human evaluators were recruited for the human assessment, each assigned to specific subsets based on their background and expertise:

• Multimodal: This subset consists of visual reasoning questions inspired by Chinese civil service examinations. Three participants with prior experience in civil service exam preparation were selected. Two held undergraduate degrees in Chinese Language and Literature, and one held a degree in Artificial Intelligence.

• Logical, Graph Analysis, File Processing, General Knowledge: These subsets involve general reasoning tasks requiring basic operational knowledge of computer tools, without demanding domain-specific expertise. Two participants evaluated these subsets: one was a master’s student in Electronic Information Engineering, and the other was a bachelor’s graduate in Artificial Intelligence.

• Math: This subset includes mathematics problems at the high school to university level. Two participants, both from mathematics backgrounds, were involved: one was a Ph.D. candidate and the other a master’s student in Mathematics.

• Science: This subset comprises physics, biology, and chemistry questions at the level of the Chinese National College Entrance Examination (Gaokao). Two participants evaluated this subset: one was an undergraduate student majoring in Architecture with a science background from Gaokao, and the other was a master’s student in Physics.

• Reading: This subset contains reading comprehension tasks based on classical and modern literature, both Chinese and foreign. Three undergraduate students majoring in Chinese Language and Literature were recruited to evaluate this subset.
"Question": "在一天的24小时之中，时钟的时针、分针和秒针完全重合在一起有几次，分别是什么时间？"
("In a 24-hour day, how many times do the hour, minute, and second hands of a clock coincide completely, and at what times?")

"Label": "Logical", "Level": "3", "Final answer": "Twice, at 12:00 and 00:00", "File name": "", ""
"Steps": "1. 问题陈述...n2. 确定角度速度...n3. 分针与时针重合的计算...n4. 时间换算...n5. 秒针重合验证...n6. 三针重合条件：n7. 三针重合的唯一时间是在正12点和0点。n8. 数学证明：...n9. 最小公倍数：... 最小公倍数为 11 × 719。n10. 重合间隔周期为 11 × 719。n11. 下一次重合时间：t = (12 + i)小时。n12. 结论：n13. 在24小时内，时针、分针和秒针完全重合的次数只有两次（12点和0点）"
("1. Problem statement...n2. Determine angular velocities ...n3. Compute the coincidence of the minute and hour hands ...n4. Convert time units ...n5. Verify second-hand alignment ... n6. Condition for complete overlap:n - The only times when all three hands coincide are exactly at 12:00 and 00:00;n7. Mathematical proof; ...n8. Compute the coincidence time: ...n9. Least common multiple : The coincident interval is 11 × 719 times (Delta hours);n10. Time coincidence:n - The three hands coincide at t = (12 + i) hours; n11. Next coincidence: t = (12 + i) hours coincides again. The next coincidence happens at t = (12 + i) hours;n12. Conclusion: ... - In a 24-hour period, the hour, minute, and second hands coincide completely only twice (12:00 and 00:00).")

o1-preview
分析：n1.时钟的时针、分针和秒针完全重合的时刻，是当它们同时指向12的位置。...尽管它们的速度不同，但每12小时都回到同一位置。因此，时针、分针和秒针完全重合的时间是12:00:00和00:00:00。
'n'答案：n一共2次，分别在凌晨12:00:00和中午12:00:00时刻。
(Analysis: Within 24 hours, the hour hand and minute hand align at the 12 o'clock position once. However, with the addition of the second hand, the three hands only coincide when the second hand also points to the same position. Since the positions of the hour and minute hands repeat every 12 hours, this happens twice—once at 12:00:00 AM and once at 12:00:00 PM.)
'n'Answer: n total of two times, specifically at 12:00:00 AM and 12:00:00 PM.)

DeepSeek-R1
分析：n24小时内，时针和分针每12小时重合11次，但加入秒针后，只有当秒针也同时指向12的位置时才完全重合。由于秒针的运动，三针完全重合的次数会增加。具体来说，每12小时秒针转一圈，此时秒针位置与分针、时针同步。只有12点整和0点整时，三针完全重合。24小时内，三针完全重合的次数为24次，分别是0点0分0秒和12点0分0秒。
(Analysis: Within 24 hours, the hour hand and minute hand align at the 12 o'clock position 11 times. However, with the addition of the second hand, the three hands only coincide when the second hand also points to the same position. Since the positions of the hour and minute hands repeat every 12 hours, this happens twice—once at 12:00:00 AM and once at 12:00:00 PM.)
'n'Answer: n total of two times, specifically at 12:00:00 AM and 12:00:00 PM.)

Hunyuan-turbo-latest
分析：n要使时针、分针和秒针完全重合，秒针必须指向12点位置。一天有24小时，因此它们会重合24次。n'答案：n24次，分别是每个整点时刻（例如：0:00:00, 1:00:00, 2:00:00, ..., 23:00:00）
(Analysis: ... For all three hands to coincide, the second hand must also be at the 12 o'clock position at the exact moment when the minute and hour hands are aligned. It is determined that within a 24-hour period, the hour, minute, and second hands align perfectly only twice: at 12:00:00 AM and 12:00:00 PM.)
'n'Answer: n24 times, occurring at the start of each hour (e.g., 00:00:00, 1:00:00, 2:00:00, ..., 23:00:00.)')

GPT-4o
分析：n因为时针每12小时转一圈，分针每小时转一圈，秒针每分钟转一圈，所以它们在每个整点时刻重合一次。一天有24小时，因此它们会重合24次。n'答案：n24次，分别是每个整点时刻（例如：0:00:00, 1:00:00, 2:00:00, ..., 23:00:00）
(Analysis: ... For all three hands to coincide, the second hand must also be at the 12 o'clock position at the exact moment when the minute and hour hands are aligned. It is determined that within a 24-hour period, the hour, minute, and second hands align perfectly only twice: at 12:00:00 AM and 12:00:00 PM.)
'n'Answer: n24 times, occurring at the start of each hour (e.g., 00:00:00, 1:00:00, 2:00:00, ..., 23:00:00.)')

Figure 22: An example from the Logical subset. Correct analyses are highlighted in green, while errors are marked in red.
Table 6: Detailed distribution of the sampled tasks used for human evaluation.

<table>
  <tr>
    <th>Task ID</th>
    <th>Category</th>
    <th>Level</th>
    <th>Avg. Score</th>
  </tr>
  <tr><td>01007</td><td>Multimodal</td><td>1</td><td>1</td></tr>
  <tr><td>01016</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01018</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01031</td><td>Multimodal</td><td>3</td><td>0.67</td></tr>
  <tr><td>01039</td><td>Multimodal</td><td>3</td><td>0.67</td></tr>
  <tr><td>01042</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01046</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01057</td><td>Multimodal</td><td>3</td><td>1</td></tr>
  <tr><td>01075</td><td>Multimodal</td><td>3</td><td>1</td></tr>
  <tr><td>01106</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01107</td><td>Multimodal</td><td>2</td><td>0.33</td></tr>
  <tr><td>01111</td><td>Multimodal</td><td>2</td><td>0.67</td></tr>
  <tr><td>01133</td><td>Multimodal</td><td>2</td><td>0.67</td></tr>
  <tr><td>01142</td><td>Multimodal</td><td>3</td><td>0.33</td></tr>
  <tr><td>01143</td><td>Multimodal</td><td>3</td><td>0.33</td></tr>
  <tr><td>01145</td><td>Multimodal</td><td>3</td><td>0.33</td></tr>
  <tr><td>02003</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>02007</td><td>Logical</td><td>1</td><td>1</td></tr>
  <tr><td>02012</td><td>Logical</td><td>1</td><td>1</td></tr>
  <tr><td>02015</td><td>Logical</td><td>1</td><td>1</td></tr>
  <tr><td>02033</td><td>Logical</td><td>1</td><td>1</td></tr>
  <tr><td>02042</td><td>Logical</td><td>2</td><td>0</td></tr>
  <tr><td>02043</td><td>Logical</td><td>2</td><td>0</td></tr>
  <tr><td>02066</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>02073</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>02081</td><td>Logical</td><td>2</td><td>1</td></tr>
  <tr><td>02107</td><td>Logical</td><td>2</td><td>1</td></tr>
  <tr><td>02108</td><td>Logical</td><td>2</td><td>1</td></tr>
  <tr><td>02130</td><td>Logical</td><td>2</td><td>1</td></tr>
  <tr><td>02142</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>02156</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>02159</td><td>Logical</td><td>3</td><td>1</td></tr>
  <tr><td>03005</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03014</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03037</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03042</td><td>Graph Analysis</td><td>2</td><td>1</td></tr>
  <tr><td>03073</td><td>Graph Analysis</td><td>2</td><td>1</td></tr>
  <tr><td>03076</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03079</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03083</td><td>Graph Analysis</td><td>2</td><td>1</td></tr>
  <tr><td>03090</td><td>Graph Analysis</td><td>2</td><td>1</td></tr>
  <tr><td>03101</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>03116</td><td>Graph Analysis</td><td>1</td><td>1</td></tr>
  <tr><td>04002</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04011</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04012</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04017</td><td>Math</td><td>3</td><td>1</td></tr>
  <tr><td>04032</td><td>Math</td><td>3</td><td>1</td></tr>
  <tr><td>04041</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04045</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04054</td><td>Math</td><td>2</td><td>1</td></tr>
  <tr><td>04082</td><td>Math</td><td>2</td><td>1</td></tr>
  <tr><td>04085</td><td>Math</td><td>3</td><td>1</td></tr>
  <tr><td>04096</td><td>Math</td><td>3</td><td>1</td></tr>
  <tr><td>04114</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04124</td><td>Math</td><td>1</td><td>1</td></tr>
  <tr><td>04130</td><td>Math</td><td>2</td><td>1</td></tr>
  <tr><td>04137</td><td>Math</td><td>2</td><td>1</td></tr>
  <tr><td>04153</td><td>Math</td><td>3</td><td>1</td></tr>
  <tr><td>05003</td><td>Science</td><td>2</td><td>1</td></tr>
  <tr><td>05007</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05014</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05026</td><td>Science</td><td>1</td><td>0.75</td></tr>
  <tr><td>05037</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05043</td><td>Science</td><td>1</td><td>0.5</td></tr>
  <tr><td>05047</td><td>Science</td><td>2</td><td>1</td></tr>
  <tr><td>05049</td><td>Science</td><td>1</td><td>1</td></tr>
  <tr><td>05060</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05069</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05091</td><td>Science</td><td>1</td><td>1</td></tr>
  <tr><td>05096</td><td>Science</td><td>1</td><td>1</td></tr>
  <tr><td>05108</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05124</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05129</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05149</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>05155</td><td>Science</td><td>3</td><td>1</td></tr>
  <tr><td>06010</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06023</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06031</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06041</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06046</td><td>File Processing</td><td>1</td><td>1</td></tr>
  <tr><td>06047</td><td>File Processing</td><td>1</td><td>1</td></tr>
  <tr><td>06050</td><td>File Processing</td><td>3</td><td>1</td></tr>
  <tr><td>06056</td><td>File Processing</td><td>3</td><td>1</td></tr>
  <tr><td>06075</td><td>File Processing</td><td>1</td><td>1</td></tr>
  <tr><td>06094</td><td>File Processing</td><td>1</td><td>1</td></tr>
  <tr><td>06102</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06112</td><td>File Processing</td><td>2</td><td>1</td></tr>
  <tr><td>06138</td><td>File Processing</td><td>3</td><td>1</td></tr>
  <tr><td>06140</td><td>File Processing</td><td>3</td><td>1</td></tr>
  <tr><td>07022</td><td>General Knowledge</td><td>1</td><td>1</td></tr>
  <tr><td>07028</td><td>General Knowledge</td><td>1</td><td>1</td></tr>
  <tr><td>07029</td><td>General Knowledge</td><td>1</td><td>1</td></tr>
  <tr><td>07035</td><td>General Knowledge</td><td>3</td><td>0</td></tr>
  <tr><td>07045</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07057</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07059</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07076</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07090</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07100</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>07102</td><td>General Knowledge</td><td>2</td><td>1</td></tr>
  <tr><td>08019</td><td>Reading</td><td>1</td><td>1</td></tr>
  <tr><td>08018</td><td>Reading</td><td>1</td><td>1</td></tr>
  <tr><td>08027</td><td>Reading</td><td>1</td><td>1</td></tr>
  <tr><td>08033</td><td>Reading</td><td>1</td><td>1</td></tr>
  <tr><td>08038</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08041</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08063</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08064</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08075</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08109</td><td>Reading</td><td>2</td><td>1</td></tr>
  <tr><td>08110</td><td>Reading</td><td>2</td><td>1</td></tr>
</table>

Reviewer paper summaries: This paper proposes a Chinese reasoning benchmark characterized by multi-step, multi-task, multimodal, and mirror real-world challenges. Experiments on the proposed benchmark C2RBench reveal critical performance gaps: even SOTA models such as GPT-4o achieve only 45.20% accuracy, suggesting the challenging nature of C2RBench for advanced LLMs.

This paper introduces C²RBench, a Chinese benchmark for evaluating multi-step, multimodal reasoning capabilities of large language models (LLMs). The benchmark comprises 1,115 tasks across 8 domains (including math, science, logical reasoning, etc.) and 3 difficulty levels based on reasoning complexity. Tasks require an average of 8.44 reasoning steps, with level-3 tasks averaging 13.03 steps.

The paper proposes a benchmark for evaluating LLMs and MMLMs specifically targeted toward Chinese language. The benchmark proposes across two axis, one around 8 different like Math, Science, General Knowledge etc and another axis around number of reasoning steps required to solve the problem.
Summary of strengths: 1. The writing and presentation are clear.

2. The experiments are thorough and cover various models.

- Addresses a clear gap: The authors identify limitations in existing benchmarks, particularly the lack of Chinese reasoning benchmarks with multi-step reasoning requirements.
- Comprehensive design: C²RBench consists of 1,115 tasks across 8 domains and 3 difficulty levels, with an average of 8.44 reasoning steps per task (up to 13.03 for level-3 tasks).
- Diverse task formats: The benchmark incorporates various modalities (text, images, tables, files) and answer types (multiple-choice, open-ended) that simulate real-world reasoning scenarios.
- Extensive evaluation: The authors tested 16 LLMs and 20 MLLMs on C²RBench

The paper curates a new benchmark by scraping from four data sources and generating synthetic data. The paper claims to follow a stringent quality guideline.
Summary of weaknesses: 1. Due to the emergence of numerous evaluation datasets for large language models, verifying the efficiency and usability of these datasets has become a critical concern.

2. I  have some questions regarding the dataset construction process. How many initial samples were collected? How many samples were removed at each data processing stage? What were the specific annotation instructions and training procedures for the annotators?

3. How was quality control specifically implemented?

4. It's crucial to assess the quality of the constructed dataset by manually verifying.

- Limited reporting of human performance: The paper doesn't benchmark human performance, which would provide a useful reference point.
- The authors only describe what types of tasks are included, but I believe they should provide appropriate explanations for each type of task. For example, it's difficult to understand exactly what the Multimodal tasks entail.
- Despite the variety of task types, the specific number of questions for each task is not clear.
- Line 305, regarding ensuring the accuracy of "LLM as judge," what are the specific experimental details? For example, how many human annotators were recruited, and how was the quality of "LLM as judge" determined?
- For Text-only LLMs, the authors seem to have only tested the logical reasoning subset. Why? I think as part of the completeness of this paper, the performance of Text-only LLMs on other text-only tasks should also be tested. I consider this a major omission, and unless the authors provide clear feedback, I will not change my rating.
- I suggest that the authors release the data and code through an anonymous GitHub link.

The benchmark isn't novel, similar to other existing benchmarks like GSM8k for mathematical reasoning and GAIA for multimodal task. Major novelty should not be limited to only a new language.
Comments, suggestions and typos: 1. What is the evaluation prompt? Please provide an example.

2. The title does not reflect several characteristics of the dataset, such as multimodal and real-world challenges.

no

The protocols adhered to data curation and selection may be bought out. Also, the mechanism for achieving quality control.

The error bars in Fig 5 could be made consistent with levels

The paper attempts to mix too many things at one go. Taking a subset and doing it well could be considered.
Overall assessment scores: 2.5

2.5

3
Metareview score: 2
Metareview: This paper introduces a small but challenging Chinese benchmark for evaluating multi-step, multimodal reasoning capabilities of LLMs. The tasks cover 8 domains and 3 difficulty levels, requiring an average of 8 reasoning steps, or 13 steps in some tasks.
Summary of reasons to publish: - No such Chinese datasets exist, although there are similar datasets in other languages.
- Diverse task formats: multiple modalities, answer types, and domains (although some reviewer regarded it as a weakness).
- Extensive evaluation.