Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models
Authors: Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Dekai Sun, Zhicheng Zheng, Wenbo Su, Bo Zheng
Accepted at: acl25

Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models

Yancheng He1*, Shilong Li1*, Jiaheng Liu1,2†, Yingshui Tan1, Weixun Wang1, Hui Huang1, Xingyuan Bu1, Hangyu Guo1, Chengwei Hu1, Boren Zheng1, Zhuoran Lin1, Xuepeng Liu1, Dekai Sun1, Shirong Lin1, Zhicheng Zheng1, Xiaoyong Zhu1, Wenbo Su1, Bo Zheng1
1Alibaba Group, China 2Nanjing University, China
heyancheng.hyc@taobao.com, liujiaheng@nju.edu.cn

Abstract
New LLM benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of LLMs to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where reference answers are static and cannot be changed over time. Third, following SimpleQA, questions and answers are very short, and the grading process is easy-to-evaluate. Based on Chinese SimpleQA, we perform a comprehensive evaluation of the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide developers to better understand the factuality abilities of their models and facilitate the growth of LLMs1.

1 Introduction
A significant challenge in AI development is to ensure language models generate factually accurate responses (Zhao et al., 2023; Liu et al., 2025; Bu et al., 2021; Li et al., 2024; Bai et al., 2024; Huang et al., 2025; Zhang et al., 2024; Liu et al., 2024). Current frontier models sometimes produce false outputs or answers that are not substantiated by evidence. This is the problem known as “hallucinations”, which greatly hinders the extensive use of general AI technologies, such as large language models (LLMs). Besides, it is difficult to evaluate the factuality abilities of the existing LLMs. For example, LLMs usually generate lengthy responses containing numerous factual claims. Recently, to address the aforementioned evaluation problem, OpenAI has released the SimpleQA benchmark (Wei et al., 2024) with 4,326 concise and fact-seeking questions, which makes measuring factuality simple and reliable.

However, the SimpleQA benchmark primarily targets the English language, resulting in a limited understanding of LLMs’ capabilities in other languages. Moreover, inspired by several recent Chinese LLM benchmarks (e.g., C-Eval (Huang et al., 2023), CMMLU (Li et al., 2023b)), to evaluate the factuality abilities of LLMs in Chinese, we present the Chinese SimpleQA benchmark, which consists of 3000 high-quality questions spanning 6 major topics, ranging from humanities to science and engineering, as shown in the left of Figure 1. Specifically, the distinct main features of our proposed Chinese SimpleQA dataset are as follows:

• Chinese: Our Chinese SimpleQA focuses on the Chinese language, which provides a comprehensive evaluation of the factuality abilities of existing LLMs in Chinese.

• Diverse: Chinese SimpleQA covers 6 major topics (i.e., “Chinese Culture”, “Humanities”, “Engineering, Technology, and Applied Sciences”, “Life, Art, and Culture”, “Society”, and “Natural Science”), and these topic includes 99 fine-grained subtopics in total, which demonstrates the diversity of our Chinese SimpleQA.

• High-quality: We conduct a comprehensive and rigorous quality control process to ensure the quality and accuracy of Chinese SimpleQA.

• Static: Following SimpleQA, to preserve the evergreen property of Chinese SimpleQA, all reference answers would not change over time.

• Easy-to-evaluate: Following SimpleQA, as the questions and answers are very short, the grading procedure is fast to run via existing LLMs.
Figure 1: Left: Overview of Chinese SimpleQA. “Chinese Cul.” and “ETAS” represent “Chinese Culture” and “Engineering, Technology, and Applied Sciences”, respectively. Right: Dataset statistics of Chinese SimpleQA.

Moreover, we have performed a comprehensive evaluation and analysis on Chinese SimpleQA, and several insightful findings are as follows:

• Chinese SimpleQA is challenging. Only o1-preview and Doubao-pro-32k achieve the passing score (63.8% and 61.9% on the correct metric), and there is a long way to go for many existing LLMs.

• Larger models lead to better results. Based on the results of the Qwen2.5 series, InternLM series, Yi-1.5 series, etc, we observe that better performance is obtained when LLM is larger.

• Larger models are more calibrated. o1-preview is more calibrated than o1-mini, and GPT-4o is more calibrated than GPT-4o-mini.

• RAG matters. Performance gaps between different LLMs decrease a lot when using RAG (Retrieval-Augmented Generation). For GPT-4o and Qwen2.5-3B, the performance gap decreases from 42.4% to 9.3% using RAG.

• Alignment tax exists. Existing alignment or post-training strategies usually decrease the factuality of language models.

• Rankings of SimpleQA and Chinese SimpleQA are different. The performance of several LLMs focusing on Chinese (Doubao-pro-32k, and GLM-4-Plus) is close to the high-performance o1-preview. In particular, in the “Chinese Culture” topic, these Chinese community LLMs are significantly better than GPT or o1 series models.

<table>
  <tr>
    <th>Question Category</th>
    <th>Number</th>
  </tr>
  <tr>
    <td>Total</td>
    <td>3000</td>
  </tr>
  <tr>
    <td>- Chinese Culture</td>
    <td>326</td>
  </tr>
  <tr>
    <td>- Humanities</td>
    <td>609</td>
  </tr>
  <tr>
    <td>- Engineering, Technology and Applied Sciences</td>
    <td>481</td>
  </tr>
  <tr>
    <td>- Life, Art and Culture</td>
    <td>601</td>
  </tr>
  <tr>
    <td>- Society</td>
    <td>453</td>
  </tr>
  <tr>
    <td>- Natural Science</td>
    <td>530</td>
  </tr>
  <tr>
    <th colspan="2">Question Length</th>
  </tr>
  <tr>
    <td>- maximum length</td>
    <td>106</td>
  </tr>
  <tr>
    <td>- minimum length</td>
    <td>9</td>
  </tr>
  <tr>
    <td>- avg length</td>
    <td>29.4</td>
  </tr>
  <tr>
    <th colspan="2">Ref. Answer Length</th>
  </tr>
  <tr>
    <td>- maximum length</td>
    <td>33</td>
  </tr>
  <tr>
    <td>- minimum length</td>
    <td>1</td>
  </tr>
  <tr>
    <td>- avg length</td>
    <td>6.0</td>
  </tr>
</table>

2 Related Works

LLM Factuality. LLM factuality is the capability of large language models to produce contents that follow factual content, including commonsense, world knowledge, and domain facts, and the factual content can be substantiated by authoritative sources (e.g., Wikipedia, Textbooks). Recent works have explored the potential of LLMs to serve as factual knowledge bases (Yu et al., 2023; Pan et al., 2023; He et al., 2025). Specifically, existing studies have primarily focused on qualitative assessments of LLM factuality (Lin et al., 2022a; Chern et al., 2023), investigations into knowledge storage mechanisms (Meng et al., 2022; Chen et al., 2023), and analyses on knowledge-related issues (Gou et al., 2023).
Factuality Benchmarks. Many factuality benchmarks (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023; Li et al., 2023b; Srivastava et al., 2023; Yang et al., 2018) have been proposed. For example, MMLU (Hendrycks et al., 2021) is to measure the multitask accuracies on a diverse set of 57 tasks. TruthfulQA (Lin et al., 2022a) focuses on assessing the truthfulness of a language model’s generated answers. Additionally, HaluEval (Li et al., 2023c) is to examine the tendency of LLMs to produce hallucinations. Recently, SimpleQA (Wei et al., 2024) has been proposed to measure the short-form factuality in LLMs. However, SimpleQA only focuses on the English domain. In contrast, our Chinese SimpleQA aims to comprehensively evaluate factuality in Chinese.

3 Chinese SimpleQA

3.1 Overview

The left of Figure 1 shows the category distribution of Chinese SimpleQA, which encompasses a comprehensive set of 6 major topics with 99 distinct subtopics. The six main topics are: “Chinese Culture”, “Humanities”, “Engineering, Technology and Applied Sciences”, “Life, Art and Culture”, “Society”, and “Natural Science”, and the 99 subcategories are detailed in Appendix I. Moreover, we specially set up the category of “Chinese Culture” to evaluate the region-specific knowledge. We also present several examples from our dataset in Appendix G. The right of Figure 1 provides detailed statistics for the Chinese SimpleQA dataset. The dataset consists of 3,000 samples, and the distribution across the six major categories is relatively balanced, facilitating a comprehensive assessment of LLMs in diverse domains. In addition, the average answer length is approximately six tokens, ensuring efficient evaluation and minimizing potential errors.

3.2 Dataset Construction

The data construction process consists of the following steps: (1) extracting and filtering relevant knowledge content, (2) manually collecting high-quality question-answer examples for each topic, (3) generating question-answer pairs using predefined criteria, and (4) verifying and revising them if requirements are not met.

Specifically, we first collect a large volume of knowledge-rich text from various fields, such as Wikipedia² and Baidu Baike³. We then apply specific filtering rules and use a trained quality assessment model to remove low-quality data, such as texts with little information content or too specific. Next, we manually create several high-quality examples for each category to provide a few-shot learning foundation for the model’s question generation. These examples are designed to demonstrate various ways of asking questions to ensure diversity in the generated outputs. Finally, we prompt the LLM to generate question-answer pairs using these high-quality knowledge contents based on predefined criteria. We have also established an automated multi-round feedback mechanism: when the model generates a question-answer pair, it triggers another model to evaluate whether the pair meets the criteria. If it does not, the secondary model provides suggestions for improvement, and the main model regenerates the pair based on this feedback.

Notably, the construction of question-answer pairs is based on the following criteria:

Questions must be based on factual knowledge. Questions should pertain to objective facts about the world and must not be influenced by subjective opinions. For example, questions that begin with “What do you think about” or “How would you evaluate” are inappropriate, as they invite personal judgment rather than factual inquiry.

Answers must be unique and unambiguous. Each question must have a single, definitive answer, eliminating any possibility for multiple correct responses. Questions that have contentious or ambiguous answers are not suitable. For instance, the question “Who is the author of Dream of the Red Chamber?” does not meet the requirements because there are different opinions.

Answers must not change over time. Answers should reflect timeless facts unaffected by the time the question is posed. For example, “What is the atomic number of carbon?” and the answer “6” remains unchanged. In contrast, questions regarding current affairs, such as “Who is the current president of a certain country?” are inappropriate, as their answers are subject to change.

Questions must be challenging. Questions should not be overly simple, and queries need to
<table>
  <tr>
    <th>Benchmark</th>
    <th>Venue</th>
    <th>Data Size</th>
    <th>Language</th>
    <th>Data Source</th>
    <th>Categories</th>
    <th>Open-ended</th>
    <th>Reasoning</th>
    <th>Short-form</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>TriviaQA (Joshi et al., 2017)</td>
    <td>ACL</td>
    <td>650K</td>
    <td>English</td>
    <td>Human Collection</td>
    <td>-</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>NQ (Kwiatkowski et al., 2019)</td>
    <td>TACL</td>
    <td>3,610</td>
    <td>English</td>
    <td>Real World</td>
    <td>-</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>MMLU (Hendrycks et al., 2021)</td>
    <td>ICLR</td>
    <td>15,908</td>
    <td>English</td>
    <td>Exams</td>
    <td>57</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>TruthfulQA (Lin et al., 2022b)</td>
    <td>ACL</td>
    <td>817</td>
    <td>English</td>
    <td>Human Writers</td>
    <td>38</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>ROUGE</td>
  </tr>
  <tr>
    <td>C-Eval (Huang et al., 2023)</td>
    <td>NeurIPS</td>
    <td>13,948</td>
    <td>Chinese</td>
    <td>Exams</td>
    <td>52</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>CMMLU (Li et al., 2023a)</td>
    <td>ACL</td>
    <td>11,528</td>
    <td>Chinese</td>
    <td>Exams</td>
    <td>67</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>HalfEval (Li et al., 2023d)</td>
    <td>EMNLP</td>
    <td>5000</td>
    <td>English</td>
    <td>General</td>
    <td>-</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>ChineseFactEval (Chen et al., 2023)</td>
    <td>Arxiv</td>
    <td>12</td>
    <td>Chinese</td>
    <td>Human Collection</td>
    <td>7</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>LLM-as-a-judge</td>
  </tr>
  <tr>
    <td>AGI-Eval (Zhang et al., 2024)</td>
    <td>NAACL</td>
    <td>8062</td>
    <td>Chinese</td>
    <td>Exams</td>
    <td>20</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SimpleQA (Wei et al., 2023)</td>
    <td>Arxiv</td>
    <td>4,226</td>
    <td>English</td>
    <td>Human Writers</td>
    <td>10</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>LLM-as-a-judge</td>
  </tr>
  <tr>
    <td><b>Chinese SimpleQA (Ours)</b></td>
    <td></td>
    <td>5,000</td>
    <td>Chinese</td>
    <td>Self-constructed & Human Writers</td>
    <td>99</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>LLM-as-a-judge</td>
  </tr>
</table>

Table 1: Comparisons between our Chinese SimpleQA and other benchmarks.

![An overview of the entire production process of Chinese SimpleQA.](page_104_682_1237_246.png)

Figure 2: An overview of the entire production process of Chinese SimpleQA.

assess the knowledge depth thoroughly.

Questions must be answerable as of 2023. Each question must be answerable by December 31, 2023, ensuring fair evaluation for models trained on data available post this date.

3.3 Quality Control

Automatic Verification. We also build an automated verification mechanism, including LLM-based verification and RAG-based verification. Specifically, we first use LLM to determine whether the question meets the criteria and is relevant to the current category. If it does not meet the requirements or is irrelevant, the question and answer pair will be deleted directly. Then, we use the built RAG system to verify the answer. Here, we build it based on the LlamaIndex \( ^4 \) framework and use Google and Bing search results as data sources. If there are contradictory answers in the retrieval results, they will be deleted directly. For more details, please refer to Appendix K.

Difficulty Filtering. In addition, we filter simple questions to discover the knowledge boundaries of LLMs and improve the difficulty of Chinese SimpleQA. Specifically, if a question could be correctly answered by all five powerful models\(^5\), it is considered as a simple question and will be discarded.

Human Verification. Following the automated data collection, human verification is employed to further enhance the quality of the dataset. Specifically, each question is independently evaluated by two human annotators. Initially, the annotators assess whether the question adheres to the predefined criteria outlined earlier. If either annotator deems the question non-compliant, it is discarded. Subsequently, both annotators use search engines to retrieve relevant information and formulate answers. At this stage, the annotators are required to use content from authoritative sources (e.g., Wikipedia, Baidu Baike), and each must provide at least two supporting URLs. In cases where the annotators’ answers are inconsistent, a third annotator reviews the sample. The final annotation is determined by the third annotator, who references the initial two assessments. Finally, the human annotation results are compared with responses generated by the large language model (LLM), and only those question-answer pairs that are fully consistent are

\( ^4 \)https://github.com/run-llama/llama_index

\( ^5 \)GPT-4o (OpenAI, 2023), Meta-Llama-3-70B-Instruct (Dubey et al., 2024), Qwen2.5-72B-Instruct (Team, 2024d), DeepSeek-7B-chat (DeepSeek-AI, 2024a) and Baichuan2-7B-chat (Baichuan, 2023).
retained. This rigorous human verification process ensures that the dataset is both accurate and meets established standards.

Desensitization. After completing the above verification, we also use our safety risk model to filter to ensure that the final questions and answers do not contain any security risks.

Preventing Data Contamination. We need to mention that our Chinese SimpleQA evaluates the factuality knowledge abilities, where the knowledge is saved in the training corpus from the website (e.g., Wikipedia). Thus, we rigorously check generated questions against our question database for highly similar or identical ones. When such similarities are detected, we rewrite the questions to ensure uniqueness, which prevents direct memorization and ensures that the evaluation reflects the model’s ability to recall knowledge.

Analysis of the Retention Rate. In the collection process, many low-quality question-answer pairs are discarded. Specifically, 10,000 pairs are initially generated. After LLM-based verification and RAG-based verification, roughly 2,840 pairs are removed. After that, another 3,690 samples are removed after difficulty evaluation through testing with different models, which means that only about 35% of the original generated data remains. Finally, after a thorough and rigorous manual review, only about 3,000 samples are kept, which is approximately 30% of the original dataset.

3.4 Comparison to other benchmarks

In Table 1, we compare Chinese SimpleQA with several mainstream benchmarks, and our dataset is the first Chinese evaluation set to adopt a generative approach to evaluate the factuality abilities comprehensively. It is worth noting that the C-Eval and CMMLU mainly adopt multiple-choice evaluation methods that may introduce option bias and reduce the difficulty of questions, making it easier for models to guess the correct answer rather than truly understand the question (We do a detailed experimental analysis in Appendix C). In contrast, the generative evaluation method used in Chinese SimpleQA is closer to real-world scenarios. In addition, compared with other similar datasets, our benchmark has the advantages of high evaluation efficiency and more comprehensive coverage.

4 Experiments

4.1 Setup

We use the same prompt format in all experiments. The temperature and sampling parameters are the official configuration or default parameters of each LLM. The judge model we use is GPT-4o. We also evaluate using their smaller models as judge models, all of which achieved a high degree of consistency(see Appendix B for more details). For more details on the experimental implementation, please refer to Appendix H.

4.2 Baseline Models

We evaluate a total of 41 models, comprising 17 closed-source models and 24 open-source models. The closed-source models include: o1-preview 6, Doubao-pro-32k7, GLM-4-Plus8, GPT-4o9, Qwen-Max (Team, 2024c), Gemini-1.5-pro (Team, 2024a), DeepSeek-V2.5 (DeepSeek-AI, 2024b), Claude-3.5-Sonnet 10, Yi-Large11, moonshot-v1-8k12, GPT-4-turbo (OpenAI, 2023), GPT-4 (OpenAI, 2023), Baichuan3-turbo13, o1-mini14, Doubao-lite-4k15, GPT-4o-mini16, GPT-3.5 (Brown et al., 2020). The open-source models include: Qwen2.5 series (Team, 2024d), InternLM2.5 series (Team, 2024b), Yi-1.5 series (AI et al., 2024), LLaMA3 (AI@Meta, 2024) series, DeepSeek Series (DeepSeek-AI, 2024a), Baichuan2 series (Baichuan, 2023), Mistral series (Jiang et al., 2023), ChatGLM3 and GLM-4 (GLM et al., 2024; Du et al., 2022).

4.3 Evaluation Metrics

Following SimpleQA, we also adopt the following five evaluation metrics. (1) Correct (CO): The predicted answer fully includes the reference answer without introducing any contradictory elements. (2) Not attempted (NA): The reference answer is not fully given in the predicted answer,

6https://openai.com/index/introducing-openai-o1-preview/
7https://www.volcengine.com/product/doubao
8https://bigmodel.cn/dev/api/normal-model/glm-4
9https://openai.com/index/hello-gpt-4o/
10https://www.anthropic.com/news/claude-3-5-sonnet
11https://platform.lingyiwanwu.com/
12https://platform.moonshot.cn/
13https://platform.baichuan-ai.com/
14https://openai.com/o1/
15https://www.volcengine.com/product/doubao
16https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="5">Overall results on 5 metrics</th>
    <th colspan="6">F-score on 6 topics</th>
  </tr>
  <tr>
    <th>CO</th>
    <th>NA</th>
    <th>IN</th>
    <th>CGA</th>
    <th>F-score</th>
    <th>CC</th>
    <th>HU</th>
    <th>ETAS</th>
    <th>LAC</th>
    <th>SO</th>
    <th>NS</th>
  </tr>
  <tr>
    <th colspan="12">Closed-Source Large Language Models</th>
  </tr>
  <tr>
    <td>o1-preview</td>
    <td>63.8</td>
    <td>12.2</td>
    <td>24.0</td>
    <td>72.7</td>
    <td><b>67.9</b></td>
    <td>45.7</td>
    <td><b>69.8</b></td>
    <td><b>72.4</b></td>
    <td><b>65.0</b></td>
    <td><b>73.5</b></td>
    <td><b>72.3</b></td>
  </tr>
  <tr>
    <td>Doubao-pro-32k</td>
    <td>61.9</td>
    <td>10.3</td>
    <td>27.8</td>
    <td>69.1</td>
    <td>65.3</td>
    <td><b>61.8</b></td>
    <td>69.3</td>
    <td>69.0</td>
    <td>56.1</td>
    <td>64.2</td>
    <td>70.4</td>
  </tr>
  <tr>
    <td>GLM-4-Plus</td>
    <td>58.7</td>
    <td>7.4</td>
    <td>33.9</td>
    <td>63.4</td>
    <td>60.9</td>
    <td>56.5</td>
    <td>64.1</td>
    <td>64.9</td>
    <td>50.7</td>
    <td>66.6</td>
    <td>62.8</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>59.3</td>
    <td>1.4</td>
    <td>39.3</td>
    <td>60.1</td>
    <td>59.7</td>
    <td>39.4</td>
    <td>64.0</td>
    <td>65.1</td>
    <td>53.3</td>
    <td><b>68.6</b></td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>54.1</td>
    <td>11.3</td>
    <td>34.6</td>
    <td>61.0</td>
    <td>57.4</td>
    <td>47.8</td>
    <td>59.9</td>
    <td>63.5</td>
    <td>49.9</td>
    <td>61.2</td>
    <td>59.3</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>54.4</td>
    <td>8.0</td>
    <td>37.6</td>
    <td>59.1</td>
    <td>56.7</td>
    <td>41.4</td>
    <td>59.1</td>
    <td>60.8</td>
    <td>52.2</td>
    <td>56.3</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>DeepSeek-V2.5</td>
    <td>54.1</td>
    <td>5.9</td>
    <td>40.0</td>
    <td>57.5</td>
    <td>55.7</td>
    <td>50.4</td>
    <td>57.6</td>
    <td>58.8</td>
    <td>50.1</td>
    <td>59.4</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>46.2</td>
    <td>27.4</td>
    <td>26.4</td>
    <td>63.6</td>
    <td>53.5</td>
    <td>28.7</td>
    <td>61.3</td>
    <td>60.4</td>
    <td>42.2</td>
    <td>59.8</td>
    <td>57.7</td>
  </tr>
  <tr>
    <td>Yi-Large</td>
    <td>47.3</td>
    <td>16.4</td>
    <td>36.3</td>
    <td>56.6</td>
    <td>51.5</td>
    <td>41.1</td>
    <td>56.5</td>
    <td>55.1</td>
    <td>41.7</td>
    <td>57.6</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>moonshot-v1-8k</td>
    <td>48.7</td>
    <td>5.4</td>
    <td>45.9</td>
    <td>51.5</td>
    <td>50.1</td>
    <td>49.8</td>
    <td>54.1</td>
    <td>56.8</td>
    <td>41.4</td>
    <td>53.0</td>
    <td>46.6</td>
  </tr>
  <tr>
    <td>GPT-4-turbo</td>
    <td>45.6</td>
    <td>14.2</td>
    <td>40.2</td>
    <td>53.1</td>
    <td>49.1</td>
    <td>24.2</td>
    <td>55.2</td>
    <td>58.9</td>
    <td>43.9</td>
    <td>52.5</td>
    <td>50.8</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>45.4</td>
    <td>8.4</td>
    <td>46.2</td>
    <td>49.6</td>
    <td>47.4</td>
    <td>25.2</td>
    <td>54.0</td>
    <td>52.8</td>
    <td>41.8</td>
    <td>52.8</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td>Baichuan3-turbo</td>
    <td>45.2</td>
    <td>9.0</td>
    <td>45.8</td>
    <td>49.6</td>
    <td>47.3</td>
    <td>32.3</td>
    <td>52.5</td>
    <td>54.0</td>
    <td>35.4</td>
    <td>54.6</td>
    <td>50.9</td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>39.5</td>
    <td>20.6</td>
    <td>39.9</td>
    <td>49.7</td>
    <td>44.1</td>
    <td>21.3</td>
    <td>49.2</td>
    <td>55.9</td>
    <td>33.8</td>
    <td>48.8</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td>Doubao-lite-4k</td>
    <td>36.7</td>
    <td>31.2</td>
    <td>32.1</td>
    <td>53.3</td>
    <td>43.4</td>
    <td>40.2</td>
    <td>44.8</td>
    <td>51.0</td>
    <td>31.1</td>
    <td>41.4</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>37.6</td>
    <td>0.9</td>
    <td>61.5</td>
    <td>37.9</td>
    <td>37.8</td>
    <td>19.0</td>
    <td>42.4</td>
    <td>46.4</td>
    <td>31.0</td>
    <td>42.2</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>29.7</td>
    <td>2.9</td>
    <td>67.4</td>
    <td>30.6</td>
    <td>30.1</td>
    <td>13.3</td>
    <td>35.8</td>
    <td>35.2</td>
    <td>25.6</td>
    <td>32.7</td>
    <td>31.7</td>
  </tr>
  <tr>
    <th colspan="12">Open-Source Large Language Models</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>39.8</td>
    <td>11.7</td>
    <td>48.5</td>
    <td>45.1</td>
    <td>42.3</td>
    <td>35.9</td>
    <td>46.0</td>
    <td>44.7</td>
    <td>27.3</td>
    <td>43.5</td>
    <td>41.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>48.4</td>
    <td>7.1</td>
    <td>44.5</td>
    <td>52.1</td>
    <td>50.2</td>
    <td>36.3</td>
    <td>56.1</td>
    <td>57.9</td>
    <td>37.1</td>
    <td>53.3</td>
    <td>56.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>38.8</td>
    <td>11.1</td>
    <td>50.1</td>
    <td>43.6</td>
    <td>41.1</td>
    <td>33.7</td>
    <td>45.8</td>
    <td>48.7</td>
    <td>27.3</td>
    <td>44.7</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>35.4</td>
    <td>9.6</td>
    <td>55.0</td>
    <td>39.2</td>
    <td>37.2</td>
    <td>30.2</td>
    <td>41.8</td>
    <td>46.1</td>
    <td>24.1</td>
    <td>38.8</td>
    <td>41.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>26.6</td>
    <td>9.9</td>
    <td>63.5</td>
    <td>29.5</td>
    <td>27.9</td>
    <td>20.1</td>
    <td>32.7</td>
    <td>33.8</td>
    <td>18.0</td>
    <td>28.6</td>
    <td>32.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>16.2</td>
    <td>12.8</td>
    <td>71.0</td>
    <td>18.6</td>
    <td>17.3</td>
    <td>13.4</td>
    <td>17.9</td>
    <td>26.1</td>
    <td>9.3</td>
    <td>15.6</td>
    <td>20.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>11.1</td>
    <td>14.6</td>
    <td>74.3</td>
    <td>13.1</td>
    <td>12.0</td>
    <td>11.0</td>
    <td>11.3</td>
    <td>18.7</td>
    <td>6.7</td>
    <td>12.2</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>GLM4-9B</td>
    <td>25.9</td>
    <td>12.5</td>
    <td>61.6</td>
    <td>29.6</td>
    <td>27.6</td>
    <td>28.8</td>
    <td>32.1</td>
    <td>32.0</td>
    <td>17.6</td>
    <td>28.9</td>
    <td>27.8</td>
  </tr>
  <tr>
    <td>ChatGLM3-6B</td>
    <td>11.2</td>
    <td>13.6</td>
    <td>75.2</td>
    <td>12.9</td>
    <td>12.0</td>
    <td>12.1</td>
    <td>13.8</td>
    <td>12.4</td>
    <td>8.8</td>
    <td>13.4</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>InternLM2.5-20B</td>
    <td>31.5</td>
    <td>7.7</td>
    <td>60.8</td>
    <td>34.1</td>
    <td>32.8</td>
    <td>32.0</td>
    <td>37.1</td>
    <td>37.7</td>
    <td>21.2</td>
    <td>35.7</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>24.7</td>
    <td>7.5</td>
    <td>67.8</td>
    <td>26.7</td>
    <td>25.7</td>
    <td>25.5</td>
    <td>29.4</td>
    <td>31.0</td>
    <td>16.4</td>
    <td>26.9</td>
    <td>25.8</td>
  </tr>
  <tr>
    <td>InternLM2.5-1.8B</td>
    <td>5.3</td>
    <td>31.1</td>
    <td>63.6</td>
    <td>7.6</td>
    <td>6.2</td>
    <td>6.1</td>
    <td>8.7</td>
    <td>7.2</td>
    <td>3.3</td>
    <td>4.5</td>
    <td>7.4</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B</td>
    <td>30.9</td>
    <td>5.8</td>
    <td>63.3</td>
    <td>32.8</td>
    <td>31.8</td>
    <td>28.2</td>
    <td>36.9</td>
    <td>36.8</td>
    <td>24.4</td>
    <td>32.8</td>
    <td>31.4</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>18.2</td>
    <td>2.9</td>
    <td>78.9</td>
    <td>18.7</td>
    <td>18.4</td>
    <td>17.2</td>
    <td>20.2</td>
    <td>24.3</td>
    <td>10.2</td>
    <td>20.1</td>
    <td>19.8</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>15.9</td>
    <td>2.8</td>
    <td>81.3</td>
    <td>16.3</td>
    <td>16.1</td>
    <td>14.2</td>
    <td>17.9</td>
    <td>21.3</td>
    <td>10.3</td>
    <td>16.8</td>
    <td>16.5</td>
  </tr>
  <tr>
    <td>LLaMA3.1-70B</td>
    <td>38.3</td>
    <td>9.4</td>
    <td>52.3</td>
    <td>42.3</td>
    <td>40.2</td>
    <td>22.9</td>
    <td>47.2</td>
    <td>49.3</td>
    <td>34.5</td>
    <td>49.6</td>
    <td>40.4</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td>16.9</td>
    <td>8.8</td>
    <td>74.3</td>
    <td>18.6</td>
    <td>17.7</td>
    <td>8.5</td>
    <td>20.7</td>
    <td>23.4</td>
    <td>9.7</td>
    <td>20.5</td>
    <td>20.7</td>
  </tr>
  <tr>
    <td>DeepSeek-67B</td>
    <td>43.5</td>
    <td>14.8</td>
    <td>41.7</td>
    <td>51.1</td>
    <td>47.0</td>
    <td>34.3</td>
    <td>54.5</td>
    <td>50.3</td>
    <td>42.3</td>
    <td>49.0</td>
    <td>46.2</td>
  </tr>
  <tr>
    <td>DeepSeek-V2-Lite-Chat</td>
    <td>33.7</td>
    <td>12.8</td>
    <td>53.5</td>
    <td>38.6</td>
    <td>36.0</td>
    <td>35.3</td>
    <td>38.5</td>
    <td>41.7</td>
    <td>32.2</td>
    <td>37.5</td>
    <td>31.2</td>
  </tr>
  <tr>
    <td>DeepSeek-7B</td>
    <td>23.2</td>
    <td>13.2</td>
    <td>63.6</td>
    <td>26.7</td>
    <td>24.8</td>
    <td>24.5</td>
    <td>27.2</td>
    <td>28.9</td>
    <td>20.6</td>
    <td>27.0</td>
    <td>21.5</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>19.1</td>
    <td>24.9</td>
    <td>56.0</td>
    <td>25.4</td>
    <td>21.8</td>
    <td>24.0</td>
    <td>25.8</td>
    <td>23.3</td>
    <td>16.8</td>
    <td>23.0</td>
    <td>18.7</td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>12.5</td>
    <td>21.8</td>
    <td>65.7</td>
    <td>16.0</td>
    <td>14.0</td>
    <td>14.6</td>
    <td>16.1</td>
    <td>15.4</td>
    <td>11.1</td>
    <td>13.8</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>Mixtral-8x22B-Instruct-v0.1</td>
    <td>27.3</td>
    <td>2.2</td>
    <td>70.5</td>
    <td>27.9</td>
    <td>27.6</td>
    <td>10.6</td>
    <td>32.3</td>
    <td>36.0</td>
    <td>21.0</td>
    <td>34.1</td>
    <td>26.9</td>
  </tr>
  <tr>
    <td>Mixtral-8x7B-Instruct-v0.1</td>
    <td>20.4</td>
    <td>7.2</td>
    <td>72.4</td>
    <td>22.0</td>
    <td>21.2</td>
    <td>5.2</td>
    <td>26.5</td>
    <td>29.0</td>
    <td>13.0</td>
    <td>25.0</td>
    <td>23.3</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.2</td>
    <td>15.0</td>
    <td>8.8</td>
    <td>76.2</td>
    <td>16.4</td>
    <td>15.6</td>
    <td>4.5</td>
    <td>18.2</td>
    <td>22.2</td>
    <td>9.5</td>
    <td>21.4</td>
    <td>15.7</td>
  </tr>
</table>

Table 2: Results of different models on Chinese SimpleQA. For metrics, CO, NA, IN, and CGA denote “Correct”, “Not attempted”, “Incorrect”, and “Correct given attempted”, respectively. For subtopics, CC, HU, ETAS, LAC, SO and NS represent “Chinese Culture”, “Humanities”, “Engineering, Technology, and Applied Sciences”, “Life, Art, and Culture”, “Society”, and “Natural Science”, respectively. Bold indicates the best results within the same group of models, while underline indicates the second best.

and there are no contradictory elements with the reference answer. (3) Incorrect (IN): The predicted answer contradicts the reference answer, even if the contradiction is solved. (4). Correct given attempted (CGA): The metric is the proportion of accurately answered questions among those attempted questions. (5). F-score: The metric represents the harmonic mean between correct and
correct given attempted.

4.4 Main Results

In Table 2, we provide the results of different LLMs on Chinese SimpleQA. Specifically, we provide the overall results on 5 evaluation metrics. Additionally, we report the F-score for 6 topics to analyze fine-grained factuality abilities. We have the following insightful observations:

• o1-preview achieves the best performance on Chinese SimpleQA, and results of several recent closed-source LLMs focusing on Chinese (Doubao-pro-32k and GLM-4-Plus) are very close to o1-preview.

• It is obvious that the “mini” series models (o1-mini, GPT-4o-mini) achieve lower results than the corresponding larger models (o1-preview, GPT-4o), which also indicates these “mini” series models do not pay attention to memorize factuality knowledge.

• A Larger LLM leads to better performance, where we can draw this conclusion based on many model series (e.g., GPT, Qwen2.5, InternLM2.5, Yi-1.5). The scatter plot in Figure 4 shows a clearer positive correlation between model scale and performance.

• Small LLMs usually lead to higher scores on “not attempted (NA)”. The NA scores for o1-mini, InternLM2.5-1.8B are 20.5 and 31.2, respectively, which are larger than the scores of corresponding larger LLMs a lot (o1-preview with 12.2, InternLM2.5-20B with 7.7).

• There is a significant performance difference among different subtopics for different LLMs. Notably, the Chinese community LLMs (e.g., Doubao-pro-32k, GLM-4-Plus, Qwen-Max) are significantly better than GPT or o1 models in the Chinese Culture (CC) subtopic. In contrast, o1 has significant advantages in science-related subtopics (e.g., ETAS and NS).

In addition, we also provide the detailed results (CO and CGA metrics) on 6 topics in Figure 3.

4.5 Further Analysis

Analysis of Calibration. For the calibration of different LLMs, following SimpleQA, we instruct the model to provide a corresponding confidence level (from 0 to 100) when answering questions to measure the model’s confidence in its answers (See the prompt in Appendix L). We know that a perfectly calibrated model’s confidence (%) should match its answers’ actual accuracy. The left plot in Figure 5 illustrates the alignment performance, which indicates that GPT-4o aligns better than GPT-4o-mini and o1-preview aligns better than o1-mini. For the Qwen2.5 series, the alignment order is Qwen2.5-72B > Qwen2.5-32B > Qwen2.5-7B > Qwen2.5-3B, which suggests that larger model sizes result in better calibration. Furthermore, for all evaluated models, their confidence in the range of confidence > 50 falls below the line of perfect alignment, which means that they all overestimate the accuracy of their responses and overconfidence exists.

Analysis of Test-Time Compute. We also evaluate the relationship between increased test-time compute and response accuracy for different LLMs. Specifically, we randomly sample 50 samples from Chinese SimpleQA, and for each sample, the model is asked to independently answer 100 times. Then, we obtain the model’s response accuracy using the Best-of-N method as the inference counts increase. The results are shown in the right plot of Figure 5. We observe that as the times of inferences increase, the response accuracy of all models improves and eventually reaches a ceiling. This is reasonable for Chinese SimpleQA to probe the boundaries of a model’s knowledge.

Analysis on the effect of RAG. We explore the effect of the Retrieval-Augmented Generation (RAG) in enhancing the factual accuracy of LLMs on Chinese SimpleQA. Specifically, we reproduce a RAG system based on LlamaIndex (Liu, 2022), incorporating Google search APIs(For an ablation study on different retrieval APIs, see Appendix F). In Figure 6, all models show a substantial improvement in accuracy with RAG. For example, the performance of Qwen2.5-3B improved more than threefold. Notably, nearly all models with RAG outperform native GPT-4o. Meanwhile, RAG also leads to a marked reduction in performance disparities among models. For example, the F-score difference between Qwen2.5-3B with RAG and Qwen2.5-72B with RAG is only 6.9%. This suggests that RAG reduces the performance gaps greatly, enabling even smaller ones to achieve high performance when augmented with RAG. Overall, this suggests that RAG serves as an effective shortcut for enhancing the factuality.

Analysis on the alignment tax. Recently, prior
Figure 3: Results (CO and CGA metrics) of different models for six topics.

Figure 4: Relationship between model scale (in billion parameters) and F-score on Chinese SimpleQA.

Figure 5: Left: Calibration of LLMs based on their stated confidence. Right: Improvement in accuracy with increased test-time compute using Best-of-N.

Figure 6: The effect of RAG strategy.

Figure 7: The effect of alignment in post-training.

studies (OpenAI, 2023; Song et al., 2023) have found that the alignment can lead to a decrease in the abilities of language models as known as the “alignment tax”. To illustrate the effect of alignment on factuality, we conduct a comparative performance analysis between pre-trained models and aligned models that are trained with Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF). As illustrated in Figure 7, different models exhibit varying trends after post-training, but most models have a significant decline. Among these, the Baichuan2 series models show the most significant decreases, with Baichuan2-7B and Baichuan2-13B experiencing F-score reductions of 47% and 28%, respectively. This reflects that the alignment training of most current LLMs still has obvious drawbacks of knowledge hallucinations, which further reflects the necessity of our Chinese SimpleQA dataset.

Comparison between Chinese SimpleQA and SimpleQA. We also compare the ranking differences of various models on the SimpleQA and the Chinese SimpleQA. In Figure 8, there are notable discrepancies in model performance across these two benchmarks. For instance, Doubao-pro-32k ranks significantly higher on the Chinese SimpleQA, moving from 12th to 2nd place (+10). Conversely, GPT-4 shows a decline in performance on the Chinese SimpleQA, dropping from 3rd to 9th
place (-6). These differences emphasize the importance of evaluating models on datasets in various languages and the need for research into optimizing model performance across different linguistic environments. In addition, most Chinese community-developed models (e.g., Qwen-Max, GLM-4-Plus, Yi-Large, Doubao-pro-32k) perform better on the Chinese SimpleQA than on the SimpleQA, showing their advantages on Chinese.

5 Conclusion

In this paper, we propose the first Chinese short-form factuality benchmark (i.e., Chinese SimpleQA), which mainly has five important features (i.e., Chinese, diverse, high-quality, static, and easy-to-evaluate). Besides, we comprehensively evaluate the performance of existing 40+ LLMs on factuality and provide detailed analysis to demonstrate the advantages of our Chinese SimpleQA.

6 Limitations

While Chinese SimpleQA provides valuable insights, it has some limitations. Its coverage of six main topics and 99 subtopics may not fully capture the diversity of all domains, particularly niche or emerging areas. As a static benchmark, it cannot reflect real-time advancements or evolving factual information. Additionally, its focus on short-form questions and reliance on simple evaluation metrics might overlook complex reasoning tasks or nuanced factuality. Addressing these limitations will be our focus in future work.

References

01. AI, ;, Alex Young, Bei Chen, Chao Li, Chengu Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. Preprint, arXiv:2403.04652.

AI@Meta. 2024. Llama 3 model card.

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762.

Baichuan. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Xingyuan Bu, Junran Peng, Junjie Yan, Tieniu Tan, and Zhaoxiang Zhang. 2021. Gaia: A transfer learning system of object detection that fits your needs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 274–283.

Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. Preprint, arXiv:2308.13198.

I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023. Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios. Preprint, arXiv:2307.13528.

DeepSeek-AI. 2024a. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954.

DeepSeek-AI. 2024b. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezong Qiu, Zhilin Yang, and Jie Tang. 2022. Glim: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Harshtron, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie
Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolidakis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Milanon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarral, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vrane, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junting Jia, Kalyan Vasudevanwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Manatt Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girhard, Rohit Patel, Romain Sauvestre, Ronnie Polidorio, Roshan Sumbally, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthi, Sheng Shen, Shengye Wan, Shruti Bhsale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaev, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenying Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuwei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yunjing Mao, Zacharie Delpierre Couedt, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsian Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Tsegugine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcueta, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Bader, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegen, Hunter Goldman, Igor Molbyog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatkov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natasha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ramchandani, Pritish Yuvraj, Qian Liang, Rachad Alao,
Rachel Rodriguez, Rafi Ayub, Raghatham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadhi Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuiqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjin Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaoqian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. arXiv preprint arXiv: 2407.21783.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen-hui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Han-lin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.

Zhibin Gou, Zhijhong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. Preprint, arXiv:2305.11738.

Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. 2025. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).

Hui Huang, Jiaheng Liu, Yancheng He, Shilong Li, Bing Xu, Conghui Zhu, Muyun Yang, and Tiejun Zhao. 2025. Muse: Improving complex instruction following with multi-granularity self-contrastive training. arXiv preprint arXiv:2502.11541.

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. Vancouver, Canada. Association for Computational Linguistics.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Tim Baldwin. 2023a. Cmmlu: Measuring massive multi-task language understanding in chinese. ArXiv, abs/2306.09212.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023b. Cmmlu: Measuring massive multi-task language understanding in chinese. Preprint, arXiv:2306.09212.

Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023c. Halueval: A large-scale hallucination evaluation benchmark for large language models. Preprint, arXiv:2305.11747.

Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023d. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Computational Linguistics.
Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. 2024. Graphreader: Building graph-based agent to enhance long-context abilities of large language models. arXiv preprint arXiv:2406.14550.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022a. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022b. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.

Jerry Liu. 2022. LlamaIndex.

Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. 2024. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157.

Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36.

OpenAI. 2023. Gpt-4 technical report. PREPRINT.

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jianpu Wang, and Xindong Wu. 2023. Unifying large language models and knowledge graphs: A roadmap. Preprint, arXiv:2306.08302.

Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. 2023. Reward collapse in aligning large language models. arXiv preprint arXiv:2305.17608.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

Gemini Team. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

InternLM2 Team. 2024b. Internlm2 technical report. Preprint, arXiv:2403.17297.

Qwen Team. 2024c. Introducing qwen1.5.

Qwen Team. 2024d. Qwen2.5: A party of foundation models.

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Preprint, arXiv:1809.09600.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Conference for Learning Representation (ICLR).

Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Ying-hao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhang Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhai Huang, and Wenhui Chen. 2024. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405.19327.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zigang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. abs/2303.18223.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. AGIEval: A human-centric benchmark for evaluating foundation models. Preprint, arXiv:2304.06364.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. AGIEval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico. Association for Computational Linguistics.
Figure 9: Model Performance Across Different CoT Strategies.

Figure 10: Robustness of Judge Models.

<table>
  <tr>
    <th>Model</th>
    <th>Single-Choice Accuracy (%)</th>
    <th>Shuffled-Options Accuracy (%)</th>
    <th>Open-Ended QA Accuracy (%)</th>
    <th>Accuracy Drop (%)</th>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>84.89</td>
    <td>83.99</td>
    <td>66.77</td>
    <td>21.35</td>
  </tr>
  <tr>
    <td>Yi-Large</td>
    <td>71.60</td>
    <td>68.88</td>
    <td>56.80</td>
    <td>20.80</td>
  </tr>
  <tr>
    <td>Baichuan3-turbo</td>
    <td>74.62</td>
    <td>73.23</td>
    <td>59.82</td>
    <td>19.84</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>60.12</td>
    <td>60.42</td>
    <td>42.30</td>
    <td>29.65</td>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>48.20</td>
    <td>46.83</td>
    <td>30.21</td>
    <td>37.32</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>73.41</td>
    <td>75.53</td>
    <td>63.14</td>
    <td>13.99</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>75.23</td>
    <td>75.83</td>
    <td>54.38</td>
    <td>27.71</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Chat</td>
    <td>48.64</td>
    <td>47.73</td>
    <td>25.08</td>
    <td>48.45</td>
  </tr>
  <tr>
    <td>ChatGLM3-6B</td>
    <td>47.73</td>
    <td>45.32</td>
    <td>19.34</td>
    <td>59.49</td>
  </tr>
</table>

Table 3: Evaluation results of various models using three methods: original multiple-choice questions, shuffled-options multiple-choice questions, and open-ended QA. The table illustrates accuracy differences across evaluation formats and highlights the impact of option bias and the challenges posed by open-ended QA.

A  Analysis on the effect of Chain-of-Thought

To evaluate the impact of Chain-of-Thought (CoT) prompting strategies on model factuality, we implemented and compared two approaches: “Recall and Answer” and “Step by Step”. As illustrated in Figure 9, the performance differences induced by these strategies are not substantial across models of varying scales. Notably, larger models, such as Qwen2.5-72B-Instruct and Qwen-Max, display a marginal improvement with CoT strategies, but the gains remain relatively modest compared to baseline performances. These findings align with conclusions drawn in similar studies, such as those in C-EVAL and CMMLU, where CoT prompting did not universally enhance accuracy, and in some cases, resulted in performance degradation. This suggests that the effectiveness of CoT strategies in improving factuality may be limited for tasks that do not require intensive reasoning.

B  Robustness of Judge Models

The figure 10 shows the evaluation results of six models selected from different performance levels of the current ranking. We use various evaluation models to evaluate these models. The evaluation models include Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-72B-Instruct, and GPT-4o used in the previous experiment. It can be seen that although the scores of different evaluation models are different, the relative scores and rankings of each model have not changed. This consistency demonstrates the robustness of our evaluation set and evaluation method, and means that using smaller-scale evaluation models (such as Qwen2.5-3B-Instruct or Qwen2.5-7B-Instruct) can also have a high consistency rate, and can be evaluated efficiently even with limited resources.

C  Biases in Choice Evaluation

The experimental results presented in Table 3 demonstrate the impact of option bias in multiple-choice evaluations and highlight the importance of open-ended question-answering (QA) evaluations. To conduct the experiment, we used GPT-4 to transform the CEval dataset into a question-answering format, applying the Chinese SimpleQA pipeline to filter the data. This process resulted in a curated set of 300 high-quality QA items for experimentation.
Figure 12: Detailed results on some selected subtopics.

Each model was evaluated using three methods: (1) the original multiple-choice question format, (2) the multiple-choice format with shuffled options, and (3) the open-ended QA format. Results show that most models exhibit a significant reduction in accuracy when evaluated with the open-ended QA format (note: the questions remained consistent across formats). For instance, the Qwen-Max model’s accuracy dropped by 21%, while GPT-3.5’s accuracy decreased by 30%. These findings suggest that the multiple-choice format simplifies the evaluation process, potentially underestimating the difficulty of the task. Additionally, some models exhibited changes in accuracy after the options were shuffled, indicating the presence of option bias. These observations raise concerns that the multiple-choice evaluation method may fail to fully capture the factual correctness of the models or adequately address hallucination tendencies. In contrast, the generation-based evaluation aligns more closely with realistic task scenarios. This format requires models to generate free-form answers, eliminating the possibility of “guessing” from predefined options and providing a clearer assessment of the model’s reasoning and comprehension abilities.

We hope that Chinese SimpleQA can provide the research community with a more accurate perspective to advance model development, rather than just following the trend of multiple-choice based evaluation.

D  Analysis Across Answer Types

In Figure 11, we show the accuracy of each model on different answer types. It can be observed that the accuracy of responses to questions involving dates and numerical information is significantly lower compared to other answer types such as organizations, places, and persons. This suggests that the models are more prone to generating hallucinations when processing numerical data, highlighting a persistent challenge in addressing arithmetic reasoning and date-related tasks within these systems.

E  Analysis on the results of subtopics

As mentioned in Section 3.2, the benchmark covers a total of 99 subtopics, which can comprehensively detect the knowledge level of the model in various domains. Figure 12 illustrates the performance
<table>
  <tr>
    <th>Model</th>
    <th>CO</th>
    <th>NA</th>
    <th>IN</th>
    <th>CGA</th>
    <th>F-score</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>59.3</td>
    <td>1.3</td>
    <td>39.3</td>
    <td>60.1</td>
    <td>59.7</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>78.4</td>
    <td>4.2</td>
    <td>13.2</td>
    <td>85.6</td>
    <td>81.8</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>79.2</td>
    <td>5.7</td>
    <td>15.1</td>
    <td>84.0</td>
    <td>81.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>48.4</td>
    <td>1.8</td>
    <td>44.5</td>
    <td>52.1</td>
    <td>50.2</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>74.9</td>
    <td>4.2</td>
    <td>13.9</td>
    <td>84.3</td>
    <td>79.4</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>77.5</td>
    <td>6.7</td>
    <td>15.8</td>
    <td>81.5</td>
    <td>79.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>38.8</td>
    <td>5.6</td>
    <td>50.1</td>
    <td>43.6</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>74.1</td>
    <td>7.2</td>
    <td>12.8</td>
    <td>85.2</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>72.9</td>
    <td>10.7</td>
    <td>15.8</td>
    <td>82.1</td>
    <td>77.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>26.6</td>
    <td>4.6</td>
    <td>63.5</td>
    <td>29.5</td>
    <td>27.9</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>71.1</td>
    <td>6.0</td>
    <td>16.9</td>
    <td>80.8</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>68.3</td>
    <td>11.2</td>
    <td>19.5</td>
    <td>77.8</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>16.2</td>
    <td>7.6</td>
    <td>71.0</td>
    <td>18.6</td>
    <td>17.3</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>68.8</td>
    <td>4.4</td>
    <td>20.9</td>
    <td>76.7</td>
    <td>72.5</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>65.4</td>
    <td>7.7</td>
    <td>26.4</td>
    <td>71.2</td>
    <td>68.2</td>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>37.6</td>
    <td>0.9</td>
    <td>61.5</td>
    <td>37.9</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>w. RAG (Google’s API)</td>
    <td>76.8</td>
    <td>2.0</td>
    <td>17.0</td>
    <td>81.9</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>w. RAG (Baidu’s API)</td>
    <td>75.0</td>
    <td>2.9</td>
    <td>22.1</td>
    <td>77.3</td>
    <td>76.1</td>
  </tr>
</table>

Table 4: Performance Comparison of Models with and without RAG using Different APIs.

comparison between the o1 model and seven notable Chinese community models within several common domains. Firstly, from an overall perspective, the o1-preview model exhibits the most comprehensive performance across these domains, with the Doubao model following closely. In contrast, the Moonshot model demonstrates the weakest overall performance. Secondly, when examining specific domains, a significant disparity emerges between the Chinese community models and the o1 model in areas such as Computer Science and Medicine. However, this gap is minimal in domains like Education and Economics. Notably, in Education, some Chinese community models outperform the o1-preview, highlighting their potential for achieving success in specific vertical domains. Lastly, when examining specific models, the Moonshot model is notably weaker in Mathematics, Law, and Entertainment, while the Baichuan model also underperforms in Entertainment. The Yi-Large model excels in Education, and the o1 model maintains the strongest performance across other domains. Evaluating the performance of the models across diverse domains within the benchmark dataset enables users to identify the most suitable model for their specific needs.

F  Performance Comparison of Different Retrieval APIs

we conduct additional experiments using Baidu as the search engine within the RAG framework. The experimental results are as follows in Table 4.

Our findings indicate that the F-score performance of most models using Baidu’s API is lower compared to those using Google’s API. Further analysis reveals that the retrieval contexts from Baidu’s API exhibit relatively higher noises, suggesting the need for more effective filtering strategies. Additionally, we observed that when the model is larger, the performance degradation is smaller. This suggests that larger models are more robust in handling noisy or inconsistent input data.

G  Chinese SimpleQA Examples

We present several examples of our dataset in Figure 13.
H  Evaluation Details

For evaluating the chat models, we use the official chat template. For the pre-trained models, we employ the few-shot learning approach, where a small set of high-quality question-answer examples is provided to guide the model in understanding the task and generating accurate, concise responses.

For evaluating models with RAG, the model is given relevant reference materials related to the user’s question. The system prompt instructs the model to prioritize information from the retrieved information, or, if no relevant information is found, to rely on its own knowledge to generate the answer. The prompt is illustrated in Figure 14.

I  Distribution of Subtopics

Figure 15 shows the distribution of the 99 categories in the SimpleQA dataset, including the count and the average question token length.

Figure 16 illustrates the difficulty distribution for questions across 99 distinct categories, measured by the number of incorrect responses from models. Firstly, many categories exhibit narrow and symmetrical violin shapes, indicating a relatively uniform distribution of question difficulty. This suggests that within these categories, questions are consistently challenging, providing a balanced assessment environment for model evaluation. Such consistency is crucial for ensuring fairness in comparisons across different models, as it minimizes biases introduced by disproportionately easy or overly difficult questions. Secondly, the overall layout shows the diversity of the dataset, while most categories maintain a reasonable difficulty distribution. This balance enables the identification of strengths and weaknesses in model performance without being affected by extreme outliers or irregular distributions.

J  Models Performance Across Difference Topics
<table>
  <tr>
    <th>primary_category</th>
    <th>secondary_category</th>
    <th>question</th>
    <th>reference answer</th>
  </tr>
  <tr>
    <td rowspan="2">中华文化(Chinese Culture)</td>
    <td>中医(Traditional Chinese Medicine)</td>
    <td>创立了调气活血的“衡法”治则的是哪一位中医学家?</td>
    <td>颜德</td>
  </tr>
  <tr>
    <td></td>
    <td>Which Chinese medicine scientist created the "Heng method" treatment principle for regulating Qi and activating blood circulation?</td>
    <td>Yan De</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>民俗(Folklore)</td>
    <td>西迁节主要是哪个少数民族的节日?</td>
    <td>锡伯族</td>
  </tr>
  <tr>
    <td></td>
    <td>Which ethnic minority primarily celebrates the Xiqian Festival?</td>
    <td>Xibe ethnic group</td>
  </tr>
  <tr>
    <td rowspan="2">人文与社会科学(Humanities)</td>
    <td>政治(Political Science)</td>
    <td>2021年国际宗教自由峰是在美国哪个城市举行?</td>
    <td>华盛顿</td>
  </tr>
  <tr>
    <td></td>
    <td>In which U.S. city was the 2021 International Religious Freedom Summit held?</td>
    <td>Washington, D.C.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>教育学(Education)</td>
    <td>先行组织者概念是哪位美国教育心理学家提出的?</td>
    <td>大卫·奥苏伯尔</td>
  </tr>
  <tr>
    <td></td>
    <td>Which American educational psychologist proposed the concept of advance organizers?</td>
    <td>David Ausubel</td>
  </tr>
  <tr>
    <td rowspan="2">自然与自然科学(Natural Science)</td>
    <td>资讯科学(Information Science)</td>
    <td>《中国图书馆分类法》第五版中规定U6表示哪一类?</td>
    <td>水路运输</td>
  </tr>
  <tr>
    <td></td>
    <td>What category does U6 represent in the 5th edition of the Chinese Library Classification?</td>
    <td>Water Transport</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>数学(Mathematics)</td>
    <td>画法几何这门学科主要是由哪位法国数学家提出的?</td>
    <td>加斯帕尔·蒙日</td>
  </tr>
  <tr>
    <td></td>
    <td>Which French mathematician is credited with the development of descriptive geometry?</td>
    <td>Gaspard Monge</td>
  </tr>
  <tr>
    <td rowspan="3">生活、艺术与文化(Life, Art, and Culture)</td>
    <td>动画(Animation)</td>
    <td>《喜羊羊与灰太狼》是哪家公司制作的原创动画作品?</td>
    <td>广东原创动力文化传播有限公司</td>
  </tr>
  <tr>
    <td></td>
    <td>Which company produced the original animated series "Pleasant Goat and Big Big Wolf"?</td>
    <td>Guangdong Original Power Culture Communication Co., Ltd.</td>
  </tr>
  <tr>
    <td>雕塑(Sculpture)</td>
    <td>著名的华尔街铜牛雕塑是由哪位意大利雕塑家创作的?</td>
    <td>阿图罗·迪·莫迪卡(Arturo Di Modica)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>Who created the famous Wall Street Bull sculpture?</td>
    <td>Arturo Di Modica</td>
  </tr>
  <tr>
    <td rowspan="2">社会(Society)</td>
    <td>传统(Tradition)</td>
    <td>根据《礼记·内则》，如果女子未许嫁，她应在多少岁进行笄礼?</td>
    <td>20</td>
  </tr>
  <tr>
    <td></td>
    <td>According to the "Liji: Neize," at what age should an unmarried woman undergo the Ji ceremony?</td>
    <td>20</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>城市(City)</td>
    <td>特拉斯卡拉城是哪个国家的城市?</td>
    <td>墨西哥</td>
  </tr>
  <tr>
    <td></td>
    <td>In which country is the city of Tlaxcala located?</td>
    <td>Mexico</td>
  </tr>
  <tr>
    <td rowspan="2">工程、技术与应用科学(Engineering, Technology, and Applied Sciences)</td>
    <td>计算机科学(Computer Science)</td>
    <td>1.7.0版本的pytorch可以兼容的最新cuda版本是多少?</td>
    <td>11</td>
  </tr>
  <tr>
    <td></td>
    <td>What is the latest compatible CUDA version for PyTorch 1.7.0?</td>
    <td>11</td>
  </tr>
  <tr>
    <td></td>
    <td>生物工程(Bioengineering)</td>
    <td>哪位中国科学家首先研制成功了转基因鱼?</td>
    <td>朱作言</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>Which Chinese scientist was the first to successfully develop genetically modified fish?</td>
    <td>Zhu Zuoyan</td>
  </tr>
</table>

Figure 13: Some examples of Chinese SimpleQA.

请结合检索材料准确地回答用户的问题，如果检索材料有相关知识，优先参考检索材料，若无则依靠自身知识作答。

Figure 14: The prompt of RAG.
<table>
  <tr>
    <th>primary_category</th>
    <th>secondary_category</th>
    <th>count</th>
    <th>question_avg_tokens</th>
    <th>primary_category</th>
    <th>secondary_category</th>
    <th>count</th>
    <th>question_avg_tokens</th>
  </tr>
  <tr>
    <td rowspan="12">Chinese Culture</td>
    <td>Traditional Chinese Medicine (中医)</td>
    <td>33</td>
    <td>32.1</td>
    <td rowspan="12">Society</td>
    <td>Industry (产业)</td>
    <td>33</td>
    <td>28.2</td>
  </tr>
  <tr><td>Chinese Folklore (中国民俗)</td><td>30</td><td>36.2</td><td>Tradition (传统)</td><td>26</td><td>32.6</td></tr>
  <tr><td>Chinese History (中国历史)</td><td>48</td><td>30.9</td><td>Country (国家)</td><td>35</td><td>25.8</td></tr>
  <tr><td>Chinese Opera (中国戏曲)</td><td>21</td><td>30.9</td><td>City (城市)</td><td>36</td><td>26.4</td></tr>
  <tr><td>Chinese Literature (中国文学)</td><td>51</td><td>29.0</td><td>Media (媒体)</td><td>24</td><td>24.0</td></tr>
  <tr><td>Chinese Martial Arts (中国武术)</td><td>26</td><td>29.3</td><td>Safety (安全)</td><td>20</td><td>32.3</td></tr>
  <tr><td>Chinese Mythology (中国神话)</td><td>20</td><td>38.6</td><td>Religion (宗教)</td><td>27</td><td>26.6</td></tr>
  <tr><td>Chinese Music (中国音乐)</td><td>22</td><td>36.5</td><td>Government (政府)</td><td>32</td><td>31.2</td></tr>
  <tr><td>Calligraphy (书法)</td><td>23</td><td>31.0</td><td>Politics (政治)</td><td>31</td><td>35.4</td></tr>
  <tr><td>Buddhism (佛教)</td><td>26</td><td>27.9</td><td>Education (教育)</td><td>23</td><td>28.5</td></tr>
  <tr><td>Taoism (道教)</td><td>26</td><td>33.7</td><td>Culture (文化)</td><td>25</td><td>27.5</td></tr>
  <tr><td>Demography (人口学)</td><td>21</td><td>32.5</td><td>Ethnic Groups (民族)</td><td>29</td><td>29.3</td></tr>
  <tr><td>Anthropology (人类学)</td><td>20</td><td>33.9</td><td>Law (法律)</td><td>28</td><td>31.6</td></tr>
  <tr><td>Communication Studies (传播学)</td><td>21</td><td>28.3</td><td>Crime (犯罪)</td><td>19</td><td>37.5</td></tr>
  <tr><td>Military Studies (军事学)</td><td>37</td><td>36.6</td><td>Organization (组织)</td><td>37</td><td>27.2</td></tr>
  <tr><td>History (历史)</td><td>50</td><td>37.1</td><td>Economy (经济)</td><td>26</td><td>27.3</td></tr>
  <tr><td>Philosophy (哲学)</td><td>36</td><td>34.4</td><td>Agriculture (农业)</td><td>35</td><td>25.0</td></tr>
  <tr><td rowspan="17">Library and Information Science (图书资讯科学)</td>
    <td>Psychology (心理学)</td><td>17</td><td>32.0</td><td>Animals (动物)</td><td>30</td><td>24.4</td>
  </tr>
  <tr><td>Political Science (政治学)</td><td>40</td><td>29.6</td><td>Chemistry (化学)</td><td>40</td><td>24.3</td></tr>
  <tr><td>Education (教育学)</td><td>30</td><td>31.3</td><td>Medicine (医学)</td><td>30</td><td>32.9</td></tr>
  <tr><td>Literature (文学)</td><td>40</td><td>32.1</td><td>Geography (地理)</td><td>44</td><td>26.6</td></tr>
  <tr><td>Journalism (新闻学)</td><td>31</td><td>25.6</td><td>Geology (地质学)</td><td>40</td><td>25.5</td></tr>
  <tr><td>Ethnology (民族学)</td><td>27</td><td>30.3</td><td>Natural Science</td><td>44</td><td>22.6</td></tr>
  <tr><td>Law (法学)</td><td>39</td><td>32.4</td><td>Cryptography (密码学)</td><td>33</td><td>25.9</td></tr>
  <tr><td>Sociology (社会学)</td><td>31</td><td>28.3</td><td>Mathematics (数学)</td><td>32</td><td>29.1</td></tr>
  <tr><td>Management (管理学)</td><td>23</td><td>27.1</td><td>Botany (植物)</td><td>32</td><td>21.7</td></tr>
  <tr><td>Economics (经济学)</td><td>35</td><td>30.4</td><td>Meteorology (气象学)</td><td>27</td><td>29.1</td></tr>
  <tr><td>Arts (艺术)</td><td>42</td><td>32.3</td><td>Physics (物理学)</td><td>41</td><td>29.6</td></tr>
  <tr><td>Linguistics (语言学)</td><td>40</td><td>25.6</td><td>Biology (生物)</td><td>36</td><td>28.3</td></tr>
  <tr><td>Transportation (交通)</td><td>22</td><td>28.1</td><td>Pharmacy (药学)</td><td>32</td><td>24.7</td></tr>
  <tr><td>Metallurgy (冶金学)</td><td>26</td><td>24.8</td><td>Information Science (资讯科学)</td><td>34</td><td>28.7</td></tr>
  <tr><td>Chemical Engineering (化学工程)</td><td>20</td><td>31.4</td><td>Animation (动画)</td><td>40</td><td>29.4</td></tr>
  <tr><td>Printing (印刷)</td><td>24</td><td>23.6</td><td>Entertainment (娱乐)</td><td>41</td><td>30.1</td></tr>
  <tr><td>Civil Engineering (土木工程)</td><td>30</td><td>28.2</td><td>Toys (玩具)</td><td>22</td><td>28.5</td></tr>
  <tr><td>Architecture (建筑学)</td><td>24</td><td>30.8</td><td>Drama (戏剧)</td><td>21</td><td>28.5</td></tr>
  <tr><td>Mechanical Engineering (机械工程)</td><td>27</td><td>28.4</td><td>Handicrafts (手工艺)</td><td>20</td><td>31.3</td></tr>
  <tr><td>Materials Science (材料科学)</td><td>26</td><td>26.4</td><td>Photography (摄影)</td><td>20</td><td>34.0</td></tr>
  <tr><td>Hydraulic Engineering (水利工程)</td><td>21</td><td>30.8</td><td>Collectibles (收藏)</td><td>27</td><td>25.3</td></tr>
  <tr><td>Surveying (测绘学)</td><td>27</td><td>24.6</td><td>Cultural Relics (文物)</td><td>28</td><td>30.3</td></tr>
  <tr><td>Environmental Science (环境科学)</td><td>24</td><td>32.4</td><td>Tourism (旅游)</td><td>22</td><td>32.1</td></tr>
  <tr><td>Bioengineering (生物工程)</td><td>21</td><td>40.8</td><td>Fashion (服装)</td><td>18</td><td>25.6</td></tr>
  <tr><td>Electrical Engineering (电气工程)</td><td>23</td><td>29.9</td><td>Life, Art, and Culture</td><td>45</td><td>30.2</td></tr>
  <tr><td>Mining (矿业)</td><td>25</td><td>26.0</td><td>Gaming (游戏)</td><td>45</td><td>27.8</td></tr>
  <tr><td>Energy Science (能源科学)</td><td>31</td><td>28.9</td><td>Comics (漫画)</td><td>42</td><td>27.8</td></tr>
  <tr><td>Aerospace (航空航天)</td><td>29</td><td>25.9</td><td>Movies (电影)</td><td>45</td><td>31.5</td></tr>
  <tr><td>Computer Science (计算机科学)</td><td>48</td><td>24.4</td><td>Television (电视)</td><td>48</td><td>28.3</td></tr>
  <tr><td>Communication Technology (信息技术)</td><td>33</td><td>25.8</td><td>Painting (绘画)</td><td>29</td><td>35.8</td></tr>
  <tr><td>Figure 15: Statistics of All Categories in the Dataset: Counts and Question Token Number Distribution.</td><td></td><td></td><td></td><td>Dance (舞蹈)</td><td>20</td><td>29.8</td></tr>
  <tr><td></td><td></td><td></td><td></td><td>Festivals (节日)</td><td>18</td><td>30.2</td></tr>
  <tr><td></td><td></td><td></td><td></td><td>Sculpture (雕塑)</td><td>30</td><td>33.8</td></tr>
  <tr><td></td><td></td><td></td><td></td><td>Music (音乐)</td><td>45</td><td>32.7</td></tr>
  <tr><td></td><td></td><td></td><td></td><td>Food and Drink (饮食)</td><td>20</td><td>24.5</td></tr>
</table>
Figure 16: Distribution of question difficulty across 99 categories in the dataset. Each violin represents the spread of difficulty within a category, measured by the number of models that failed to answer the question correctly. Narrow and symmetrical violins indicate more uniform difficulty distributions, while wider violins or those with elongated tails suggest greater variability in question difficulty.
Top 10 Models for Metallurgy
Model: DeepSeek-V2.5, Doubao-Pro-32K, GLM-4-Plus, O1-Preview, GPT-4o

Top 10 Models for Chemical Engineering
Model: O1-Preview, GLM-4, OPT-4o, Doubao-Jio-4K, Doubao-Pro-32K

Top 10 Models for Printing
Model: GLM-4-Plus, Doubao-Pro-32K, OPT-4o, O1-Preview, DeepSeek-V2.5

Top 10 Models for Civil Engine
Model: O1-Preview, Doubao-Pro-32K, GLM-4-Plus, DeepSeek-V2.5, GPT-4o

Top 10 Models for Architecture
Model: GLM-4-Plus, Gemini-1.5-Pro, DeepSeek-V2.5, DeepSeek-LM-679, DeepSeek-V2.5

Top 10 Models for Mechanical Engineering
Model: Doubao-Pro-32K, O1-Preview, Baiduhan-3-Turbo, GLM-4-Plus, Gemini-1.5-Pro

Top 10 Models for Materials Science
Model: Doubao-Pro-32K, O1-Preview, DeepSeek-V2.5, Claude-3.5-Sonnec, O1-Mini

Top 10 Models for Hydraulic Engi
Model: DeepSeek-V2.5, GLM-4-Plus, Yi-Large, Doubao-Pro-32K, GLM-4

Top 10 Models for Surveying
Model: Doubao-Pro-32K, O1-Preview, Claude-3.5-Sonnec, GPT-4o, Gemini-1.5-Pro

Top 10 Models for Traditional Chinese Medicine
Model: Doubao-Pro-32K, GLM-4-Plus, Moonshot-V1-8K, DeepSeek-V2.5, DeepSeek-LM-679

Top 10 Models for Chinese Folklore
Model: Doubao-Pro-32K, GLM-4-Plus, DeepSeek-V2.5, DeepSeek-V2.5, DeepSeek-LM-679

Top 10 Models for Chinese His
Model: InterLM2-520B, DeepSeek-LM-679, Yi-15-349, Owen-2.5-148, OPT-4

Top 10 Models for Chinese Opera
Model: Doubao-Pro-32K, GLM-4-Plus, DeepSeek-V2.5, DeepSeek-V2.5, O1-Preview

Top 10 Models for Chinese Literature
Model: Doubao-Pro-32K, GLM-4-Plus, DeepSeek-V2.5, Gemini-1.5-Pro, Moonshot-V1-8K

Top 10 Models for Chinese Martial Arts
Model: Doubao-Pro-32K, GLM-4-Plus, Gemini-1.5-Pro, O1-Preview, DeepSeek-V2.5

Top 10 Models for Chinese Myth
Model: GLM-4-Plus, Doubao-Pro-32K, Moonshot-V1-8K, DeepSeek-V2.5, DeepSeek-V2.5

Top 10 Models for Chinese Music
Model: Doubao-Pro-32K, DeepSeek-V2.5, Moonshot-V1-8K, GLM-4-Plus, DeepSeek-LM-679

Top 10 Models for Calligraphy
Model: Doubao-Pro-32K, DeepSeek-V2.5, Moonshot-V1-8K, GLM-4-Plus, Gemini-1.5-Pro

Top 10 Models for Transportation
Model: Doubao-Pro-32K, O1-Preview, Moonshot-V1-8K, GLM-4-Plus, GLM-4

Top 10 Models for Industry
Model: O1-Preview, DeepSeek-V2.5, GLM-4-Plus, DeepSeek-V2.5, GPT-4o

Top 10 Models for Demography
Model: O1-Preview, Claude-3.5-Sonnec, Yi-Large, GPT-4o, GLM-4

Top 10 Models for Anthropology
Model: GLM-4-Plus, Doubao-Pro-32K, DeepSeek-V2.5, DeepSeek-V2.5, DeepSeek-LM-679

Top 10 Models for Communication Studies
Model: GLM-4-Plus, Gemini-1.5-Pro, Doubao-Pro-32K, O1-Preview, Yi-Large

Top 10 Models for Tradition
Model: O1-Preview, GLM-4-Plus, GPT-4o, DeepSeek-V2.5, GLM-4
Top 10 Models for Buddhism
O1-Preview
GLM-4-Plus
Doubao-Pro-32K
Doubao-Lite-4K
DeepSeek-LLM-6FB

Top 10 Models for Military Studies
O1-Preview
GLM-4-Plus
Doubao-Pro-32K
GPT-4c
Baidubian-Turbo

Top 10 Models for Agriculture
GLM-4
O1-Preview
DeepSeekV2.5
GPT-4c
Gemini-1.5-Pro

Top 10 Models for Animal
Doubao-Pro-32K
O1-Preview
Doubao-Lite-4K
GPT-4c
DeepSeekV2.5

Top 10 Models for Animation
O1-Preview
DeepSeek-V2.5
Doubao-Pro-32K
GPT-4c

Top 10 Models for Chemistry
Doubao-Pro-32K
O1-Preview
GPT-4c
Gemini-1.5-Pro
GLM-4-Plus

Top 10 Models for Medicine
O1-Preview
Owen-3.5-72B
GPT-4c
GLM-4-Plus
Doubao-Pro-32K

Top 10 Models for History
Owen-2.5-72B
Owen-2.5-14B
DeepSeek-LLM-6FB
GPT-4c-Turbo
O1-Mini

Top 10 Models for Philosophy
O1-Preview
GLM-4-Plus
Doubao-Pro-32K
Claude-3.5-Sonnet
GPT-4c

Top 10 Models for Country
GLM-4-Plus
GPT-4c
Yi-Large
O1-Preview
GLM-4

Top 10 Models for Library and Information Science
Doubao-Pro-32K
O1-Preview
Claude-3.5-Sonnet
DeepSeekV2.5
GLM-4

Top 10 Models for Geology
Doubao-Pro-32K
O1-Preview
DeepSeek-V2.5
GPT-4c-Turbo

Top 10 Models for City
O1-Preview
GLM-4-Plus
GPT-4c-Turbo
Doubao-Pro-32K

Top 10 Models for Astronomy
O1-Preview
GPT-4c
Gemini-1.5-Pro
Doubao-Pro-32K
Llama-3-170B-Instruct

Top 10 Models for Entertainment
O1-Preview
Gemini-1.5-Pro
GPT-4c
GLM-4-Plus
Doubao-Pro-32K

Top 10 Models for Media
GLM-4-Plus
Claude-3.5-Sonnet
Yi-Large
Mooshid-V1-8K
GPT-4c

Top 10 Models for Safety
O1-Preview
Claude-3.5-Sonnet
GPT-4c
GLM-4-Plus

Top 10 Models for Religion
O1-Preview
Doubao-Pro-32K
GLM-4-Plus
GLM-4
Yi-Large

Top 10 Models for Cryptography
O1-Preview
GPT-4
Gemini-1.5-Pro
O1-Mini
Doubao-Pro-32K

Top 10 Models for Tools
O1-Preview
GPT-4c
GPT-4
Claude-3.5-Sonnet
Doubao-Pro-32K

Top 10 Models for Psychology
O1-Preview
Doubao-Pro-32K
Claude-3.5-Sonnet
DeepSeekV2.5
GLM-4-Plus

Top 10 Models for Drama
Gemini-1.5-Pro
DeepSeekV2.5
GLM-4-Plus
GLM-4
Doubao-Pro-32K

Top 10 Models for Handicrafts
Doubao-Pro-32K
Gemini-1.5-Pro
O1-Preview
GPT-4c
DeepSeek-LLM-6FB
Top 10 Models for Photography
Model
O1-Preview
GLM-4
GPT-4c
Gemini-1.5-Pro
DeepSeek-LLM-6FB

Top 10 Models for Collectibles
Model
O1-Preview
GPT-4c
Gemini-1.5-Pro
DeepSeekV2.5
DeepSeekV2.5

Top 10 Models for Government
Model
GPT-4c
O1-Preview
GLM-4-Plus
Doubao-Pro-32K
GLM-4

Top 10 Models for Politics
Model
O1-Preview
Doubao-Pro-32K
Qwen-2.5-T2B
GPT-4c
GLM-4-Plus

Top 10 Models for Political Science
Model
GPT-4c
O1-Preview
Claude-3.5-Sonnet
Doubao-Pro-32K
GPT-4

Top 10 Models for Education
Model
O1-Preview
Claude-3.5-Sonnet
O1-Preview
GLM-4-Plus
Yi-Large

Top 10 Models for Education
Model
Yi-Large
Doubao-Pro-32K
GLM-4-Plus
GPT-4c
O1-Preview

Top 10 Models for Mathematics
Model
O1-Preview
Doubao-Pro-32K
DeepSeek-V2.5
GPT-4c
DeepSeek-V2.5

Top 10 Models for Culture
Model
Doubao-Pro-32K
GPT-4c
GLM-4-Plus
DeepSeek-V2.5
Yi-Large

Top 10 Models for Literature
Model
Doubao-Pro-32K
GLM-4-Plus
DeepSeekV2.5
O1-Preview
DeepSeekV2.5

Top 10 Models for Cultural Relics
Model
O1-Preview
Doubao-Pro-32K
GLM-4-Plus
DeepSeek-V2.5
DeepSeek-V2.5

Top 10 Models for Journalism
Model
GLM-4-Plus
Doubao-Pro-32K
O1-Preview
DeepSeek-V2.5
Gemini-1.5-Pro

Top 10 Models for Tourism
Model
O1-Preview
Doubao-Pro-32K
GLM-4-Plus
GPT-4c
GPT-4-Turbo

Top 10 Models for Ethnic Groups
Model
O1-Preview
GPT-4c
GLM-4-Plus
Gemini-1.5-Pro
Doubao-Pro-32K

Top 10 Models for Fashion
Model
Gemini-1.5-Pro
GPT-4c
Doubao-Pro-32K
DeepSeek-V2.5
MoonshotV1-8K

Top 10 Models for Botany
Model
Doubao-Pro-32K
Gemini-1.5-Pro
GLM-4-Plus
Doubao-Lite-4K
O1-Preview

Top 10 Models for Ethology
Model
O1-Preview
Claude-3.5-Sonnet
DeepSeek-V2.5
DeepSeek-V2.5
GPT-4

Top 10 Models for Meteorology
Model
O1-Preview
GPT-4c
GLM-4-Plus
Gemini-1.5-Pro
O1-Mini

Top 10 Models for Law
Model
O1-Preview
Doubao-Pro-32K
GLM-4-Plus
GPT-4c
Gemini-1.5-Pro

Top 10 Models for Law
Model
O1-Preview
Doubao-Pro-32K
GPT-4c
DeepSeek-V2.5
Claude-3.5-Sonnet

Top 10 Models for Ethology
Model
O1-Preview
GPT-4c
Gemini-1.5-Pro
DeepSeek-V2.5
GLM-4-Plus

Top 10 Models for Gaming
Model
O1-Preview
GPT-4c
Gemini-1.5-Pro
DeepSeek-V2.5
GLM-4-Plus

Top 10 Models for Comics
Model
O1-Preview
GPT-4c
Gemini-1.5-Pro
DeepSeekV2.5
DeepSeekV2.5

Top 10 Models for Physics
Model
Doubao-Pro-32K
O1-Preview
Gemini-1.5-Pro
DeepSeek-V2.5
Balchuan3-Turbo

Top 10 Models for Crime
Model
O1-Preview
DeepSeek-V2.5
Gemini-1.5-Pro
Doubao-Pro-32K
GPT-4-Turbo
Top 10 Models for Environmental Science
Model: Gemini-1.5-Pro, GPT-4c, Doubao-Pro-32K, O1-Preview, GPT-4

Top 10 Models for Biology
Model: Doubao-Pro-32K, DeepSeek-V2.5, Gemini-1.5-Pro, GLM-4-Plus, O1-Preview

Top 10 Models for Bioengineering
Model: O1-Preview, GLM-4-Rus, Gemini-1.5-Pro, DeepSeek-V2.5, Moonshot-V1-8K

Top 10 Models for Movies
Model: DeepSeek-LLM-67B, Owen-2.5-72B, GPT-4c, GPT-4c-Mini, O1-Mini

Top 10 Models for Electrical Engineering
Model: O1-Preview, Claude-3.5-Sonnet, GPT-4, GPT-4-Turbo, Doubao-Pro-32K

Top 10 Models for Television
Model: O1-Preview, DeepSeek-V2.5, GLM-4-Plus, Doubao-Pro-32K, DeepSeek-V2.5

Top 10 Models for Mining
Model: Doubao-Pro-32K, O1-Preview, GLM-4, GLM-4-Plus, Batchuan3-Turbo

Top 10 Models for Sociolog
Model: O1-Preview, GPT-4c, Claude-3.5-Sonnet, Doubao-Pro-32K, GLM-4-Plus

Top 10 Models for Management
Model: Doubao-Pro-32K, GLM-4-Rus, DeepSeek-V2.5, O1-Preview, GPT-4c

Top 10 Models for Organization
Model: O1-Preview, GLM-4, GLM-4-Plus, Claude-3.5-Sonnet, Doubao-Pro-32K

Top 10 Models for Economy
Model: O1-Preview, DeepSeek-V2.5, Claude-3.5-Sonnet, GPT-4c, Doubao-Pro-32K

Top 10 Models for Economics
Model: GPT-4-Turbo, GPT-4c, Claude-3.5-Sonnet, Doubao-Pro-32K, Gemini-1.5-Pro

Top 10 Models for Painting
Model: O1-Preview, Doubao-Pro-32K, Gemini-1.5-Pro, GPT-4c, GLM-4-Plus

Top 10 Models for Energy Science
Model: O1-Preview, O1-Mini, Claude-3.5-Sonnet, GPT-4, GLM-4-Plus

Top 10 Models for Dance
Model: Doubao-Pro-32K, O1-Preview, GLM-4, DeepSeek-V2.5, GPT-4c

Top 10 Models for Aerospace
Model: O1-Preview, Llama-3-170B-Instruct, GPT-4c, Gemini-1.5-Pro, Doubao-Pro-32K

Top 10 Models for Arts
Model: O1-Preview, Doubao-Pro-32K, GPT-4c, DeepSeek-V2.5, DeepSeek-V2.5

Top 10 Models for Festivals
Model: Doubao-Pro-32K, O1-Preview, GPT-4c, GLM-4-Plus, Owen-2.5-72B

Top 10 Models for Pharmacy
Model: Doubao-Pro-32K, O1-Preview, Batchuan3-Turbo, GPT-4c, GLM-4-Rus

Top 10 Models for Computer Science
Model: O1-Preview, Gemini-1.5-Pro, GPT-4c, Doubao-Pro-32K, GPT-4

Top 10 Models for Linguistics
Model: Doubao-Pro-32K, O1-Preview, Gemini-1.5-Pro, GLM-4-Plus

Top 10 Models for Information Science
Model: O1-Preview, Doubao-Pro-32K, Gemini-1.5-Pro, Owen-2.5-72B, Claude-3.5-Sonnet

Top 10 Models for Communication Technology
Model: O1-Preview, GPT-4c, GPT-4-Turbo, GLM-4-Rus, Gemini-1.5-Pro

Top 10 Models for Taoism
Model: DeepSeek-V2.5, Doubao-Pro-32K, O1-Preview, Yi-Large, DeepSeek-V2.5
Figure 17: Bar charts showcasing the top 10 models ranked by F1 score across diverse topics.

Figure 17 shows the best performing models in each topic, which are sorted by F1 score. It can be seen that the strongest models in different fields may not be the same. The results emphasize that model performance varies with the field, which can help researchers gain a deeper understanding of the applicability of each model to a specific field.

K  Generation and validation of question-answer pairs

The generation and validation of question-answer pairs both use OpenAI’s gpt-4o-0806. The specific prompts are shown in Figures 18, 19, and 20.

L  Analysis of Model Calibration
现在需要你根据给定的文档生成一个事实类问题和对应的标准答案，需要满足下列要求:
1. 生成的问题必须关联到客观世界的知识，例如可以询问“2024年诺贝尔物理学奖的获得者是谁？”不得构造涉及个人观点或感受相关的主观问题，如“你如何看待xxx？”。
2. 所提出的问题应该有且只有一个明确且无争议的实体作为答案，且问题表述中不应存在任何形式的模糊性或歧义。例如，避免提问“巴拉克和米歇尔·奥巴马在哪里会面？”因为无法确定是指哪一次会面；同样不要问“白民国人身体的特点是什么？”因为这个问题过于模糊，没有明确的答案。“周汝昌最为人熟知的著作是哪个？”也是不合格问题，因为“最熟知”可能是有争议的。
3. 问题的答案应当是时间不变的，不会随着时间的推移而改变。例如，“美国现任总统是谁？”就不是一个合适的问题，因为总统身份会随选举结果改变。
4. 问题应该具有一定的难度，以体现出一定的挑战性。例如：电影《脱衣舞娘》是由同名小说改编的，该小说的作者是谁？
5. 如果问题的答案为英文人名，请给出中文翻译后的名字和括号里带上英文原名，格式如：雅各布·福格（Jakob Fugger）。
6. 生成的问题需要与给定的类目相关
请将生成的问题和答案以JSON格式返回，具体格式如下：
{"question": "这里填写生成的问题","answer": "这里填写对应的标准答案"}
###以下是一些示例###
### 示例一
类目：娱乐
文档内容：2022年国际足联世界杯为第22届国际足联世界杯，于2022年11月20日至12月18日在卡塔尔举行[2][3]。成为全球爆发防疫后首个终结限制的大型国际体育盛事。考量到气候因素，本届世界杯亦是首次于11月至12月北半球秋季[注 1]举行之世界杯。决赛于卡塔尔的卢赛尔体育场举行，由阿根廷队对阵卫冕冠军法国队。双方先于比赛正规时间踢至加时赛以3-3赛和，后在点球大战中阿根廷以4–2击败法国，赢得了此届世界杯，也是阿根廷继1986年世界杯后，相隔36年再度于世界杯夺冠，继巴西、意大利及德国后第四支三次冠军的球队，也成为继巴西后第二支在亚洲夺冠的南美洲球队。克罗地亚则以点球1击败该年黑马摩洛哥赢得季军[4][5]。
返回结果：{"question": "2022年世界杯决赛点球大战中阿根廷队是以多少击败法国队？","answer": "4–2"}
### 示例二
类目：政治
文档内容：中国与世界贸易组织（英语：China and the World Trade Organization）指中华人民共和国与世界贸易组织的关系。在部长级会议达成协议后，中国于2001年12月11日成为世界贸易组织成员。[1][2]在承认这一点之前，双方进行了漫长的谈判，并且需要对中国经济进行重大改革。世贸组织的成员资格一直存在争议，对其它国家产生了重大的经济和政治影响(也被称为“中国冲击”)，对世贸组织框架与中国经济模式之间的不匹配也存在争议。[3][4]评估和执行合规已成为中美贸易关系中的问题，包括中国的不合规行为如何为本国经济创造利益。[5][6]
返回结果：{"question": "中国是哪一年正式成为世界贸易组织成员？ ","answer": "2001"}
###
让我们开始吧！
你是一个数据质量检查员，现在需要你检查下面生成的问题是否满足以下要求：
1. 生成的问题必须对客观世界的知识的提问。例如可以询问“2024年诺贝尔物理学奖的获得者是谁？”不得构造涉及个人观点或感受相关的主观问题，如“你如何看待xxx？”。
2. 问题应该有且只有一个明确且无争议的实体作为答案，且问题表述中不应存在任何形式的模糊性或歧义。例如，避免提问“巴拉克和米歇尔·奥巴马在哪里会面？”因为无法确定是指哪一次会面；同样不要问“白民国人身体的特点是什么？”因为这个问题过于模糊，没有明确的答案。注意如果回答是多个实体也不满足要求，例如：“软体动物、腕足动物及被囊动物”
3. 问题的答案应当是时间不变的，不会随着时间的推移而改变。例如，“美国现任总统是谁？”就不是一个合适的问题，因为总统身份会随选举结果改变。
如果问题不合格则解释并输出“【否】”， 如果问题合格则直接输出“【是】”
#### 以下是一些示例：
问题：《黄帝内经》中, 援物比类思维方式包括哪些核心概念?
评价：该问题不是只有一个确切答案，【否】

问题：建筑理论主要研究什么内容?
评价：该问题不具体，回答不是只有一个实体，【否】

问题：成立了程派高氏八卦掌的高义盛原籍是哪里?
评价：回答范围不明确，不清楚是回答到城市还是省份，【否】

问题：自由恋爱主义最初的目标是将哪些事务与国家分离?
评价：该问题不是只有一个答案，【否】

问题：汉十高速公路连接的武汉市和哪两个城市?
评价：【是】
#### 如果问题不合格则输出原因并最后输出“【否】”， 如果问题合格则直接输出“【是】”，注意如果认为问题不合格需要输出原因
让我们开始吧！

Figure 19: The prompt for validating criteria.

请结合检索材料判断给定的问题是否满足以下要求以及答案是否正确:
1. 问题应该有且只有一个明确且无争议的实体作为答案，且问题表述中不应存在任何形式的模糊性或歧义。例如，避免提问“巴拉克和米歇尔·奥巴马在哪里会面？”因为无法确定是指哪一次会面；同样不要问“白民国人身体的特点是什么？”因为这个问题过于模糊，没有明确的答案。注意如果回答是多个实体也不满足要求，例如：“软体动物、腕足动物及被囊动物”
2. 问题的答案应当是时间不变的，不会随着时间的推移而改变。例如，“美国现任总统是谁？”就不是一个合适的问题，因为总统身份会随选举结果改变。
3. 检索材料应该能够验证给定答案的正确性，并且这些材料中不应存在其他与之相矛盾的答案。如果检索材料中存在不同的答案，则该问题被视为无法验证或答案不明确，不满足要求。
请依据上述标准，审查并确保提出的问题及其答案符合要求，如果不满足上面任一要求，则输出原因，最后输出“【不合格】”，否则输出检索材料对应的片段，最后输出“【合格】”

Figure 20: The prompt for verifying factual accuracy based on RAG.
请阅读以下问题:
{question}
请基于此问题提供你的最佳答案，并用0到100的分数表示你对该答案的信心（置信度）。请以如下的JSON格式给出回复:
{
    "answer": "你的答案",
    "confidence_score": 你的置信度
}

Figure 21: The prompt for guiding the model to output confidence.

Reviewer paper summaries: This paper introduces Chinese SimpleQA, a counterpart to English SimpleQA, designed to evaluate factual accuracy in Chinese. The dataset consists of six major categories, including Chinese culture. It consists of 3,000 short-answer QA pairs created through a combination of LLM generation and human verification. The dataset is used to evaluate both open and closed LLMs, showing that the task remains challenging, even for closed models. The authors also provide in-depth analyses to support their findings.

The paper introduces Chinese SimpleQA, a Chinese benchmark designed to evaluate the factual accuracy of LLMs in answering short-form factual questions. It covers six major topics and 99 subtopics. The dataset consists of 3,000 question-answer pairs with quality control involving automatic and human verification. With experiments on over 40 LLMs, the authors reveal trends such as the influence of model size, alignment tax, and the effectiveness of RAG.

This paper introduces Chinese SimpleQA, the first Chinese short-form factuality benchmark for evaluating LLMs. The dataset consists of 3000 questions across 6 major topics and 99 subtopics and was constructed through automated generation with hybrid quality control by LLMs and human annotators. Compared to existing Chinese benchmarks like C-Eval and CMMLU, which primarily use multiple-choice questions, Chinese SimpleQA better reflects real-world scenarios.

Experiments demonstrated that the benchmark is challenging, with even o1-preview only achieving a passing score (63.8% on the correctness metric). The authors also conducted experiments to analyze the effects of model size, calibration, RAG, and alignment.
Summary of strengths: - The dataset includes a wide range of topics, with Chinese culture as one of the unique categories not covered by English SimpleQA.
- Using short-answer generation as the task format makes it more practical and realistic compared to multiple-choice questions.
- The dataset is well-curated and challenging, providing a solid benchmark for current models.

- The paper fills a significant gap by providing a Chinese-specific benchmark, addressing the lack of factuality evaluation tools tailored to the Chinese language. 
- The dataset construction process incorporates automated filtering, multi-round feedback, and dual human verification. This ensures the high-quality of the benchmark.
- Comprehensive evaluation on over 40 LLMs and several interesting Insights.

- As a benchmark paper, this work is highly thorough in its engineering details, including desensitization and data contamination prevention during dataset construction, comprehensive evaluations on mainstream LLMs, and analyses on calibration, test-time computation, and the effect of RAG.
- The proposed Chinese SimpleQA dataset is a valuable contribution to Chinese NLP community, addressing a crucial gap left by existing datasets like C-Eval and CMMLU, which primarily rely on multiple-choice questions, potentially introducing option bias in LLM assessments.
- The paper confirms that the "alignment tax" affects LLM factuality, highlighting an important direction for future research on instruction tuning. For instance, incorporating more factually enriched conversations into SFT datasets might help mitigate this issue and enhance LLM performance.
Summary of weaknesses: - The reliability of the LLM judge used for evaluation is not fully convincing, especially for assessing answers in Chinese. It would be helpful to provide justification for the LLM judge's performance. Additionally, the limitations should acknowledge that the LLM judge might not be 100% accurate, and the results should be interpreted with caution.
- This is a minor point, but some details about the dataset construction process need clarification, particularly regarding the models used. For example, what is the "trained quality assessment model" mentioned in L169-170? Also, what specific model was used for QA generation and verification? How accurate is the model?

- The paper primarily adapts the existing SimpleQA framework for Chinese, without introducing significant methodological innovations. Its novelty is constrained to language localization.
- By focusing exclusively on short-form, static, and unambiguous factual questions, the benchmark overlooks more complex scenarios that require reasoning, multi-document understanding, or handling ambiguous questions. It may suffer from unintended data contamination or cheating in the future.

- Compared to the 10 topics in SimpleQA, Chinese SimpleQA has only 6 main topics, which might introduce more potential bias into the dataset. Additionally, this difference in distribution makes the ranking comparison in Figure 8 less meaningful.
Comments, suggestions and typos: - Please provide an English translation for the prompts in Figure 18-21.
- Include details about the annotators' demographics. Are the two annotators and one final verifier part of the authors, or are they external contributors?
- Specify the license for the dataset.
- For Section 3.2, consider aligning the explanation more closely with the flow in Figure 2. Dividing the explanation based on the six steps illustrated in the figure would make it easier to follow.

- there is an extra top line and bottom line in table 1.
- consider add some bold or underlined numbers in table 2, to improve its readability.
- consider add dynamic to such dataset to avoid data contamination.

- In Section 4.5, the authors noted that incorporating RAG improves the factual accuracy of LLMs, using Google as the search engine. Given the unique characteristics of the Chinese internet environment, it would be interesting to compare performance when replacing Google with local Chinese search engines like Baidu.
- The authors stated that the dataset primarily draws from Wikipedia and Baidu Baike. It would be insightful to evaluate how well an LLM fine-tuned on Wikipedia and Baidu Baike performs on this dataset. If the model's performance remains inferior even after fine-tuning, it would further validate the value of this dataset.
Overall assessment scores: 3.5

3.5

4
Metareview score: 4
Metareview: This paper describes "Chinese SimpleQA," a benchmark explicitly designed to evaluate the factual accuracy of Large Language Models (LLMs) in answering short-form factual questions in Chinese. The dataset consists of 3,000 question-answer pairs spanning six significant topics and 99 subtopics. The authors constructed the dataset through automated question generation, LLM-based filtering, and human verification. The benchmark aims to provide a more realistic assessment of LLM factuality than existing Chinese multiple-choice-based benchmarks like C-Eval and CMMLU. The paper also presents extensive evaluations across more than 40 LLMs, analyzing factors such as model size, alignment effects, retrieval-augmented generation (RAG), and calibration techniques.
Summary of reasons to publish: 1. The paper addresses an important need for a factuality benchmark for Chinese.
2. The dataset is well-curated, featuring multiple layers of automated and human verification, helping ensure high quality and reducing contamination risks.
3. The authors conduct an extensive study involving over 40 LLMs, presenting valuable insights and analysis.