See the World, Discover Knowledge: A Chinese Factuality Evaluation for Large Vision Language Models
Authors: Jihao Gu, Yingyao Wang, Pi Bu, wang chen, Ziming Wang, Tengtao Song, Donglai Wei, Jiale Yuan, Yingxiu Zhao, Jun Song, Bo Zheng, Yancheng He, Shilong Li, Jiaheng Liu, Meng Cao, Xiaoyong Zhu, Yingshui Tan, 李享, Wenbo Su
Accepted at: acl25

“See the World, Discover Knowledge”:
A Chinese Factuality Evaluation for Large Vision Language Models

Jihao Gu*, Yingyao Wang†, Pi Bu,
Chen Wang, Ziming Wang, Tengtao Song, Donglai Wei, Jiale Yuan,
Yingxiu Zhao, Yancheng He, Shilong Li, Jiaheng Liu, Meng Cao, Jun Song‡,
Yingshui Tan, Xiang Li, Wenbo Su, Xiaoyong Zhu, Bo Zheng,
Alibaba Group
{gujihao.gjh, wangyingyao.wyy, jsong.sj}@taobao.com

Abstract

The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models’ knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 35 advanced open-source and closed-source models, revealing critical performance gaps within this field. Our evaluation-friendly code 1 and data 2 have already been open-sourced.

1 Introduction

In the multimodal field, ensuring the factual accuracy of responses generated by Large Vision Language Models (LVLMs) presents a significant challenge. Currently, cutting-edge models still produce erroneous outputs that do not align with the image content or provide answers lacking support from knowledge evidence. This issue is known as the “visual hallucination”, primarily arising from difficulties in multimodal alignment and the lack of visual knowledge. In the realm of Large Language Models (LLMs), the factuality assessment of linguistic hallucination has garnered extensive attention. For instance, OpenAI introduced the SimpleQA benchmark (Wei et al., 2024), and Alibaba launched a Chinese SimpleQA (He et al., 2024), both of them efficiently evaluate the lengthy responses generated by LLMs containing numerous factual claims through simple questions.

However, the factuality assessment concerning visual hallucination has not received adequate attention (Fu et al., 2024). Researchers have extensively employed benchmarks such as OK-VQA (Marino et al., 2019) and A-OKVQA (Schwenk et al., 2022) to assess the factuality capabilities of LVLMs, but both of them have a narrow knowledge coverage (e.g., A-OKVQA only includes 7 categories). Furthermore, we emphasize that the evaluation of visual factuality should be decoupled into two parts: seeing the world (i.e., object recognition) and discovering knowledge. As illustrated in Figure 2, a

![The overview of ChineseSimpleVQA, which evaluates visual factuality covering 56 subtopics.](page_1012_320_410_410.png)

Figure 1: The overview of ChineseSimpleVQA, which evaluates visual factuality covering 56 subtopics.
<table>
  <tr>
    <th>Benchmark</th>
    <th>Size</th>
    <th>Lang</th>
    <th>Source</th>
    <th>Categories</th>
    <th>Vision</th>
    <th>Open-ended</th>
    <th>Short-form</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>OK-VQA (Marino et al., 2019)</td>
    <td>8,062</td>
    <td>en</td>
    <td>Exams</td>
    <td>20</td>
    <td>✓</td>
    <td>×</td>
    <td>×</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>A-OKVQA (Schwenk et al., 2022)</td>
    <td>125</td>
    <td>zh</td>
    <td>Human Collection</td>
    <td>7</td>
    <td>✓</td>
    <td>✓</td>
    <td>×</td>
    <td>LLM-as-a-Judge</td>
  </tr>
  <tr>
    <td>SimpleQA (Wei et al., 2024)</td>
    <td>4,326</td>
    <td>en</td>
    <td>Human Writers</td>
    <td>10</td>
    <td>×</td>
    <td>✓</td>
    <td>✓</td>
    <td>LLM-as-a-Judge</td>
  </tr>
  <tr>
    <td>Chinese SimpleQA (He et al., 2024)</td>
    <td>3,000</td>
    <td>zh</td>
    <td>Self-constructed & Human Writers</td>
    <td>99</td>
    <td>×</td>
    <td>✓</td>
    <td>✓</td>
    <td>LLM-as-a-Judge</td>
  </tr>
  <tr>
    <td><b>ChineseSimpleVQA (Ours)</b></td>
    <td>2,200</td>
    <td>zh</td>
    <td>Self-constructed & Human Writers</td>
    <td>56</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>LLM-as-a-Judge</td>
  </tr>
</table>

Table 1: Comparisons between our ChineseSimpleVQA and other knowledge-based VQA benchmarks.

model must first comprehend that the image depicts a “chestnut” to answer the next question, “What family does it belong to?” This decoupling of object recognition and knowledge discovery aids in a deeper analysis of the capability boundaries and execution mechanisms of LVLMs.

In this paper, to comprehensively assess the factual knowledge of LVLMs, we present a ChineseSimpleVQA benchmark\footnote{https://chinesesimplevqa.github.io/ ChieseSimpleVQA.github.io/}, which consists of a dataset containing 2,200 high-quality questions across 56 topics, spanning from the humanities to science and engineering, as depicted in Figure 1. Specifically, the key distinguishing features of our proposed ChineseSimpleVQA are as follows:

• <b>Multi-hop:</b> Visual factuality inquiries are decomposed into two steps: object recognition and knowledge assessment. This multi-hop strategy allows us to analyze the capability boundaries and execution mechanisms of LVLMs.

• <b>Diverse:</b> ChineseSimpleVQA emphasizes the Chinese language and covers 8 major topics (i.e., “Nature”, “Sciences”, “Engineering”, “Humanities & Society”, “Modern Architecture”, “Ancient Architecture”, “Geography & Meteorological” and “Life Culture & Art”). These topics encompass 56 fine-grained subtopics.

• <b>High-quality:</b> We implement a rigorous pipeline for the benchmark construction, including automatic verification, difficulty filtering, and human verification.

• <b>Static:</b> To maintain the enduring quality of ChineseSimpleVQA, all reference answers will remain unchanged over time.

• <b>Easy to evaluate:</b> All of the questions and answers are in a short format for quick evaluation.

Furthermore, we have open-sourced all datasets and code, providing ready-to-run scripts to assist researchers in their endeavors.

Subsequently, we conducted a comprehensive evaluation of 35 advanced LVLMs, covering 8 major topics and 56 sub-topics. Here are some insightful findings:

• <b>ChineseSimpleVQA is challenging:</b> The closed-source model o1-preview achieved the best performance in terms of visual factuality, surpassing the top open-source models by approximately 20 points. This indicates that open-source LVLMs still have a long way to go in this area.

• <b>Larger models lead to better results:</b> Within the same series of models, a larger model size generally yields better results. This trend holds for nearly all series of models.

• <b>Larger models exhibit better calibration:</b> We observe that o1-preview is better calibrated than o1-mini, and GPT-4o is more calibrated than GPT-4o-mini.

• <b>More sampling increase performance:</b> Increasing the sampling attempts (Best-of-N) can improve the model’s performance, but this tends to stabilize after around 30 times.

• <b>See the world, but not discover knowledge:</b> Multi-hop questions demonstrate that even the LVLMs can correctly recognize objects, the final performance may still fall short.

2 Related Works

<b>VQA Benchmarks:</b> The problem of Visual Question Answering (VQA) was originally conceptualized as a Visual Turing Test (Geman et al., 2015). Addressing this test necessitates that computational systems replicate human faculties, including real-world visual recognition, comprehension
Figure 2: The production pipeline of ChineseSimpleVQA consists of automated verification, difficulty filtering, and human verification. This process generates multi-hop questions (i.e. object recognition Q&A and merged Q&A).

of language, rudimentary reasoning, and factual knowledge. Certain datasets emphasize the evaluation of perceptual and linguistic understanding capabilities (Antol et al., 2015; Malinowski and Fritz, 2014; Goyal et al., 2017; Liu et al., 2024, 2025; Yu et al., 2023; Li et al., 2024b), whereas others delve into the intricacies of complex reasoning processes (Johnson et al., 2017; Yue et al., 2024). Notwithstanding these advancements, assessing factuality remains a formidable challenge, an aspect that has received limited attention within the scope of VQA datasets.

Factuality Benchmarks: Several datasets also necessitate factual or commonsense knowledge (Marino et al., 2019; Wang et al., 2015, 2017; Zellers et al., 2019; Tan et al., 2024). For instance, OK-VQA (Marino et al., 2019) evaluates a model’s capability to answer questions using open-domain knowledge. Building upon this, A-OKVQA (Schwenk et al., 2022) integrates diverse external knowledge and reasoning capabilities, while S3VQA (Jain et al., 2021) introduces a dataset featuring questions that require object detection within images. Recently, factuality benchmarks have become increasingly important for LLMs: SimpleQA introduced by OpenAI (Wei et al., 2024) assesses short-form factuality, and a Chinese SimpleQA (He et al., 2024) is proposed by Alibaba Group.

Comparison to other knowledge-based VQA benchmarks: In Table 1, we present a comparative analysis of ChineseSimpleVQA alongside several mainstream knowledge-based VQA benchmarks. Our dataset represents the first Chinese multimodal evaluation set designed to comprehensively assess factual abilities, thereby addressing a significant gap in the multimodal domain. Moreover, the multi-hop questioning in our dataset queries the image’s content and related facts, effectively evaluating the model’s object recognition and factual knowledge handling.

3 ChineseSimpleVQA

3.1 Overview

As shown in Figure 2, ChineseSimpleVQA is collected through a rigorous processing pipeline including automatic verification, difficulty filtering, and human verification to ensure high quality and appropriate difficulty levels. Ultimately, we obtained 1,100 images and 2,200 Q&A pairs covering 56 knowledge subtopics. Moreover, drawing from human cognitive processes, each image is designed to include two questions: the first question is solely for visual object recognition (i.e., seeing the world), while the second question requires both object recognition and factual knowledge (i.e., discovering knowledge). In other words, these two questions are used to evaluate the model’s ability to “recognize everything” and “understand encyclopedias”, respectively. Data examples can be found in Appendix C.2.

3.2 Knowledge Collection

To create visually oriented Q&A pairs based on encyclopedic knowledge, we collected a substantial number of images along with their corresponding knowledge texts from Wikipedia’s “Picture of the
Day” column4. Notably, since our goal is to build a Chinese benchmark, we specifically used the Chinese version of the column as our data source. In total, we gathered 6,546 image-text pairs, with the text having an average length of 723 characters.

3.3 Q&A Construction

Our benchmark focuses on both the visual recognition and knowledge discovery capabilities of LVLMs. Therefore, we construct Q&A pairs through three stages: (1). Building Image Object Recognition Questions: Identify the main objects within the images and formulate questions. (2). Generating Knowledge-Based Questions on Image Objects: Develop questions that investigate deeper into the knowledge related to the identified objects based on their knowledge texts. (3). Merging the Previous Two Steps to Create Two-Hop Visual Knowledge Questions: Integrate the visual recognition and knowledge-based questions to create more complex, two-hop questions that are grounded in visual content.

In the final version of our dataset, we retain the Q&A pairs from the first and last stages for each image to allow users to analyze the capability boundaries and execution mechanisms of LVLMs. Notably, the Q&A pairs at all stages are initially generated by prompting the LLM5 based on the criteria outlined in the following paragraphs. Notably, all Prompts are provided in Appendix C.4.

Criteria of Object Recognition Q&A. 1) Questions should be answerable using only information in the image, without needing external knowledge. For example, the question “Who designed the poster in the image?” is avoided as this requires extra information. 2) Each question must have a single, clear answer. For example, “What is in the image?” is unsuitable as it can refer to multiple entities. 3) Each question should be clear and specific, not divisible into separate inquiries.

Criteria of Knowledge-Based Q&A. 1) Questions should rely on factual knowledge, avoiding subjective opinions or personal views. 2) Answers must be clear and distinct, similar to the previous point. 3) Do not ask two questions at once, consistent with the previous guideline. 4) Answers should be timeless and not change over time, reflecting enduring facts. 5) Questions should be challenging, and in-depth knowledge should be assessed, avoiding overly simple queries. 6) Keep questions concise by including only essential information for a clear answer. For example, instead of asking, “In which country is the Desen’ka train station located in Vinnytsia City?” ask, “In which country is the Desen’ka train station located?”

Criteria of Final Merged Q&A. 1) Questions must rely entirely on the answer to the first Q&A pair, meaning the next question cannot be answered correctly without the first answer.2) Ensure that new questions are clear and concise, maintaining fluency and grammatical accuracy.

<table>
  <tr>
    <th>Question Category</th>
    <th>Proportion</th>
  </tr>
  <tr>
    <td>Total Number</td>
    <td>2,200</td>
  </tr>
  <tr>
    <td>Nature</td>
    <td>26.2%</td>
  </tr>
  <tr>
    <td>Life, Culture & Art</td>
    <td>14.5%</td>
  </tr>
  <tr>
    <td>Geography & Meteorological</td>
    <td>12.4%</td>
  </tr>
  <tr>
    <td>Modern Architecture</td>
    <td>11.8%</td>
  </tr>
  <tr>
    <td>Human & Society</td>
    <td>9.8%</td>
  </tr>
  <tr>
    <td>Ancient Architecture</td>
    <td>9.7%</td>
  </tr>
  <tr>
    <td>Sciences</td>
    <td>8.4%</td>
  </tr>
  <tr>
    <td>Engineering</td>
    <td>7.2%</td>
  </tr>
  <tr>
    <th colspan="2">Recognition Question Length</th>
  </tr>
  <tr>
    <td>- Maximum Length</td>
    <td>28</td>
  </tr>
  <tr>
    <td>- Minimum Length</td>
    <td>6</td>
  </tr>
  <tr>
    <td>- AVG Length</td>
    <td>11.1</td>
  </tr>
  <tr>
    <th colspan="2">Final Question Length</th>
  </tr>
  <tr>
    <td>- Maximum Length</td>
    <td>46</td>
  </tr>
  <tr>
    <td>- Minimum Length</td>
    <td>8</td>
  </tr>
  <tr>
    <td>- AVG Length</td>
    <td>16.1</td>
  </tr>
  <tr>
    <th colspan="2">Recognition Answer Length</th>
  </tr>
  <tr>
    <td>- Maximum Length</td>
    <td>16</td>
  </tr>
  <tr>
    <td>- Minimum Length</td>
    <td>1</td>
  </tr>
  <tr>
    <td>- AVG Length</td>
    <td>5.2</td>
  </tr>
  <tr>
    <th colspan="2">Final Answer Length</th>
  </tr>
  <tr>
    <td>- Maximum Length</td>
    <td>33</td>
  </tr>
  <tr>
    <td>- Minimum Length</td>
    <td>1</td>
  </tr>
  <tr>
    <td>- AVG Length</td>
    <td>4.5</td>
  </tr>
</table>

Table 2: Dataset statistics.

4https://en.wikipedia.org/wiki/Wikipedia:Picture_of_the_day
5GPT-4o(0806)(OpenAI, 2023) is used for the generation of Q&A pairs, automatic quality verification, and evaluation.

3.4 Quality Control

We design strict quality control processes, including automatic verification and human annotation, to identify and correct corresponding issues.

Automatic Verification. We developed a multi-round automated method to verify the quality of synthetic Q&A pairs, aiming to regenerate or filter out questions that do not meet the above criteria. Upon completion of the automatic verification, we retained a total of 5,009 images with 10,018 qualified Q&A pairs.
Difficulty Filtering. Subsequently, we filtered out simple samples to identify the knowledge boundaries of LLMs and enhance the difficulty of the benchmark. Specifically, if a question can be correctly answered by all of the four powerful models, i.e., GPT-4o (0806), Claude 3.5 Sonnet (Enis and Hopkins, 2024), Gemini 1.5 Pro (Team et al., 2024), and Qwen-VL-Max (Bai et al., 2023), it is considered simple and thus discarded. As a result of this process, a total of 3,058 images with 6,116 Q&A pairs were retained.

Human Verification. During this parse, a team of 23 annotators performed data verification and rewriting, while 6 engineers reviewed and selected the high-quality data.

Annotators must ensure question-answer pairs meet the standards by the following operations: 1) verifying and rewriting the questions or answers that do not meet the standards in Section3.3; 2) replacing unqualified or unrepresentative images with new ones from online sources; and 3) verifying answer accuracy via search engines and Baidu Baike. Pairs that cannot be improved are discarded. Finally, manual annotations are compared with LLM-verified results, retaining only fully consistent Q&A pairs. This rigorous process ensures accuracy and adherence to established standards.

Data Desensitization. To mitigate potential security risks during evaluation, we submitted the final dataset to six security auditors for a comprehensive security review. Each sample of data was cross-checked by at least two auditors, and only the data that passed all scrutiny was retained.

After the above annotations, 2,411 images with 4,822 Q&A pairs were retained. Afterward, the algorithm experts carefully selected 1,100 images with 2,200 QA pairs as the final dataset.

3.5 Dataset Statistics

As shown in Table 2, ChineseSimpleVQA consists of 8 major topics and 56 subtopics, which are based on manually labeled tags. Among these topics, natural knowledge is the most common, comprising 26.2% of the total. Furthermore, since ChineseSimpleVQA primarily evaluates short-form answers, the average answer lengths for these two stages of questions are 5.2 words and 4.5 words, respectively, enabling a quicker evaluation.

4 Experiment

4.1 Setup

We maintain a consistent prompt format throughout all experiments, ensuring uniformity in our benchmark. The temperature and sampling parameters are set to the official default settings for each model. For evaluation, we use the “GPT-4o-0806” as a judge model.

4.2 Baseline Models

We evaluate 13 closed-source LVLMs (i.e., o1-preview (0901) , o1-mini (0901) , GPT-4o (0806) , GPT-4o-mini (0708) 6 , Gemini-1.5-pro-flash, Gemini-1.5-pro (Team et al., 2024), Gemini-2.0-flash7, Claude-3.5-sonnet28, Claude-3.5-sonnet, Doubao-1.5-vision-pro9, Doubao-vision-pro/lite, Qwen-VL-max (Bai et al., 2023)) and 22 open-source LVLMs (i.e., Qwen2-VL series (Wang et al., 2024), Qwen2.5-VL series10, Deepseek-VL-2 series (Wu et al., 2024), LLaVA-one-vision-0.6B/72B (Li et al., 2024a), GLM-4V-9B11, Llama-vision-11B/90B12, InternVL series (Chen et al., 2024)).).

4.3 Evaluation Metrics

We use the following metrics to evaluate ChineseSimpleVQA: (1) Correctness (CO): The predicted answer to the final question should fully encompass the reference answer without contradictions. (2) Not attempted answers(NA): The predicted answer to the final question is not related to the reference answer without any contradictions. (3) Incorrectness (IN): The predicted answer to the final question contradicts the reference answer, as does the answer to the first question. (4) Correctness among Given Attempted (CGA): The proportion of questions accurately answered among those attempted. (5) F-score: The harmonic mean of Correct and Correct Given Attempted.

6https://openai.com/index/
7https://deepmind.google/technologies/gemini/flash/
8https://www.anthropic.com/news/clause-3-5-sonnet
9https://team.doubao.com/en/direction/vision
10https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api
11https://open.bigmodel.cn/dev/api/normal-model/glm-4v
12https://ollama.com/library/llama3.2-vision
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Overall results of Merged Q&A</th>
    <th colspan="6">Overall results of Recognition Q&A</th>
  </tr>
  <tr>
    <th>CO</th>
    <th>IN↓</th>
    <th>NA↓</th>
    <th>CGA</th>
    <th>F-score</th>
    <th>CO</th>
    <th>IN↓</th>
    <th>NA↓</th>
    <th>CGA</th>
    <th>F-score</th>
  </tr>
  <tr>
    <th colspan="11">Closed-Source Large Vision Language Models</th>
  </tr>
  <tr>
    <td>o1-preview</td>
    <td><b>68.8</b></td>
    <td>24.6</td>
    <td>6.5</td>
    <td>73.6</td>
    <td><b>71.1</b></td>
    <td>79.1</td>
    <td>13.6</td>
    <td>7.3</td>
    <td><b>85.3</b></td>
    <td><b>82.1</b></td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>52.7</td>
    <td>38.1</td>
    <td>9.2</td>
    <td>58.1</td>
    <td>55.3</td>
    <td>64.8</td>
    <td>24.9</td>
    <td>10.3</td>
    <td>72.2</td>
    <td>68.3</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>59.1</td>
    <td>35.5</td>
    <td>5.4</td>
    <td>62.4</td>
    <td>60.7</td>
    <td>77.5</td>
    <td>15.5</td>
    <td>7.0</td>
    <td>83.4</td>
    <td>80.4</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>51.0</td>
    <td>43.6</td>
    <td>5.4</td>
    <td>53.9</td>
    <td>52.4</td>
    <td>70.8</td>
    <td>23.1</td>
    <td>6.1</td>
    <td>75.4</td>
    <td>73.0</td>
  </tr>
  <tr>
    <td>Gemini-2.0-flash</td>
    <td>64.5</td>
    <td>29.5</td>
    <td>5.9</td>
    <td>68.6</td>
    <td>66.5</td>
    <td>76.7</td>
    <td>19.6</td>
    <td>3.7</td>
    <td>79.7</td>
    <td>78.2</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro-flash</td>
    <td>56.5</td>
    <td>34.6</td>
    <td>8.8</td>
    <td>62.0</td>
    <td>59.2</td>
    <td>70.3</td>
    <td>25.9</td>
    <td>3.8</td>
    <td>73.1</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>66.2</td>
    <td>31.4</td>
    <td><b>2.5</b></td>
    <td>67.8</td>
    <td>67.0</td>
    <td>77.5</td>
    <td>20.7</td>
    <td><b>1.7</b></td>
    <td>78.9</td>
    <td>78.2</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet2</td>
    <td>63.8</td>
    <td>30.6</td>
    <td>5.5</td>
    <td>67.6</td>
    <td>65.6</td>
    <td>77.6</td>
    <td>17.2</td>
    <td>5.2</td>
    <td>81.9</td>
    <td>79.7</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td>
    <td>59.5</td>
    <td>26.4</td>
    <td>14.2</td>
    <td>69.4</td>
    <td>64.0</td>
    <td>69.5</td>
    <td>20.2</td>
    <td>10.3</td>
    <td>77.5</td>
    <td>73.3</td>
  </tr>
  <tr>
    <td>Qwen-VL-max</td>
    <td>56.5</td>
    <td>39.6</td>
    <td>3.8</td>
    <td>58.8</td>
    <td>57.6</td>
    <td>72.9</td>
    <td>24.6</td>
    <td>2.5</td>
    <td>74.7</td>
    <td>73.8</td>
  </tr>
  <tr>
    <td>Doubao-1.5-vision-pro</td>
    <td>53.2</td>
    <td>36.4</td>
    <td>10.5</td>
    <td>59.4</td>
    <td>56.1</td>
    <td>50.4</td>
    <td>39.1</td>
    <td>10.5</td>
    <td>56.3</td>
    <td>53.2</td>
  </tr>
  <tr>
    <td>Doubao-vision-pro</td>
    <td>52.0</td>
    <td>43.2</td>
    <td>4.8</td>
    <td>54.6</td>
    <td>53.3</td>
    <td>44.0</td>
    <td>51.9</td>
    <td>4.1</td>
    <td>45.9</td>
    <td>44.9</td>
  </tr>
  <tr>
    <td>Doubao-vision-lite</td>
    <td>31.9</td>
    <td>30.0</td>
    <td>38.1</td>
    <td>51.5</td>
    <td>39.4</td>
    <td>30.6</td>
    <td>43.5</td>
    <td>25.8</td>
    <td>41.3</td>
    <td>35.2</td>
  </tr>
  <tr>
    <th colspan="11">Open-Source Large Vision Language Models</th>
  </tr>
  <tr>
    <td>Deepseek-VL2</td>
    <td>33.4</td>
    <td>60.3</td>
    <td>6.4</td>
    <td>35.6</td>
    <td>34.5</td>
    <td>32.0</td>
    <td>58.5</td>
    <td>9.5</td>
    <td>35.4</td>
    <td>33.6</td>
  </tr>
  <tr>
    <td>Deepseek-VL2-small</td>
    <td>37.0</td>
    <td>58.5</td>
    <td>4.5</td>
    <td>38.8</td>
    <td>37.9</td>
    <td>33.1</td>
    <td>58.4</td>
    <td>8.5</td>
    <td>36.2</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>Deepseek-VL2-tiny</td>
    <td>22.5</td>
    <td>72.3</td>
    <td>5.2</td>
    <td>23.8</td>
    <td>23.1</td>
    <td>23.0</td>
    <td>69.9</td>
    <td>7.1</td>
    <td>24.8</td>
    <td>23.8</td>
  </tr>
  <tr>
    <td>LLaVA-onevision-72B</td>
    <td>34.7</td>
    <td>60.6</td>
    <td>4.6</td>
    <td>36.4</td>
    <td>35.6</td>
    <td>29.5</td>
    <td>59.6</td>
    <td>10.9</td>
    <td>33.1</td>
    <td>31.2</td>
  </tr>
  <tr>
    <td>LLaVA-onevision-7B</td>
    <td>18.0</td>
    <td>57.7</td>
    <td>24.3</td>
    <td>23.8</td>
    <td>20.5</td>
    <td>14.5</td>
    <td>61.2</td>
    <td>24.4</td>
    <td>19.1</td>
    <td>16.5</td>
  </tr>
  <tr>
    <td>LLaVA-onevision-0.5B</td>
    <td>7.9</td>
    <td>82.3</td>
    <td>9.8</td>
    <td>8.8</td>
    <td>8.3</td>
    <td>8.5</td>
    <td>70.6</td>
    <td>20.8</td>
    <td>10.8</td>
    <td>9.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B</td>
    <td>49.0</td>
    <td><b>42.8</b></td>
    <td>8.2</td>
    <td><b>53.4</b></td>
    <td>51.1</td>
    <td>45.7</td>
    <td><b>48.5</b></td>
    <td>5.7</td>
    <td>48.5</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>39.5</td>
    <td>54.2</td>
    <td>6.4</td>
    <td>42.1</td>
    <td>40.8</td>
    <td>41.3</td>
    <td>51.2</td>
    <td>7.5</td>
    <td>44.6</td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-3B</td>
    <td>31.4</td>
    <td>61.3</td>
    <td>7.4</td>
    <td>33.9</td>
    <td>32.6</td>
    <td>32.9</td>
    <td>62.0</td>
    <td>5.1</td>
    <td>34.7</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td><b>50.6</b></td>
    <td>46.6</td>
    <td>2.7</td>
    <td>52.1</td>
    <td><b>51.3</b></td>
    <td><b>48.0</b></td>
    <td>49.1</td>
    <td><b>2.9</b></td>
    <td><b>49.4</b></td>
    <td><b>48.7</b></td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>38.2</td>
    <td>60.3</td>
    <td><b>1.5</b></td>
    <td>38.8</td>
    <td>38.5</td>
    <td>39.5</td>
    <td>57.1</td>
    <td>3.4</td>
    <td>40.9</td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>Qwen2-VL-2B</td>
    <td>29.0</td>
    <td>66.5</td>
    <td>4.5</td>
    <td>30.4</td>
    <td>29.7</td>
    <td>34.2</td>
    <td>63.1</td>
    <td>2.7</td>
    <td>35.1</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>GLM-4v</td>
    <td>34.8</td>
    <td>51.7</td>
    <td>13.5</td>
    <td>40.2</td>
    <td>37.3</td>
    <td>37.1</td>
    <td>54.5</td>
    <td>8.4</td>
    <td>40.5</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>Llama-vision-90B</td>
    <td>46.2</td>
    <td>51.6</td>
    <td>2.2</td>
    <td>47.2</td>
    <td>46.7</td>
    <td>37.2</td>
    <td>57.5</td>
    <td>5.3</td>
    <td>39.3</td>
    <td>38.2</td>
  </tr>
  <tr>
    <td>Llama-vision-11B</td>
    <td>29.4</td>
    <td>66.4</td>
    <td>4.3</td>
    <td>30.7</td>
    <td>30.0</td>
    <td>27.4</td>
    <td>64.3</td>
    <td>8.4</td>
    <td>29.9</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>InterVL2.5-78B</td>
    <td>41.1</td>
    <td>51.7</td>
    <td>7.2</td>
    <td>44.3</td>
    <td>42.6</td>
    <td>36.2</td>
    <td>56.7</td>
    <td>7.1</td>
    <td>38.9</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>InterVL2.5-38B</td>
    <td>35.5</td>
    <td>54.3</td>
    <td>10.2</td>
    <td>39.6</td>
    <td>37.5</td>
    <td>30.0</td>
    <td>56.4</td>
    <td>13.6</td>
    <td>34.7</td>
    <td>32.2</td>
  </tr>
  <tr>
    <td>InterVL2.5-26B</td>
    <td>32.4</td>
    <td>52.9</td>
    <td>14.7</td>
    <td>38.0</td>
    <td>34.9</td>
    <td>29.9</td>
    <td>55.3</td>
    <td>14.8</td>
    <td>35.1</td>
    <td>32.3</td>
  </tr>
  <tr>
    <td>InterVL2.5-8B</td>
    <td>28.4</td>
    <td>60.5</td>
    <td>11.1</td>
    <td>31.9</td>
    <td>30.0</td>
    <td>24.5</td>
    <td>63.5</td>
    <td>12.1</td>
    <td>27.8</td>
    <td>26.0</td>
  </tr>
  <tr>
    <td>InterVL2.5-4B</td>
    <td>24.5</td>
    <td>73.7</td>
    <td>1.8</td>
    <td>24.9</td>
    <td>24.7</td>
    <td>22.2</td>
    <td>72.5</td>
    <td>5.4</td>
    <td>23.4</td>
    <td>22.8</td>
  </tr>
  <tr>
    <td>InterVL2.5-2B</td>
    <td>13.9</td>
    <td>73.9</td>
    <td>12.2</td>
    <td>15.8</td>
    <td>14.8</td>
    <td>16.8</td>
    <td>74.0</td>
    <td>9.2</td>
    <td>18.5</td>
    <td>17.6</td>
  </tr>
  <tr>
    <td>InterVL2.5-1B</td>
    <td>14.7</td>
    <td>70.7</td>
    <td>14.5</td>
    <td>17.2</td>
    <td>15.9</td>
    <td>16.8</td>
    <td>69.3</td>
    <td>13.9</td>
    <td>19.5</td>
    <td>18.1</td>
  </tr>
</table>

Table 3: Performance comparison of closed source and open source LVLMs on multi-hop QAs (i.e. Merged Q&A and Recognition Q&A). For metrics, CO, NA, IN, and CGA denote “Correct”, “Not attempted”, “Incorrect”, and “Correct given attempted”, respectively. The highest scores among models in each metric are highlighted in green.

4.4 Main Results

We present the performance results of various LVLMs on the ChineseSimpleVQA dataset in Table 3. The evaluation metrics are introduced in Section 4.3. We report the models’ performance on recognition and final merged questions, and summarize the following findings:

• o1-preview is the best performer for both recognition and final questions among all models. The next best are Gemini-2.0-flash and Gemini-1.5-pro, which follow closely behind. For Chinese ability, Qwen family performs the best.

• Performance on final questions is positively correlated with the performance on recognition questions, though accuracy is typically higher for recognition questions. This suggests that models can identify objects but sometimes fail to grasp deeper knowledge. Some models (e.g. Deepseek-VL2) perform better on final questions because correct and incorrect object predictions can point to the same final answer. For instance, both “Misumena vatia” and “Thomisidae” belong to the order “Araneae.”13

• The larger the model size within the same series of models, the better the result. Taking the Qwen2-VL series as an example, when the model size increases from 3B to 72B, the accuracy on the final questions increases from 29.0% to 50.6%. More detailed analysis can be found

13Examples are the scientific names of two spider species.
Figure 3: Correctness(CO) metric for eight topics. To conserve space, only the top 10 models are displayed.

Figure 4: The number of incorrectly answered final questions, and the number of samples where the image object was correctly recognized.

in Appendix B.1.

• The IN rate of the models is much higher than their NA rate (in addition to Doubao-vision-lite), indicating that the models are more likely to confidently provide incorrect factual information. Mitigating this hallucination problem remains a significant challenge. In Section 4.6, we will conduct a detailed analysis focusing on model confidence.

4.5 Performance on Different Topics

We select the top 10 models with the highest CO rate on the final questions and evaluated their performance across different topics. The results are shown in Figure 3. For all topics, the trends in model performance for both recognition and final questions align with the conclusions presented in the previous section. It is obvious that these models are more adept at topics such as modern architecture, engineering, and science. However, their performance is slightly lower on natural topics, as these questions often need more specialized knowledge than common sense can provide.

From this figure, we can observe the differences in the model’s performance on recognition questions versus final questions across different topics. For example, Claude-3.5-Sonnet exhibits a small performance gap in the topic of modern architecture. This indicates that once the model accurately identifies the object of an image, it is more likely to provide accurate responses based on relevant knowledge. Conversely, if the model demonstrates a large performance gap on a particular topic, it suggests that the model’s knowledge of that topic is limited.

4.6 Further Analysis

Analysis of Knowledge Scope. In the design of ChineseSimpleQA, the final question requires deeper, extended knowledge beyond the recognition question. To explore at which stage the real knowledge limitation arises when models fail to answer the final question, we collect examples where different models provided incorrect answers to the final questions and conduct analysis by calculating the percentage of corresponding recognition questions that were answered correctly. The results are presented in Figure 4.

For o1-preview, Gemini-1.5-pro, Caude3.5-sonet2 and Qwen-vl-max, the error rate caused by image object recognition is less than 50%. This shows that for these models, the inability to correctly answer the final question is mainly limited by the lack of more complex extended knowledge. For the other models, it is impossible to correctly recognize the image object from the initial stage, especially GLM-4v-9B.

Analysis of Calibration. The confidence level (%) of a perfectly calibrated model should precisely match the accuracy of its predictions. From this
Figure 5: Up: Calibration of LLMs based on their stated confidence for Recognition and final Q&A. Down: Improvement in accuracy with increased test-time compute using Best-of-N for Recognition and final Q&A.

Figure 6: Robustness of Judge Models of Final Q&A.

perspective, we prompt the models to provide their confidence levels (ranging from 0 to 100) while answering questions. In Figure 5, we present the accuracy of the models’ answers at different confidence intervals (with an interval size of 10).

The results illustrate the alignment performance of the tested models. Specifically, o1-preview exhibits the best performance on both types of questions. While o1-mini ranks second in recognition questions, closely following o1-preview, it performs slightly worse on final questions. Overall, most models fall below the perfect alignment line, indicating that they tend to be overconfident, even when their answers are incorrect.

Analysis of Inference Attempt. Furthermore, we analyze the changes in the accuracy of the model’s answers as the inference attempts increased. Specifically, we sample 100 instances from each of the two types of questions in ChineseSimpleVQA. For each instance, the model is asked to independently generate an answer 100 times. For the results of multiple inferences, we use the Best-of-N method to obtain the accuracy of the model responses.

Figure 5 shows that within the range of 1-30 inference times, the accuracy of the tested model increases with the increase of inference times. However, when the number of inferences is greater than 30, the performance of the model tends to be stable. Beyond this point, the model does not acquire significantly more accurate knowledge even with additional attempts. These findings can serve as a reference for determining the boundaries of the model’s knowledge capabilities.

5 Robustness of Judge Models

Moreover, we select two models, i.e., Gemini-1.5-pro and Claude-3.5-sonnet2, as the judge models and compare the evaluation results of GPT-4o-0806. The results are shown in Figure 6. It can be seen that when using different evaluation models, the ranking of the tested models remains consistent. This further proves the robustness of the evaluation method of ChineseSimpleVQA.
6 Conclusion

We introduce the first short-form VQA benchmark, named Chinese SimpleVQA, to evaluate the visual factuality abilities of existing LVLMs. This benchmark is characterized by important features such as multi-hop, diverse, high-quality, static, and easy-to-evaluate. Based on Chinese SimpleVQA, we conduct a comprehensive evaluation of the performance of nearly 30 models regarding visual factuality, providing a detailed analysis of the capability boundaries to demonstrate the advantages and necessity of ChineseSimpleVQA.

7 Limitation

From the perspective of topic distribution, the coverage of topics discussed in this paper remains relatively limited. A broader topic coverage would contribute to a more comprehensive assessment of visual knowledge. In terms of data volume, after rigorous quality filtering and difficulty balancing, the final version of our dataset represents only a tiny fraction of the overall data. Therefore, expanding the dataset will be a key focus of our future efforts. Furthermore, as highlighted by this work (Jiang et al., 2025), datasets to evaluate the model against potential jailbreak attacks is crucial. Finally, regarding model structure, improving the visual factuality of the model to enhance its performance on ChineseSimpleVQA will become increasingly important. However, both performance improvement and security aspects may already be beyond the scope of this paper.

8 Ethic Statement

We respect intellectual property rights and comply with relevant laws and regulations. The source of our dataset is publicly available, and we have taken careful measures to ensure that the dataset does not contain any personal sensitive information. In addition, our work is only for research purposes, not for commercial purposes.

References

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Maxim Enis and Mark Hopkins. 2024. From llm to nmt: Advancing low-resource machine translation with claude. arXiv preprint arXiv:2404.13813.

Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. 2024. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296.

Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. 2015. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623.

Yash Goyal, Tejas Khot. Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. 2024. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140.

Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2021. Select, substitute, search: A new benchmark for knowledge-augmented visual question answering. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2491–2498.

Yilei Jiang, Xinyan Gao, Tianshuo Peng, Yingshui Tan, Xiaoyong Zhu, Bo Zheng, and Xiangyu Yue. 2025. Hiddendetec: Detecting jailbreak attacks against large vision-language models via monitoring hidden states. arXiv preprint arXiv:2502.14744.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava-oncevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024b. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mm-bench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer.

Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. Advances in neural information processing systems, 27.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204.

R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5).

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer.

Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiangyong Zhu, et al. 2024. Chinese safetyqa: A safety short-form factuality benchmark for large language models. arXiv preprint arXiv:2412.15265.

Gemini Team, Petko Georgiev, Ying Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427.

Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2015. Explicit knowledge-based reasoning for visual question answering. arXiv preprint arXiv:1511.02570.

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. ArXiv, abs/2411.04368.

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731.

A Overview of Appendix

We have over 10 pages of this appendix, comprising the following subsections for the convenience of readers:

More additional experiments

• Appendix B.1: Additional experimental analysis of model scale.

• Appendix B.2: Additional experimental analysis of performance comparison among models.

• Appendix B.3: Additional experimental analysis of rankings on ChineseSimpleVQA.

• Appendix B.4: Additional experimental analysis between popularity and difficulty.
Figure 7: Relationship between model scale (in billion parameters) and F-score on Recognition and final Q&A.

Figure 8: Example of model responses of where Deekseep-vl2-small’s answer is correct but Deepseep-VL2’s answer is wrong.

• Appendix B.5: Additional analysis of Human Calibration.

• Appendix B.6: Additional experimental comparison with other Datasets.

• Appendix B.7: Additional experimental comparison with Knowledge-Based Q&A

• Appendix B.8: More results.

More visualization and cases

• Appendix C.1: Detailed question distribution.

• Appendix C.2: Dataset visualization of our ChineseSimpleVQA.

• Appendix C.3: Visualization of performance comparison among models.

• Appendix C.4: Details of all prompts in the dataset generation and validation.

We hope that our efforts will serve as a source of inspiration for more readers!

B More Experiments

B.1 F-score v.s. Model Scale
In Figure 7, we illustrate the trend of F-score as the model size increases. Generally, there is a positive correlation between these two factors. How-
ever, this relationship weakens in the intermediate range, especially for vision recognition tasks, where larger models may perform worse. This observation contrasts with findings from ChineseSimpleQA, which analyzed large language models (LLMs). For large vision-language models (LVLMs), while increased size suggests greater knowledge capacity, accurate image object recognition hinges more on the quality of image-text alignment training, critically affecting factual accuracy. Additionally, Table 3 shows the performance of same-series models typically improves with size, but Deepseek-VL and Deepseek-VL-small deviate from this pattern. Figure 8 provides examples where the smaller model correctly answers the questions that the larger Deepseek-VL fails, highlighting the discrepancy between them.

B.2 CGA Results on 8 Topics

In this section, we present more results of Correctness among Given Attempted (CGA). In principle, the models are encouraged to do not attempt to answer when they are unable to determine the knowledge. Therefore, we further show the accuracy of the models’ responses on 8 topics after excluding the questions that the models do not attempt to answer, namely CGA. In general, GCA has higher indicators than CO. The detail results are shown in Figure 9.

B.3 Rankings on ChineseSimpleVQA

We also compare the ranking differences of various models on the recognition questions and the final questions. The rankings are shown in Figure 12.

![CGA of the top 10 models for eight topics.](page_153_120_1142_377.png)

Figure 9: CGA of the top 10 models for eight topics.

B.4 Analysis between Popularity and Difficulty

We select the first 100 samples from our dataset and monitor their page views on Wikipedia over the past 30 days to assess their popularity. We categorize the popularity into three ranges based on the number of views: (0-10), (10-50), and (50, +inf). We then evaluate the experimental performance of the following 10 closed-source models across these different levels of popularity as shown in Table 4. The results show that models generally perform better with higher popularity data, particularly in Recognition Q&A tasks. For data with over 50 views in the past 30 days, most models can achieve an accuracy rate of 100%.

B.5 Analysis of Human Calibration

We manually extract 110 samples of the main results from both the merged and recognition Q&A for verification and find that agreement with human assessments reached 95.5% and 97.3%, respectively. This proves the validity of using LLM-as-judge for evaluation.

B.6 Comparison with other Datasets

In order to explore whether models that perform well in other datasets continue to perform well in this dataset. We compare the evaluation performance of models on our ChineseSimpleVQA and text-based (no vision) short-form factuality benchmarks——ChineseSimpleQA (He et al., 2024). The experimental results are shown in Table 5: As can be seen from the CO metric, the o1-preview model achieves the best results in the merged Q&A and Recognition Q&A of ChineseSimpleVQA, as well as in ChineseSimpleQA. In contrast, the gpt-
4o-mini model performs the worst in the merged Q&A of ChineseSimpleVQA and in ChineseSimpleQA, demonstrating consistency. For the Recognition Q&A of ChineseSimpleVQA, the model that performed the worst was o1-mini. This indicates that our benchmark is a qualified factuality evaluation set that can also assess large vision language models’ image recognition capabilities. It holds broad evaluative applications for LVLMs.

B.7 Performance Comparison including Knowledge-Based Q&A

To assist us in analysis, we conduct an evaluation on text-only Knowledge-Based Q&A. The experimental results are shown in Table 6. From the CO metric perspective, we can find that the o1 preview still shows the best performance, which is consistent with the other two Q&A tasks. Additionally, the performance of Knowledge-Based Q&A is similar to Merged Q&A, and for most models, Knowledge-Based Q&A performs better. This indicates that for LVLMs, it is more challenging to simultaneously recognize images as well as answer Knowledge-Based questions, and compared to image recognition, Knowledge-Based Q&A is the more difficult part.

B.8 More Results

We present the CO and CGA results of all models in Figure 10 and 11, respectively.

C More Details of ChineseSimpleQA

C.1 Detailed Question Distribution

In Table 7, we further give the detailed distribution (number of questions) of ChineseSimpleVQA under 56 sub-topics. In addition, we counted the average length of recognition questions and final questions under each topic.

C.2 ChineseSimpleVQA Examples

We provide examples from the 8 topics in ChineseSimpleVQA to illustrate our benchmark’s focus on multi-hop QAs. Figures 13 show that our benchmark covers various knowledge topics and uses multi-hop questioning to better evaluate different models’ capabilities.

C.3 Model Response Examples

In this section, we visualize the answers provided by different models to the same set of questions, to further investigate the capability boundaries of these models. As illustrated in Figures 14 through 16, similar to the conclusion presented in Section 4.4, almost all models fail to achieve satisfactory performance on our ChineseSimpleVQA. Addition-

![CO of all other different models for eight topics.](page_98_120_1292_670.png)

Figure 10: CO of all other different models for eight topics.
Figure 11: CGA of all other different models for eight topics.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">CO of Merged Q&A</th>
    <th colspan="4">CO of Recognition Q&A</th>
  </tr>
  <tr>
    <th>0 - 10</th>
    <th>10 - 50</th>
    <th>50 - +inf</th>
    <th>+inf</th>
    <th>0 - 10</th>
    <th>10 - 50</th>
    <th>50 - +inf</th>
    <th>+inf</th>
  </tr>
  <tr>
    <td>o1-preview</td>
    <td>68.00</td>
    <td>84.62</td>
    <td>80.00</td>
    <td>84.00</td>
    <td>90.38</td>
    <td>100.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>o1-mini</td>
    <td>68.00</td>
    <td>73.08</td>
    <td>60.00</td>
    <td>72.00</td>
    <td>76.92</td>
    <td>80.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>72.00</td>
    <td>73.08</td>
    <td>80.00</td>
    <td>76.00</td>
    <td>86.54</td>
    <td>100.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>48.00</td>
    <td>69.23</td>
    <td>80.00</td>
    <td>68.00</td>
    <td>92.31</td>
    <td>80.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemini-2.0-flash</td>
    <td>72.00</td>
    <td>78.85</td>
    <td>60.00</td>
    <td>84.00</td>
    <td>90.38</td>
    <td>80.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro-flash</td>
    <td>80.00</td>
    <td>71.15</td>
    <td>70.00</td>
    <td>88.00</td>
    <td>78.85</td>
    <td>80.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td>
    <td>80.00</td>
    <td>76.92</td>
    <td>100.00</td>
    <td>92.00</td>
    <td>86.54</td>
    <td>100.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet2</td>
    <td>84.00</td>
    <td>73.08</td>
    <td>80.00</td>
    <td>88.00</td>
    <td>84.62</td>
    <td>100.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td>
    <td>68.00</td>
    <td>67.31</td>
    <td>80.00</td>
    <td>92.00</td>
    <td>78.85</td>
    <td>90.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen-VL-max</td>
    <td>88.00</td>
    <td>73.08</td>
    <td>90.00</td>
    <td>80.00</td>
    <td>78.85</td>
    <td>90.00</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Different popularity’s difficulty of closed source LVLMs on multi-hop QAs (i.e. Merged Q&A and Recognition Q&A).

ally, the differences in performance between the various models are significant.

C.4 Prompts of Dataset Generation and Validation

The generation, validation, and evaluation of question-answer pairs all use OpenAI’s gpt-4o-0806. We provide the specific prompts in the section. The data generation prompt are shown in Figure 17, Figure 18 and Figure 19; The validation prompt is shown in Figure 20; The evaluation prompt for ChineseSimpleVQA is shown in Figure 21. All construction and evaluation processes of Chinese SimpleVQA use the Chinese version of these prompts.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">Overall results of Merged Q&A</th>
    <th colspan="5">Overall results of Recognition Q&A</th>
    <th colspan="5">Overall results of ChineseSimpleQA</th>
  </tr>
  <tr>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
  </tr>
  <tr>
    <td>o1-preview</td><td style="background-color:#b3e5b3;">68.8</td><td style="background-color:#b3e5b3;">24.6</td><td>6.5</td><td style="background-color:#b3e5b3;">73.6</td><td style="background-color:#b3e5b3;">71.1</td><td style="background-color:#b3e5b3;">79.1</td><td style="background-color:#b3e5b3;">13.6</td><td>7.3</td><td style="background-color:#b3e5b3;">85.3</td><td style="background-color:#b3e5b3;">82.1</td><td style="background-color:#b3e5b3;">63.8</td><td style="background-color:#b3e5b3;">24.0</td><td>12.2</td><td style="background-color:#b3e5b3;">72.7</td><td style="background-color:#b3e5b3;">67.9</td>
  </tr>
  <tr>
    <td>o1-mini</td><td>52.7</td><td>38.1</td><td>9.2</td><td>58.1</td><td>55.3</td><td>64.8</td><td>24.9</td><td>10.3</td><td>72.2</td><td>68.3</td><td>39.5</td><td>39.9</td><td>20.6</td><td>49.7</td><td>44.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>59.1</td><td>35.5</td><td>5.4</td><td>62.4</td><td>60.7</td><td>77.5</td><td>15.5</td><td>7.0</td><td>83.4</td><td>80.4</td><td>59.3</td><td>39.3</td><td>1.4</td><td>60.1</td><td>59.7</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td>51.0</td><td>43.6</td><td>5.4</td><td>53.9</td><td>52.4</td><td>70.8</td><td>23.1</td><td>6.1</td><td>75.4</td><td>73.0</td><td>37.6</td><td>61.5</td><td style="background-color:#b3e5b3;">0.9</td><td>37.9</td><td>37.8</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td><td>66.2</td><td>31.4</td><td style="background-color:#b3e5b3;">2.5</td><td>67.8</td><td>67.0</td><td>77.5</td><td>20.7</td><td style="background-color:#b3e5b3;">1.7</td><td>78.9</td><td>78.2</td><td>37.4</td><td>37.6</td><td>8.0</td><td>59.1</td><td>56.7</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td><td>59.5</td><td>26.4</td><td>14.2</td><td>69.4</td><td>64.0</td><td>69.5</td><td>20.2</td><td>10.3</td><td>77.5</td><td>73.3</td><td>45.2</td><td>26.4</td><td>27.4</td><td>63.6</td><td>53.5</td>
  </tr>
</table>

Table 5: Performance comparison between ChineseSimpleVQA and ChinsesSimpleQA (He et al., 2024). For metrics, CO, NA, IN, and CGA denote “Correct”, “Not attempted”, “Incorrect”, and “Correct given attempted”, respectively. The highest scores among models in each metric are highlighted in green.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">Overall results of Merged Q&A</th>
    <th colspan="5">Overall results of Recognition Q&A</th>
    <th colspan="5">Overall results of Knowledge-Based Q&A</th>
  </tr>
  <tr>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
    <th>CO</th><th>IN↓</th><th>NA↓</th><th>CGA</th><th>F-score</th>
  </tr>
  <tr>
    <td>o1-preview</td><td style="background-color:#b3e5b3;">68.8</td><td style="background-color:#b3e5b3;">24.6</td><td>6.5</td><td style="background-color:#b3e5b3;">73.6</td><td style="background-color:#b3e5b3;">71.1</td><td style="background-color:#b3e5b3;">79.1</td><td style="background-color:#b3e5b3;">13.6</td><td>7.3</td><td style="background-color:#b3e5b3;">85.3</td><td style="background-color:#b3e5b3;">82.1</td><td style="background-color:#b3e5b3;">72.9</td><td style="background-color:#b3e5b3;">21.0</td><td>6.6</td><td style="background-color:#b3e5b3;">77.5</td><td style="background-color:#b3e5b3;">74.8</td>
  </tr>
  <tr>
    <td>o1-mini</td><td>52.7</td><td>38.1</td><td>9.2</td><td>58.1</td><td>55.3</td><td>64.8</td><td>24.9</td><td>10.3</td><td>72.2</td><td>68.3</td><td>50.9</td><td>34.2</td><td>14.9</td><td>59.8</td><td>55.0</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>59.1</td><td>35.5</td><td>5.4</td><td>62.4</td><td>60.7</td><td>77.5</td><td>15.5</td><td>7.0</td><td>83.4</td><td>80.4</td><td>66.5</td><td>31.6</td><td>1.8</td><td>67.8</td><td>67.2</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td><td>51.0</td><td>43.6</td><td>5.4</td><td>53.9</td><td>52.4</td><td>70.8</td><td>23.1</td><td>6.1</td><td>75.4</td><td>73.0</td><td>47.1</td><td>51.2</td><td>1.2</td><td>47.9</td><td>47.2</td>
  </tr>
  <tr>
    <td>Gemini-2.0-flash</td><td>64.5</td><td>29.5</td><td>5.9</td><td>68.6</td><td>66.5</td><td>76.7</td><td>19.6</td><td>3.7</td><td>79.7</td><td>78.2</td><td>69.4</td><td>30.3</td><td style="background-color:#b3e5b3;">0.5</td><td>69.6</td><td>69.4</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro-flash</td><td>56.5</td><td>34.6</td><td>8.8</td><td>62.0</td><td>59.2</td><td>70.3</td><td>25.9</td><td>3.8</td><td>73.1</td><td>71.6</td><td>57.4</td><td>38.5</td><td>4.4</td><td>59.8</td><td>58.5</td>
  </tr>
  <tr>
    <td>Gemini-1.5-pro</td><td>66.2</td><td>31.4</td><td style="background-color:#b3e5b3;">2.5</td><td>67.8</td><td>67.0</td><td>77.5</td><td>20.7</td><td style="background-color:#b3e5b3;">1.7</td><td>78.9</td><td>78.2</td><td>66.5</td><td>32.5</td><td>1.5</td><td>67.0</td><td>66.5</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet2</td><td>63.8</td><td>30.6</td><td>5.5</td><td>67.6</td><td>65.6</td><td>77.6</td><td>17.2</td><td>5.2</td><td>81.9</td><td>79.7</td><td>62.4</td><td>30.0</td><td>7.9</td><td>67.4</td><td>64.6</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td><td>59.5</td><td>26.4</td><td>14.2</td><td>69.4</td><td>64.0</td><td>69.5</td><td>20.2</td><td>10.3</td><td>77.5</td><td>73.3</td><td>56.5</td><td>26.7</td><td>16.7</td><td>67.9</td><td>61.7</td>
  </tr>
  <tr>
    <td>Qwen-VL-max</td><td>56.5</td><td>39.6</td><td>3.8</td><td>58.8</td><td>57.6</td><td>72.9</td><td>24.6</td><td>2.5</td><td>74.7</td><td>73.8</td><td>58.0</td><td>39.5</td><td>1.7</td><td>59.9</td><td>59.3</td>
  </tr>
</table>

Table 6: Performance comparison of closed source and open source LVLMs on multi-hop QAs (i.e. Merged Q&A, Knowledge-Based Q&A and Recognition Q&A). For metrics, CO, NA, IN, and CGA denote “Correct”, “Not attempted”, “Incorrect”, and “Correct given attempted”, respectively. The highest scores among models in each metric are highlighted in green.
Figure 12: Rankings of different models on ChineseSimpleVQA.
Table 7: Avg. question length of all subtopics. "ChemE." represents “Chemical Engineering”.

<table>
  <tr>
    <th>Primary Category</th>
    <th>Secondary Category</th>
    <th>Count</th>
    <th>Q1 avg. length</th>
    <th>Q2 avg. length</th>
  </tr>
  <tr>
    <td rowspan="6">Geography & Meteorological</td>
    <td>Meteorology & Climate（大气、气象与气候）</td>
    <td>10</td>
    <td>13.5</td>
    <td>22.1</td>
  </tr>
  <tr>
    <td>Hydrology（水文）</td>
    <td>25</td>
    <td>11.1</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>Human Geography（人文地理）</td>
    <td>38</td>
    <td>12.5</td>
    <td>16.2</td>
  </tr>
  <tr>
    <td>Mountains（山脉）</td>
    <td>19</td>
    <td>10.8</td>
    <td>14.7</td>
  </tr>
  <tr>
    <td>Landforms & Topography（地貌与地形）</td>
    <td>30</td>
    <td>11.9</td>
    <td>16.8</td>
  </tr>
  <tr>
    <td>Geology（地质）</td>
    <td>7</td>
    <td>12.3</td>
    <td>19.7</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>7</td>
    <td>13.0</td>
    <td>22.9</td>
  </tr>
  <tr>
    <td rowspan="6">Modern Architecture</td>
    <td>Commercial Buildings（商业建筑）</td>
    <td>14</td>
    <td>12.1</td>
    <td>16.2</td>
  </tr>
  <tr>
    <td>Public Cultural Buildings（公共文化建筑）</td>
    <td>39</td>
    <td>11.1</td>
    <td>16.5</td>
  </tr>
  <tr>
    <td>Funerary and Memorial Buildings（墓葬与纪念建筑）</td>
    <td>10</td>
    <td>12.9</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>13</td>
    <td>10.9</td>
    <td>15.2</td>
  </tr>
  <tr>
    <td>Religious Buildings（宗教建筑）</td>
    <td>21</td>
    <td>10.4</td>
    <td>15.0</td>
  </tr>
  <tr>
    <td>Transportation Buildings（交通建筑）</td>
    <td>34</td>
    <td>11.2</td>
    <td>15.6</td>
  </tr>
  <tr>
    <td rowspan="10">Nature</td>
    <td>Insects（昆虫）</td>
    <td>49</td>
    <td>9.4</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>Reptiles（爬行类）</td>
    <td>30</td>
    <td>9.6</td>
    <td>12.6</td>
  </tr>
  <tr>
    <td>Birds（鸟类）</td>
    <td>37</td>
    <td>9.5</td>
    <td>14.4</td>
  </tr>
  <tr>
    <td>Mammals（哺乳类）</td>
    <td>39</td>
    <td>9.8</td>
    <td>14.2</td>
  </tr>
  <tr>
    <td>Angiosperms（被子植物）</td>
    <td>43</td>
    <td>10.3</td>
    <td>13.3</td>
  </tr>
  <tr>
    <td>Arthropods（non-insect）（节肢动物(非昆虫)）</td>
    <td>19</td>
    <td>9.9</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>Gymnosperms（裸子植物）</td>
    <td>6</td>
    <td>11.7</td>
    <td>13.7</td>
  </tr>
  <tr>
    <td>Fish（鱼类）</td>
    <td>21</td>
    <td>9.1</td>
    <td>11.2</td>
  </tr>
  <tr>
    <td>Amphibians（两栖类）</td>
    <td>11</td>
    <td>9.4</td>
    <td>13.6</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>12</td>
    <td>11.3</td>
    <td>14.0</td>
  </tr>
  <tr>
    <td>Mollusks（软体动物）</td>
    <td>21</td>
    <td>10.1</td>
    <td>11.2</td>
  </tr>
  <tr>
    <td rowspan="10">Life Culture & Art</td>
    <td>Sculpture（雕塑）</td>
    <td>23</td>
    <td>11.6</td>
    <td>14.3</td>
  </tr>
  <tr>
    <td>Cuisine & Festivals（饮食与节庆）</td>
    <td>14</td>
    <td>11.1</td>
    <td>20.4</td>
  </tr>
  <tr>
    <td>Film & Television（影视剧）</td>
    <td>8</td>
    <td>10.8</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>Sports（体育）</td>
    <td>23</td>
    <td>12.6</td>
    <td>22.8</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>18</td>
    <td>12.0</td>
    <td>17.2</td>
  </tr>
  <tr>
    <td>Painting（绘画）</td>
    <td>38</td>
    <td>11.9</td>
    <td>16.0</td>
  </tr>
  <tr>
    <td>Literature & Mythology（文学与神话）</td>
    <td>11</td>
    <td>11.8</td>
    <td>20.1</td>
  </tr>
  <tr>
    <td>Music（音乐）</td>
    <td>17</td>
    <td>9.3</td>
    <td>15.8</td>
  </tr>
  <tr>
    <td>Handicrafts（手工艺）</td>
    <td>7</td>
    <td>13.3</td>
    <td>19.9</td>
  </tr>
  <tr>
    <td rowspan="6">Humanities & Society</td>
    <td>Classical & Medieval History（古典与中世纪史）</td>
    <td>14</td>
    <td>10.6</td>
    <td>17.6</td>
  </tr>
  <tr>
    <td>Contemporary History（近代史）</td>
    <td>25</td>
    <td>10.0</td>
    <td>17.4</td>
  </tr>
  <tr>
    <td>Modern History（现代史）</td>
    <td>20</td>
    <td>12.1</td>
    <td>19.5</td>
  </tr>
  <tr>
    <td>Religious Studies（宗教学）</td>
    <td>10</td>
    <td>12.8</td>
    <td>20.3</td>
  </tr>
  <tr>
    <td>Politics & Economics（政治学与经济学）</td>
    <td>15</td>
    <td>11.6</td>
    <td>18.1</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>9</td>
    <td>14.0</td>
    <td>20.9</td>
  </tr>
  <tr>
    <td>Ancient History（历史学）</td>
    <td>15</td>
    <td>12.5</td>
    <td>21.8</td>
  </tr>
  <tr>
    <td rowspan="5">Sciences</td>
    <td>Biology（生物学）</td>
    <td>26</td>
    <td>10.0</td>
    <td>14.1</td>
  </tr>
  <tr>
    <td>Physics（物理学）</td>
    <td>9</td>
    <td>13.3</td>
    <td>20.9</td>
  </tr>
  <tr>
    <td>Astronomy（天文学）</td>
    <td>39</td>
    <td>11.7</td>
    <td>16.7</td>
  </tr>
  <tr>
    <td>Medicine & Chemistry（医学与化学）</td>
    <td>13</td>
    <td>12.1</td>
    <td>19.7</td>
  </tr>
  <tr>
    <td>Mathematics（数学）</td>
    <td>5</td>
    <td>12.6</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td rowspan="6">Ancient Architecture</td>
    <td>Residences & Palaces（住宅与宫殿建筑）</td>
    <td>13</td>
    <td>11.7</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td>Funerary and Memorial Buildings（墓葬与遗址）</td>
    <td>13</td>
    <td>11.4</td>
    <td>16.8</td>
  </tr>
  <tr>
    <td>Public Cultural Buildings（公共文化建筑）</td>
    <td>11</td>
    <td>10.6</td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>Transportation Buildings（交通建筑）</td>
    <td>7</td>
    <td>10.4</td>
    <td>14.9</td>
  </tr>
  <tr>
    <td>Religious Buildings（宗教建筑）</td>
    <td>38</td>
    <td>10.9</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>Other（其他）</td>
    <td>25</td>
    <td>10.7</td>
    <td>16.2</td>
  </tr>
  <tr>
    <td rowspan="4">Engineering</td>
    <td>Military, Materials, Energy and ChemE.（军事、材料、能源、化工）</td>
    <td>11</td>
    <td>11.4</td>
    <td>19.7</td>
  </tr>
  <tr>
    <td>Mechanical Engineering（机械）</td>
    <td>30</td>
    <td>12.8</td>
    <td>17.4</td>
  </tr>
  <tr>
    <td>Aerospace（航空航天）</td>
    <td>30</td>
    <td>12.2</td>
    <td>18.1</td>
  </tr>
  <tr>
    <td>Electronic Information（电子信息）</td>
    <td>8</td>
    <td>10.6</td>
    <td>19.3</td>
  </tr>
</table>
<table>
  <tr>
    <th>Topic</th>
    <th>Image</th>
    <th>Recognition Q&amp;A</th>
    <th>Final Q&amp;A</th>
  </tr>
  <tr>
    <td>自然 (Nature)</td>
    <td>![图片中所示的蓝色动物是什么？](page_184_232_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中所示的蓝色动物是什么?<br>
      <b>A:</b> 蓝海星<br>
      <b>Q:</b> What is the blue animal shown in the picture?<br>
      <b>A:</b> Linckia laevigata
    </td>
    <td>
      <b>Q:</b> 图片中所示的蓝色动物属于哪个门?<br>
      <b>A:</b> 棘皮动物门<br>
      <b>Q:</b> To which phylum does the blue animal shown in the picture belong?<br>
      <b>A:</b> Echinodermata
    </td>
  </tr>
  <tr>
    <td>科学 (Sciences)</td>
    <td>![图片中展示的几何形状是什么？](page_184_352_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中展示的几何形状是什么?<br>
      <b>A:</b> 圆锥抛物面<br>
      <b>Q:</b> What geometric shape is shown in the picture?<br>
      <b>A:</b> Circular parabola
    </td>
    <td>
      <b>Q:</b> 图片中展示的几何形状的方程是什么?<br>
      <b>A:</b> \( z = -a(x^2 + y^2),\ a > 0 \)<br>
      <b>Q:</b> What is the equation of the geometric shape shown in the picture?<br>
      <b>A:</b> \( z = -a(x^2 + y^2),\ a > 0 \)
    </td>
  </tr>
  <tr>
    <td>地理与气象 (Geography & Meteorological)</td>
    <td>![图片中展示的自然灾害是什么？](page_184_472_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中展示的自然灾害是什么?<br>
      <b>A:</b> 龙卷风<br>
      <b>Q:</b> What natural disaster is shown in the picture?<br>
      <b>A:</b> Tornado
    </td>
    <td>
      <b>Q:</b> 图片中展示的自然灾害的强度破坏划分来源于哪个标准?<br>
      <b>A:</b> 藤田级数<br>
      <b>Q:</b> Which standard is used to classify the intensity and damage of natural disasters shown in the picture?<br>
      <b>A:</b> Fujita Scale
    </td>
  </tr>
  <tr>
    <td>现代建筑 (Modern Architecture)</td>
    <td>![图片中的建筑是什么？](page_184_592_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中的建筑是什么?<br>
      <b>A:</b> 慕尼黑奥林匹克体育场<br>
      <b>Q:</b> What is the building in the picture?<br>
      <b>A:</b> Munich Olympic Stadium
    </td>
    <td>
      <b>Q:</b> 图片中的建筑曾举办哪一年的世界杯足球赛决赛?<br>
      <b>A:</b> 1974<br>
      <b>Q:</b> In which year did the building in the picture host the World Cup final?<br>
      <b>A:</b> 1974
    </td>
  </tr>
  <tr>
    <td>古代建筑 (Ancient Architecture)</td>
    <td>![图片中的这座寺庙叫什么名字？](page_184_712_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中的这座寺庙叫什么名字?<br>
      <b>A:</b> 空中宫殿<br>
      <b>Q:</b> What is the name of the temple in the picture?<br>
      <b>A:</b> Phimeanakas
    </td>
    <td>
      <b>Q:</b> 图片中的这座寺庙位于柬埔寨的哪个古城中?<br>
      <b>A:</b> 吴哥窟古城<br>
      <b>Q:</b> In which ancient city in Cambodia is this temple located?<br>
      <b>A:</b> Angkor Wat
    </td>
  </tr>
  <tr>
    <td>生活、文化与艺术 (Life Culture & Art)</td>
    <td>![这幅画作的名称是什么？](page_184_832_267_120.png)</td>
    <td>
      <b>Q:</b> 这幅画作的名称是什么?<br>
      <b>A:</b> 《母性》<br>
      <b>Q:</b> What is the building in the picture?<br>
      <b>A:</b> Motherhood
    </td>
    <td>
      <b>Q:</b> 这幅画作收藏于哪个博物馆?<br>
      <b>A:</b> 克拉科夫国家博物馆<br>
      <b>Q:</b> Which museum houses this painting?<br>
      <b>A:</b> National Museum in Krakow
    </td>
  </tr>
  <tr>
    <td>工程技术 (Engineering)</td>
    <td>![图片中展示的机器是什么？](page_184_952_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中展示的机器是什么?<br>
      <b>A:</b> 电话机<br>
      <b>Q:</b> What is the machine shown in the picture?<br>
      <b>A:</b> Telephone
    </td>
    <td>
      <b>Q:</b> 第一个申请图片中展示的机器专利的人是谁?<br>
      <b>A:</b> 亚历山大·格拉汉姆·贝尔<br>
      <b>Q:</b> Who was the first person to patent the machine shown in the picture?<br>
      <b>A:</b> Alexander Graham Bell
    </td>
  </tr>
  <tr>
    <td>人文社会 (Humanities & Society)</td>
    <td>![图片中的是什么雕像？](page_184_1072_267_120.png)</td>
    <td>
      <b>Q:</b> 图片中的是什么雕像?<br>
      <b>A:</b> 希湿奴<br>
      <b>Q:</b> What statue is in the picture?<br>
      <b>A:</b> Vishnu
    </td>
    <td>
      <b>Q:</b> 图片中的雕像属于哪个宗教?<br>
      <b>A:</b> 印度教<br>
      <b>Q:</b> To which religion does the state in the picture belong?<br>
      <b>A:</b> Hinduism
    </td>
  </tr>
</table>

Figure 13: Examples of ChineseSimpleVQA on 8 topics.
Q: 图片中的建筑物于何时正式启用?
Ref A: 2007年4月26日

O1-preview-0912: ✗
图片中的建筑物是香港的中环9号码头（Central Pier No.9）。该号码头于**2007年8月20日**正式启用，作为连接香港岛与离岛及其他地区的渡轮服务设施之一。

GPT-4o-0806: ✓
根据提供的图片链接，图片中的建筑物是香港中环码头的第9号码头。中环码头新址的各号码头分别在2007年启用。因此，第9号码头也是在2007年正式启用的。

Qwen2.5-VL-72B: ✗
图片中的建筑物是香港的中环码头，位于维多利亚港畔。中环码头正式启用的时间是1981年。

Deepseek-VL2: ✗
图片中展示的是香港的尖沙咀钟楼，正式启用日期是1998年。这座建筑是香港文化中心的一部分，以其独特的建筑风格和地理位置而闻名。钟楼不仅是一个地标性建筑，还经常举办各种文化活动，是游客和当地居民的热门去处。

Doubao1.5-vision-pro: ✗
图中近处的建筑物是香港尖沙咀九号码头。该码头于2013年3月31日正式启用。\n\n香港尖沙咀九号码头是维多利亚港的一个重要码头，为乘客提供了前往香港不同区域的渡轮服务。码头的启用进一步完善了香港的水上交通网络，方便了市民和游客的出行。

Q: 图片中的汽车首次亮相于哪个车展?
Ref A: 法兰克福车展

O1-preview-0912: ✗
图片中的汽车是奥迪e-tron。这款车于2018年在日内瓦国际汽车展（Geneva Motor Show）上首次亮相。

GPT-4o-0806: ✗
图片中显示的是奥迪e-tron。奥迪e-tron首次亮相于2018年的日内瓦车展。

Qwen2.5-VL-72B: ✗
图片中的汽车是奥迪e-tron quattro概念车，它首次亮相于2010年的巴黎车展。这款概念车展示了奥迪在电动化和未来设计方面的愿景，具有独特的外观和先进的技术配置。

Deepseek-VL2: ✗
图片中的汽车是奥迪e8 e-tron概念车，首次亮相于2012年的日内瓦国际汽车展。这款车展示了奥迪在电动和混合动力技术方面的创新，是奥迪在电动化领域的重要一步。

Doubao1.5-vision-pro: ✓
图中汽车是奥迪e-tron概念车，它于2009年法兰克福车展首次亮相。\n\n这款标志着奥迪在电动汽车领域的重要探索，展示了奥迪在电动驱动技术和未来汽车设计方面的理念。法兰克福车展作为全球最重要的汽车展会之一，为奥迪提供了一个绝佳的平台来展示其创新技术和未来愿景。

Figure 14: Model response examples of ChineseSimpleVQA: part 1.
Q: 图片中的湿壁画是哪位艺术家的作品？
Ref A: 保罗·查勒（Paul Troger）

![A painting of a religious scene](page_184_320_579_320.png)

O1-preview-0912: ✗
图片中的湿壁画是由巴托洛梅奥·阿尔托蒙特（Bartolomeo Altomonte）创作的。他是一位意大利裔奥地利画家，以其在巴洛克时期的湿壁画作品著称。这幅湿壁画位于奥地利赛滕施泰因修道院（Stift Seitenstetten）的大理石大厅（Marmorsaal）天花板上，展示了他精湛的艺术技巧和丰富的想象力。

GPT-4o-0806: ✓
这幅湿壁画位于Stift Seitenstetten修道院的Marmorsaal（大理石大厅）中，由巴洛克时期艺术家保罗·特洛格尔（Paul Troger）创作。

Qwen2.5-VL-72B: ✗
这幅湿壁画是德国巴洛克画家约翰·巴普蒂斯特·齐默尔曼（Johann Baptist Zimmermann）的作品。这幅作品位于德国维尔茨堡的维尔茨堡主教宫（Würzburg Residence）的楼梯间天花板上，是齐默尔曼最著名的作品之一。这幅湿壁画描绘了众多天使和神话人物，展现了巴洛克艺术的华丽和动感。

Deepseek-VL2: ✗
这幅湿壁画是意大利巴洛克时期著名艺术家天花板的杰作，通常 attributed to 皮埃特罗·达·科尔托纳（Pietro da Cortona）。他以其宏伟的湿壁画和巴洛克风格的装饰而闻名，这幅作品可能位于意大利的某个教堂或宫殿。

Doubao1.5-vision-pro: ✗
图中的湿壁画是由**乔瓦尼·巴蒂斯塔·提埃坡罗**创作的《圣·克莱门特的荣耀》。这幅作品创作于1739年，位于意大利威尼斯圣洛克大会堂的天顶。画面通过生动的色彩和动态的构图，展现了宗教题材的庄严与神圣。提埃坡罗以其精湛的技艺和富有想象力的场景设计，在艺术史上留下了深刻的印记。

Q: 图片中的动物属于哪一目？
Ref A: 食肉目（Carnivora）

![A seal on a beach](page_1012_320_579_320.png)

O1-preview-0912: ✓
图片中的动物是**海狮**，属于**食肉目**。

GPT-4o-0806: ✗
这张图片中的动物是加拉帕戈斯海狮，它属于鳍足目（Pinnipedia）。

Qwen2.5-VL-72B: ✓
图片中的动物是一只海狮，海狮属于哺乳纲（Mammalia）食肉目（Carnivora）鳍足亚目（Pinnipedia）。

Deepseek-VL2: ✗
图片中的动物是一只海狮，属于哺乳纲（Mammalia）。海狮是海洋哺乳动物，通常生活在沿海地区，以鱼类为食。它们具有流线型的身体和强壮的鳍状肢，适应于游泳和潜水。海狮的社交行为非常有趣，它们常常在岩石或沙滩上聚集，形成群体生活。

Doubao1.5-vision-pro: ✓
图中的动物是海狮，属于**食肉目**。\n\n海狮是食肉目海狮科的哺乳动物，主要生活在海洋中，以鱼类和头足类动物为食。
Q: 图片中的建筑是什么？
Ref A: 巴哈伊世界中心国际档案馆

O1-preview-0912: ×
图片中的建筑是位于法国布列塔尼大区伊勒-维莱讷省圣马洛市的**“法国国家档案馆”**（Archives nationales）。这座建筑临近港口，是圣马洛的著名地标之一。

GPT-4o-0806: ×
图片中的建筑是法国马赛的《马赛档案馆》。这是马赛的一个重要的文化和历史建筑，位于老港附近，是城市的重点地标之一。

Qwen2.5-VL-72B: ×
图片中的建筑是巴哈伊世界中心的一部分，位于以色列的海法市。这座建筑以其宏伟的白色大理石柱和金色的圆顶而闻名，是巴哈伊信仰的重要圣地。建筑周围的花园和景观也十分美丽，吸引了许多游客和信徒前来参观。

Deepseek-VL2: √
图片中的建筑是巴哈伊世界中心，位于以色列的海法。巴哈伊世界中心是巴哈伊信仰的圣地，由巴哈伊信仰的创始人巴哈欧拉设计，于1953年开始建造，1963年完工。建筑风格融合了古典和现代元素，具有独特的圆顶和柱廊设计。

Doubao1.5-vision-pro: √
图中建筑是**“国际档案馆”**，位于以色列海法的卡梅尔山。它是巴哈伊教的一座重要建筑，用于保存巴哈伊教的重要文献和档案。建筑风格融合了古典元素，正面有一排宏伟的石柱，顶部为绿色的屋顶，整体显得庄严肃穆。周围的景观设计精美，有繁茂的绿植和鲜艳的花卉，背景中还能看到蓝色的大海和远处的船只，营造出一种宁静而神圣的氛围。

Q: 图片中的鸟类的学名是什么？
Ref A: 峭壁霸鹟 (Hirundinea ferruginea)

O1-preview-0912: ×
图片中的鸟类叫做**“鸣鹭”**（学名：*Syrigma sibilatrix*）。鸣鹭是一种中型涉禽，属于鹭科。它们主要分布在南美洲的开阔草原、湿地和沼泽地区，以优雅的外形和独特的鸣叫声而闻名。

GPT-4o-0806: ×
图片中的鸟类是美洲鹭（Ciconia maguari）

Qwen2.5-VL-72B: ×
图片中的鸟类是兀鹫，学名为Gyps fulvus。

Deepseek-VL2: ×
图片中的鸟类看起来像是一只红尾伯劳（Lanius cristatus）。红尾伯劳是一种小型食虫鸟类，以其鲜艳的红色尾巴和独特的捕食行为而闻名。它们通常栖息在开阔的草原、农田和灌木丛中，以昆虫为食。红尾伯劳的学名是Lanius cristatus，属于伯劳科（Laniidae）。

Doubao1.5-vision-pro: ×
图中的鸟类是褐头牛鹂，学名*Molothrus ater*。它属于拟鹂科牛鹂属，是一种广泛分布于北美洲的鸟类。褐头牛鹂以其独特的繁殖习性著称，它们常常将卵产在其他鸟类的巢穴中，让其他鸟类代为孵化和养育幼鸟。这种行为被称为巢寄生。褐头牛鹂的羽毛主要呈棕色和黑色，雄性和雌性在外观上有一定的差异。雄性通常有更鲜艳的羽毛，而雌性则较为暗淡。它们的栖息地包括开阔的田野、草地和森林边缘等。

Figure 16: Model response examples of ChineseSimpleVQA: part 3.
Now, I will provide you with an image along with related documents and supplementary knowledge. Please generate an associated image-based question and its corresponding standard answer based on the image. The generated question should be one that can be asked about the image, and its answer must strictly be an entity that appears both in the image and within the related documents or supplementary knowledge.

The generated question must adhere to the following requirements:

1. The question must have an answer that is an entity present in the image itself, requiring no association with any knowledge outside of what is depicted in the image to answer. For example, avoid asking questions like "In which country is the canal shown in the picture located?" because the answer is not an entity found in the image. Also, avoid asking "Who designed the poster in the picture?" since answering this requires information beyond what's visible in the image. Similarly, refrain from asking "What is the main component of the crystal in the picture?" as the answer cannot be determined from the image alone.
2. The question posed must have exactly one clear and uncontested entity as its answer, and there should be no form of ambiguity or vagueness in the question's formulation. For instance, avoid posing questions like "What is in the picture?" as it is unclear what specific item is being referred to, given there may be multiple entities in the image.
3. Avoid asking two questions at once; the question should consist of one specific inquiry only and should not be divisible into two separate questions, such as "What are the objects in the top left corner and at the bottom of the picture?" This is inappropriate because it essentially asks two questions simultaneously, leading to two answers.
4. If the answer to the question is in English, provide the Chinese translation of the answer followed by the English name in parentheses, formatted as: Jacob Fugger (Jakob Fugger).
5. Return the result in Simplified Chinese, not Traditional Chinese.
Please return the generated question and answer in the following format:
{"question": "Generated Question Here", "answer": "Answer Here"}

(***Examples***)**
Let's begin!

Figure 17: The prompt for generating the object recognition question-answer pairs in stage 1.
Now I will provide you with a question-answer pair along with related documents and supplementary knowledge. Based on this information, you need to generate a new factual question and its corresponding standard answer. The generated question must start with the answer provided in the given question-answer pair, and the answer should be derived from the provided documents and supplementary knowledge.

The generated question must adhere to the following requirements:

1. The question must relate to objective facts about the world, such as asking "Which movie did Robert De Niro star in during 1974?" Avoid constructing subjective questions involving personal opinions or feelings, like "What do you think about xxx?"
2. The question must have exactly one clear and uncontested entity as its answer, and there should be no ambiguity or vagueness in the formulation of the question. For example, avoid asking "Which movie did Robert De Niro star in?" because it is unclear which specific movie and year are being referred to; "Which work is Zhou Ruchang best known for?" is also an inappropriate question because "best known" can be debatable. Also, avoid asking "What is the typical length of an American crocodile in inches?" because the answer is usually a range and thus vague. Similarly, avoid asking "In which regions are night herons primarily distributed?" because there are usually multiple answers, making it unclear. Also, do not ask "From where to where does the locomotive run?" because trains can travel bidirectionally, leading to potentially two answers.
3. The answer to the question should be timeless and not subject to change over time. For instance, "Who is the President of the United States?" is not a suitable question because the identity of the president changes over time.
4. Avoid asking two questions at once; the question should consist of one specific inquiry only and should not be divisible into two separate questions, such as "Who were the architects of St. Peter's Basilica and the square separated by it?" is inappropriate because it essentially asks two questions simultaneously, leading to two answers.
5. The question should have a certain level of difficulty to present a challenge. For example, "The film Stripease was adapted from a novel of the same name. Who is the author of that novel?"
6. Do not provide extra information when posing the question. Ensure that the question provides as little information as possible while still allowing for a definitive answer. Absolutely do not pose the question like this: "In which country is Desanka railway station located in Vinnitsia Oblast?" Since 'Vinnitsia Oblast' provides additional information. The question should instead be "In which country is Desanka railway station located?"
7. If the answer to the question is in English, provide the Chinese translation of the answer followed by the English name in parentheses, formatted as: Jacob Fugger (Jakob Fugger).
8. Return the result in Simplified Chinese, not Traditional Chinese.
Please return the generated question and answer in the following format:
{"question": "Generated Question Here", "answer": "Answer Here"}

(***Example***)

Let's begin!

Figure 18: The prompt for generating the knowledge-based question-answer pairs in stage 2.
现在我给你提供两个问答对，他们是递进的关系，第二个问答对是根据第一个问答对得到的。你需要修改并合成这两个问答对的问题来生成一个新的流畅的事实性问题，以第一个问题的一部分开头，以第二个问题的一部分结尾。生成的问题必须不能出现第一个问答对的答案，能以第二个问答对的答案作为回答。

生成的问题必须满足以下要求：
1. 生成的问题必须严格和第一个问答对的答案相关，需要达到得不到第一个问答对的答案就无法正确回答问题的程度。例如可以询问“图片中的湖泊位于土耳其的哪个省？”而不能够询问“图片中的湖泊所在的七湖国家公园位于土耳其的哪个省？”因为这个问题不需要知道第一个问答对的答案就能够回答。
2. 确保新生成的问题流畅而简洁。

请按照下面的格式返回生成的问题和答案：
{"question": "此处为生成问题", "answer": "此处为答案"}

####以下是一些示例####

（**示例**）
####

让我们开始吧！

Now, I will provide you with two question-answer pairs that are progressively related; the second pair is derived based on the information from the first pair. You need to modify and combine the questions from these two pairs to generate a new, coherent factual question. The new question should start with part of the first question and end with part of the second question. The generated question must not include the answer from the first question-answer pair but should be answerable using the answer from the second question-answer pair.

The generated question must adhere to the following requirements:

The generated question must be strictly related to the answer of the first question-answer pair, to the extent that it cannot be correctly answered without knowing the answer from the first pair. For example, you may ask "In which province of Turkey is the lake in the picture located?" but not "In which province of Turkey is the Seven Lakes National Park, where the lake in the picture is located?" because the latter can be answered without knowing the answer from the first question-answer pair.
Ensure that the newly generated question is fluent and concise.
Please return the generated question and answer in the following format:
{"question": "Generated Question Here", "answer": "Answer Here"}

（**Example**）
Let's begin!

Figure 19: The prompt for merging the object recognition question-answer pairs and the knowledge-based question-answer pairs in stage 3.
你是一个数据质量检查员，现在需要你检查下面根据图片生成的问题是否满足以下要求：
1. 生成的问题必须严格和图片相关，需要达到没有图片就无法正确回答问题的程度。例如可以询问“图中的动物习惯生活在什么环境中？”而不能够询问“鸵鸟习惯生活在什么环境中？”，因为这个问题不需要看图片就能够回答。
2. 生成的问题不能够仅仅依靠图片就能够回答。例如可以询问“图中展现的蝴蝶属于哪个科？”但是不能询问“图中展现的蝴蝶翅膀是什么颜色？”因为这个问题仅仅依靠图片就能够回答，我们要让问题需要结合外部知识才能回答。
3. 生成的问题必须关联到客观世界的知识，例如可以询问“图片中的人物在1974年出演了哪一部电影？”不得构造涉及个人观点或感受相关的主观问题，如“你如何看待xxx？”。
4. 所提出的问题必须且只有一个明确且无争议的实体作为答案。且问题表述中不应存在任何形式的模糊性或歧义。例如，避免提问“图片中的人物出演了哪一部电影”因为无法确定是指哪一年出演的哪一部电影；同样不要问“图片中的人类身体的特点是什么？”因为这个问题过于模糊，没有明确的答案。“图中作家最为人熟知的作品是哪个？”也是不严格问题，因为“最熟知”可能是有争议的；也不要问“图中的这种动物通常长度是多少英寸？”因为这个问题的答案通常是-一个范围，比较模糊。同样不要提问“图中的鸟类主要分布在哪些区域？”因为这个问题通常会有多个答案，不清楚。还要提问“图中的机车是从哪里往哪里的？”因为机车可能是双向行驶的，答案可能有两个。
5. 问题的答案应当是时间不变的，不会随着时间的推移而改变。例如，“图片中这个国家的总统是谁？”就不是一个合适的问题，因为总统身份会随时间改变。
6. 避免同时询问两个问题。所提出的问题应该且仅有一个具体的问题，不能够被拆分成两个问题，比如“图片中这座大殿的建筑和广场设计者分别是谁。”不合适，因为这个问题相当于同时问了两个问题，也会有两个答案。
如果问题不合格则解释并输出“【否】”，如果问题合格则直接输出“【是】”

#### 以下是示例：

#### 如果问题不合格则输出原因并最后输出“【否】”，如果问题合格则直接输出“【是】”，注意如果认为问题不合格需要输出原因
（**示例**）
让我们开始吧！

You are a data quality inspector. Now, you need to inspect whether the questions generated based on images meet the following criteria:

1. The generated question must be strictly related to the image, such that it cannot be correctly answered without the image. For example, you may ask "In what environment does the animal in the picture typically live?" but not "In what environment does the ostrich typically live?" because the latter can be answered without looking at the image.
2. The generated question should not be answerable solely based on the image. For instance, you may ask "To which family does the butterfly shown in the picture belong?" but not "What color are the wings of the butterfly shown in the picture?" because the latter can be answered just by looking at the image. We want questions that require external knowledge.
3. The generated question must relate to objective facts about the world. For example, you may ask "Which movie did the person in the picture star in during 1974?" but avoid constructing subjective questions involving personal opinions or feelings, such as "What do you think about xxx?"
4. The question must have exactly one clear and uncontested entity as its answer, and there should be no ambiguity or vagueness in the formulation of the question. For example, avoid asking "Which movie did the person in the picture star in?" because it is unclear which specific movie and year are being referred to; similarly, avoid asking "What are the characteristics of people like the one shown in the picture?" because this is too vague and lacks a clear answer. Questions like "Which work is the writer in the picture best known for?" are also inappropriate because "best known" can be debatable. Also, avoid asking "What is the typical length of the animal shown in the picture in inches?" because the answer is usually a range and thus vague. Similarly, avoid asking "In which regions are the birds shown in the picture primarily distributed?" because there are usually multiple answers, making it unclear. Also, do not ask "From where to where does the locomotive in the picture run?" because trains can travel bidirectionally, leading to potentially two answers.
5. The answer to the question should be timeless and not subject to change over time. For instance, "Who is the president of the country shown in the picture?" is not a suitable question because the identity of the president changes over time.
6. Avoid asking two questions at once; the question should consist of one specific inquiry only and should not be divisible into two separate questions, such as "Who were the architects of the building and the square shown in the picture separately?" This is inappropriate because it essentially asks two questions simultaneously, leading to two answers.
If the question does not meet the criteria, explain why and output "【否】". If the question meets all the criteria, directly output "【是】"

Here are some examples:
If the question does not meet the criteria, provide the reason and then output "【否】". If the question meets all the criteria, directly output "【是】". Note that if you believe the question does not meet the criteria, you must provide the reason.

（**Example**）
Let's begin!
You are an expert in evaluating the correctness of image-based question-answer pairs. Your task is to assess whether the model's predicted answer is correct based on the given question, image, standard answer, and the model's predicted answer. You should classify the result as: [Correct], [Incorrect], or [No Attempt].
First, we will list examples for each evaluation category, then you will evaluate new predictions accordingly.
Here are examples of [Correct] responses:
(**Example**)
These responses are classified as [Correct] because:
- They fully include the key information from the standard answer.
- They do not contain any information that contradicts the standard answer.
- Focus solely on semantic content; variations in language (English vs. Chinese), capitalization, punctuation, grammar, and order are not considered.
- It is acceptable for responses to include vague statements or guesses, provided they contain the standard answer and do not introduce incorrect information or contradictions.
Here are examples of [Incorrect] responses:
(**Example**)
These responses are classified as [Incorrect] because:
- They contain factual statements that contradict the standard answer. Even if the statement includes qualifiers (e.g., "might be", "although I'm not sure, I think"), it is still considered incorrect.
Here are examples of [No Attempt] responses:
(**Example**)
These responses are classified as [No Attempt] because:
- They do not include key information from the standard answer.
- There are no contradictory statements in the response.
- Here is a new example question. Please respond with only one letter: "A", "B", or "C" without apologies or corrections, just the evaluation of the response.
Evaluate the predicted answer for this new question as one of the following:
A: [Correct]
B: [Incorrect]
C: [No Attempt]

Return only the letter "A", "B", or "C" without additional text.

Figure 21: The prompt for evaluation.

Reviewer paper summaries: The paper presents ChineseSimpleVQA, the first factuality-based visual question-answering benchmark in Chinese, designed to evaluate the factuality of LVLMs across various topics. Key contributions include a rigorous data construction pipeline, a strategy for decoupling visual factuality into object recognition and knowledge discovery, and a comprehensive evaluation of 34 LVLMs. Overall, this work makes meaningful contributions. However, there is still room for improvement in the evaluation methodology and the depth of insights drawn from the assessment.

The paper introduced a novel visual question-answering benchmark ChineseSimpleVQA, which designed to evaluate the factual accuracy of Large Vision Language Models (LVLMs) in Chinese. The benchmark focuses on multi-hop questions that decouple visual recognition ("seeing the world") from knowledge discovery ("discovering knowledge") and cover 8 major topics and 56 subtopics.
Summary of strengths: + Given the rapid development of LVLMs, establishing more comprehensive and challenging multimodal factuality benchmarks is crucial. ChineseSimpleVQA is a meaningful contribution, particularly for the Chinese domain.

+ The authors design a rigorous data construction pipeline, integrating large-model-assisted automatic data generation with human verification, which can serve as a reference for future benchmark development.

+ The paper includes extensive experiments and analyzes several key observations on model performance.

The author propose a benchmark to evaluate models’ knowledge in Chinsese.
The benchmark support 34 state-of-the-art models (13 closed-source, 21 open-source) provides actionable insights.
Summary of weaknesses: - It is not clear how the evaluation of factuality in object recognition and knowledge discovery is effectively decoupled. The authors evaluate model performance on both the visual recognition QA and the Merged QA. Intuitively, the model’s performance on visual recognition QA reflects factuality in object recognition, while the performance on the Merged QA reflects overall factuality. However, it remains unclear how the factuality in knowledge discovery is specifically evaluated. I wonder why the authors did not include an evaluation of model performance on text-only knowledge-based QA.

- It would be helpful if the authors provided comprehensive comparisons between different models in object recognition and knowledge discovery, respectively. This would allow readers to better understand the capabilities of modality-specific components of existing LVLMs.

- This work focuses on questions that specify which object needs to be recognized, which simplifies the annotation of the visual recognition QA. However, some multimodal factuality questions do not specify which object needs to be recognized. For example, given a picture of a place with a landmark building in one part of the image, the question might be: "Which country does this place belong to?"

1. As the dataset is a major contribution of this paper. However, authors do not release the correspond datasets and evaluation, which is one of a main problem of this paper.
2. Model evaluation uses only large language models for scoring, lacks manual calibration, and obtains experimental results that appear to be correct.
3. Lack of comparison between the dataset and other datasets, and more specifically whether models that perform well in other datasets continue to perform well in this dataset.
Comments, suggestions and typos: - The authors remain the samples that four leading LVLMs can not answer to increase the difficulty of the benchmark. However, this method will waste a lot of samples. It would be beneficial to explore rules to help annotate hard samples.

- In Table 1, the authors categorize the metric into Accuracy and LLM-as-a-Judge. However, I think these two are concepts of different dimensions.

See the second and third points of Weakness, where the essay lacks necessary components.
Overall assessment scores: 3

3