CDTP: A Large-Scale Chinese Data-Text Pair Dataset for Comprehensive Evaluation of Chinese LLMs

## Abstract

Abstract Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks. However, Chinese LLMs face unique challenges, primarily due to the dominance of unstructured free text and the lack of structured representations in Chinese corpora. While existing benchmarks for LLMs partially assess Chinese LLMs, they are still predominantly English-centric and fail to address the unique linguistic characteristics of Chinese, lacking structured datasets essential for robust evaluation. To address these challenges, we present a C omprehensive B enchmark for E valuating C hinese L arge L anguage M odels (CB-ECLLM) based on the newly constructed Chinese Data-Text Pair (CDTP) dataset. Specifically, CDTP comprises over 7 million aligned text pairs, each consisting of unstructured text coupled with one or more corresponding triples, alongside a total of 15 million triples spanning four critical domains. The core contributions of CDTP are threefold: (i) enriching Chinese corpora with high-quality structured information; (ii) enabling fine-grained evaluation tailored to knowledge-driven tasks; and (iii) supporting multi-task fine-tuning to assess generalization and robustness across scenarios, including Knowledge Graph Completion, Triple-to-Text generation, and Question Answering. Furthermore, we conduct rigorous evaluations through extensive experiments and ablation studies to assess the effectiveness, Supervised Fine-Tuning (SFT), and robustness of the benchmark. To support reproducible research, we offer an open-source codebase and outline potential directions for future investigations based on our insights.

## 1 Introduction

Large Language Models (LLMs) (Pan et al., 2024; Chang et al., 2024) have demonstrated exceptional performance across a broad range of Natural Language Processing (NLP) tasks (Wang et al., 2024a; Yan et al., 2025; 2024), positioning themselves as pivotal milestones on the path toward Artificial General Intelligence (AGI) (Pei et al., 2019; Fei et al., 2022).
However, achieving AGI requires mastering not only proficiency in widely studied languages like English, but also mastery of complex linguistic systems such as Chinese, which present unique and substantial challenges.

Chinese poses distinctive challenges for LLMs, largely because its corpora are dominated by unstructured free text and lack explicit structured annotations such as entity–attribute–value triples, which are essential for learning reliable entity–relation mappings (Liu et al., 2022; Zhu et al., 2022). Table [1](https://arxiv.org/html/2510.06039v1#S1.T1) illustrates this issue: a single Chinese expression can yield multiple plausible interpretations depending on context. This absence of structured signals constrains performance on a wide range of knowledge-intensive tasks, including knowledge retrieval (Li et al., 2025), relation inference (Wang et al., 2024b), and text generation (Li et al., 2024a). In these settings, models are required to extract or memorize facts from lengthy unstructured text, a process that increases training difficulty, propagates errors, weakens generalization across domains, and ultimately reduces interpretability.

**Table 1: Examples illustrating polysemy and structural ambiguity in Chinese text. Each case shows one surface form with two possible interpretations.**
| Chinese Expression | Interpretation 1 | Interpretation 2 |
| --- | --- | --- |
| 行 | 行动 | 行业 |
|  | to do/go | profession/industry |
| 武汉市长江大桥 | 武汉市长/江大桥 | 武汉市/长江大桥 |
|  | the mayor of Wuhan named Jiang Daqiao | the Yangtze River Bridge in Wuhan |

To bridge this gap, we construct the Chinese Data-Text Pair (CDTP) dataset, which tightly aligns

Figure: Figure 1: The example of our proposed Chinese Data-Text Pair (CDTP) dataset.
Refer to caption: CDTP_DATA.png

structured triples with corresponding natural language text. CDTP is developed through rigorous collection and cleaning procedures to guarantee strict alignment, high-quality content, and broad domain coverage. In total, it comprises over $7$ million text samples and their corresponding $15$ million triples, spanning four major domains: History and Politics, Humanities and Society, Technology and Economics, and Nature and Environment. This large-scale, systematically curated resource provides a solid foundation for evaluating the interplay between structured knowledge and unstructured text, and its overall format is illustrated in Figure [1](https://arxiv.org/html/2510.06039v1#S1.F1).

Moreover, existing benchmarks for evaluating LLMs, such as CMMLU (Li et al., 2023), C-Eval (Huang et al., 2023), and SuperCLUE (Xu et al., 2023), are largely shaped by English-centric task design and fail to capture challenges unique to Chinese, including the scarcity of structured annotations and the prevalence of ambiguous expressions. Consequently, they provide only an incomplete assessment of Chinese LLMs in knowledge-intensive tasks requiring relational reasoning and factual accuracy. To overcome these limitations, we further present the Comprehensive Benchmark for Evaluating Chinese Large Language Models (CB-ECLLM), constructed on top of CDTP to provide systematic and knowledge-driven evaluation. CB-ECLLM targets three representative tasks, including Knowledge Graph Completion(KGC) (Wang et al., 2024a; 2023a), Triple-to-Text Generation(T2T) (Li et al., 2021), and Question Answer(QA) (He et al., 2024). Together, these tasks capture the interplay between structured knowledge and natural language, providing a more targeted evaluation of Chinese LLMs.

Building on these tasks, we conducted experiments to evaluate Chinese LLMs across three key dimensions: effectiveness, Supervised Fine-Tuning (SFT) (Dong et al., 2024; Lu et al., 2024), and robustness. For effectiveness, we evaluate the performance of $8$ Chinese LLMs across $3$ tasks on $4$ distinct sub-datasets. This evaluation assesses how well the Chinese LLMs generate accurate responses by integrating structured knowledge, and how they perform across different domains and task-specific challenges. Regarding SFT, we conduct further experiments to evaluate the impact of incorporating additional labeled data on the performance of Chinese LLMs. In terms of robustness, we assess the stability of the SFT Chinese LLMs by testing their performance on Out-Of-Distribution (OOD) data. This aims to explore how well LLMs can maintain accuracy and reliability when faced with data that deviates from the training distribution. To summarize, the primary contributions of this paper is three-fold:

- •
Comprehensive Benchmark. We introduce a comprehensive benchmark tailored to the unique linguistic and cultural complexities of Chinese, such as context-dependent disambiguation, morphological segmentation, and idiomatic expression comprehension, addressing a critical gap in existing evaluations;
- •
High-Quality Aligned Dataset. To the best of our knowledge, we have constructed the largest Chinese Data-to-Text Pair (CDTP) dataset, comprising 7 million meticulously aligned text pairs, each consisting of unstructured text paired with one or more corresponding triples, totaling 15 million triples across four major domains;
- •
Multi-Dimensional Evaluation. We systematically evaluate Chinese LLMs from multiple perspectives, including performance on diverse tasks and datasets, generalizability across domains and tasks, and robustness against Out-Of-Distribution (OOD) data.

## 2 Related Work

Comprehensive evaluation of LLMs typically encompasses three core dimensions: linguistic competence, factual accuracy, and domain-specific knowledge. Existing benchmarks can be broadly categorized based on their target language into Chinese-oriented and English-oriented benchmarks.

Chinese-oriented Benchmarks. Benchmarks such as CMMLU (Li et al., 2023) and C-Eval (Huang et al., 2023) provide large-scale multiple-choice evaluations covering various disciplines. SuperCLUE (Xu et al., 2023) extends this by including agent-centric and safety-sensitive tasks. More recently, GraphEval (Liu et al., 2024) proposes to evaluate the factuality of LLMs by comparing model outputs against a large-scale Chinese knowledge graph, avoiding expensive human annotations. Similarly, KG-SFT (Zhao et al., 2024) demonstrates that fine-tuning LLMs with structured knowledge graph explanations enhances their performance in QA tasks, especially in low-resource domains. Beyond evaluation, several efforts have explored knowledge graph construction in Chinese. For instance, TechGPT-2.0 (Wang et al., 2025) focuses on entity and relation extraction using LLMs in Chinese, targeting complex domains such as law and medicine. Another line of work explores the effect of fine-tuning vs. prompting strategies for transforming unstructured text into structured triples (Chen et al., 2024).

However, none of these works offer a large-scale, domain-diverse, and strictly aligned triple-text resource that can serve both as a benchmark and as supervised data for structured generation and reasoning in Chinese.

English-oriented Benchmarks. In English, datasets such as WebNLG (Gardent et al., 2017), KELM (Agarwal et al., 2021), and KGTEXT (Annasamy et al., 2023) provide aligned triple-text pairs for text generation, supporting evaluation of LLMs’ capabilities in knowledge-grounded generation.
Recent work, such as KG-LLM (Yao et al., 2024), proposes treating knowledge graph triples as sequences and uses LLMs for tasks like triple classification and relation prediction, demonstrating strong results even with mid-sized models. A recent survey by Khorashadizadeh et al. (2024) categorizes the evolving interplay between LLMs and KGs, covering tasks like KG question answering, validation, and descriptive text generation.

While these English-centric benchmarks and models offer valuable insights, they do not account for the linguistic and cultural challenges unique to Chinese, such as character-level ambiguity, idiomatic expressions, and implicit subjects. Moreover, no existing English benchmark provides Chinese-equivalent scale and alignment rigor tailored to LLM evaluation.

## 3 Chinese Data-to-Text Pair (CDTP) Dataset

The Chinese Data-to-Text Pair (CDTP) dataset is constructed through a systematic methodology consisting of two sequential phases: data collection and data processing. In the data collection phase, we compile a raw data corpus of approximately 27 million entries from various online encyclopedic sources, such as Baidu Baike and Sogou Baike. In the data processing phase, we refine this data through data cleaning and quality improvement to ensure consistency and factual accuracy. Furthermore, we adopt targeted strategies to increase data diversity and ensure balanced coverage across multiple domains. This section provides a detailed description of the CDTP dataset construction pipeline, including statistical analysis and category-level organization.

### 3.1 Data Collection

The data collection pipeline is designed to construct a large-scale KG-Text pair dataset from heterogeneous sources, ensuring broad coverage and high-quality alignment. The pipeline consists of two main stages: seed entry generation, followed by data crawling and expansion. This process yields approximately 30 million triples involving around 15 million unique entities.

Seed Entry Generation.
The data collection process begins with aggregating a set of seed entries from three sources: OwnThink (Li et al., 2024b), KgClue (Wang et al., 2023b), and WudaoCorpora (Yuan et al., 2021). This step yields a total of 56 million seed entries, which serve as the foundation for subsequent data crawling. The diversity of these sources ensures broad coverage across various domains and contexts.

#### Data Crawling and Expansion.

Baike websites typically offer concise textual descriptions and structured relation tables, such as infoboxes containing attribute-value pairs, for individual entries. These tables provide a rich source of high-quality relational information that can be directly converted into triples, making them particularly valuable for structured knowledge extraction. Leveraging these features, we develop a customized Baike crawler that systematically retrieves both textual descriptions and triples for each seed entry from a precompiled list. To ensure comprehensive coverage and semantic diversity, the crawler employs a depth-first search strategy: it initiates from the Baike page of a seed entry and recursively traverses hyperlinked entities, capturing a wide array of related concepts and relations. This recursive expansion enables the system to uncover latent semantic connections across entries, effectively transforming a relatively small seed set into a large-scale, rich, and diverse KG with extensive entity and relation coverage.

### 3.2 Data Processing

The data processing pipeline refines raw collections to ensure accuracy, consistency, and relevance, comprising two main stages: data cleaning and quality enhancement. In the cleaning stage, we remove noise and inconsistencies using string matching, position-based filtering, and rule-based deduplication, while the enhancement stage improves reliability through statistical filtering, expert validation, and external knowledge verification, ensuring precise alignment between structured triples and textual descriptions with high relational accuracy.

Data Cleaning. Given the encyclopedic nature of Baike-sourced text data, we specifically target two prevalent types of noise in our cleaning process: (1) redundant triples—relation assertions unsupported by textual evidence, and (2) redundant descriptions—repetitive or irrelevant content that deviates from the corresponding structured knowledge, thus ensuring better alignment between text and triples.

Redundant Triples.
Redundant triples refer to triples that are not mentioned in the accompanying text

**Table 2: Illustrative examples of redundant triples (highlighted in red).**
|  | (90电影金曲精选，专辑语言，粤语) |
| --- | --- |
| Triple 1 | (Golden Songs in 90s Movies, album_language, Cantonese) |
|  | (90电影金曲精选，专辑歌手，许冠杰) |
| Triple 2 | (Golden Songs in 90s Movies, allbum_singer, Guanjie Xu) |
|  | (90电影金曲精选，曲日数量，12首) |
| Triple 3 | (Golden Songs in 90s Movies, number_of_songs, 12) |
|  | 《90电影金曲精选》是许冠杰发行的音乐专辑，共收录 |
|  | 了12首歌曲。 |
|  | Golden Songs in 90s Movies is a music album released by |
| Text | Guanjie Xu, which contains 12 songs. |

description. As illustrated in Table [2](https://arxiv.org/html/2510.06039v1#S3.T2), the triple “(
90电影金曲精选, 专辑语言, 粤语
) (Golden Songs in 90s Movies, album language, Cantonese)” conveys factual information about the album but is not explicitly reflected in the corresponding textual description, and is therefore treated as a redundant triple. To ensure alignment, we examine each data pair by checking whether the semantic content of each triple is supported by the accompanying text. Triples that lack textual grounding are removed. If all triples in a pair are filtered out, the entire data pair is discarded. This filtering procedure ensures strong semantic alignment between structured knowledge and natural language, laying a reliable foundation for downstream tasks. Additionally, discarded triples may serve as auxiliary data for text generation or data augmentation.

Redundant Descriptions.
Redundant descriptions contain information present in the text but absent

**Table 3: Examples of redundant descriptions (marked in red).**
|  | (黑莓 9220，操作系统，Blackberry OS 7) |
| --- | --- |
| Triple 1 | (BlackBerry 9220, operating_system, Blackberry OS 7) |
|  | (黑莓 9220，上市日期，2012年) |
| Triple 2 | (BlackBerry 9220, launch_date, 2012) |
|  | 黑莓 9220是2012年上市的一款支持Blackberry OS 7操作系统 |
|  | 的2.4英寸的3G手机。 |
|  | BlackBerry 9220 is a 2.4-inch 3G mobile phone that supports |
| Text | BlackBerry Os 7 operating system, which was launched in 2012. |

from the relation triples.
As shown in Table [3](https://arxiv.org/html/2510.06039v1#S3.T3), text descriptions often contain additional factual details, such as “2.4 inches” or “3G mobile phone”, that are not represented in the associated triples. To identify and eliminate such redundancy, we compute a position index by locating the character offset of the last matched triple and normalizing it by the total length of the text. Pairs with an index above 0.75, indicating that triples appear predominantly towards the end—are retained; otherwise, they are discarded. For the retained pairs, we inspect the remaining text beyond the last triple to detect unmatched relation mentions by cross-referencing with all Baike-extracted relations. Additionally, in enumerated structures separated by punctuation, we require that all items are grounded in the relation triples. If any listed item lacks a corresponding triple, the data pair is excluded to maintain strict alignment.

Data Quality Improvement. To improve data quality and address the long-tail distribution in KGs, we employ three complementary strategies: Relation Filtering. We retain only high-frequency relation types to reduce noise and emphasize semantically meaningful connections (Appendix [B.1](https://arxiv.org/html/2510.06039v1#A2.SS1)). Manual Verification. Seven trained annotators independently review each relation type, with double annotation ensuring over 90% agreement and filtering out low-quality triples. Search Validation. For each candidate triple, we retrieve the top 10 external web results and retain it only if both the head and tail entities co-occur, ensuring contextual accuracy. The final data is categorized into four domains—History and Politics, Humanities and Society, Technology and Economics, and Nature and Environment—to support domain-specific evaluation.
These measures collectively improve dataset quality and establish a reliable benchmark for evaluating Chinese LLMs. Further details on data statistics can be found in Appendix [B.1](https://arxiv.org/html/2510.06039v1#A2.SS1) and  [B.2](https://arxiv.org/html/2510.06039v1#A2.SS2).

## 4 Benchmark Design

In this section, we provide a comprehensive overview of CB-ECLLM, covering benchmark settings (Sec. [4.1](https://arxiv.org/html/2510.06039v1#S4.SS1)), and evaluation metrics $\&$ implementation details (Sec. [4.2](https://arxiv.org/html/2510.06039v1#S4.SS2)).

### 4.1 Benchmark Settings

Benchmark Datasets. To evaluate model performance across diverse domains, we construct our benchmark using four representative sub-datasets drawn from the full CDTP collection (7M pairs): CDTP_HP (History and Politics), CDTP_HS (Humanities and Society), CDTP_TE (Technology and Economics), and CDTP_NE (Nature and Environment). For the experiments in this paper, each sub-dataset is split into 80$\%$ training and 20$\%$ evaluation. To ensure tractable and reproducible experiments while preserving data diversity, we report results on a stratified 10K subset drawn from the full CDTP dataset. This subset preserves key distributions (domain, entity type, predicate frequency, and text length), is sampled with a fixed pseudorandom seed, and follows the same 80$\%$/20$\%$ split per domain.

**Table 4: Evaluation Results of Eight Chinese LLMs on Four Datasets for Three Tasks. The best three results are highlighted by 1st, 2nd, and 3rd.**
| Dataset | Task | Metrics | GLM-4-9B | Yi-9B | Qwen1.5-7B | InternLM2-7B | Llama-3-8B | Baichuan2-7B | DeepSeek-7B | Phi-2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | MRR | 0.4417 | 0.6659 | 0.5458 | 0.4906 | 0.4538 | 0.4607 | 0.5364 | 0.3305 |
|  |  | Hits@1 | 0.3445 | 0.5250 | 0.4005 | 0.3210 | 0.2720 | 0.2385 | 0.3710 | 0.1911 |
|  | KGC | F1 Score | 0.5125 | 0.6885 | 0.5719 | 0.4860 | 0.4277 | 0.3851 | 0.5412 | 0.3209 |
|  |  | MRR | 0.2594 | 0.4818 | 0.3469 | 0.3712 | 0.2911 | 0.3236 | 0.2942 | 0.3078 |
|  |  | ACC | 0.1870 | 0.3695 | 0.2425 | 0.2475 | 0.1835 | 0.1695 | 0.2005 | 0.1970 |
|  | QA | F1 Score | 0.3151 | 0.5396 | 0.3903 | 0.3968 | 0.3101 | 0.2899 | 0.3340 | 0.3292 |
|  |  | BLEU | 0.2405 | 0.2647 | 0.1471 | 0.2211 | 0.2058 | 0.1542 | 0.2502 | 0.1089 |
|  |  | ROUGE_1 | 0.7073 | 0.7253 | 0.5516 | 0.6968 | 0.6784 | 0.5569 | 0.6767 | 0.5680 |
|  |  | ROUGE_L | 0.6448 | 0.6655 | 0.4969 | 0.6223 | 0.6099 | 0.5100 | 0.6246 | 0.5235 |
| CDTP_HP | T2T | METEOR | 0.6972 | 0.7030 | 0.4029 | 0.6608 | 0.6462 | 0.4872 | 0.6442 | 0.4556 |
|  |  | MRR | 0.4927 | 0.6805 | 0.5363 | 0.4613 | 0.4449 | 0.4457 | 0.5610 | 0.2959 |
|  |  | Hits@1 | 0.3937 | 0.5430 | 0.3975 | 0.2925 | 0.2640 | 0.2070 | 0.4140 | 0.1621 |
|  | KGC | F1 Score | 0.5650 | 0.7038 | 0.5689 | 0.4526 | 0.4177 | 0.3430 | 0.5856 | 0.2790 |
|  |  | MRR | 0.2319 | 0.4535 | 0.3072 | 0.3555 | 0.2915 | 0.3049 | 0.3012 | 0.2932 |
|  |  | ACC | 0.1600 | 0.3310 | 0.1960 | 0.2210 | 0.1675 | 0.1352 | 0.1990 | 0.1715 |
|  | QA | F1 Score | 0.2759 | 0.4974 | 0.3278 | 0.3620 | 0.2869 | 0.2382 | 0.3319 | 0.2928 |
|  |  | BLEU | 0.2235 | 0.2556 | 0.1361 | 0.2228 | 0.1964 | 0.1481 | 0.2230 | 0.1012 |
|  |  | ROUGE_1 | 0.6992 | 0.7174 | 0.5527 | 0.7009 | 0.6694 | 0.5533 | 0.6628 | 0.5706 |
|  |  | ROUGE_L | 0.6408 | 0.6586 | 0.5025 | 0.6307 | 0.6079 | 0.5054 | 0.6128 | 0.5181 |
| CDTP_HS | T2T | METEOR | 0.6910 | 0.7086 | 0.4052 | 0.6696 | 0.6411 | 0.4771 | 0.6201 | 0.4614 |
|  |  | MRR | 0.4693 | 0.7249 | 0.6002 | 0.5105 | 0.4527 | 0.4602 | 0.6413 | 0.2805 |
|  |  | Hits@1 | 0.3900 | 0.6055 | 0.4735 | 0.3460 | 0.2690 | 0.1940 | 0.5145 | 0.1490 |
|  | KGC | F1 Score | 0.5612 | 0.7543 | 0.6427 | 0.5141 | 0.4240 | 0.3249 | 0.6794 | 0.2594 |
|  |  | MRR | 0.3261 | 0.5106 | 0.3149 | 0.4215 | 0.3385 | 0.4023 | 0.3532 | 0.3028 |
|  |  | ACC | 0.2650 | 0.4235 | 0.2320 | 0.3145 | 0.2245 | 0.2805 | 0.2750 | 0.1925 |
|  | QA | F1 Score | 0.4190 | 0.5950 | 0.3766 | 0.4785 | 0.3667 | 0.4381 | 0.4314 | 0.3229 |
|  |  | BLEU | 0.2146 | 0.2213 | 0.1213 | 0.1786 | 0.2079 | 0.1209 | 0.2153 | 0.0851 |
|  |  | ROUGE_1 | 0.6910 | 0.6912 | 0.5399 | 0.6667 | 0.6789 | 0.5412 | 0.6508 | 0.5502 |
|  |  | ROUGE_L | 0.6535 | 0.6505 | 0.5029 | 0.6147 | 0.6367 | 0.5076 | 0.6210 | 0.5108 |
| CDTP_NE | T2T | METEOR | 0.6791 | 0.6705 | 0.4008 | 0.6225 | 0.6502 | 0.4693 | 0.6136 | 0.4259 |
|  |  | MRR | 0.4856 | 0.6705 | 0.4973 | 0.4619 | 0.4424 | 0.4488 | 0.5305 | 0.3112 |
|  |  | Hits@1 | 0.3910 | 0.5375 | 0.3620 | 0.2840 | 0.2610 | 0.2240 | 0.3675 | 0.1856 |
|  | KGC | F1 Score | 0.5622 | 0.6992 | 0.5316 | 0.4424 | 0.4140 | 0.3660 | 0.5375 | 0.3130 |
|  |  | MRR | 0.3070 | 0.5076 | 0.3486 | 0.4045 | 0.3147 | 0.3620 | 0.3514 | 0.3130 |
|  |  | ACC | 0.2199 | 0.4060 | 0.2425 | 0.2775 | 0.1910 | 0.2075 | 0.2585 | 0.1900 |
|  | QA | F1 Score | 0.3605 | 0.5775 | 0.3903 | 0.4344 | 0.3207 | 0.3437 | 0.4108 | 0.3193 |
|  |  | BLEU | 0.2600 | 0.2740 | 0.1665 | 0.2395 | 0.2446 | 0.1550 | 0.2552 | 0.1321 |
|  |  | ROUGE_1 | 0.7169 | 0.7295 | 0.5875 | 0.7077 | 0.7031 | 0.5409 | 0.6886 | 0.6032 |
|  |  | ROUGE_L | 0.6623 | 0.6715 | 0.5364 | 0.6415 | 0.6440 | 0.5003 | 0.6406 | 0.5584 |
| CDTP_TE | T2T | METEOR | 0.7165 | 0.7244 | 0.4608 | 0.6795 | 0.6854 | 0.4643 | 0.6638 | 0.4949 |

**Table 5: Evaluation Results of Eight Chinese LLMs on Four Datasets for Three Tasks After Supervised Fine-Tuning (SFT). The best three results are highlighted by 1st, 2nd, and 3rd.**
| Datasets | Tasks | Metrics | Supervised Fine-Tuning (SFT) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GLM-4-9B | Yi-9B | Qwen1.5-7B | InternLM2-7B | Llama-3-8B | Baichuan2-7B | DeepSeek-7B | Phi-2 |  |  |  |
|  |  | MRR | 0.6105 | 0.7965 | 0.7877 | 0.7877 | 0.7553 | 0.7823 | 0.7927 | 0.6742 |
|  |  | Hits@1 | 0.4516 | 0.7170 | 0.7044 | 0.7029 | 0.6595 | 0.6853 | 0.7100 | 0.5420 |
|  | KGC | F1 Score | 0.6222 | 0.8352 | 0.8266 | 0.8255 | 0.7948 | 0.8133 | 0.8304 | 0.7030 |
|  |  | MRR | 0.6331 | 0.6295 | 0.6083 | 0.6368 | 0.5954 | 0.6084 | 0.6354 | 0.4832 |
|  |  | ACC | 0.5625 | 0.5810 | 0.5525 | 0.5865 | 0.5370 | 0.5415 | 0.5860 | 0.3622 |
|  | QA | F1 Score | 0.7200 | 0.7350 | 0.7118 | 0.7394 | 0.6988 | 0.7026 | 0.7390 | 0.5318 |
|  |  | BLEU | 0.4604 | 0.4720 | 0.4672 | 0.4531 | 0.4608 | 0.4523 | 0.4622 | 0.3665 |
|  |  | ROUGE_1 | 0.7848 | 0.7891 | 0.7859 | 0.7778 | 0.7864 | 0.7733 | 0.7848 | 0.7542 |
|  |  | ROUGE_L | 0.7549 | 0.7601 | 0.7551 | 0.7467 | 0.756 | 0.7436 | 0.7544 | 0.7148 |
| CDTP_HP | T2T | METEOR | 0.7722 | 0.7688 | 0.7647 | 0.7551 | 0.7738 | 0.7480 | 0.7650 | 0.7456 |
|  |  | MRR | 0.5268 | 0.8051 | 0.7966 | 0.7566 | 0.7683 | 0.7845 | 0.8043 | 0.6471 |
|  |  | Hits@1 | 0.3537 | 0.7265 | 0.7140 | 0.6783 | 0.6760 | 0.6875 | 0.7230 | 0.5058 |
|  | KGC | F1 Score | 0.5225 | 0.8416 | 0.8331 | 0.8083 | 0.8067 | 0.8148 | 0.8392 | 0.6718 |
|  |  | MRR | 0.6184 | 0.6457 | 0.6437 | 0.6479 | 0.6051 | 0.6381 | 0.6516 | 0.5094 |
|  |  | ACC | 0.5470 | 0.5900 | 0.5858 | 0.5890 | 0.5388 | 0.5730 | 0.5940 | 0.4032 |
|  | QA | F1 Score | 0.7072 | 0.7421 | 0.7388 | 0.7413 | 0.7003 | 0.7285 | 0.7453 | 0.5747 |
|  |  | BLEU | 0.3610 | 0.3822 | 0.3735 | 0.3608 | 0.3192 | 0.3667 | 0.3740 | 0.2329 |
|  |  | ROUGE_1 | 0.7479 | 0.7555 | 0.7513 | 0.7440 | 0.7059 | 0.7405 | 0.7501 | 0.6967 |
|  |  | ROUGE_L | 0.7055 | 0.7146 | 0.7109 | 0.7023 | 0.6640 | 0.6982 | 0.7120 | 0.6423 |
| CDTP_HS | T2T | METEOR | 0.7279 | 0.7329 | 0.7239 | 0.7107 | 0.6840 | 0.7035 | 0.7199 | 0.6631 |
|  |  | MRR | 0.6168 | 0.8246 | 0.8245 | 0.7997 | 0.7947 | 0.8131 | 0.8238 | 0.6848 |
|  |  | Hits@1 | 0.4705 | 0.7615 | 0.759 | 0.7285 | 0.7147 | 0.7355 | 0.7585 | 0.5712 |
|  | KGC | F1 Score | 0.6399 | 0.8646 | 0.863 | 0.8429 | 0.8336 | 0.8476 | 0.8627 | 0.7271 |
|  |  | MRR | 0.6331 | 0.6485 | 0.6284 | 0.6418 | 0.6016 | 0.6066 | 0.6431 | 0.5393 |
|  |  | ACC | 0.5904 | 0.6140 | 0.5853 | 0.6055 | 0.5521 | 0.5515 | 0.6035 | 0.4697 |
|  | QA | F1 Score | 0.7424 | 0.7608 | 0.7384 | 0.7543 | 0.7114 | 0.7109 | 0.7527 | 0.6392 |
|  |  | BLEU | 0.3755 | 0.4041 | 0.4033 | 0.3806 | 0.3978 | 0.3899 | 0.3984 | 0.2672 |
|  |  | ROUGE_1 | 0.7223 | 0.7516 | 0.7481 | 0.7348 | 0.7498 | 0.7385 | 0.7501 | 0.7070 |
|  |  | ROUGE_L | 0.6959 | 0.7291 | 0.7249 | 0.7105 | 0.7271 | 0.7144 | 0.7278 | 0.6783 |
| CDTP_NE | T2T | METEOR | 0.7056 | 0.7246 | 0.7226 | 0.6967 | 0.7275 | 0.7055 | 0.7246 | 0.6800 |
|  |  | MRR | 0.5719 | 0.7929 | 0.7856 | 0.7498 | 0.7600 | 0.7752 | 0.7854 | 0.6652 |
|  |  | Hits@1 | 0.4047 | 0.7135 | 0.6965 | 0.6725 | 0.6680 | 0.6800 | 0.7020 | 0.5277 |
|  | KGC | F1 Score | 0.5762 | 0.8328 | 0.8211 | 0.8042 | 0.8010 | 0.8095 | 0.8249 | 0.6909 |
|  |  | MRR | 0.4363 | 0.6539 | 0.6563 | 0.6206 | 0.6248 | 0.6438 | 0.6517 | 0.5365 |
|  |  | ACC | 0.3273 | 0.6051 | 0.6048 | 0.5695 | 0.5663 | 0.5820 | 0.5990 | 0.4445 |
|  | QA | F1 Score | 0.4932 | 0.7540 | 0.7537 | 0.7257 | 0.7231 | 0.7358 | 0.7492 | 0.6154 |
|  |  | BLEU | 0.3690 | 0.4188 | 0.4117 | 0.3574 | 0.4122 | 0.4061 | 0.4115 | 0.3003 |
|  |  | ROUGE_1 | 0.7426 | 0.7690 | 0.7661 | 0.7260 | 0.7686 | 0.7552 | 0.7652 | 0.7331 |
|  |  | ROUGE_L | 0.7081 | 0.7365 | 0.7349 | 0.6914 | 0.7359 | 0.7230 | 0.7328 | 0.6839 |
| CDTP_TE | T2T | METEOR | 0.7262 | 0.7452 | 0.7448 | 0.6567 | 0.7493 | 0.7252 | 0.7402 | 0.7111 |

Evaluation Models.
To evaluate the performance of Chinese LLMs on our proposed CDTP dataset, we selected eight state-of-the-art models with diverse architectures and training paradigms, including both general-purpose and Chinese-focused LLMs. The selected LLMs are GLM-4-9B (GLM et al., 2024), Yi-9B (Young et al., 2024), Qwen1.5-7B (Bai et al., 2023), InternLM2-7B (Cai et al., 2024), Llama-3-8B (Grattafiori et al., 2024), Baichuan2-7B (Yang et al., 2023), Phi-2 (Javaheripi et al., 2023), and DeepSeek-7B (Bi et al., 2024). Details of these Chinese LLMs are provided in Appendix [C.1](https://arxiv.org/html/2510.06039v1#A3.SS1).

Benchmark Tasks.
To fully unlock the potential of the CDTP dataset, we conduct a comprehensive evaluation of Chinese LLMs based on three tasks, including Question Answer (QA), Knowledge Graph Completion (KGC), and Triple to Text Generation (T2T). For QA and KGC, each query is cast as a multiple-choice problem with one correct answer and nine distractors. Distractors are drawn from KG neighbors and type-/frequency-matched entities, filtered for semantic feasibility, and enriched with polysemy-targeted candidates to capture Chinese-specific ambiguities. This design enables these tasks to assess both relational reasoning and robustness under contextual ambiguity. Detailed task definitions and construction procedures are provided in Appendix [C.2](https://arxiv.org/html/2510.06039v1#A3.SS2).

### 4.2 Evaluation Metrics and Implementation Details

Evaluation Metrics.
Several commonly used metrics are employed to evaluate the performance of Chinese LLMs across the three tasks: QA, KGC, and T2T. For QA and KGC, we use Mean Reciprocal Rank (MRR), Hits@1, and F1 Score. For the T2T task, BLEU, ROUGE, and METEOR are applied. Detailed descriptions and calculation methods for these metrics are provided in Appendix [C.3](https://arxiv.org/html/2510.06039v1#A3.SS3).

Supervised Fine-Tuning (SFT) Parameters Settings.
To validate the quality of our proposed CDTP dataset, SFT experiments were conducted on $8$ LLMs using DeepSpeed (Rasley et al., 2020).
Each dataset category was fine-tuned independently with the following configurations: {Training Framework: DeepSpeed with ZeRO Stage 2; Batch Size: 8; Learning Rate: $9.65\times 10^{-6}$ with a cosine learning rate scheduler; Max Sequence Length: 2048; Mixed Precision: BF16; Number of Epochs: 3}. All models are fine-tuned on the SFT datasets using 8 NVIDIA H100 GPUs.

## 5 Experimental Results

In this section, we introduce the experimental setup and discuss the experimental results in CU-ECLLM benchmark. Specifically, we aim to answer the following research questions:
⚫ RQ1 (Effectiveness): How do different Chinese LLMs perform under various datasets for three tasks? (Sec. [5.1](https://arxiv.org/html/2510.06039v1#S5.SS1))
⚫ RQ2 (SFT): How effective are different Chinese LLMs in fine-tuning Chinese LLMs on these labeled data improves their ability to handle the tasks? (Sec. [5.2](https://arxiv.org/html/2510.06039v1#S5.SS2))
⚫ RQ3 (Robustness): How well LLMs can maintain accuracy when faced with out-of-distribution (OOD) data? (Sec. [5.3](https://arxiv.org/html/2510.06039v1#S5.SS3)),Appendix [D.1](https://arxiv.org/html/2510.06039v1#A4.SS1) is the experiment of LLM with different parameters.

### 5.1 RQ1: Performance Comparison

Experiment Design.
We comprehensively evaluate and compare the performance of eight Chinese LLMs across three tasks—Knowledge Graph Completion (KGC), Question Answer (QA), and Triple to Text Generation (T2T)—using four distinct datasets.

Experimental Results.
Table [4](https://arxiv.org/html/2510.06039v1#S4.T4) and Table [9](https://arxiv.org/html/2510.06039v1#A4.T9) show the performance comparison in terms of three tasks on four datasets, and Figure [2](https://arxiv.org/html/2510.06039v1#S5.F2) provides a line graph to compare the performance between QA and KGC. Moreover, the bar graph illustrating the performance rankings of Chinese LLMs with respect to their parameter sizes is shown in Figure [3](https://arxiv.org/html/2510.06039v1#S5.F3). We have the following observations.

Observation ❶: Performance varies significantly across tasks and datasets.
As shown in Table [4](https://arxiv.org/html/2510.06039v1#S4.T4), some Chinese LLMs excel in T2T tasks but underperform in KGC or QA tasks. For instance, InternLM2-7B performs well in T2T tasks, achieving METEOR scores above 0.66 on CDTP_TE and CDTP_HS, but its performance in QA tasks, such as Hits@1 and MRR, is less impressive. Furthermore, the performance varies across different datasets. For example, in the KGC task, some models perform well on CDTP_HP but show weaker results on CDTP_TE, with similar trends observed in QA and T2T tasks. This indicates that the specific requirements of each task and dataset can significantly influence model performance. Consequently, a comprehensive evaluation framework that incorporates multiple tasks and metrics is crucial for fully assessing Chinese LLMs.

Observation ❷: Comparison between QA and KGC tasks. As illustrated in Figure [2](https://arxiv.org/html/2510.06039v1#S5.F2), all Chinese

Figure: Figure 2: Performance comparison between KGC and QA on CDTP_HP.
Refer to caption: KGC_QA.png

LLMs consistently outperform in KGC compared to QA tasks within the MRR metric. This observation stands in contrast to trends commonly seen in English, where LLMs typically perform better on unstructured QA tasks, largely due to their pretraining on vast amounts of unstructured natural language corpora (Achiam et al., 2023). The divergence highlights the unique linguistic challenges of Chinese—such as polysemy, syntactic ambiguity, and script variation—which can impair model performance on unstructured tasks. In comparison, the structured nature of KGC provides explicit semantic relationships that help alleviate these challenges, resulting in more stable and reliable outcomes. These findings underscore the importance of developing evaluation methodologies that are sensitive to the linguistic characteristics of Chinese.

Observation ❸: The larger-scale LLMs exhibit better performance. As illustrated in Figure [3](https://arxiv.org/html/2510.06039v1#S5.F3), the performance of base models appears to improve with an increase in model size, especially for tasks such as T2T and KGC. Larger models, such as Yi-9B and InternLM2-7B, consistently outperform smaller models on these tasks. This observation suggests that increasing the parameters

Figure: Figure 3: Ranking of 8 base models on CDTP_HP across three tasks. Lager values indicating better performance.
Refer to caption: model_rank.png

enhances its ability to capture more complex patterns and semantic relationships, which are crucial for tasks that require deeper language understanding and structured knowledge. However, this trend does not hold as strongly for QA tasks. Despite having a large number of parameters, Yi-9B does not significantly outperform smaller models like DeepSeek-7B in the QA task. This suggests that the QA task, which relies more on open-ended, unstructured questions, may not benefit as directly from an increase in model size, as performance improvement in QA tasks depends more on the model’s ability to handle complex reasoning and contextual understanding rather than just the number of parameters. For the ranking experiments of evaluating other base and SFT models, see Appendix [D.2](https://arxiv.org/html/2510.06039v1#A4.SS2).

### 5.2 RQ2: Supervised Fine-Tuning (SFT) Experiments

Experiment Design. We perform Supervised Fine-Tuning (SFT) on these base models and subsequently reassess their performance on these datasets to measure improvements resulting from fine-tuning.

Experimental Results. Table [5](https://arxiv.org/html/2510.06039v1#S4.T5) and Figure [4](https://arxiv.org/html/2510.06039v1#S5.F4) show the performance comparison in terms of three tasks on four datasets after SFT. We have the following key observations.

Observation ❹: Improvements Across Datasets and Tasks. As shown in Figure [4](https://arxiv.org/html/2510.06039v1#S5.F4), SFT models

Figure: Figure 4: Performance comparison of SFT and Base model on different datasets in T2T task.
Refer to caption: SFT.png

consistently outperform their base counterparts on the T2T task across all datasets.
This highlights the significant generalization gains introduced by SFT. Notably, Yi-9B and GLM-4-9B exhibit strong and stable generalization capabilities, with their SFT variants achieving consistently high METEOR scores and showing minimal performance degradation across different datasets. This suggests these models possess high adaptability to varying linguistic inputs and task nuances, making them reliable across domains. In contrast, models like Phi-2 and Qwen1.5-7B show lower base performance and larger performance gaps between their base and SFT versions, indicating weaker generalization in the absence of fine-tuning. Nevertheless, after SFT, these models demonstrate substantial performance gains, further validating the effectiveness of fine-tuning in enhancing text generation quality. For the effects of other task-based and SFT models, see Appendix [D.3](https://arxiv.org/html/2510.06039v1#A4.SS3) and [D.1](https://arxiv.org/html/2510.06039v1#A4.SS1).

Observation ❺: Narrowed Performance Gaps After Fine-Tuning.
Fine-tuning improves the performance by reducing the performance gaps between different models, leading to more balanced results across multiple tasks. Specifically, fine-tuning enhances generalization and stability, allowing models of various sizes and architectures to perform more consistently. After fine-tuning, most models approach near-perfect scores, indicating that the fine-tuning data is both broad and high-quality, making it applicable to a wide range of tasks and model types. These findings emphasize the significance of our dataset in optimizing model performance across different tasks.

### 5.3 RQ3: Robustness under Out-Of-Distribution (OOD) Data

Experiment Design. We conduct the robustness experiments across three tasks with specialized datasets to verify the robustness of Chinese LLMs Yi-9B on OOD data (Liu et al., 2023), before and after SFT using our proposed data. For the KGC task, we select YAGO3-10 (Wang et al., 2024c) as the OOD dataset and evaluate performance using the MRR metric.
For the QA task, HotpotQA (Yang et al., 2018) is employed, with evaluation based on the F1 Score.
For the T2T task, we adopt the WebNLG dataset (Gardent et al., 2017), using METEOR as the evaluation metric.

Experimental Results. Figure [5](https://arxiv.org/html/2510.06039v1#S5.F5) presents the performance comparison across three tasks on three OOD datasets, before and after SFT. We have the following key observations.

Observation ❻: Improved Performance After Fine-Tuning. As illustrated in Figure [5](https://arxiv.org/html/2510.06039v1#S5.F5), SFT

Figure: Figure 5: Comparison of robustness between base models and SFT models under the out-of-distribution (OOD) data.
Refer to caption: ood.png

significantly enhances model performance across all tasks (KGC, QA, T2T) when tested with OOD data. For example, in the QA task, the SFT model achieves a notable improvement over the base model, indicating enhanced reasoning capabilities under distributional shifts. Similar trends are observed in KGC and T2T tasks, where SFT helps the models generalize better to unseen entities and text patterns. These results validate that Chinese LLMs possess a certain level of inherent robustness when facing OOD data. More importantly, the consistent performance gains after SFT further demonstrate the effectiveness of our proposed CDTP datasets.

Observation ❼: Task-Specific Robustness. While SFT consistently enhances performance across all OOD tasks, the extent and nature of the improvements vary by task type. The QA task exhibits the most substantial gains, reflecting improved reasoning and contextual understanding. KGC, despite its lower baseline, benefits markedly from SFT, suggesting enhanced generalization of structured knowledge. In contrast, T2T demonstrates more moderate yet stable improvements, indicating better alignment in text generation. These task-specific variations highlight that robustness gains are not uniform and underscore the necessity of targeted fine-tuning strategies to adapt Chinese LLMs effectively to diverse real-world scenarios.

## 6 Conclusion and Future Directions

In this paper, we propose a Comprehensive Benchmark for Evaluating Chinese Large Language Models (CB-ECLLM) based on the newly constructed Chinese Data-Text Pair (CDTP) dataset. Specifically, CDTP comprises over 7 million meticulously aligned text pairs, each consisting of unstructured text coupled with one or more corresponding triples, alongside a total of 15 million triples spanning four critical domains: History and Politics, Humanities and Society, Technology and Economics, and Nature and Environment. CDTP supports tasks like KGC, T2T, and QA, with a special emphasis on the unique linguistic challenges of Chinese. Furthermore, we conduct rigorous evaluations through extensive experiments and ablation studies to comprehensively assess the effectiveness, Supervised Fine-Tuning (SFT), and robustness of the benchmark. This dataset offers a comprehensive benchmark for evaluating reasoning, generalization, and structured knowledge comprehension in Chinese LLMs, paving the way for AGI. One limitation is the proposed CDTP dataset only contains textual inputs and structured triples, without incorporating cross-modal signals such as images or audio, which constrains its applicability in multimodal settings.

Building on these promising results and limitations (Details of limitations see Appendix [E](https://arxiv.org/html/2510.06039v1#A5)), we outline critical research directions to address remaining challenges:

- •
Enhanced Domain and Entity Coverage. We plan to expand the CDTP dataset by incorporating data from more diverse sources, including underrepresented domains and long-tail entities. This will improve model robustness and adaptability in specialized or low-resource scenarios.
- •
Cross-Domain Generalization. Future work will explore domain-adaptive pretraining and meta-learning approaches to enhance generalization across unseen domains and heterogeneous data modalities, such as tabular, visual, or conversational inputs.
- •
Linguistic and Cultural Enrichment. Future work will extend CDTP beyond factual triples by incorporating linguistically and culturally rich phenomena, including metaphorical expressions, pragmatic markers, and culturally embedded references.

## References

- Pan et al. [2024]
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu.
Unifying large language models and knowledge graphs: A roadmap.
*IEEE Transactions on Knowledge and Data Engineering*, 36(7):3580–3599, 2024.
- Chang et al. [2024]
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.
A survey on evaluation of large language models.
*ACM Transactions on Intelligent Systems and Technology*, 15(3):1–45, 2024.
- Wang et al. [2024a]
Jiapu Wang, Sun Kai, Linhao Luo, Wei Wei, Yongli Hu, Alan Wee-Chung Liew, Shirui Pan, and Baocai Yin.
Large language models-guided dynamic adaptation for temporal knowledge graph reasoning.
*Advances in Neural Information Processing Systems*, 37:8384–8410, 2024a.
- Yan et al. [2025]
Zhichao Yan, Jiapu Wang, Jiaoyan Chen, Yanyan Wang, Hongye Tan, Jiye Liang, Xiaoli Li, Ru Li, and Jeff Z Pan.
Prompting large language models with partial knowledge for answering questions with unseen entities.
*arXiv preprint arXiv:2508.01290*, 2025.
- Yan et al. [2024]
Zhichao Yan, Jiapu Wang, Jiaoyan Chen, Xiaoli Li, Ru Li, and Jeff Z Pan.
Atomic fact decomposition helps attributed question answering.
*arXiv preprint arXiv:2410.16708*, 2024.
- Pei et al. [2019]
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al.
Towards artificial general intelligence with hybrid tianjic chip architecture.
*Nature*, 572(7767):106–111, 2019.
- Fei et al. [2022]
Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al.
Towards artificial general intelligence via a multimodal foundation model.
*Nature Communications*, 13(1):3094, 2022.
- Liu et al. [2022]
Pan Liu, Yanming Guo, Fenglei Wang, and Guohui Li.
Chinese named entity recognition: The state of the art.
*Neurocomputing*, 473:37–53, 2022.
- Zhu et al. [2022]
Peng Zhu, Dawei Cheng, Fangzhou Yang, Yifeng Luo, Dingjiang Huang, Weining Qian, and Aoying Zhou.
Improving chinese named entity recognition by large-scale syntactic dependency graph.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:979–991, 2022.
- Li et al. [2025]
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou.
From matching to generation: A survey on generative information retrieval.
*ACM Transactions on Information Systems*, 43(3):1–62, 2025.
- Wang et al. [2024b]
Jiapu Wang, Zheng Cui, Boyue Wang, Shirui Pan, Junbin Gao, Baocai Yin, and Wen Gao.
Ime: Integrating multi-curvature shared and specific embedding for temporal knowledge graph completion.
In *Proceedings of the ACM Web Conference 2024*, pages 1954–1962, 2024b.
- Li et al. [2024a]
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.
Pre-trained language models for text generation: A survey.
*ACM Computing Surveys*, 56(9):1–39, 2024a.
- Li et al. [2023]
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.
CMMLU: Measuring massive multitask language understanding in chinese.
*arXiv preprint arXiv:2306.09212*, 2023.
- Huang et al. [2023]
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
*Advances in Neural Information Processing Systems*, pages 62991–63010, 2023.
- Xu et al. [2023]
Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan.
Superclue: A comprehensive chinese large language model benchmark.
*arXiv preprint arXiv:2307.15020*, 2023.
- Wang et al. [2023a]
Jiapu Wang, Boyue Wang, Meikang Qiu, Shirui Pan, Bo Xiong, Heng Liu, Linhao Luo, Tengfei Liu, Yongli Hu, Baocai Yin, et al.
A survey on temporal knowledge graph completion: Taxonomy, progress, and prospects.
*arXiv preprint arXiv:2308.02457*, 2023a.
- Li et al. [2021]
Ziran Li, Zibo Lin, Ning Ding, Hai-Tao Zheng, and Ying Shen.
Triple-to-text generation with an anchor-to-prototype framework.
In *Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence*, pages 3780–3786, 2021.
- He et al. [2024]
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi.
G-retriever: Retrieval-augmented generation for textual graph understanding and question answering.
*Advances in Neural Information Processing Systems*, 37:132876–132907, 2024.
- Dong et al. [2024]
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou.
How abilities in large language models are affected by supervised fine-tuning data composition.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, pages 177–198, 2024.
- Lu et al. [2024]
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou.
# instag: Instruction tagging for analyzing supervised fine-tuning of large language models.
In *International Conference on Learning Representations*, pages 1–10, 2024.
- Liu et al. [2024]
Yuchen Liu, Wenlong Zhang, Haoyu Wang, et al.
Evaluating the factuality of large language models using large-scale knowledge graphs.
*arXiv preprint arXiv:2403.08122*, 2024.
- Zhao et al. [2024]
Qing Zhao, Lei Chen, Xiaozhi Wang, et al.
Knowledge graph finetuning enhances knowledge manipulation in large language models.
*arXiv preprint arXiv:2403.12012*, 2024.
- Wang et al. [2025]
Jing Wang, Xuefeng Liu, Yuanqi Li, et al.
Techgpt-2.0: A large language model project to solve the task of knowledge graph construction.
[https://huggingface.co/TechGPT-2.0](https://huggingface.co/TechGPT-2.0), 2025.
Technical Report.
- Chen et al. [2024]
Minghao Chen, Ruiqi Zhao, Renjie Tan, et al.
Fine-tuning vs. prompting: Evaluating the knowledge graph construction with llms.
*arXiv preprint arXiv:2402.09280*, 2024.
- Gardent et al. [2017]
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini.
The webnlg challenge: Generating text from rdf data.
In *Proceedings of the International Conference on Natural Language Generation*, pages 124–133, 2017.
- Agarwal et al. [2021]
Shikhar Agarwal, Antoine Bosselut, Ari Holtzman, et al.
Knowledge-enriched linked open data generation.
*arXiv preprint arXiv:2010.00796*, 2021.
- Annasamy et al. [2023]
Nikhil Annasamy, Bill Yuchen Lin, and Xiang Ren.
Kgtext: A benchmark for knowledge graph text generation.
*arXiv preprint arXiv:2301.11467*, 2023.
- Yao et al. [2024]
Meng Yao, Jitao Tang, Shumin Chen, et al.
Exploring large language models for knowledge graph completion.
*arXiv preprint arXiv:2403.03074*, 2024.
- Khorashadizadeh et al. [2024]
Amir Khorashadizadeh, Yanwei Zhang, Wei Liu, et al.
Research trends for the interplay between large language models and knowledge graphs.
*arXiv preprint arXiv:2401.05983*, 2024.
- Li et al. [2024b]
Zhuoyang Li, Liran Deng, Hui Liu, Qiaoqiao Liu, and Junzhao Du.
Unioqa: A unified framework for knowledge graph question answering with large language models.
*arXiv preprint arXiv:2406.02110*, 2024b.
- Wang et al. [2023b]
Chen Wang, Min Hua, Jiale Song, and Xue-song Tang.
Knowledge graphs enhanced large language model prompt for electric power question answering.
In *Proceedings of the International Conference on Electronic Information Technology and Computer Engineering*, pages 24–29, 2023b.
- Yuan et al. [2021]
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang.
Wudaocorpora: A super large-scale chinese corpora for pre-training language models.
*AI Open*, 2:65–68, 2021.
- GLM et al. [2024]
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al.
Chatglm: A family of large language models from glm-130b to glm-4 all tools.
*arXiv preprint arXiv:2406.12793*, 2024.
- Young et al. [2024]
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al.
Yi: Open foundation models by 01. ai.
*arXiv preprint arXiv:2403.04652*, 2024.
- Bai et al. [2023]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
*arXiv preprint arXiv:2309.16609*, 2023.
- Cai et al. [2024]
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, et al.
Internlm2 technical report, 2024.
- Grattafiori et al. [2024]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Yang et al. [2023]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al.
Baichuan 2: Open large-scale language models.
*arXiv preprint arXiv:2309.10305*, 2023.
- Javaheripi et al. [2023]
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al.
Phi-2: The surprising power of small language models.
*Microsoft Research Blog*, 1(3):3, 2023.
- Bi et al. [2024]
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al.
Deepseek llm: Scaling open-source language models with longtermism.
*arXiv preprint arXiv:2401.02954*, 2024.
- Rasley et al. [2020]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.
In *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining*, pages 3505–3506, 2020.
- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Liu et al. [2023]
Yixin Liu, Kaize Ding, Huan Liu, and Shirui Pan.
Good-d: On unsupervised graph out-of-distribution detection.
In *Proceedings of the ACM International Conference on Web Search and Data Mining*, pages 339–347, 2023.
- Wang et al. [2024c]
Jiapu Wang, Boyue Wang, Junbin Gao, Shirui Pan, Tengfei Liu, Baocai Yin, and Wen Gao.
MADE: Multicurvature adaptive embedding for temporal knowledge graph completion.
*IEEE Transactions on Cybernetics*, 54(10):5818–5831, 2024c.
- Yang et al. [2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.
HotpotQA: A dataset for diverse, explainable multi-hop question answering.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2369–2380, 2018.

## Appendix A Use of Large Language Models

We employed LLMs exclusively as editing assistants to enhance grammar, clarity, and conciseness of the manuscript. All technical contributions, including experimental design, data processing, evaluation, and conclusions, were conceived, implemented, and validated by the human authors. Edits suggested by LLMs were carefully reviewed and either accepted or modified by the authors; no numerical results, figures, or analyses were generated or approved solely by the LLM.

## Appendix B Dataset

### B.1 TOP 20 Relations

Knowledge graphs (KGs) are inherently long-tailed, with a small number of relations occurring very frequently while the majority appear only sparsely. Such skewness can introduce noise and reduce the reliability of downstream evaluation. To mitigate this effect, we analyzed the frequency distribution of relations in the triples and retained only high-frequency ones. This filtering step emphasizes commonly occurring and semantically meaningful relations while suppressing rare or noisy edges. Figure [6](https://arxiv.org/html/2510.06039v1#A2.F6) illustrates the top 20 most frequent relations in the dataset.

Figure: Figure 6: Triple relationship frequency top 20.
Refer to caption: top20.png

Figure: Figure 7: Distribution of data pairs by the number of associated relation triples. A 1-1 pair indicates one text aligned with a single triple; 1-2 and 1-3 pairs follow the same convention. The 1-3$+$ category denotes cases where one text is aligned with more than three triples.
Refer to caption: statistics.png

### B.2 Data Statistics

The proposed CDTP dataset is organized into data pairs, each consisting of a main entity, a set of
relation triples, and a corresponding text description. The number of triples associated with each text ranges from 1 to 14, and the description is designed to comprehensively capture the information expressed by the triples. To examine the distribution of triples per data pair, we conduct a statistical analysis, summarized in Figure [7](https://arxiv.org/html/2510.06039v1#A2.F7).

The results reveal that the dataset is dominated by pairs containing no more than three triples, which account for roughly 96% of all entries. Among these, the 1-2 pair (one text aligned with two triples) is the most frequent, contributing about 3 million instances. In contrast, pairs with higher triple counts are rare; for example, only three 1-14 pairs (one text aligned with fourteen triples) are observed, underscoring the sparsity of such cases.

Given this skewed distribution, our analysis and downstream experiments primarily focus on 1-3$\mathbf{+}$ pairs. This choice not only simplifies the presentation but also reflects the dataset’s practical utility, as most real-world applications involve relatively few relation triples per text.

## Appendix C Model Details Appendix

### C.1 Details of Evaluation Models

To comprehensively evaluate the performance of large language models (LLMs) on our proposed dataset, we selected eight state-of-the-art models across diverse architectures and training paradigms. These models represent a mix of general-purpose and Chinese-focused LLMs, providing a robust benchmark for comparison. The evaluated models are as follows:

- •
GLM-4-9B (^1^11Available at: huggingface.co/THUDM/glm-4-9b): A highly capable general-purpose LLM optimized for multiple languages and tasks,supporting 26 languages.
- •
Yi-9B (^2^22Available at: huggingface.co/01-ai/Yi-9B): A Chinese-focused, high-quality natural language understanding and generation model designed as a bilingual language model, trained on a 3T multilingual corpus, and excelling in coding and math within the Yi series of models.
- •
Qwen1.5-7B (^3^33Available at: huggingface.co/Qwen/Qwen1.5-7B): A powerful and efficient model, designed for reasoning and text generation tasks, featuring substantial improvements in aligning chat models with human preferences and allowing for the handling of long contexts.
- •
InternLM2-7B (^4^44Available at: huggingface.co/internlm/internlm2-7b): The latest iteration of the InternLM series is optimized for Chinese-centric datasets and offers effective support for ultra-long contexts of up to 200,000 characters.
- •
Llama-3-8B (^5^55Available at: huggingface.co/meta-llama/Meta-Llama-3-8B): The Llama 3 instruction-tuned models, recognized as a leading general-purpose LLM for multilingual applications, are specifically optimized for dialogue use cases, all while emphasizing helpfulness and safety in their design.
- •
Baichuan2-7B (^6^66Available at: huggingface.co/baichuan-inc/Baichuan2-7B-Base): A model focused on the Chinese language, demonstrates exceptional capabilities in text completion and knowledge-intensive tasks, having attained the highest performance on various authoritative benchmarks in Chinese, English, and multilingual general and domain-specific domains.
- •
DeepSeek-7B (^7^77Available at: huggingface.co/deepseek-ai/deepseek-llm-7b-base): Known for its ability to handle complex reasoning and domain-specific queries, this model places in the top 3 on AlignBench, ranks as a top-tier performer on MT-Bench, specializes in math, code, and reasoning, and supports 128K context length.
- •
Phi-2 (^8^88Available at: huggingface.co/microsoft/phi-2B): A next-generation LLM designed for efficiency and high-quality text generation, augmented with a new data source comprising various NLP synthetic texts and curated websites to enhance safety and educational value.

These models were chosen to ensure a diverse evaluation across different architectural designs and training objectives. By evaluating their performance across the 12 tasks, derived from combining four dataset categories with three distinct task types, we aim to gain insights into the strengths, limitations, and task-specific capabilities of each model. This comprehensive evaluation provides valuable benchmarks for future model development and research in LLM performance optimization.

### C.2 Task Implementation Details

To effectively evaluate Chinese LLMs, we transform the CDTP dataset into formats tailored for specific tasks. For constructing challenging single-choice questions (one gold answer and nine distractors), we first build a candidate pool for each gold entity by combining its $1$-hop and $2$-hop neighbors in KGs with entities from the same type and similar corpus-frequency bins. A semantic-feasibility filter using a pretrained Chinese sentence-embedding model is then applied to remove near-synonyms of the gold answer and candidates that are semantically unrelated to the question context.

Distractors are selected based on a heuristic that prioritizes entity-type consistency, high semantic confusability, and coverage across frequency/subdomain to avoid homogeneous sets. The ten choices are then shuffled per item using a pseudorandom permutation for evaluation.

To specifically address Chinese language challenges, such as polysemy and ambiguity, we include distractors derived from KG neighbors and same-type entities that are semantically confusable with different senses of the target token. This targeted construction allows us to assess model robustness under Chinese semantic ambiguity within the same standardized evaluation framework.

The downstream tasks are employed as follows:

- •
Question Answering (QA):
The QA task involves generating questions based on two components of a triple (head entity, relation, or tail entity) and predicting the third component as the answer. For example, given the triple
“(±)反式菊酸, 分子式, C10H16O2”((±) trans-chrysanthemic acid,molecular formula,C10H16O2),
the model is asked
“What is the molecular formula of “(±)反式菊酸” ((±) trans-chrysanthemic acid?)”,
and select the most appropriate option from ten choices as shown in Table [6](https://arxiv.org/html/2510.06039v1#A3.T6).
Table 6: The example of the QA tasks.
Query
(±)反式菊酸的分子式是什么？
What is the molecular formula of “(±) trans-chrysanthemic acid"?
Options
A. C9H16O2, B. C10H16O2, C. C11H16O2, D. C11H16O3, E. C12H18O2,
F. C10H15O2, G. C10H16O3, H. C9H15O2, I. C11H15O2, J. C10H17O2.
Knowledge Graph Completion (KGC):
The KGC task evaluates a model’s ability to predict the missing element of an incomplete triple. For instance, given an incomplete triple
(Aisin-Gioro Xibao, date of birth, ?)
“(爱新觉罗·锡保, 出生日期, ?)”
, the model is required to choose the option that best fits the missing part of the triple from a set of options as shown in Table [7](https://arxiv.org/html/2510.06039v1#A3.T7).
Table 7: The example of the KGC tasks.
Triple
(爱新觉罗·锡保, 出生日期, ?)
(Aisin-Gioro Xibao, date of birth, ?)
Options
A. 1680, B. 1681, C. 1682, D. 1683, E. 1684, F. 1685, G. 1686, H. 1687, I. 1688, J. 1689.
- •
Triple-to-Text Generation (T2T):
In the T2T task, the model generates a coherent and fluent natural language description based on a given triple, as shown in Table [8](https://arxiv.org/html/2510.06039v1#A3.T8).
Table 8: The example of the T2T tasks.
Triples
[“延庆宫, 作者, 勾台符”, “延庆宫, 创作年代, 宋代”]
["Yanqing Palace, author, Gou Taifu", "Yanqing Palace, creation period, Song Dynasty"]
Text
《延庆宫》是宋代诗人勾台符的作品之一。
"Yanqing Palace" is one of the works of Gou Taifu, a poet of the Song Dynasty.

### C.3 Details of Evaluation Metrics

To comprehensively evaluate the performance of the models across the three tasks, different metrics are employed based on the specific characteristics of each task:

#### Question Answering (QA) Task:

For the QA task, the following evaluation metrics are employed:

- •
MRR (Mean Reciprocal Rank): Computes the mean of the reciprocal ranks of the first correct answer across all queries. A higher MRR value indicates superior performance in ranking the correct answer earlier in the result list.
- •
ACC (Accuracy): Denotes the proportion of questions for which the model produces the correct answer, serving as a direct measure of prediction correctness.
- •
F1 Score: Represents the harmonic mean of precision and recall, offering a balanced evaluation that accounts for both false positives and false negatives in classification.

#### Knowledge Graph Completion (KGC) Task:

This task assesses the model’s capability to infer missing entities or relations within knowledge graph triples. The evaluation metrics include:

- •
MRR (Mean Reciprocal Rank) and F1 Score: These metrics, also utilized in the QA task, are used to evaluate the ranking quality of the predicted entities and the overall classification performance, respectively.
- •
Hits@k: Measures the proportion of cases where the correct entity appears within the top-$k$ ranked candidates. This metric provides insight into the model’s effectiveness in prioritizing correct predictions.

#### Triple-to-Text (T2T) Task:

The T2T task focuses on generating coherent and semantically accurate natural language descriptions from structured knowledge graph triples. The following automatic evaluation metrics are adopted:

- •
BLEU (Bilingual Evaluation Understudy): Assesses the degree of n-gram overlap between the generated text and reference text, and is widely used in machine translation and text generation tasks.
- •
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Evaluates the overlap of unigrams, bigrams, and longer sequences between the generated and reference texts, primarily measuring recall-based performance.
- •
METEOR (Metric for Evaluation of Translation with Explicit ORdering): Incorporates precision, recall, stemming, synonym matching, and word order to provide a more comprehensive assessment of textual similarity and fluency.

The specific calculation formulas for each evaluation metric are presented below:

- •
MRR (Mean Reciprocal Rank): MRR evaluates the average reciprocal rank of the correct answers:
$\mathrm{MRR}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{\mathrm{rank}_{i}},$
where $N$ is the total number of queries, and $\mathrm{rank}_{i}$ denotes the rank position of the first correct answer for the $i$-th query. If the correct answer is not found in the returned results, $\frac{1}{\mathrm{rank}_{i}}$ is treated as 0.
- •
ACC (Accuracy): Accuracy is a straightforward metric that measures the proportion of correctly answered questions over the total number of questions:
$\mathrm{Accuracy}=\frac{N_{\text{correct}}}{N_{\text{total}}}$
where $N_{\text{correct}}$ denotes the number of questions for which the model predicted the correct answer, and $N_{\text{total}}$ is the total number of questions. In QA tasks, a prediction is considered correct if it exactly matches the ground truth answer.
- •
F1 Score: The F1 Score is the harmonic mean of precision and recall, providing a single metric that balances both false positives and false negatives:
$\text{F1\ Score}\ =\ 2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}},$
where $\text{Precision}\ =\ \frac{\text{True\ Positives}}{\text{True\ Positives\ +\ False\ Positives}}$ is how many of the samples predicted to be positive are true positive samples; $\text{Recall}\ =\ \frac{\text{True\ Positives}}{\text{True\ Positives\ +\ False\ Negatives}}$ is how many of the real positive class samples were successfully predicted to be positive classes.
- •
Hits@k: Hits@k measures the proportion of correct answers that appear within the top-$k$ predicted results:
$\mathrm{Hits@}k=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left(\mathrm{rank}_{i}\leq k\right),$
where $\mathbb{I}(\cdot)$ is the indicator function that returns 1 if the condition is true, and 0 otherwise. A higher Hits@k indicates better recall among the top-$k$ predictions.
- •
BLEU (Bilingual Evaluation Understudy): BLEU is a widely used metric for evaluating the quality of machine-generated text by comparing n-gram overlap between the generated text and reference texts. It is particularly effective for tasks like machine translation and text generation, where word order and n-gram similarity are important:
$\mathrm{BLEU}=\exp\left(\min\left(1,\frac{\text{count}_{\text{match}}(n)}{\text{count}_{\text{generated}}(n)}\right)\cdot\text{BP}\right)$
Here, $\text{count}_{\text{match}}(n)$ refers to the number of matching n-grams between the generated and reference texts, $\text{count}_{\text{generated}}(n)$ is the total number of n-grams in the generated text, and BP is the brevity penalty that penalizes overly short translations. BLEU is typically calculated using unigram, bigram, and higher-order n-grams.
- •
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): ROUGE measures the recall-based overlap between n-grams, word sequences, or word pairs in the generated text and reference texts. It is commonly used for evaluating summarization and text generation tasks. The most common variants are ROUGE-N (measuring n-gram overlap) and ROUGE-L (measuring longest common subsequence overlap):
$\mathrm{ROUGE-N}=\frac{\sum_{n=1}^{N}\text{count}_{\text{match}}(n)}{\sum_{n=1}^{N}\text{count}_{\text{reference}}(n)}$
In the case of ROUGE-L, the score is computed based on the longest common subsequence (LCS) between the generated and reference texts, reflecting the fluency and coherence of the generated content.
- •
METEOR (Metric for Evaluation of Translation with Explicit ORdering):
$\text{METEOR}\ =\ \frac{1}{n}\sum_{i=1}^{n}{\frac{\text{precision}_{i}\cdot\text{recall}_{i}}{\alpha\cdot\text{precision}_{i}+\left(1-\alpha\right)\cdot\text{recall}_{i}}},$
where $n$ is the number of words matched and $\alpha$ is a modulating parameter (usually 0.9).

## Appendix D Experiments

### D.1 Experiments on LLMs with Different Parameter Sizes

**Table 9: LLM with different parameter sizes Experiments**
| Datasets | Tasks | Metrics | Qwen1.5-7B-Base | Qwen3-32B-Base | Qwen1.5-7B-SFT | Qwen3-32B-SFT |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | MRR | 0.7004 | 0.5458 | 0.7877 | 0.8140 |
|  |  | Hits@1 | 0.5645 | 0.4005 | 0.7044 | 0.7375 |
|  | KGC | F1 Score | 0.7216 | 0.5719 | 0.8266 | 0.8489 |
|  |  | MRR | 0.5308 | 0.3469 | 0.6083 | 0.6448 |
|  |  | ACC | 0.426 | 0.2425 | 0.5525 | 0.6033 |
|  | QA | F1 Score | 0.5975 | 0.3903 | 0.7118 | 0.7526 |
|  |  | BLEU | 0.5875 | 0.1471 | 0.4672 | 0.7088 |
|  |  | ROUGE_1 | 0.7163 | 0.5516 | 0.7859 | 0.7957 |
|  |  | ROUGE_L | 0.6594 | 0.4969 | 0.7551 | 0.7663 |
| CDTP_HP | T2T | METEOR | 0.706 | 0.4029 | 0.7647 | 0.7834 |
|  |  | MRR | 0.7017 | 0.5363 | 0.7966 | 0.8068 |
|  |  | Hits@1 | 0.5655 | 0.3975 | 0.7140 | 0.7282 |
|  | KGC | F1 Score | 0.7225 | 0.5689 | 0.8331 | 0.8427 |
|  |  | MRR | 0.5279 | 0.3072 | 0.6437 | 0.6640 |
|  |  | ACC | 0.4147 | 0.1960 | 0.5858 | 0.6101 |
|  | QA | F1 Score | 0.5863 | 0.3278 | 0.7388 | 0.7578 |
|  |  | BLEU | 0.5879 | 0.1361 | 0.3735 | 0.6678 |
|  |  | ROUGE_1 | 0.7129 | 0.5527 | 0.7513 | 0.7659 |
|  |  | ROUGE_L | 0.6486 | 0.5025 | 0.7109 | 0.7289 |
| CDTP_HS | T2T | METEOR | 0.7017 | 0.4052 | 0.7239 | 0.7495 |
|  |  | MRR | 0.7808 | 0.6002 | 0.8245 | 0.8416 |
|  |  | Hits@1 | 0.6855 | 0.4735 | 0.759 | 0.7870 |
|  | KGC | F1 Score | 0.8134 | 0.6427 | 0.863 | 0.8808 |
|  |  | MRR | 0.5951 | 0.3149 | 0.6284 | 0.6370 |
|  |  | ACC | 0.5175 | 0.2320 | 0.5853 | 0.6030 |
|  | QA | F1 Score | 0.682 | 0.3766 | 0.7384 | 0.7523 |
|  |  | BLEU | 0.5375 | 0.1213 | 0.4033 | 0.6678 |
|  |  | ROUGE_1 | 0.6781 | 0.5399 | 0.7481 | 0.7602 |
|  |  | ROUGE_L | 0.6403 | 0.5029 | 0.7249 | 0.7373 |
| CDTP_NE | T2T | METEOR | 0.6606 | 0.4008 | 0.7226 | 0.7431 |
|  |  | MRR | 0.7201 | 0.4973 | 0.7856 | 0.8145 |
|  |  | Hits@1 | 0.6003 | 0.3620 | 0.6965 | 0.7433 |
|  | KGC | F1 Score | 0.7502 | 0.5316 | 0.8211 | 0.8527 |
|  |  | MRR | 0.5554 | 0.3486 | 0.6563 | 0.6637 |
|  |  | ACC | 0.452 | 0.2425 | 0.6048 | 0.6205 |
|  | QA | F1 Score | 0.6226 | 0.3903 | 0.7537 | 0.7658 |
|  |  | BLEU | 0.5943 | 0.1665 | 0.4117 | 0.6769 |
|  |  | ROUGE_1 | 0.7203 | 0.5875 | 0.7661 | 0.7754 |
|  |  | ROUGE_L | 0.6645 | 0.5364 | 0.7349 | 0.7423 |
| CDTP_TE | T2T | METEOR | 0.7113 | 0.4608 | 0.7448 | 0.7594 |

Experimental Design. We comprehensively evaluate and compare the performance of two Chinese LLMs, Qwen1.5-7B and Qwen3-32B, both in their base form and after Supervised Fine-Tuning (SFT). The evaluation spans three knowledge-driven tasks, such as Knowledge Graph Completion (KGC), Question Answering (QA), and Triple-to-Text Generation (T2T), across four domain-specific datasets.

Model Selection. We evaluate four Qwen variants: Qwen1.5-7B-Base and Qwen3-32B-Base (base models with 7B and 32B parameters), together with their supervised fine-tuned counterparts, Qwen1.5-7B-SFT and Qwen3-32B-SFT.

Experimental Results and Analysis. Table [9](https://arxiv.org/html/2510.06039v1#A4.T9) presents the comprehensive results. Our analysis yields the following key findings:

- 1.
Effectiveness of SFT: Supervised fine-tuning (SFT) consistently boosts performance across all four datasets and three task types (KGC, QA, and T2T). Both 7B and 32B SFT models show clear gains over their base counterparts.
- 2.
Performance Ranking: The overall hierarchy of model performance is:
Qwen3-32B-SFT > Qwen1.5-7B-SFT > Qwen1.5-7B-Base > Qwen3-32B-Base.
This indicates that while larger model size provides potential advantages, task-specific fine-tuning is essential for fully unlocking that potential.
- 3.
Task-Specific Gains: Performance improvements from SFT are systematically larger in KGC than in QA tasks, suggesting that structured knowledge completion benefits more substantially from fine-tuning than open-ended question answering.
- 4.
Scalability of Larger Models: The Qwen3-32B-SFT achieves the strongest overall results, demonstrating that larger models, when properly fine-tuned, can deliver superior capabilities beyond the reach of smaller models.

### D.2 Ranking of Different Base and SFT Models

Experiment Design.
We comprehensively rank the performance of eight base and SFT Chinese LLMs across three tasks—Knowledge Graph Completion (KGC), Question Answer (QA), and Triple to Text Generation (T2T)—using four distinct datasets.

Experimental Results. Figure [8](https://arxiv.org/html/2510.06039v1#A4.F8) and Figure [9](https://arxiv.org/html/2510.06039v1#A4.F9) provide the bar graphs to illustrate the performance rankings of Chinese LLMs with respect to their parameter sizes. We have the following observations.

Figure: (a) CDTP_HP
Refer to caption: Base_History_and_Politics.png

Figure: (a) CDTP_HP
Refer to caption: SFT_History_and_Politics.png

Figure: (a) Performance comparison of SFT and Base model on different datasets in KGC task.
Refer to caption: KGC.png

Observation ❽: The larger-scale LLMs exhibit better performance. As shown in Figure [8](https://arxiv.org/html/2510.06039v1#A4.F8), the performance of the eight base models generally improves with the increase in model size. Notably, models with a larger number of parameters, such as Yi-9B, consistently outperform smaller-scale models across the three benchmark tasks. This trend suggests that increasing parameter count can effectively enhance a model’s capacity to capture complex patterns and semantic relationships. However, as previously analyzed, this effect is less pronounced in the QA task, which primarily involves open-ended and unstructured questions. For such tasks, performance relies more heavily on the model’s abilities in complex reasoning and contextual understanding rather than on parameter size alone.

Observation ❾: Substantial Enhancement After SFT. As illustrated in Figure [9](https://arxiv.org/html/2510.06039v1#A4.F9), all SFT LLMs demonstrate improved performance compared to their corresponding base versions. However, unlike in the base models, the performance gains from increasing parameter size are less significant across the three tasks. Overall, SFT LLMs exhibit more stable and consistent performance, with models of similar sizes achieving comparable results after fine-tuning on the same dataset. This indicates that supervised fine-tuning can mitigate performance differences arising from parameter scale to some extent, leading to convergence in task performance among similarly sized models.

### D.3 Supervised Fine-Tuning (SFT) Experiments

Experiment Design. We perform Supervised Fine-Tuning (SFT) on these base models and subsequently reassess their performance on these datasets to measure improvements resulting from fine-tuning.

Experimental Results. Table [5](https://arxiv.org/html/2510.06039v1#S4.T5) and Figure [10](https://arxiv.org/html/2510.06039v1#A4.F10) show the performance comparison in terms of three tasks on four datasets after SFT. We have the following key observations.

Observation ❿: Improvements Across Datasets and Tasks. As shown in Figure [10](https://arxiv.org/html/2510.06039v1#A4.F10), SFT LLMs demonstrate consistent superiority over base LLMs across all four datasets (CDTP_HP/HS/NE/TE) in both KGC and QA tasks, with GLM-4-9B and Yi-9B achieving the highest MRR scores while maintaining minimal performance variance, indicating robust cross-domain adaptability. In contrast, smaller models (Phi-2, Qwen1.5-7B) exhibit significant SFT-induced improvements, revealing their stronger dependency on task-specific fine-tuning. The performance gaps between base and SFT versions are systematically larger in KGC tasks than in QA tasks, suggesting that structured knowledge completion benefits more substantially from supervised fine-tuning compared to open-ended question answering.

## Appendix E Limitations

While our work significantly advances Chinese data-to-text generation, it also presents several limitations:

- •
Domain and Entity Coverage. Although the CDTP dataset is large and high-quality, its coverage is constrained by the inherent biases of the source data. Underrepresented domains and infrequent entities may not be adequately captured, potentially limiting model performance on specialized or low-resource tasks.
- •
Domain Generalization. CDTP spans four major categories, but achieving comprehensive domain diversity remains challenging. While models fine-tuned on CDTP perform well on in-domain and selected out-of-distribution tasks, their ability to generalize to entirely novel domains or modalities is yet to be fully validated.
- •
Linguistic and Cultural Nuance. The dataset focuses primarily on factual knowledge expressed through structured triples, which limits its capacity to evaluate deeper linguistic competencies such as metaphorical reasoning, pragmatic inference, or cultural context understanding. Although CDTP addresses morphological ambiguity and polysemy through precise triple-text alignment, it does not explicitly model sociolinguistic or culturally embedded phenomena—factors essential for comprehensive evaluation of advanced Chinese language understanding.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: - How is CDTP compared with other benchmarking datasets?
- How can the paper address errors and biases in this dataset? Especially, the four domains (history/politics/humanities/society) are relatively restrictive in Chinese; thus, evaluation results might not be generalisable.
- The evaluated models are relatively small (<30B). Given a large dataset, would this be a limiting factor? It would be interesting to see how larger models deal with the dataset. Also, how about GPT? Mistral? Claude? as they may perform well on multilingual tasks.
- Four domains are insufficient - will there be a concrete plan to extend to more domains? Also, can the authors analyse the impacts of linguistic and structural features of the dataset/LLMs
- Can the dataset be rigorously validated by humans?
soundness: 3
strengths: - The dataset offers a large amount of structured knowledge, 7M text pairs, and 15M triples; which would be useful for many down stream tasks in NLP and others.
- It highlights the need for structural knowledge alignment, potentially linguistic richness and other modalities. It is interesting to see how this translates into higher performance in KGC rather than QA.
- The paper comes with a comprehensive evaluation using GLM, Yi, QiWen, ....
summary: The paper presents a Chinese dataset comprising text pairs and triples across four domains. It supports several tasks, including question answering, triple-to-text, and knowledge graph completion, which are useful for supervised fine-tuning of LLMs with structured knowledge data.
weaknesses: - No comparison with other Chinese datasets and benchmarks. It is also good to provide a decision tree, indicating when, why and how this dataset can be used.
- Limited domains (History, Politics, Humanities, Society) and task coverages (KGC, QA, T2T) - These domains seem to be restrictive, especially for English-centric LLMs
- Limited information on human validations

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see weakness
soundness: 4
strengths: **1**  Scale and alignment quality: The combination of large scale (7M pairs, 15M triples) and careful alignment via human checks and retrieval-based validation creates a rare and valuable resource for the Chinese community.

**2**  Well-scoped task design: The unified benchmark spans KGC/QA/T2T and explicitly accounts for Chinese ambiguity and polysemy, leading to targeted and discriminative evaluations.

**3**  Thorough empirical study: Experiments cover eight mainstream Chinese/general LLMs, include SFT and OOD setups, and demonstrate consistent gains in effectiveness and robustness.
summary: The paper presents CDTP, a large-scale Chinese Data-to-Text Pair dataset consisting of over 7 million aligned text–triple pairs (15 million triples) across four domains, together with the CB-ECLLM benchmark evaluating Chinese LLMs on Knowledge Graph Completion (KGC), Triple-to-Text (T2T), and Question Answering (QA). The authors document dataset construction, cleaning, and validation, and conduct extensive multi-model experiments, including Supervised Fine-Tuning (SFT) and Out-of-Distribution (OOD) robustness analysis.
weaknesses: **1**  Evaluation Fairness and Prompt Design: The benchmark employs eight LLMs, but their instruction formats, decoding strategies, and prompt templates are not clearly standardized. Without unified prompt calibration, the performance comparison might reflect prompt sensitivity rather than intrinsic model ability. Clarifying or releasing prompt templates would improve reproducibility.

**2**  Missing Error Analysis and Case Studies – The paper reports large tables but lacks discussion of common failure patterns or qualitative examples illustrating where Chinese-specific ambiguities challenge models.

**3**  Minor Writing and Presentation Issues: Some figures and tables (e.g., Table 4 and 5) are dense and difficult to read; summarizing relative gains or including visualization of error types could improve readability.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see weakness.
soundness: 4
strengths: 1. **High impact and originality.**

   The paper addresses an urgent need for structured Chinese evaluation resources, complementing existing English-centric benchmarks such as WebNLG and KELM. The integration of triple-text alignment at such scale and quality is genuinely novel.

2. **Rigorous dataset construction.**

   The authors clearly describe multi-stage collection, cleaning, and validation pipelines, with redundancy filtering and external verification. The inclusion of four balanced domains ensures representativeness and cross-domain generalization.

3. **Comprehensive benchmark design.**

   The benchmark spans three core tasks (KGC, QA, T2T), enabling both structured-to-text and text-to-structure reasoning evaluation. The metrics and task formulation are well-motivated and comparable to international standards.

4. **Extensive experimental analysis.**

   Evaluation over eight diverse LLMs and detailed post-SFT/OOD robustness studies show impressive thoroughness. The visualizations (Figures 2–5) and interpretation of observations (1-7) demonstrate strong empirical insight.
summary: This paper presents **CDTP**, a large-scale and meticulously aligned **Chinese Data-to-Text Pair dataset** with over **7 million pairs and 15 million triples**, accompanied by a **Comprehensive Benchmark for Evaluating Chinese LLMs (CB-ECLLM)**. It evaluates multiple representative LLMs across **KGC, T2T, and QA tasks**, including both base and SFT settings, and provides extensive analysis on **effectiveness, generalization, and robustness**. The work fills a critical gap in Chinese LLM evaluation and provides valuable infrastructure for the community.
weaknesses: 1. **Limited modality coverage.**
   The dataset focuses solely on textual and structural information; integrating visual or multimodal signals (e.g., image-caption-triple) could further enhance generalization.

2. **Cross-benchmark comparison missing.**
It would be informative to include a cross-evaluation against existing Chinese benchmarks (C-Eval, CMMLU, SuperCLUE, GraphEval) to highlight complementary strengths and illustrate where CDTP provides unique coverage.

3. **Error and bias analysis.**
   A more detailed qualitative study of typical failure cases (e.g., entity ambiguity, idiomatic errors) would enrich the interpretability of the results.

---

confidence: 5
contribution: 1
initial_rating: 4
presentation: 3
questions: See `Weakness' section.
soundness: 1
strengths: 1. The dataset is large, it might be useful for Chinese NLP community.

2. The paper provides a detailed data construction process, and promises to open source code and data to facilitate reproducibility.
summary: This paper introduces CDTP, a dataset of aligned Chinese text–triple pairs. It also porpose a CB-ECLLM benchmark to evaluate three representative tasks: Knowledge Graph Completion (KGC), Triple-to-Text Generation (T2T), and Question Answering (QA). The experiment evaluates 8 LLMs to assess effectiveness, SFT gains, and OOD robustness.
weaknesses: 1. The paper claims that Chinese language poses a challenge to LLM (L46), so this dataset was constructed. However, there is no discussion on why this dataset can help Chinese LLM training. The example in Tab. 1 could be reproduced in alternative languages.

2. L245 `we selected eight state-of-the-art models'. These models are not representative because they only include 7-9b open source models and lack other scale and closed source commercial models as baseline. Also, these models are not state-of-the-art, some of them are published in 2023.

3. The effectiveness of the evaluation tasks. The paper proposes three tasks: KGC, T2T, and QA, but does not discuss the justification for these task designs. For example, why should LLMs be evaluated using triples instead of natural language? These tasks also fail to represent the challenges LLM faces when constructing knowledge graphs. For example, in the example L805, only a person's name is provided and the model is asked to complete the birth date, while the options include different years. This question does not clarify the person being referred to and cannot constitute a valid evaluation. If this question provides more context in the form of natural language, it seems to be more effective.