AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024)

Abstract: Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. To fill in this gap, we introduce ALIGNBENCH, a comprehensive multidimensional benchmark for evaluating LLMs’ alignment in Chinese. We tailor a humanin-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge (Zheng et al., 2023) approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation codes, data, and LLM generations are available at https://github.com/THUDM/ AlignBench.

heading: 1 Introduction
text: Large Language Models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022; Zeng et al., 2022; Touvron et al., 2023) have experienced a surge in development thanks to popular products such as ChatGPT (OpenAI, 2022). During the period, alignment (Ouyang et al., 2022; Bai et al., 2022), including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and related techniques, has been justified as a key strategy to endow pre-trained
∗ Xiao and Xuanyu made equal contributions. † Work done while these authors interned at Zhipu AI. ‡ Corresponding Authors: Yuxiao and Jie.
LLMs (which can hardly follow instructions) with strong grasping of human intentions and preferences. After training, aligned LLMs have not only mastered a wide array of established NLP tasks (Wang et al., 2019; Liang et al., 2022) but also versatile language-grounded missions (Cobbe et al., 2021; Chen et al., 2021; Liu et al., 2023a). As a result, LLMs have paced a firm step towards practical applications in the wild.
Meanwhile, reliably benchmarking the broad and strong competence of LLMs has also become a significant challenge. In English, there have been MMLU (Hendrycks et al., 2021), Big-Bench (Srivastava et al., 2023), and HELM (Wang et al., 2019; Liang et al., 2022); in Chinese, there are C-Eval (Huang et al., 2023b) and CMMLU (Zeng, 2023). However, prior arts hardly examine aligned LLMs’ fulfillment of user intention and human preference in real-world conditions, and even fall short to tell the difference between aligned and base LLMs. Consequently, dedicated benchmarks are crucial for development and meaningful comparisons of aligned LLMs.
Nevertheless, designing a comprehensive and reliable benchmark for LLM alignment is nontrival. An alignment benchmark should meet several important requirements, which correspond to the unique strengths of LLMs and their applications for users:
• Real-World Scenarios: Query forms and topics should be diverse and derived from real scenarios to reflect the authentic usages of LLMs.
• Open-Ended: As aligned LLMs usually produce long open-ended replies, the benchmark should judge the correctness of detailed responses without specified forms.
• Challenging: LLMs are improving so rapidly on various aspects beyond estimation. The benchmark thus has to ensure its difficulty to identify subtle capability gaps between LLMs.
11621
• Automatic Judging: Benchmark construction and evaluation should be as automatic as possible to provide scalable, reproducible, and intime feedback to facilitate LLM development.
There have been recent attempts to introduce LLM-as-Judge (Li et al., 2023; Zheng et al., 2023) for evaluating the general alignment of LLMs. AlpacaEval (Li et al., 2023) compares target LLM’s replies against text-davinci-003’s, but has been shown unstable and uninterpretable due to its direct and pairwise scoring. MT-Bench (Zheng et al., 2023) harnesses point-wise scoring with Chain-ofThought (CoT) (Wei et al., 2022) explanations for better accuracy and transparency. However, it employs only 80 test samples and a scoring prompt that judges queries of different tasks and domains uniformly. Lastly, both benchmarks are designed only in English and cannot well reflect the level of alignment of many emerging Chinese LLMs.
In light of all mentioned issues, in this work we present ALIGNBENCH, a comprehensive multidimensional benchmark for evaluating LLMs’ alignment in Chinese. Based on observations from an online LLM service (anonymous due to review policy), we set up a semi-automatic data curation pipeline with human in the loop to create high-quality queries to construct ALIGNBENCH. ALIGNBENCH summarizes a taxonomy comprising 8 major categories of queries (Cf. Figure 1) to comprehensively cover and align with real-scenario usages. In order to make the judge model generate objective and fair evaluations, each sample is accompanied with a human-corrected LLMgenerated reference. To ensure reference correctness for knowledge intensive queries (which accounts for 66.5%), we ask annotators to search on the web, provide evidences including urls and quotations, and finally synthesize references.
To enhance the automation of the evaluation, similar to MT-Bench (Zheng et al., 2023), ALIGNBENCH distinctly leverages GPT-4 (OpenAI, 2023) as the major model evaluator in its development, which serves to discern the data samples and evaluate by referenced point-wise scoring with CoT. Differently, ALIGNBENCH further highlights strategies of rules-calibration and task-specific multidimensional judgement in the scoring. Our experiments demonstrate that these strategies contribute to ALIGNBENCH’s better consistency with human judgement and better explanation quality. Based on ALIGNBENCH, we evaluate 17 popular API-based or open-sourced LLMs that support Chinese, providing detailed comparisons of these LLMs across fine-grained capabilities on Chinese alignment.
In summary, the contributions of our work are: • We construct ALIGNBENCH, a systematic
benchmark rooted in real-scenario usages to evaluate Chinese alignment of LLMs. We also tailor a human-in-the-loop pipeline to allow accurate and sustainable benchmark maintenance.
• Targeting accurate and automatic evaluation of LLMs, we design a rule-calibrated multidimensional point-wise LLM-as-judge method for grading. Human evaluations justified its applicability compared to existing LLM-as-Judge methods (Zheng et al., 2023).
• We systematically benchmark 17 LLMs’ Chinese alignment on ALIGNBENCH. On top of their performance, we provide deep insights into status quo of Chinese LLMs’ development and highlight future directions.

heading: 2 Dataset
text: In this section, we introduce the data composition and construction pipeline of ALIGNBENCH.
Benchmark Dataset Information Evaluation Method
Data Size Language Data Source Domain Open-ended MultiDimensional Metric
MMLU (Hendrycks et al., 2021) 15,908 English Exams & Textbooks Knowledge Accuracy GSM8k (Cobbe et al., 2021) 8,000 English Human Writers Math ✓ Accuracy HumanEval (Chen et al., 2021) 164 Python Human Writers Code ✓ Pass@k CMMLU (Zeng, 2023) 11,528 Chinese Exams & Textbooks Knowledge Accuracy AGI-Eval (Zhong et al., 2023) 8,062 Chi. & Eng. Exams Knowledge Accuracy C-Eval (Huang et al., 2023b) 13,948 Chinese Exams Knowledge Accuracy
2.1 Dataset Composition
In this section, we introduce the overall composition of ALIGNBENCH. To perform a systematic evaluation, we frame a comprehensive taxonomy of the LLMs’ abilities based on real-user instructions. We inspect and summarize these use-cases into 8 main categories and 683 samples in total as shown in Table 2. Fundamental Language Ability. This category focuses on the basic language tasks derived from traditional NLP tasks such as information extraction (Etzioni et al., 2008), text classification (Wang and Manning, 2012), and commonsense knowledge (Talmor et al., 2019). They reflect common users’ practical needs of LLMs to conduct traditional tasks under zero-shot or few-shot settings with customized prompts and formats, such as text classification, information extraction, and short summarization. Thus we select high-quality diverse queries relevant to each traditional task in this category. Advanced Chinese Understanding. This category aims to evaluate LLMs’ ability to understand cultural and historical background in Chinese-specific tasks. In Chinese context, a large percentage of real-user needs are related to Chinese culture, characters, and history. However, without deliberate optimization on Chinese, cutting-edge LLMs (e.g., GPT-4 (OpenAI, 2023)) would fail to understand
and answer these questions correctly. As ALIGNBENCH targets Chinese alignment, the category plays a vital role in our overall design. Open-ended Questions. The category represents a common usage of LLMs to answer subjective questions in an open-ended manner. Users may seek for advice, recommendations, and tutoring for many daily questions concerning their work, study, travel, and lives. The key to good open-ended responses is about catering to human preference, featuring long, detailed, and highly related content. Writing Ability. Writing, one of the most frequently used LLM function, plays a vital role in LLMs’ applications. We select challenging writing instructions, which require not only an excellent mastering of language but also a high level of instruction-following (e.g., specific formatting conditions), consistency (e.g., argumentative topics), and creativity (e.g., fictions or poems). Logical Reasoning. The ability to process complicated problems with step-by-step reasoning and LLMs’ inherent knowledge is highlighted for current strong LLMs. The category aims to evaluate LLMs’ abilities to understand, analyze, and produce correct responses given intricate logical problems, using questions that require deductive, abductive, multi-hop, or commonsense reasoning. Mathematics. Math problems are challenging but also widely-adopted for LLMs’ ability testing due to practical applications. We collect math problems in different difficulty levels from elementary to advanced mathematics and in different formats, including calculations, simple problem solving, concept explanation, theorem-proof, etc. Task-oriented Role Play. Lots of users request the model to play as a specific identity to perform corresponding tasks, which is summarized as taskoriented role play. In order to evaluate the fulfillment of users’ instructions and the quality of
responses when role play, we collect role play instructions of high complexity for the category. Professional Knowledge. LLMs have proven their competence in solving domain-specific problems that require professional knowledge. The category aims to evaluate LLMs’ abilities in specific domains, including physics, history, music, law, etc. Additionally, the queries we select are generative open-ended questions that allow LLMs to generate freely and provide sufficient details.

heading: 2.2 Dataset Construction
text: Each sample in ALIGNBENCH contains a taskoriented query, a high-quality referenced answer, and its category in our taxonomy. The detailed construction pipeline is described as follows. Query Curation. To ensure the diversity and authenticity of the queries, we mainly refer to two sources, namely the scenarios from an online chat service and challenging problems written by researchers. Given the inherently noisy nature, we go through a high-standard data curation pipeline following rules described below. 1) Task-oriented: The query should represent human intentions and instruct LLMs to complete the specified task. 2) Clarity & Fluency: The query should be clear, and easy-to-understand and the demands should be smoothly expressed. 3) Complexity & Difficulty: The query should be hard for most LLMs, requiring them to utilize their capabilities to solve it comprehensively. 4) Desensitization: Ensure that the queries are safe and insensitive. Reference Acquisition & Improvement. For point-wise grading for ALIGNBENCH (Cf. Section 3 for analysis), scoring with a pivotal reference answer has been found beneficial to improve the reliability of the LLM-as-Judge (Zheng et al., 2023; Zhang et al., 2020). Thus we decide to provide human-curated reference answers, serving to assist evaluators in determining the correctness of the answer, and act as a scoring pivot.
However, because ALIGNBENCH has been designed to be difficult and of wide coverage, it turns out quite challenging for human annotators to provide answers from scratch in our preliminary trial. As a result, we first utilize GPT-4 to generate answers, and then ask human annotators to meticulously review, revise, and refine them as reference answers for ALIGNBENCH. To ensure reference quality, especially for knowledge-intensive queries from categories such as Professional Knowledge,
Mathematics, and Logical Reasoning, we explicitly ask annotators to conduct web search during the verification. During the search, webpage urls and quotations of contents for writing the references are required to be recorded, as shown in Table 3. Filtering & Classification. To enhance distinguishment in scores between strong LLMs, it is necessary to filter more challenging samples for evaluation. Consequently, we engage three relatively advanced Chinese-supported LLMs, including GPT-3.5-turbo (OpenAI, 2022), ChatGLM (Du et al., 2022; Zeng et al., 2022) APIs and Sparkdesk to serve as difficulty filters within our construction procedure. We subject these models to evaluation, analyzing their responses to the processed queries and subsequently scoring the answers utilizing GPT-4. By computing the average score across responses and utilizing it as a signal, we discard 50% of the queries that garnered the highest average scores, indicative of their lower difficulty levels. The approach ensures a meticulous and discerning selection of samples, effectively distinguishing between strong LLMs of varying capacities.

heading: 3 Methods
text: To effectively evaluate the quality of responses, ALIGNBENCH employs GPT-4 (OpenAI, 2023) as a major evaluator to analyze and subsequently grade the responses following adopted practices (Zheng et al., 2023; Li et al., 2023; Liu et al.,
https://xinghuo.xfyun.cn/
2023b). However, a significant designing space still exists regarding prompting, score calibration, critique explanability, and evaluation dimensions, which have been hardly explored.
Therefore, in ALIGNBENCH we design a novel rule-calibrated multi-dimensional point-wise LLMas-Judge method. The detailed prompts are in Appendix A.2 and an example is displayed in Fig 3.
Point-wise Grading & Chain-of-Thought (Wei et al., 2022). When LLM-as-Judge is leveraged, two grading methods have been previously implemented: point-wise (Zheng et al., 2023) or pairwise grading (Li et al., 2023). Nevertheless, previous study has indicated that the point-wise grading possessed comparable agreement with humans than the pair-wise grading, which suffers from position bias (Zheng et al., 2023). Additionally, considering the evaluating efficiency, compared to pair-wise grading’s quadratic number of comparisons, the point-wise grading has advantages in terms of expenses and time. Therefore ALIGNBENCH adopts point-wise grading either. During the evaluation, the inputs are the query, the model’s response, and a human-curated reference answer, and the output is an multi-dimensional analytical explanation and a final rating, ranging from 1 to 10. As the task of grading involves complex reasoning, introducing Chain-of-Thought in the scoring process has also been proved useful to augment both the score reliability and interpretability (Zheng et al., 2023). Specifically, GPT-4 is instructed to generate explanations from multiple dimensions before providing a final grade on a scale of 1 to 10.
Rule-calibrated Referencing. Given that many of the questions in ALIGNBENCH are of significant complexity and difficulty even for GPT-4, we provide a high-quality reference answer, which is primarily generated by GPT-4 and modified by human annotators to ensure its correctness and improve its quality. To guide the evaluator to compare the answer with the reference and generate more controllable scores, we provided detailed grading rules elaborating the relationship between score intervals and the answer’s quality compared to the reference. Additionally, we set the reference answer to score 8 as a reference scoring pivot.
We plot the cumulative distribution of human judge, general judge and rule-calibrated judge in Figure 2 to show that rule-calibration judge has a narrower gap to human evaluation’s cumulative distribution. Typically, rule-calibrated judge scores
much fewer top scores (9 and 10) than general judge, aligned with human scoring habits and therefore enhance the discrimination of ALIGNBENCH. Multi-dimensional Analysis. As tasks vary in their nature and characteristics, applying the same evaluation criteria to all tasks would be unjust. For instance, writing tasks should prioritize creativity, whereas logical reasoning tasks primarily require logical coherence. As a solution, we propose a multi-dimensional scoring approach to evaluate LLMs’ responses, tailoring the evaluation to the specific task at hand, promising a more comprehensive and organized explanation. Specifically, we set up different evaluation dimensions based on different types of questions, as shown in Table 7 and we instructed the evaluator to analyze the model answer from specified dimensions and provide dimensional scores. Furthermore, we found that our multi-dimensional method could effectively balance different dimensions, reducing verbosity bias, with an example shown in Table 11. The categorical information is also useful for conditioning generation temperature for target LLMs to generate reply (Zheng et al., 2023). For tasks that has a relatively fixed answers (e.g., Mathematics, Professional Knowledge, etc.), we set temperature to 0.1 to ensure more deterministic and reproducible generation; for other tasks (e.g., Writing, Task-oriented Role Play, etc.) that may need more creativity, a high temperature (e.g., 0.7) is adopted to encourage longer and more diverse generation.

heading: 4 Human Evaluation on ALIGNBENCH
text: To justify the rule-calibrated multi-dimensional point-wise LLM-as-Judge method we design for ALIGNBENCH, we conduct extensive human evaluation over ALIGNBENCH’s selected queries. We especially focus on two aspects: the method’s agreement with human judging, and the method’s critique quality for more human-interpretable results.

heading: 4.1 Agreement Evaluation
text: Previous studies (Zheng et al., 2023) have executed comprehensive agreement experiments, demonstrating that GPT-4 (OpenAI, 2023) evaluator concur excellently with humans within English contexts. However, such agreement remains considerably under-investigated in Chinese contexts, thereby warranting further exploration. We have
Since the translation subcategory contains open-ended questions that requires creativity, it’s treated as Generative Question.
conducted a comprehensive human annotation experiment, aiming to measure the agreement between evaluations adjudicated by human annotators and our method. Dataset. We randomly sample a subset of 400 queries from the complete ALIGNBENCH dataset. To make sure each category consists of enough samples to produce reliable results, smaller categories are upsampled. To cover LLMs with a wider levels of capability, we adopt answers from 8 LLMs, including GPT-4 (OpenAI, 2023), three versions of ChatGLM series (Zeng et al., 2022; Du et al., 2022), Sparkdesk, Qwen-plus-v1-search(Bai et al., 2023a), InternLM-7B-Chat (Team, 2023) and Chinese-Llama2-7B-Chat, producing a total of 3200 question-answer pairings. Subsequent to the compilation of the evaluation set, the questionanswer-reference triples are delivered to human annotators, tasked with assigning quality ratings to the answers according to the references. Given the inherent limitations bound to human cognition, annotators are instructed to employ a rating on a scale from 1 to 5. The scores are indicative of response quality, with higher scores epitomizing superior quality and profound satisfaction. In particular, a score of 1 marks irrelevant, incorrect, or potentially harmful responses. Baselines. The experiment incorporated two ro-
https://xinghuo.xfyun.cn/
bust baseline comparisons to benchmark our evaluation approach. Note that all the methods use GPT-4 to evaluate for fairness. 1) General grading: which leverages a translated and then slightly modified Chinese version of the evaluation prompt employed in MT-bench (Zheng et al., 2023). 2) Rule-calibrated grading: To better instruct the model to compare the model answer and reference answer and reduce score variances, we incorporate grading rules into the evaluation process. The method comprises five scoring intervals, each associated with a specific quality description. The reference answer is anchored to the score of 8, serving as a relative scoring pivot.
Metrics. To comprehensively measure the agreement between the GPT-4 judges and human evaluators, we adopt several metrics. 1) Sample-level Pearson Correlation: measures correlations at sample level, by averaging the Pearson score of each sample. 2) System-level Pearson Correlation: measures correlations at system level. It calculates the Pearson coefficient between humanjudge and model-judge average scores of LLMs. 3) Pairwise Agreement (w/o tie): For each response, human-judge and model-judge scores were converted into pairwise comparisons (excluding ties).
Analysis. Results of agreement experiment are presented in Table 4. It shows that our point-wise multi-dimensional rules-calibrated LLM-as-Judge method performs best, particularly on the Samplelevel Pearson metric and the Pairwise Agreement (w/o tie) metric, thereby substantiating the excellent agreement with human judges. Furthermore, it is noteworthy that all methods considered demonstrates impeccable performance on the Systemlevel Pearson metric, indicating the reliability and robustness of the LLM-as-judge.

heading: 4.2 Quality Evaluation
text: Previous study (Zheng et al., 2023) mainly focuses on the agreement between model judges and human judges when evaluating LLM-as-Judge methods. However, considering the interpretability and readability as an evaluation process and the clarity and helpfulness as a feedback-providing approach, it is also of great significance to evaluate the quality of the explanation generated by the model judge before grading. To assess the quality of explanations generated by the methods, as well as to ascertain the final rating, we conduct a pairwise quality comparison experiment. Experiment Settings. To compare the quality of the explanations given by our method and the two baselines mentioned above, we sample 500 question-answer pairs from the ALIGNBENCH dataset and generated explanations using the three LLM-as-Judge methods. Then, we pair three explanations under each sample in pairs, producing 1,500 samples, subsequently delivered into human preference comparisons.
Given a question, a model answer, a reference answer, and two explanations (denoted as A and B) given by GPT-4 judges, human annotators are instructed to compare the quality of explanations. To guide the human evaluators on the comparison, we make a quality-judgement guideline as follow: 1) Rationality: if the explanation is reasonable,
correct, and fair. 2) Readability: if the explanation is logical, well-organized, comprehensive, and detailed. 3) Consistency: if the explanation and the final rating are consistent. which also serves as a brief standard of high-quality explanations. Analysis. Results of quality evaluations are presented in Table 5. Results indicate that our method generate the most high-quality explanation and helpful feedback, defeating two baselines in pairwise comparisons with high win rates. Furthermore, it’s worth noting that rules-based grading outperforms general grading in terms of explanation quality, proving that the grading rules can provide a clear reference-based standard and, therefore contribute to the clear comparison of the reference answer and the model answer.

heading: 5 ALIGNBENCH: Benchmarking Results
text: Based on the validness of ALIGNBENCH’s LLMas-Judge scoring, we systematically benchmark a wide array of LLMs on Chinese alignment with the help of ALIGNBENCH. We use gpt-4-0613 (OpenAI, 2023) as the judge model to evaluate model performances. Main Results. Results are shown in Table 6. The results highlight that most of the evaluated close-sourced LLMs successfully achieve commendable scores (above or near 6 points). This demonstrates the potent capability of these ad-
vanced LLMs in fulfilling user intents with highquality responses, showing a relatively excellent level of alignment. It is particularly promising for the Chinese LLM community to note that certain innovative Chinese-developed LLMs display performance either equivalent to or marginally surpassing that of gpt-3.5-turbo (OpenAI, 2022), drawing closer to the recognized leading model, gpt-4-1106-preview (OpenAI, 2023).
Analysis: Reasoning Drawbacks. The evaluation has revealed some drawbacks in reasoning abilities amongst Chinese-oriented LLMs, which require further attention and development. Given the leading capability of gpt-4-1106-preview (OpenAI, 2023) in Logical Reasoning and Mathematics (with scores of 7.66 and 7.80 respectively), there is substantial room for improvement in light of their significantly weaker performance.
Analysis: Chinese Abilities. Furthermore, certain categories, such as Advanced Chinese Understanding, underline the necessity for an LLM to possess a proficient understanding of the Chinese language, culture, and history. Our findings suggest that while gpt-4-1106-preview (OpenAI, 2023) performs relatively well (ranks the second) in these areas, the best-performing Chinesedeveloped LLMs achieved comparable or even better performance in Advanced Chinese Understanding category, potentially attributed to incorporating more culture-relevant and high-quality Chinesespecific instruction-tuning data in the alignment of these Chinese LLMs.
Analysis: Open-source Achievements. The results indicated that the top-tier Chinese opensourced LLMs such as Qwen-14B-Chat (Bai et al., 2023a), Baichuan2-13B-Chat (Yang et al., 2023), have shown great performance in terms of instruction following and high-quality generation, approaching the performance of some close-sourced LLMs. Based on our evaluation, Chinese opensourced LLMs have demonstrated their potential to become alternatives and even competitors with close-sourced LLMs, showing the inspiring dynamism of the Chinese LLM community.
Analysis: Dimensions. From Table 10, we observe that gpt-4-1106-preview (OpenAI, 2023) has achieved the highest scores in all dimensions. Generally, the dimensional results are aligned with the overall results in Table 6. Furthermore, in terms of correctness and user satisfaction, there is a significant gap between gpt-4 series (OpenAI, 2023)
and other LLMs. We show complete results in Appendix A.3 and hope the evaluation can better assist the researchers to understand and enhance Chinese LLMs’ alignment on multiple dimensions.

heading: 6 Related Work
text: Evaluation of LLMs. Self-supervisedly (Liu et al., 2021) pre-trained LLMs (OpenAI, 2023; Anil et al., 2023) exhibit excellent performance in language tasks, bringing severe challenges to the effective and comprehensive evaluation of LLMs (Chang et al., 2023; Zhuang et al., 2023; Xu et al., 2023b). Faced with the challenges, researchers have proposed benchmarks (Hendrycks et al., 2021; Zhong et al., 2023; Huang et al., 2023a; Cobbe et al., 2021; Chen et al., 2021; Bai et al., 2023b; Zhang et al., 2023) focused on measuring atomic abilities, which fall short to consider real-scenario usages enough. LLM-as-Judge methods. LLMs have shown great potential in evaluating the text quality with high agreement with human judges (Li et al., 2023; Wang et al., 2023a; Liu et al., 2023b; Zheng et al., 2023) and providing helpful feedbacks (Wang et al., 2023c; Cui et al., 2023) serving as guides for improvement. However, some potential bias and risks are also identified (Zheng et al., 2023; Wang et al., 2023b; Ke et al., 2023) when using LLMs-as-judge. Alignment of LLMs. Alignment, including following human instructions and providing helpful assistance, is crucial for LLMs’ pragmatic applications (Liu et al., 2023c). To aligning LLMs with humans, related methods include supervisedly finetune LLMs (Wang et al., 2023d; Xu et al., 2023a; Sun et al., 2023) and improve further through reinforcement learning from human feedback (Stiennon et al., 2020; Ouyang et al., 2022; Glaese et al., 2022; Rafailov et al., 2023). However, it’s challenging to effectively evaluate the capabilities of alignment since the high expense to acquire human preferences and the open-ended reference-free feature in real application scenarios.

heading: 7 Conclusion
text: We introduce ALIGNBENCH, a comprehensive multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese. A sustainable humanin-the-loop data curation pipeline and a better LLM-as-Judge method has been tailored to enable ALIGNBENCH’s high-quality automatic evaluation of LLMs’ Chinese alignment. Benchmarking results of 17 Chinese-supported LLMs are reported.

heading: Limitations
text: Here we discuss several limitations of this work. To improve automatic LLM-as-Judge. While we show that leveraging GPT-4 as judge could achieve relatively high correlation with human evaluation, there is a large room for improving this correlation and granularity. Additionally, it is shown that LLM-as-Judge has some potential biases (Zheng et al., 2023), including position, verbosity, and selfenhancement. These biases could harm the correctness of evaluation on certain models. We leave these open problems for future research. To cover more topics and queries. Despite ALIGNBENCH has a relatively large query set in its class, it still needs to be enriched for a more stable and reliable LLM evaluation. Also, ALIGNBENCH does not include evaluation over long-text queries, which could be added in its future version.

heading: Acknowledgements
text: This work is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD0118600, Natural Science Foundation of China (NSFC) 62276148 and 62425601, the New Cornerstone Science Foundation through the XPLORER PRIZE.

heading: A Appendix

heading: A.1 Detailed Description of the Dataset
text: In this section, we will further elaborate the overall composition of ALIGNBENCH. In order to perform a systematic evaluation, we conducted a thorough real-user query analysis in our LLM-based chat service. We framed a comprehensive taxonomy of the LLMs’ abilities based on the Chinese real-user demands. The overall taxonomy consists of 8 categories, namely Fundamental Language Ability, Advanced Chinese Understanding, Open-ended Questions, Writing Ability, Mathematics, Task-oriented Role Play, and Professional Knowledge. Each category contains several subcategories, effectively serving as a complementary materials to deeper understand the composition of our dataset.

heading: A.1.1 Fundamental Language Ability
text: This category focuses on the basic language understanding and processing tasks, which are derived from traditional NLP tasks such as entity extraction, text classification, and commonsense knowledge.
• Commonsense knowledge. This subcategory mainly evaluates the model’s ability to master commonsense knowledge and fundamental facts, explain primary concepts, and form a basic understanding of the physical world.
• Reading Comprehension. This subcategory mainly evaluates the model’s ability to process and understand the provided textual material and answer users’ questions based on that.
• Translation. This subcategory requires models to master different kinds of languages and understand the interactions between them. Currently, this subcategory mainly covers English and Chinese.
• Text Classification. This subcategory tests model’s ability to classify text data into given categories with different characteristics.
• Information Extraction. This subcategory measures model’s ability to extract knowledge from text data, which lays a solid foundation for more challenging tasks.

heading: A.1.2 Advanced Chinese Understanding
text: This category aims to evaluate the abilities to understand, analyze and produce reasonable and correct responses faced with Chinese-specific problems, including Chinese characters, history and culture.
• Character-wise Understanding. This subcategory focuses on the real queries related to Chinese characters and phrases, requiring LLMs to understand the complex structure and contextual meaning of certain Chinese characters and utilize them to form reasonable phrases and sentences.
• Cultural Understanding. This subcategory is intended to measure models’ abilities to master the Chinese language at a higher level, including understanding implicit meaning, mastering rhetorical techniques such as humor and sarcasm, utilizing Chinese-specific idioms and phrases, and mastering knowledge related to Chinese culture and history.

heading: A.1.3 Open-ended Questions
text: This category represents an important role for the LLMs to become a critic or an advisor for the users. Equipped with sufficient knowledge and advanced reasoning ability, LLMs are capable of providing fresh thoughts, creative perspectives, feasible advice, and comprehensive recommendations
for users. Therefore, this category measures the LLMs’ ability to offer opinions and suggestions.
• Opinion Expression. This subcategory requires the model to offer reasonable thoughts to certain subjects, people, events, or circumstances following users’ instructions. Containing open-ended questions without groundtruths, the logical smoothness and the information correctness were of high priority during evaluation.
• Suggestion Offering. This subcategory aims to evaluate the ability to analyze users’ intentions and offer some feasible suggestions or recommendations. Being selected carefully, the questions cover a wide range of topics from daily life to professional advice.

heading: A.1.4 Writing Ability
text: Regarded as one of the most frequently used capabilities, writing abilities play a vital role in LLMs’ applications. Therefore, we systematically framed this category into 4 subcategories and selected typical real-user writing instructions, which require not only an excellent mastery of language but also a high level of thought formulation and creativity.
• Practical Writing. Practical writing is a practical style of articles formed in long-term social practice activities and is often used when dealing with public and private affairs in people’s daily work, including speech scripts, workrelated emails, personal statements, regulations, etc. Additionally, practical writing assistance can play a vital part in alleviating workloads and boosting productivity.
• Creative Writing. Creative writing is a writing style that requires a high level of creativity, emotions, aesthetic taste, and delicate design, including writing novels, essays, poems, lyrics, and even jokes. It represents not only higher requirements for the LLMs to fulfill users’ writing instructions as well as generating high-quality outputs but also a promising attempt at AI creativity.
• Professional Writing. Professional writing usually contains domain-specific materials and has high requirements for professional format and content. Considering the demands to use LLMs as professional writing assistants, this subcategory includes instructions like academic reports, comprehensive surveys, legal documents, financial analysis, etc.
• Custom Writing. With lots of the writing queries classified into the above 3 subcategories, there exist other highly personalized and context-related instructions that require our attention. Therefore, we collected them into this subcategory, for instance, rewriting with casual style, correction of grammar errors, expansion of the given text, imitation writing, etc.

heading: A.1.5 Logical Reasoning
text: This category aims to evaluate the abilities to understand, analyze, and produce reasonable and correct responses faced with Chinese-specific textual logic problems. Therefore, this category plays a vital role in our overall design.
• Proof Proof problems are a widely used technique for determining the correctness of arguments and propositions. It involves verifying a proposed proposition through a series of legitimate reasoning steps to ensure that it is valid and reasonable within a given logical framework. This subcategory focuses on several intellectual qualities such as LLMs’ ability to reason logically, reverse thinking, mathematical symbolic representation, and clarity of reasoning.
• Reasoning In logical reasoning, reasoning methods are required to solve a variety of complex problems related to arguments and propositions. Reasoning problems require LLMs to use a combination of logical reasoning skills, creative thinking, mathematical symbolic representation, and clear reasoning processes in order to solve a variety of complex logical puzzles effectively. Compared with Proof subcategory, this subcategory doesn’t provide ground-truths, increasing difficulty and complexity.

heading: A.1.6 Mathematics
text: Considering its logical complexity and a large proportion, math problems are regarded as a necessary field to evaluate LLMs. We collected math problems in different difficulty levels from elementary mathematics to advanced mathematics and in different formats, including calculations, simple problem solving, concept explanation, theorem-proof, etc.
• Elementary Mathematics. The elementary mathematics subcategory is a branch of mathematics that examines the capability to master basic mathematical concepts, techniques, and methods by LLMs. It typically includes but is not limited to arithmetic, Algebra, geometry,
probability, and statistics. • Advanced Mathematics. This subcategory
is from a branch of mathematics that covers a wide range of mathematical topics and concepts and is designed to provide students with in-depth knowledge of mathematical theory and problem-solving skills. Advanced mathematics typically includes calculus, differential equations, linear algebra, probability and statistics, mathematical proofs, functions of a complex variable, and linear programming. These elements form the core of advanced mathematics and provide the basis for solving complex mathematical problems in a variety of fields.
• Applied Mathematics. This subcategory focuses on how mathematical theories and methods can be applied to solve real-life problems and challenges. Distinguished from higher and elementary mathematics, applied mathematics not only requires a certain mathematical ability but also tests the linguistic comprehension and mathematical modeling skills of large language models.

heading: A.1.7 Task-oriented Role Play
text: Our real-user scenario analysis indicates that lots of users request the model to play as a specific identity to perform corresponding tasks, which is summarized as task-oriented role play. In order to evaluate the fulfillment of users’ instructions and the quality of responses when role-play, we collected role-play instructions and constructed this category.
• Celebrity. The instructions in this subcategory perform the role-play of realistic celebrities, such as politicians, athletes, etc.
• Emotional. The instructions in this subcategory assign the identity of real-life roles such as friends, relatives, pets, etc., and provide emotional support for the users.
• Entertainment. The instructions in this subcategory involves games played between the user and the LLM, requiring imagination and creativity.
• Functional. The instructions in this subcategory assign the identity of roles with different occupations, experience, or knowledge and achieve some specific tasks.
• Daily life. The instructions in this subcategory perform the role-play of different kinds of activities in a more realistic life scenario.

heading: A.1.8 Professional Knowledge
text: With their advanced knowledge abilities, LLMs have proven their competence in solving domainspecific problems that require professional knowledge. This category aims to evaluate LLMs’ abilities in specific domains, for instance, physics, history, music, law, etc.
• Physics. Physics is the natural science that studies the nature and properties of matter and energy.
• Chemistry. Chemistry is the study of the nature, composition, structure, and patterns of change of substances. The study of chemistry involves the interrelationships between substances or the correlation between matter and energy.
• Computer Science. Computer science is the systematic study of the theoretical foundations of information and computation and the practical techniques of how they are implemented and applied in computer systems.
• Biology or Medicine. Biology consists of the empirical and extensive study of all aspects of life. Medicine is an applied science that aims at and studies the protection of human health and the enhancement of physical and mental fitness.
• Economics. Economics is the social science discipline that studies the relationship between goods and services, including all purchasing, production, distribution, and consumption behaviors therein.
• Astronomy. Astronomy is a natural science that studies celestial bodies and astronomical phenomena.
• Sociology. Sociology uses a variety of research methods of empirical investigation and critical analysis to develop and refine a body of knowledge about the structure of human societies, social action, or social relations, and to apply that knowledge.
• History. History, the study of human history as an object of study, is a form of knowledge in which human beings sift and combine their historical materials
• Music. Music, broadly speaking, is any art that consists of sound. All human cultures have music, which means that the performance of music is a universal phenomenon in all cultures.
• Law. Law is a system of rules, the implementa-
tion of which is ensured by the coercive power of the state, which regulates the behavior of individuals.
• Sport. Sports or sports competition is a social activity that aims at strengthening physical fitness, improving skills, and enriching cultural life through physical exercise, technology, training, and competition.
• Geography. Geography is the study of the Earth and its features, inhabitants, and phenomena, the study of the Earth’s surface circles interacting with each other, and its spatial differences and the process of change.
• Literature. Literature, in a narrow sense, is a kind of language art, i.e., an art that uses language and writing as a means to visualize and reflect objective social life and to express the thoughts and feelings of the subjective author.
• Others. This subcategory contains questions that not covered by the above domains.

heading: A.2 Prompts and Details of Methods
text: In our multi-dimensional analysis, we select different dimensions for different categories to provide a more comprehensive and reasonable evaluation. The detailed selections of the dimensions are described in Table 8 and the descriptions of the dimensions are described in Table 9.
All the prompts used in our experiments are displayed as follows. General LLM-as-Judge is displayed in Figure 6. Rule-calibrated LLM-as-Judge is displayed in Figure 5. Our Multi-dimensional Rule-calibrated LLMas-Judge is displayed in Figure 4.

heading: A.3 Dimensional Performance
text: Our multi-dimensional rule-calibrated LLM-asJudge method provides multi-dimensional analysis and scores. Therefore, we also calculated dimensional average scores and reported them in Table 10. For each dimension, the dimensional score was averaged across all the samples that evaluate the dimension ability, since each category could be used for evaluating several but not all dimensions according to the features of the category.

heading: A.4 Case Study

heading: A.4.1 Misleading.
text: Upon meticulously verifying and modifying the reference answers to ensure correctness, we ob-
served several instances of incorrect referencing. This underscores the inherent challenges in utilizing LLMs for evaluative tasks in practical settings, where reference information may be fraught with inaccuracies and confusion. Therefore, it is imperative to delve into the LLM evaluator’s response to such ambiguous or misleading references.
As shown in Figure 7, we highlight a discrepancy where the correct answer should be India but the provided reference erroneously states Japan. Notably, the GPT-4 (OpenAI, 2023) evaluator failed to detect this error, evaluating based on the incorrect reference. Conversely, when posed the question directly, GPT-4 (OpenAI, 2023) accurately generated India as the correct answer, demonstrating its profound knowledge to the question. This dichotomy suggests that while LLMs can provide accurate responses independently, their evaluative capabilities can be compromised when presented with incorrect references or ambiguous materials. This raises concerns regarding the reliability of using LLMs as evaluators in large-scale applications and implies that the alignment process of LLMs may inadvertently reduce the model’s ability to rectify user errors.

heading: A.4.2 Reference-free Judgements
text: During the evaluation, we noted that certain LLMs are prone to producing extensive details pertinent to the query posed. This complexity poses a significant challenge for the LLM evaluator, particularly when attempting to ascertain the accuracy of the information provided in scenarios where reference materials are scarce or incomplete.
As shown in Figure 8, The provided information(more than 30) regarding the number of countries participating(in fact 13) is inaccurate. In the absence of corresponding reference information, the LLM evaluator was unable to assess the factual correctness of the response accurately. This underscores a discernible weakness in the evaluation capabilities of LLMs when operating in settings devoid of sufficient reference material, resulting in suboptimal performance in discerning factual inaccuracies.
Addressing this issue may necessitate the integration of an autonomous factual verification tool, supported by a robust and dynamically updated information database. We acknowledge the complexity of this challenge and propose it as an avenue for future research endeavors.
∫ 1

heading: Future Work
text: We acknowledge the complexity of this challenge and propose it as an avenue for future research endeavors. ∫ 1