TMMLU+: An Improved Traditional Chinese Evaluation Suite for Foundation Models

Authors: zhi rui tam
ya-ting  pai
yen-wei  lee
jun-da  chen
wei-min  chu
sega  cheng
hong-han  shuai
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 03:19:13

Abstract: We present TMMLU+, a new benchmark designed for Traditional Chinese language understanding. TMMLU+ is a multi-choice question-answering dataset with 66 subjects from elementary to professional level. It is six times larger and boasts a more balanced subject distribution than its predecessor, Taiwan Massive Multitask Language Understanding (TMMLU). We also benchmark closed-source models and 26 open-weight Chinese large language models (LLMs) of parameters ranging from 1.8B to 72B on the proposed TMMLU+. Our findings reveal that (1.) Traditional Chinese models still trail behind their Simplified Chinese counterparts, highlighting a need for more focused advancements in LLMs catering to Traditional Chinese. (2.) Current LLMs still fall short of human performance in average scores, indicating a potential need for future research to delve deeper into social science and humanities subjects. (3.) Among all the tokenization compression metrics examined, we identify that only the fertility score uniquely demonstrates strong correlations with our benchmark results. We foresee that TMMLU+ will pinpoint areas for future model improvement, thereby narrowing the gap between machine and human linguistic capabilities and supporting researchers in developing Traditional Chinese LLMs. Our dataset, along with the benchmark source code, is accessible at huggingface.co/datasets/ikala/tmmluplus.

Main paper: 
 introduction: 
 LLMs are a group of transformer models, typically trained in the autoregressive method. They can be used to solve a wide range of tasks in Natural Language Understanding (NLU) (Rajpurkar et al., 2016; Wang et al., 2019) , vision processing (Goyal et al., 2017; Fu et al., 2023) , and robotics (Padalkar et al., 2023) . As the abilities of LLMs continue to improve, existing benchmarks start to saturate and supersede the human baseline. For example, Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021a ) is created to test knowledge understanding across a wide range of topics related to English and US culture. Since then, many subsequent works (Dac Lai et al., 2023; Son et al., 2024; Yin & Kawahara, 2024) have tried to extend the same multi-choice question-answering format with a wide variety of topics into other languages.Recent efforts have sought to adapt this approach to the Chinese language. For example, as outlined in Huang et al. (2023) ; Li et al. (2023) , the primary focus is on Simplified Chinese and the cultural milieu of mainland China. However, the Traditional Chinese domain, which is used by 30M population around the world, is nowhere to be found. Given that there are stark differences between these 2 languages despite sharing the same root. A primary distinction between Simplified and Traditional Chinese lies in their divergent vocabularies. For instance, the concept of "computer memory" is expressed as "内存" in Simplified Chinese, while in Traditional Chinese it is "記憶體". A direct character-bycharacter conversion of the Traditional term to Simplified Chinese would yield "记忆体", which occupies an entirely different semantic space. This lexical divergence is further complicated by cases where identical characters carry disparate meanings across the two writing systems. A salient example is the character "丑", which in Simplified Chinese can denote either "ugly" or the period from 1 AM to 3 AM, depending on context. In contrast, Traditional Chinese employs "醜" to convey "ugly," while reserving "丑" exclusively for the aforementioned time designation. Hsu et al. (2023) proposes TC-Eval, which features TMMLU, a Traditional Chinese multichoice question-answering dataset. TMMLU covers 55 subjects and includes a total of 3,300 questions. Despite its advancements, it presents several limitations: Firstly, its scope is relatively narrow, with only 55 subjects and some featuring less than 100 questions, leading to potential gaps in subject diversity and coverage. Secondly, there are inconsistencies in the question formats, with 5% of questions deviating from the multi-choice format and 1% including more than four options. Lastly, the absence of a dedicated development set restricts the exploration of few-shot prompting techniques. Addressing these issues is crucial for advancing LLMs' understanding and applicability in linguistically and culturally diverse contexts.We propose TMMLU+, an enhanced version of TMMLU, which aims to address the issues highlighted above. TMMLU+ contains 22,690 questions across 66 subjects, ranging from primary, secondary, undergraduate, and professional levels of education. Similar to Hendrycks et al. (2021a) ; Huang et al. (2023) , we include STEM subjects: physics, chemistry, and computer science with mentions of Traditional Chinese culture. TMMLU+ benchmark specifically addresses the terminology differences between Traditional and Simplified Chinese, ensuring that models can accurately understand and respond to questions in the context of Taiwan. The terminology used in Traditional Chinese often differs from that in Simplified Chinese, reflecting variations in character forms, vocabulary, and usage. All subjects contain at least 110 questions, with 5 development questions for few-shot prompting.We conduct experiments on 23 open-weight language models ranging from 1.8B to 72B from 5 different architectures of different pretraining datasets. To our surprise, Simplified Chinese language models outperform Traditional Chinese models in the same parameter level by at least 19%. The top-performing open-weight models are Qwen-72B and Qwen-14B, followed by Yi-34B-Chat model, with scores of 64.3%, 50.5%, and 49.6%, respectively. These models are predominantly used for Simplified Chinese. Despite the impressive results of these models, there is still a gap between their performance and that of human test-takers. On average, exam takers can score around 68% on the compared subjects. We have found that LLMs struggle in math-heavy subjects such as engineering math and accounting.To dive deeper into the differences between Simplified Chinese language models and Traditional Chinese language models, we conduct translation experiments on TMMLU+ where we convert the Chinese characters (Traditional Chinese to Simplified Chinese) and repeat the evaluation on the newly converted version. We found that most Chinese language models perform better in simplified versions, with some showing a smaller margin than others. We also conduct experiments on few-shot prompting and chain-of-thought (CoT) prompting (Wei et al., 2022) . We have found only few-shot prompting increases performance in most evaluated models while CoT prompting degrades. We also study open-weight Chinese language models from various architecture with different vocabulary sizes. Our analysis has found a strong correlation between the fertility scores and benchmark results. 
datasets: 
 We introduce TMMLU+, a benchmark dataset to evaluate Traditional Chinese LLMs' multitask language understanding capabilities.The dataset comprises 22,690 multi-choice questions from 66 subjects ranging from primary to professional level. These questions cover a wide range of subjects, including humanities, social science, and STEM fields. It includes universally applicable topics like physics, chemistry, and macroeconomics. Furthermore, some questions will require test-takers to choose the correct combination of options, such as 'ABCD', as their answer. In addition, we include questions where test-takers must provide comparison results, as shown in Figure 1 .下列電磁輻射的頻率大小順序排列何 者正確？ (Which of the following is the correct order of electromagnetic radiation frequencies?) I: microwave II: γ-rays Additionally, the dataset features questions requiring specific knowledge of Taiwan, sourced from standardized tests in subjects such as Taiwanese Hokkien, the culture of the aboriginal peoples of Taiwan, and the geography of Taiwan as shown in Figure 2 . Questions from the Taiwanese Trivia Question Answering (TTQA) (Ennen et al., 2023) dataset are also included. TTQA is a multi-choice question evaluation about culture, geography, and history specific to Taiwan. To correctly answer these questions, a model needs to have an understanding of Taiwan culture. To view the list of included subjects, please refer to Appendix B.III: visible IV: IR V: UV A. I > IV > III > V > II B. II > V > III > IV > I C. V > II > IV > III > I D. V > III > IV > II > IWe have gathered previous exams between 1998 and 2023 from various public exams such as certificate exams, comprehensive assessment programs for junior students, and college entrance examinations. This results in 962 exams from different standardized test resources and PDF documents available online. In addition, we have also included subjects from TMMLU (Hsu et al., 2023) and TTQA (Ennen et al., 2023) . Subjects related to culinary arts have been consolidated into a unified culinary category, and questions about dentistry have been incorporated into our preliminary set of questions. To ensure that the testing set contains more than 100 questions, we have manually added 29 more questions to the 103 questions from TTQA. We have named this new subject TTQAv2, as it is a distinct set of questions from TTQA.questions are finalized, we follow the same practice from Hendrycks et al. (2021a) ; Huang et al. (2023) ; Li et al. (2023) by randomly shuffling the choices of each question to ensure all subjects' answers are uniformly distributed. This results in a random guess accuracy of 25%. After refining the question pool, we establish human baseline scores for 44 of the 66 subjects. These baselines are instrumental for conducting a nuanced ablation study on human performance benchmarks in subsequent phases of our research.Each subject is then split into test, dev, and validation. We ensure that each subject contains at least 110 questions, with 5 development questions for few-shot prompting, and at least a minimum of 5 validation questions. Every question has four choices: A, B, C, or D, and the model is prompted to select one answer.All 66 subjects can be grouped under 4 categories: STEM, Humanities, Social Science, and Others. For example, STEM subjects would test the arithmetic ability of LLMs, while humanities and social science focus more on the knowledge brevity of the models. For all subjects, we calculate the average accuracy score first followed by averaging all subjects by their subject category. Finally, an average score is calculated based on the 4 categories' average scores.Given the age of our questions, we suspect they may be present in the LLMs' training data, potentially compromising our goal to benchmark the LLMs' abilities. We use Min-K%++ (Zhang et al., 2024) which is the current SOTA in pretraining data detection for LLMs. We use Qwen-7B as the model and find that only 0.48% questions exist in its pretraining data. 
experiments: 
 We conduct evaluations on 23 distinct LLMs, focusing on multilingual and Chinese-specific models. Each model varies in aspects such as tokenization, pretraining datasets, size of parameters, and architectural design. Furthermore, we compare the performance of these LLMs against datasets comprising human-generated answers to assess their relative effectiveness. Additionally, we investigate several factors that could influence the performances of LLMs on these datasets.In our approach, we formulate the prompts for the LLMs in both zero-shot and few-shot settings. For the few-shot setting, we utilize the development splits, specifically employing the provided set of five examples as context examples. The complete prompts can be found in Appendix D. (Huang et al., 2023) for non-conversation LLMs. Responses are parsed using a modified regular expression by Huang et al. (2023) with Traditional Chinese characters added. For close-source models, we obtain the response via APIs and parse using the same set of parsers used by open-weight models.To provide an extensive overview of the current landscape of Chinese LLMs, we conduct benchmarks not only on Traditional Chinese-specific models (Lin & Chen, 2023 ) but also on Simplified Chinese LLMs (Bai et al., 2023; Du et al., 2022) . Our evaluation encompasses a range of well-regarded closed-source LLMs, including those from OpenAI GPT series (Brown et al., 2020) , Gemini (Team, 2023) , and Claude (Anthropic, 2023; 2024) . Within the OpenAI GPT suite, our tests incorporate GPT-4 from the gpt-4-0613 model and GPT-3.5turbo from the gpt-3.5-turbo-0613 model. For Gemini models, we use the Gemini-pro from the gemini-1.0-pro-001 model, the only publicly accessible Gemini model at the current point of writing. Regarding Claude models, we include Claude-1.3, Claude-2.0, and Claude-3.0 (Anthropic, 2024) in our evaluation where Claude-1.3 is solely trained in the supervised fine-tuned (SFT) (Lowe & Leike, 2022) , whereas Claude-2.0 undergoes training in the SFT stage, followed by reinforcement learning from human feedback (Bai et al., 2022) . The full models' detail description can be found in Appendix E.In our benchmark, we examine a spectrum of open-weight models, with parameter counts ranging from 1.8B to 72B (Bai et al., 2023; Lin & Chen, 2023; Ennen et al., 2023; Du et al., 2022) , as well as proprietary models that are evaluated via API calls (OpenAI, 2023; Team, 2023; Anthropic, 2023; 2024) . Disparities in performance are noted even among models with comparable parameter sizes, which can be attributed to differing developmental approaches. Such discrepancies highlight the importance of model architecture and training regimen over the mere scale of parameters. We intend to further investigate these differences, focusing on how factors such as model tokenization contribute to the final accuracy in Section 3.5.4. Detailed comparisons across all models are available in Appendix F.In this analysis, we compare human test-takers and a range of LLMs, including Qwen-72B, Qwen-14B, and the proprietary GPT-4, GPT-3.5-turbo, Gemini-Pro. The human baseline scores, as represented in Table 3 , have been revised to reflect the passing rate based on the proportion of correct responses to total responses for each question. To illustrate, if a question offered four possible answers and the majority of human respondents selected the correct one, the passing rate for that question would be calculated by dividing the number of correct answers by the total number of responses.Our result indicates that human test-takers maintain a lead over LLMs in zero-shot learning tasks, with an average accuracy of 68.2% across all domains. Notably, humans exhibit the highest proficiency in Social Sciences and Humanities, areas where nuanced understanding and contextual knowledge play significant roles. In comparison, the highest-performing model, Qwen-72B, shows an average accuracy of 62.1%, with particular strength in Social Science and STEM-related tasks.Based on results from Tables 2, we find subjects that necessitate mathematical reasoning, such as those in the STEM category, consistently show lower performance metrics compared to subjects that are more reliant on background knowledge and linguistic comprehension, such as Social Sciences, Humanities, and Other categories. This trend is evident in both prompting strategies, where models like GPT-4 and Qwen-72B while performing admirably across all categories, exhibit a noticeable dip in their performance in the STEM field.This study aims to investigate the impact of the linguistic differences between Traditional and Simplified Chinese characters on the performance of LLMs. Traditional and Simplified Chinese characters share only about 30% of their vocabulary, suggesting that LLMs trained in Traditional Chinese may encounter difficulties when processing Simplified Chinese text.Understanding the linguistic flexibility and limitations of LLMs within the context of these distinct yet interconnected writing systems is crucial. We utilized the widely recognized opencc (Kuo, 2023) In this section, we compare our dataset against established Simplified Chinese benchmarks, namely CMMLU (Li et al., 2023) and C-Eval (Huang et al., 2023) , using a selection of 8 language models: Qwen series, both Traditional Chinese LLMs Breeze and Taiwan LLaMA as well as GPT-4. The comparison in Figure 3 revealed that the models tended to show a consistent level of accuracy, answering questions from both CMMLU and C-Eval with a similar accuracy. The figure highlights that all chinese language models generally performed better on the Simplified Chinese benchmarks, with a trend of results moving toward the upper left corner. Comprehensive outcomes for these models can be found in the Appendix G. In our study, we identify a notable correlation between the size of a model's vocabulary and its performance in TMMLU+ tasks. Despite the broad range of pretraining corpora sizes, from 10 billion to 2 trillion tokens, which typically remain undisclosed, our findings highlight the significant impact of tokenizer design on model effectiveness. Specifically, in the context of machine translation, we reference the study by (Zouhar et al., 2023) , which demonstrates a strong correlation between Rényi entropy-a measure reflecting the diversity and distribution of tokens-and the BLEU score, a widely recognized metric for evaluating translation accuracy. Another work (Ali et al., 2023) has found that a low fertility score for tokenizers across different languages is crucial for multilingual performance. A tokenizer with a low fertility score, when presented with the same sentence, generates a shorter sequence of tokens compared to a tokenizer with a higher fertility score. In the case of Taiwan-LLaMA, 59.5% of Traditional Chinese characters in TMMLU+ are encoded with 256 bytes. This poses a challenge for the transformer module as it increases the input sequence while operating on a smaller vocab subset. To validate our hypothesis, we aggregate data from approximately 15 LLMs around 7B parameters in LLaMA (Touvron et al., 2023a; b; Jiang et al., 2023) , Qwen (Bai et al., 2023) , Bloom (Scao et al., 2022) and GPT-Neo (Black et al., 2022) architecture, and plot the correlation between their tokenizer fertility scores and average accuracy. Our result, as shown in Figure 4 , reveals a significant Pearson correlation coefficient of -0.742 between fertility score and average accuracy, indicating that high fertility negatively impacts downstream performances.In this section, we undertake a thorough examination of the error patterns exhibited by various LLMs in comparison to those made by human test-takers. Our goal is to discern whether the mistakes made by these advanced language models are akin to those typically encountered in human responses.We partition the questions into three distinct categories based on their difficulty levels: easy, medium, and hard. The difficulty level of each question is determined through an empirical approach. First, we compute an average threshold for the correct response rates of humans across all questions. This threshold is used as a benchmark to categorize each question. Specifically, a question is classified as 'hard' if its correct response rate is lower than or equal to 25%, 'medium' if the rate is above 25% but lower than or equal to the average threshold, and 'easy' if the rate is higher than the average threshold. Second, the categorized data is then used to populate arrays corresponding to each difficulty level, thereby enabling a comparative analysis of error patterns across the different categories. Finally, we analyze the distribution of errors in various categories and find that Qwen-72B outperforms humans in difficult questions across all major categories. However, most questions are dominated by easy difficulty, resulting in an average Qwen-72B score that falls short compared to humans. Existing evaluation libraries for multi-choice question answering, such as lm-eval-harness (Gao et al., 2023) , typically measure the negative log-likelihood of choice terms and select the option with the highest log probability as the final answer. In contrast, our evaluation relies on string matching to extract the final answer, more closely mimicking real-world user interactions. However, this method can yield scores lower than random choice if the parser fails to extract a final answer. We compared these two methods across three evaluated models, as shown in  
related works: 
 Many benchmark datasets have been created to evaluate the abilities of models in NLU tasks. Some of these benchmarks focus on specific subjects. For instance, the GLUE benchmark (Wang et al., 2019 ) is a collection of NLU tasks that mainly concentrate on language skills. AI2 Reasoning Challenge (Clark et al., 2018) is a dataset comprising multi-choice questions from science exams at various grade levels. On the other hand, MATH (Hendrycks et al., 2021b) is designed to evaluate LLMs' ability to solve mathematical problems. MMLU (Hendrycks et al., 2021a ) is a benchmark used to assess models' knowledge comprehension in different domains. It provides broad and in-depth coverage of subjects, making it widely used for evaluating LLMs' abilities in knowledge understanding. The proliferation of benchmarks is increasing and expanding; however, it is noteworthy that most of these benchmarks have been developed in English.With Chinese being the language having the largest user base worldwide, there also has been a significant increase in the number of benchmark datasets available in Chinese recently.For example, a bilingual benchmark including both English and Chinese has been released by Zhong et al. (2023) which focuses on standardized tests. Furthermore, some datasets are tailored to specific fields, such as CMATH (Wei et al., 2023) , which evaluates the ability of LLMs to solve mathematical problems at the elementary level. Additionally, Zeng (2023) proposed MMCU, which covers four key areas: medicine, law, psychology, and education.Researchers have also created benchmarks that cover a wide range of subjects and difficulty levels from elementary school to college, such as M3KE (Liu et al., 2023) , CMMLU (Li et al., 2023) , and C-Eval (Huang et al., 2023) .However, considering the discrepancies in idiomatic usage and cultural nuances between Traditional Chinese and Simplified Chinese users, the benchmark datasets in Simplified Chinese might not be entirely suitable for accurately assessing LLMs' NLU capabilities in Traditional Chinese. Despite this scarcity, there are a few existing benchmarks specifically designed for Traditional Chinese. These include DRCD (Shao et al., 2019) focusing on reading comprehension, TTQA (Ennen et al., 2023) emphasizing Taiwanese-related knowledge extracted from Wikipedia pages. Also, Hsu et al. (2023) released a benchmark encompassing a diverse range of knowledge in 55 subjects from Taiwanese standardized examinations. 
conclusion: 
 In this work, we introduce TMMLU+, an improved benchmark for evaluating large language models understanding in Traditional Chinese through multi-choice question answering.Our preliminary results have found that existing Traditional Chinese LLMs still fall behind their Simplified Chinese counterparts in terms of understanding Traditional Chinese. Furthermore, our analysis indicates that none of the current LLMs achieve an average score surpassing human performance, highlighting the challenging nature of our dataset and its potential to drive forward the development of more sophisticated language understanding systems. Creation of year Due to concern of memorization, we try to use as many questions from 2023 as shown in Figure 6 . Our earliest questions can be found in as early as 1998. 
 
 Appendix: 1 9 9 8 1 9 9 9 2 0 0 0 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9 2 0 1 0 2 0 1 1 2 0 1 2 2 0 1 3 2 0 1 4 2 0 1 5 2 0 1 6 2 0 1 7 2 0 1 8 2 0 1 9 2 0 2 0 2 0 2 1 2 0 2 2 2 0 2 3 Year Tables 13 provide a detailed breakdown of how answer options are distributed across a wide range of subjects in TMMLU+. Each value in the table represents a percentage that corresponds to the portion of each option (A, B, C, D) within the specific subject. Our prompts for zero-shot prompting can be found in Figure 8 . The prompt for five shot and chain of thought prompt can be found in Figure 9 and Figure 10 respectively. The part where LLMs response is marked in red. In CoT prompting, we specifically ended the sentence in "Let's think step by step" such that the model would first reason about all four choices.Here are the multi-choice questions about the Taiwan regional linguistic exam, please choose the correct answer. It is not always necessary to pay anti-dumping duties when declaring imports.答案：讓我們一步一步思考，首先，首先，選項A並不完全正確，因為並非所有的進 出口貨物都需要辦理檢驗及報關，這取決於貨物的種類和目的地國家的法規。其次， 選項B的敘述也不對，一般來說，信用調查應該在報價之前進行，以確保買家有足夠的 信用來完成交易。再者，選項C中的中國輸出入銀行免費為廠商辦理國外徵信，這個 說法並不正確，因為徵信通常是需要費用的。因此，選項D是正確的，進口報關時不 一定要繳反傾銷稅，這取決於貨物的種類和目的地國家的法規。反傾銷稅是為了保護 國內產業不受到外國廉價商品的傾銷影響，但並非所有的進口商品都需要繳納。所以 答案是D。Answer: Let's think step by step. Firstly, Option A is not entirely correct, as not all import and export goods need to undergo inspection and customs declaration, which depends on the type of goods and the regulations of the destination country. Secondly, the statement in Option B is also incorrect; generally, credit investigations should be conducted before quoting to ensure that the buyer has sufficient credit to complete the transaction. Furthermore, the statement in Option C that the Export-Import Bank of China handles overseas credit information for manufacturers for free is not accurate, as credit investigation usually requires a fee. Therefore, Option D is correct; paying anti-dumping duties is not always necessary when declaring imports, depending on the type of goods and the regulations of the destination country. Anti-dumping duties are meant to protect domestic industries from the impact of cheap foreign goods being dumped, but not all imported goods are required to pay them. Therefore, the answer is D Figure 10 : An example of CoT in answer-only scenarios. The English translation is given beneath the corresponding Chinese text. The English translation appears below the corresponding Chinese text. The text highlighted in blue serves as our prompt, while the text in red was generated by GPT-4.This section outlines the specifics of the models utilized in our investigation.GPT-4 and GPT-3.5-turbo represent OpenAI's advancements in conversational models. Initially, these models undergo supervised fine-tuning leveraging a dataset comprising userassistant dialogues. Subsequently, reinforcement learning from human feedback (RLHF) further refines their capabilities. For this study, we employ versions of these models that were current as of June 13th, marking the latest updates available to us.LLaMA and LLaMA 2 epitomize Meta's efforts in enhancing large, decoder-only language models. Both iterations employ Rotary Positional Embedding (RoPE) for positional encoding and Root Mean Square Layer Normalization (RMSNorm) for model normalization. LLaMA 2 distinguishes itself by undergoing an extended pretraining phase on a diversified and more comprehensive dataset.Qwen is pretrained on up to 3 trillion tokens, including Chinese, English, multilingual texts, code, and mathematics, covering general and professional fields.Qwen1.5-14B-Chat incorporate SwiGLU activation, RoPE, and multi-head attention mechanisms. It offers enhanced capabilities for managing multiple languages and facilitating chat interactions, capable of supporting up to 32K tokens in context length.Atom-7B is developed from the LLaMA 7B foundation, augmenting its vocabulary with an additional 32,000 Simplified Chinese tokens. This model benefits from further pretraining on a diverse and unspecified corpus, with a focus on sequences up to 4,000 tokens in length.DeepSeek-LLM-7B-Base is a fresh instantiation of the LLaMA architecture, pretrained from the ground up on an extensive 2 trillion token dataset, encompassing both English and Chinese languages.DeepSeek-LLM-7B-Chat evolves from the Deepseek-llm-7b-base, undergoing fine-tuning on a proprietary instructional dataset, the specifics of which remain undisclosed.Yayi-7b builds upon the BLOOMZ 7B model, engaging in continuous pretraining on a selectively curated corpus of Simplified Chinese and English texts.Yayi-7b-llama2 mirrors the approach of Yayi-7b, albeit starting from the LLaMA 2 7B model, to undertake continuous pretraining on a similarly filtered corpus of Simplified Chinese and English texts.Breeze-7B-Instruct-v0.1 enhances the original Mistral-7B model by incorporating an additional 30,000 Traditional Chinese tokens into its vocabulary. This expanded model then undergoes further pretraining on a substantial, though unspecified, volume of Traditional Chinese data.Mistral-7b-80k extends the Mistral 7B framework, focusing its additional pretraining efforts on Simplified Chinese datasets.BlueLM-7B-Base is GPT-Neo architecture language model pre-trained on high-quality data with 2.6 trillion tokens in English and Simplified Chinese.Claude-3.0 Opus developed by Anthropic, is trained using a combination of unsupervised learning and a technique called Constitutional AI. It involves pretraining the model on a massive dataset of text and image data to acquire language and vision capabilities. Subsequently, Constitutional AI guides the model's alignment with human values by explicitly specifying ethical and behavioral principles.Gemini-pro is a multimodal model developed by Google AI, trained on a massive and versatile dataset to seamlessly process a combination of text, audio, and visual inputs. It is capable of generating outputs that integrate both images and text.Yi-6B-Chat and Yi-34B-Chat are pre-trained on 3 trillion tokens, with the training data updated through June 2023. Both of them are trained with 4K sequence length and can be increased to 32K tokens during inference.Yi-9B undergoes continuous training, building upon the Yi-6B model with 0.8T tokens.ChatGLM3-6B retains the foundational structure of its predecessors, introducing an newly designed prompt format to support function calls, code interpretation, and the handling of intricate scenarios such as agent tasks.Taiwan-LLM-7B-v2.1-chat is designed specifically for Traditional Chinese, emphasizing the unique linguistic and cultural aspects of Taiwan. Originating from an extensive foundational model, it incorporates a wide range of Taiwanese text sources and is enhanced through supervised fine-tuning.Chinese-Llama-2-7b enhances the original Llama 2 by undergoing additional pretraining with Chinese text data around 120G and incorporating a more extensive vocabulary of size 55,296 for broadening coverage. It accommodates a 4K context size, with the capability to extend beyond 18K through the application of the NTK method.Table 14 presents a comprehensive accuracy breakdown across numerous subjects for Qwen-72B, GPT-4, Qwen-14B, and Gemini-Pro.GPT-4 Qwen-14B Qwen-72B Gemini-pro engineeringPrevious work (Kojima et al., 2022; Wei et al., 2022) that prompt-based parsers often achieve higher scores compared to rule-based parsers. While a prompt-based parser is much more versatile than a rule-based parser as prompt response format may differ significantly between LLMs, due to cost consideration, this former method is not widely adopted as the default answer extraction method. In this section, we compare the difference between these two as shown in Table 16 . For the prompt-based parser, we utilize gemini-1.5-flash to extract the final answer. 問題：在氯氣(Cl2)和乙烷(ethane)的UV 照光的反應中，下面哪一步反應是傳播步驟 (propagation event)？ Question: In the UV-irradiated reaction between chlorine gas (Cl2) and ethane, which of the following steps is a propagation event? 問題：給定兩向量u = [1 1 2] T 及v = [2 -1 1] T ，下列選項何者錯誤？ Question: Given two vectors u = [1 1 2] T and v = [2 -1 1] T ,I) Cl • +CH 3 -CH 3 → CH 3 -CH 3 -Cl + H• II) Cl • +CH 3 -CH 3 → CH 3 -H 2 C • +HCl III) Cl • +CH 3 -H 2 C• → CH 3 -CH 2 -Cl IV) Cl 2 + CH 3 -H 2 C• → CH 3 -CH 2 -Cl + Cl• V) Cl 2 + UV light → C l • +Cl• A. II 與IV A. IIQuestion: There is a AC generator with a coil area of 0.03 square meters, a total of 20 turns of the coil, and rotates at a fixed speed of 600 revolutions per minute in a uniform magnetic field of 0.2 Tesla. What is the approximate maximum induced electromotive force of this generator in volts? 問題：核反應時，有10 -3 公克的質量損失可以轉換成多少熱能？ Question: In a nuclear reaction, how much heat can be transformed from a mass loss of 10 -3 grams? A. 9 × 10 9 卡 A. 9 × 10 9 cal B. 9 × 10 10 卡 B. 9 × 10 10 cal C. 2.14 × 10 10 卡 C. 2.14 × 10 10 cal D. 2.14 × 10 9 卡 D. 2.14 × 10 9 cal 答案：C Answer: C Figure 15: A "secondary physics" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.問題：取一質量10 kg 材質均勻的合金，將其分成兩塊，其中一塊製成一個邊長為10 cm 的實心正立方體，另一塊製成一個質量為2 kg 的實心球，則此實心球的體積應為多 少？ Question: Take a 10 kg uniform alloy, divide it into two pieces, one is made into a solid cube with a side length of 10 cm, and the other is made into a solid sphere with a mass of 2 kg. What should be the volume of this solid sphere? 問題：陸羽《茶經》論水云：「山水上，石泉又上，江水次而井水下。」 又「江水取去人遠者，井取汲多者。」其說止於此，未嘗品第天下之水味。 張又新《煎茶水記》云劉伯芻謂水有七等，以揚子江為第一，惠山石泉為第二，虎丘 井第三，淮水居末。 又載羽為李季卿論水次第有二十種，江水居山水上，井水居江水上。二說皆與《茶 經》不合。 水味有美惡而已，欲求天下之水一一而次第之者，謬說也。 羽之論水，惡渟浸１而喜泉源，故井取多汲者。江雖長，然眾水雜聚，故次山水。惟 此說近物理云。 --改寫自歐陽脩〈大明水記〉 根據本文，下列敘述何者最恰當？ Question: Lu Yu states in "The Tea Classic" that "mountain water is on top, followed by stone spring, with river water next and well water at the bottom." Also "river water is drawn from afar, and well water is drawn more. Hence, it ends here, he never tasted all the waters in the world. In Zhang Youxin's "Record of Boiling Tea Water", it says Liu Bochao classified water into seven categories, ranking the Yangtze River as the first, Huishan Stone Spring as the second, Tiger Hill Well as the third, and Huai River as the last. Also, it says, Lu Yu discussed with Li Jiqing that there were twenty types of water, with river water towering over mountain water, and well water standing above river water. These two theories contradict The Tea Classic. The taste of water is just good or bad, and it is absurd to rank all the waters in the world one by one. Lu Yu's discussion of water dislikes cloudy and stagnant water and prefers spring sources, so wells are drawn more. Though the river is long, it is a gathering of multiple waters, hence it is next to mountain water. This view is closer to physical reason. -Rewritten from Ouyang Xiu's "Record of Daming Water" According to the text, which of the following statements is most suitable?  


Decision: The paper presents a traditional Chinese benchmark, TMMLU+, covering 66 subjects with a total of 22,690 questions, along with an in-depth evaluation of many representative LLMs, offering valuable insights into their traditional Chinese performance. The reviewers were all supportive; this will be a very valuable benchmark.

Reviews: review: paper_summary: paper_summary: This paper introduces TMMLU+, an expanded and enhanced benchmark suite designed to assess language model performance on Traditional Chinese language understanding tasks. Compared to its predecessor, TMMLU, it offers a more comprehensive array of multiple-choice questions across a wider range of subjects, with improved balance in subject distribution. The authors conduct extensive analyses examining various model features, such as performance differences between simplified and traditional characters, the use of Chain-of-Thought (CoT), tokenizer vocabulary size, models and human performance.
strength_weakness: strength_weakness: The benchmark primarily focuses on knowledge-based questions, leading to significant overlap in terminology between Mainland Mandarin and Taiwanese Mandarin. While some questions reflect real language usage differences (e.g., Figure 8 in Appendix D and the Taiwanese Hokkien questions), they constitute a small portion of the overall dataset.
questions: questions: In Figure 8, the English prompt is not correctly translated.
paper_summary: paper_summary: The paper introduces TMMLU+, an enhanced Traditional Chinese Evaluation Suite designed to assess the capabilities of LLMs in traditional Chinese. The benchmark is divided into four categories with a total of 22,690 multi-choice questions. The paper conducts a thorough evaluation of several representative LLMs, such as close-source LLMs LLMs (e.g., chatGPT, GPT4) and many open-weight LLMs, including those tailored for simplified and traditional Chinese.
strength_weakness: strength_weakness: 1. The benchmark in this work has not well distinguished itself among the many existing benchmarks for Chinese. It shares similarities with benchmarks tailored for simplified Chinese, such as C-Eval.

2. The dataset's license should be explicitly specified in the paper.
questions: questions: 1. Appendix E lists many LLMs. Why Table 2 doesnot report the performance of all LLMs? Additionally, Figure 4 includes additional LLMs not featured in Table 2. In abstract it says 26 open-weight Chinese LLMs, it seems not ture.

2. How are examples chosen for the development and validation sets? Table 2 observes decline in performance for many LLMs with few-shot prompting. Is there a potential correlation with the examples in the development set?

3. LLMs exhibit significantly reduced performance under the CoT setting compared to without. What are the reasons behind?

4. How much money you spent on GPT family models?
paper_summary: paper_summary: This paper introduces TMMLU+, an improved benchmark dataset for evaluating the language understanding capabilities of large language models (LLMs) in Traditional Chinese. TMMLU+ consists of 22,690 multi-choice questions across 66 subjects, ranging from elementary to professional levels. The authors conduct experiments on 26 open-weight language models and several closed-source models, comparing their performance against human test-takers. The findings reveal that Traditional Chinese models still lag behind their Simplified Chinese counterparts, and current LLMs fall short of human performance in average scores, particularly in social science and humanities subjects.
strength_weakness: strength_weakness: - While the paper compares the performance of Traditional and Simplified Chinese language models, it does not provide the underlying reasons for the performance gap between the two. Further investigation into the linguistic and cultural differences between Traditional and Simplified Chinese could provide more insights. (especially for readers who are not familiar with the Chinese.)
- The process of creating the CoT demonstration has been overly simplified, and there are concerns about whether the quality has been adequately ensured. Additionally, there is no explanation for why the performance of CoT is lower compared to direct methods.

Comments: title: Thanks for pointing out weak points
comment: We are grateful for bAwS's keen observations on sections needing improvement, which will guide changes to our revisions.
Thank you for the clarifications. After considering the authors' response, I've modified my score accordingly.