Measuring Taiwanese Mandarin Language Understanding

Authors: po-heng  chen
sijia  cheng
wei-lin  chen
yen-ting  lin
yun-nung  chen
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 12:00:11

Abstract: The evaluation of large language models (LLMs) has drawn substantial attention in the field recently.
This work focuses on evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks.
We present TMLU, a comprehensive evaluation suit tailored for assessing the advanced knowledge and reasoning capability in LLMs, under the context of Taiwanese Mandarin.
TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels.
In addition, we curate chain-of-thought-like few-shot explanations for each subject to facilitate the evaluation of complex reasoning skills.
To establish a comprehensive baseline, we conduct extensive experiments and analysis on 24 advanced LLMs.
The results suggest that Chinese open-weight models demonstrate inferior performance comparing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chinese counterparts.
The findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Taiwanese-Mandarin LLMs.
We release the benchmark and evaluation scripts for the community to promote future research.

Main paper: 
 introduction: 
 The emergence of large language models (LLMs) has revolutionized the field of natural language processing (NLP). As the core that drives the trajectory of AI development, evaluation stands a crucial role in the era of LLMs. Conventional NLP benchmarks (Wang et al., 2018; 2019) have been widely adopted to evaluate natural language understanding (NLU) abilities in LMs. However, the applicability of these benchmarks has decreased as the ever larger models are demonstrating human-level performance, leaving little headroom for research developments (Hendrycks et al., 2020; Goyal et al., 2022; Liu et al., 2023) .Towards appropriately benchmarking LLMs, probing advanced world knowledge and measuring complex reasoning capabilities are the main focus LLM evalution in recent days (Clark et al., 2018; Hendrycks et al., 2021; Cobbe et al., 2021; Wang et al., 2023) . Additionally, evaluation benchmarks for languages beyond English have also been introduced (Li et al., 2023; Huang et al., 2023) , in parallel to the rise of multilingual LLMs (Le Scao et al., 2022; Muennighoff et al., 2022; Wei et al., 2023) and LLMs that are optimized for different regional languages (Lee et al.; Nguyen et al., 2023) .Specifically, a number of benchmarks (Huang et al., 2023; Li et al., 2023; Xu et al., 2023) have been proposed for assessing Chinese LLMs (Zeng et al., 2022; Yang et al., 2023; Bai et al., 2023; AI et al., 2024) . However, these benchmarks and models are all developed under the context of Simplified Chinese used in Mainland China. On the other hand, Traditional Chinese, often referred to as Traditional Mandarin, has been notably underrepresented. Predominantly used in Taiwan, Hong Kong, and Macao, Traditional Mandarin carries different language usage between countries or locations. Specifically, Taiwanese Mandarin posses linguistic nuances, cultural intricacies, and a different written form which diverge from Chinese used in China and present unique challenges (Wang & Tang, 2021; Lin & Chen, 2023) . For example, the word "土豆 (Tudou)" typically means "potato" in Mainland China, while it refers to "peanut" in Taiwan (Zhou & Zhou, 2019) . Consequently, the need of building benchmarks organically in Taiwanese Mandarin is necessitated to facilitate the development of localized, Taiwanese-Mandarin LLMs.In this work, we bridge the gap and present TMLU -a comprehensive evaluation suit tailored for assessing advanced knowledge and reasoning capability in LLMs under Taiwanese-Mandarin context. TMLU consists of a broad spectrum of 37 subjects, spanning social science, STEM, humanities, Taiwan-specific content, and others, from middle school to professional level. Including high school and college entrance exams, civil service exams, and other localized knowledge in Taiwan. Furthermore, TMLU includes manually curated, Chainof-Thought-inspired (Wei et al., 2022) explanations to facilitate the evaluation of reasoning capability in LLMs. See Figure 1 for an overview of TMLU.To establish a comprehensive baseline, we evaluate 24 advanced LLMs, including both open-weight and proprietary models, on TMLU. The experimental results indicate that, in general, proprietary black-box LLMs with multilingual capability outperform openweight models developed by Chinese communities. Also, open-weight models tailored for Taiwanese Mandarin demonstrate suboptimal scores comparing to models build predominantly in Simplified-Chinese background. Overall, the findings suggest that there still exist ample rooms for improvement, underscoring our goal to facilitate the progress of Taiwanese-Mandarin LLMs. In addition, we present a number of analysis and conduct data contamination test to further validate the reliability and robustness of TMLU. 
related work: 
 Prior to the bloom of LLMs, standard NLP benchmarks such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) have been widely adopted to evaluate linguistic proficiency and natural language understanding (NLU) abilities in LMs. However, the practicality of these benchmarks in the era of LLMs have saturated as new models are demonstrating Table 1 : A comparison of existing Taiwanese Mandarin evaluation benchmarks and our proposed TMLU. Notably, TMLU is superior to TC-Eval in the degree of localization, and is more robust to dataset contamination compared to TMMLU-plus. We present a detailed discussion in Section 2.superhuman performance (Goyal et al., 2022; Liu et al., 2023) . In light of this, benchmarks which focus on assessing the world knowledge and complex reasoning abilities in LLMs have been introduced recently (Hendrycks et al., 2020; Srivastava et al., 2022; Hendrycks et al., 2021; Suzgun et al., 2022; Liang et al., 2022) . The MMLU benchmark (Hendrycks et al., 2020) covers an array of subjects across STEM and social sciences, from elementary to expert levels, for comprehensive multitask evaluation. The BIG-bench benchmark (Srivastava et al., 2022) contains a large collection of diverse tasks that are designed to be beyond the current ability of LLMs. However, these benchmarks are primarily constructed in English, which limit their usage in assessing LLMs developed in the context of other languages.To this end, several benchmarks have been proposed to facilitate evaluation of Chinese LLMs. C-Eval (Huang et al., 2023) presents one of the first comprehensive evaluation suits built to assess LLMs' advanced knowledge and reasoning capabilities in Chinese context, with subjects across social science, STEM, and humanities. Another concurrent benchmark, CMMLU (Li et al., 2023) , shares a similar vision and includes questions relevant to Chinese users' culture and daily life, in addition to standardized examinations. However, these existing benchmarks are designed in Simplified Chinese, thus, unsuitable for evaluating LLMs developed in the context of Traditional Chinese.To the best of our knowledge, only two benchmarks-TC-Eval (Hsu et al., 2023) and TMMLU-plus (Tam et al., 2024) -have been developed towards the goal of evaluating LLMs' capabilities for Traditional Chinese. TC-Eval consists of a collection of existing datasets, mainly from conventional NLP tasks, e.g. reading comprehension, summarization, and sentiment classification (Shao et al., 2018; STPI, 2020; Narayan et al., 2018; Maas et al., 2011) , and a newly introduced, MMLU-like multiple-choice dataset, termed TMMLU. Recently, TMMLU-plus is proposed to focus on the assessment of advance knowledge in Taiwanese-Mandarin LLMs. Compared to TMMLU, TMMLU-plus is six times larger and contains questions spanning a broader range of subjects.Our proposed TMLU differs from previous benchmarks in the following ways. Firstly, all questions within TMLU are originally in Traditional Chinese. On the other hand, a sizeable portion of TC-Eval is translated from English datasets (Narayan et al., 2018; Maas et al., 2011; Srivastava et al., 2023) , which misaligns the goal of evaluating LLMs in the Taiwanese-Mandarin environment due to potential western geographical biases. For example, the XSum dataset (Narayan et al., 2018) adopted by TC-Eval is sourced from BBC articles, which are likely grounded in western economic and social context. Although TMMLU-plus is tailored specifically for Taiwanese Mandarin, we still observe instances in Simplified Chinese (Figure 6b ).Secondly, compared to TMLU, TMMLU-plus exhibits several potential issues that could compromise the reliability and robustness of the benchmark. In our preliminary investigation, we find that most questions in TMMLU-plus could be found on one single website. 2In addition, we observe the presence of the website's content in mC4 (Xue et al., 2021) , a widely adopted corpus for pre-training LMs. More quantitatively, we randomly sample 100 instances from TMMLU-plus and discover 91 of them are on the website. We also examine the full mC4 corpus and find 28,685 entries have URLs under the website. Overall, this greatly exposes its robustness to test data contamination (Sainz et al., 2023b; a; Golchin & Surdeanu, 2023; Shi et al., 2023) . To minimize such risks, we follow Huang et al. (2023) by collecting data sourced in PDF and Microsoft Word documents from the internet, instead of plain texts directly from the web. Moreover, TMMLU-plus contains unanswerable questions that require information (e.g., images or tables) not accompanied in the dataset (Figure 6b ).Lastly, TMLU provides two distinct features not available in TC-Eval and TMMLU-plus:(1) manual-constructed few-shot CoT demonstrations to elicit reasoning; (2) standardized mathematical expressions (e.g., symbols and equations) in L A T E X format for better clarity (Figure 6a ). In sum, our proposed TMLU is better for the community to objectively evaluate the model capability due to its better transparency3 , localization, and robustness. 
overview: 
 The TMLU benchmark is an evaluation suit tailored for assessing advanced knowledge and reasoning capability in LLMs under Taiwanese Mandarin, in the format of multiplechoice questions. TMLU contains a wide range of subjects spanning social science, STEM, humanities, Taiwan-specific content, and others, across middle school to professional levels.The goal of TMLU is to provide an evaluation suit, which is easy-to-use and accessible, for developers to gauge how their models would likely perform in the real-world Taiwanese Mandarin context.One of the main concerns of NLP evaluation in the age of LLMs is test data contamination (Sainz et al., 2023b; a; Golchin & Surdeanu, 2023; Shi et al., 2023) , sometimes referred to as data leakage. That is, the models are (pre-)trained on data that belongs to the test split of a benchmark and subsequently evaluate on the same test split, making the evaluation results questionable. As these is generally no guarantee of building contamination-free evaluation benchmark, especially given how little information about the training data are disclosed by proprietary models, mitigating the risk of contamination is equally important.To this end, we follow the design principle of Huang et al. 2023 5. Highway Bureau Questions Bank (HB) -the theory test questions from the Highway Bureau of Taiwan, including questions such as car and motorcycle driving regulations for the driving license test.All of our data is in the format of PDF or Microsoft Word document initially and processed into text format subsequently, by manual annotation with the aided of OCR toolkits. 5Mathematical expressions are standardized into L A T E X format for STEM subjects that involve substantial symbols and equations, following prior works (Hendrycks et al., 2021; Huang et al., 2023) . The produced L A T E X formulas are compiled and verified by annotators to ensure their correctness. In addition, all instances are validated by human for quality assurance to exclude unsuitable instances (e.g., unanswerable questions that require accompanied images or tables which are not presented in the text body).Our final resulting benchmark, TMLU, consists of a collection of 2,981 multiple-choice questions across 37 subjects, covering difficulty levels from middle school, high school, to professional. We categorize the subjects into five disciplines -social science, STEM, humanities, Taiwan specific, and other. Each subject is further divided into a development set consisting of five instances and a test set consisting of the remaining instances. Statistics of TMLU grouped by difficulty disciplines and levels are provided in Table 2 .To facilitate the evaluation of reasoning capability in Taiwanese-Mandarin LLMs, we manually construct Chain-of-Thought-like explanations for each development set instances. Specifically, the main source of our explanations (55.68%) are curated by leveraging the textbook website. 6 It provides corresponding textbook-level explanations for CAP, GSAT, and AST exam questions included in TMLU, in the format of PDF documents. Other sources for explanation curation include (1) question-and-answer websites or forum (26.49%);(2) GPT-4-preview (3.78%), where explanations are adopted only if the predicted answer is correct;(3) fully handcrafted (4.86%), which is composed by thoroughly searching the Web;(4) others (9.19%), where the exact source is either ambiguous or untraceable.  
experimental setups: 
 Models. We comprehensively benchmark 24 LLMs on TMLU. The adopted models are capable of processing Traditional-Chinese content and vary in size and developer organizations. Concretely, we conduct experiment on 6 closed-source, proprietary models via APIs, including models from OpenAI,7 Anthropic, and Google. For open-weight models, we focus on models developed for Chinese, English, and multilingual purposes. This included Taiwan-LLM (Lin & Chen, 2023) and Breeze (Hsu et al., 2024) for Taiwanese Mandarin, as well as Yi (AI et al., 2024) and Qwen (Bai et al., 2023) for Simplified Chinese. For a complete list of the models evaluated, please refer to Table 3 .Few-shot Evaluation. The evaluation is conducted in few-shot with setups: direct answer and CoT. For each subject, the five instances from the development split are utilized as demonstrations (i.e., five shots). The adoption of few-shot evaluation is consistent with benchmarks such as C-Eval (Huang et al., 2023) and MMLU (Hendrycks et al., 2020) , and the convention in prior works (Touvron et al., 2023; Team et al., 2023; Achiam et al., 2023) for measuring LLM performance. The setting is believe to capture the inherent, underlying potential of LLMs more robustly and extrapolate to downstream task adaptation (Huang et al., 2023) .The evaluation prompt consists of the following components: the problem definition, a set of few-shot examples, and the input instance to query. For instruction-tuned (chat-oriented) models, we apply the accompanied chat templates if available. Compared with direct answer, CoT prompting includes an explanation (i.e., rationale) triggered with "Let's think step by step", before the final answer prediction. Example prompts with TMLU instances are provided in Figure 2 (direct answer) and Figure 7 (CoT).Answer Extraction. As show by Fourrier et al., 2023 , different flavors of implementation in extracting predictions from models bring nontrivial impact to the evaluation results.To ensure a fair evaluation and address variations in model accessibility and prompting settings, we adopt two methods for answer extraction: (1) Likelihood-based method. The method aligns with the original implementation in MMLU (Hendrycks et al., 2021) , which involves examining a set of candidate answer symbols (i.e., {"A", "B", "C", "D"}) and selecting the symbol with the highest assigned token probability as the model's prediction.(2) Generation-based method. When direct determination of candidate token probabilities is impractical, we opt for the first token generated greedily that can be interpreted as an option code to derive the answer prediction.For proprietary models, as access to the probability of each candidate answer symbol is restricted, we utilize the generation-based method for both direct and CoT prompting scenarios. In the case of open-source models, we employ the likelihood-based method for direct prompting. However, when employing CoT prompting, where the model must first generate an explanation for the queried instance, we are constrained to using the generationbased method. In instances of CoT prompting, we adjust the number of shots based on the model context length to avoid failure to produce the answer.The experimental results are presented in Table 3a and Table 3b for direct answer and CoT prompting, respectively. In general, flagship proprietary models demonstrate performance superior to open-weight counterparts, and models trained on Chinese data (i.e., Simplified Chinese, Taiwanese Mandarin, and Multilingual models) exhibit substantial improvement over models focusing on English context. For direct answer prompting, the Breeze modeltailored for Taiwanese Mandarin with only 7 billions parameters -outperforms GPT-3.5. The top-performing open-weight model Yi, with 34 billions parameters, outperforms proprietary models such as Gemini-pro and Claude-instant, and is comparable to GPT-4.In CoT prompting, proprietary models outperform the best open-source model by a significant margin of 7%∼27%. The results suggest an ample opportunity for enhancing reasoning capabilities via CoT prompting in open-source models. We provide more analysis between direct answer and CoT prompting in Section 5.In sum, the established baseline suggests that TMLU offers a proper ground, with headroom for improvements, for evaluating LLMs in the context of Taiwanese Mandarin. Furthermore, the exhibited performance of Taiwanese-Mandarin LLMs still lack behind the Simplified-Chinese counterparts, substantiating the goal of TMLU to foster the development of localized, Taiwanese-Mandarin LLMs in the future. 
robustness to test data contamination: 
 To further investigate the possibility of test data contamination mentioned in Section 2 and 3.2, we employ MIN-K% PROB (Shi et al., 2023) , a reference-free method for detecting pre-training data from LLMs. Based on the hypothesis that an example seen by the model before (i.e., during pre-training) is less likely to include words with low probability (i.e., high negative log-likelihood), MIN-K% PROB selects a set of k% of tokens from the input text with minimum token probability and average their log-likelihood as an indicator of whether the input text is in the pre-training data. Specifically, given an input example x = (x 1 , x 2 , ..., x n ), where Min-K(x) is the set of k% of tokens in x with the minimum token probability. The lower the MIN-K% PROB is, the more likely that the input example x has been seen during pre-training.MIN-K% PROB(x) = 1 |Min-K(x)| ∑ x i ∈Min-K(x) -log p(x i |x 1 , x 2 , ..x i-1 )Taiwan-LLM-7B-base Breeze-7B-Base Yi-6B Qwen1.5-7B Llama-2-7B Mistral-7B (Shi et al., 2023) of six base models on TMMLU-plus (Tam et al., 2024) and our dataset TMLU. The lower the MIN-K% PROB is, the more likely the input instances of the datasets are in the model's pre-training data.We apply MIN-K% PROB on 2000 sampled instances from TMLU and TMMLU-plus with 6 different base models. We present results in Table 3 and discuss more implementation details in Appendix B. As shown, TMLU achieves higher MIN-K% PROB than TMMLU-plus across all tested models, which implies that TMLU is less likely to contain data that has been seen during pre-trained. In addition, the finding also aligns with our hypothesis that TMMLU-plus might be collected by crawling plain text from the web, and further validates the effectiveness of our design principle (Section 3.2) that sourcing data from downloaded documents could reduce the risk of data contamination.To further investigate the potential capability of LLMs, we leverage our curated explanations and employ CoT prompting. CoT prompting elicits step-by-step reasoning chains towards answer derivations, which has been shown to improve tasks that require complex, multihop reasoning significantly. We present results comparing CoT with standard answer-only prompting in Figure 4 for STEM subjects. Full results are reported in Table 9 . As shown, models that are able to benefit from CoT are mainly large, proprietary ones, exemplified by GPT-4 (15%) and Claude-2 (23%). The finding may be attribute to the emergent nature of reasoning capability in LLMs. Interestingly, Taiwan-LLM-7B-Chat is the only model exhibiting improvement with CoT at its parametric scale. Here we investigate the performance of models on the temporal dimension. We group questions from different years as subsets and evaluate the average accuracy, as presented in Figure 5 . Questions from before 2017 (2013∼2016) are group with 2017 as one subset. The results show that, for most models, no consistent trends or behaviors are exhibited. Yet, for GPT-4-turbo and Yi-34B-Chat, the scores imply a rising trend where models perform better on questions from more recent years. Future endeavors could offer a more precise implication and rigorous explanation for the results. 
conclusion: 
 In this work, we introduce TMLU -an evaluation suite designed to assess the advanced knowledge and reasoning abilities of LLMs in the context of Taiwanese Mandarin. Experimental results and analysis suggest great opportunities for future developers, in particular, for open-source models. Besides addressing the scarcity of benchmark for Traditional Chinese community, we envision TMLU to serve as a grounded testbad, fostering development of localized Taiwanese-Mandarin LLMs.[TMLU] -The instance from "AST physics" test split. Here choices are omitted for brevity. Figure 6a shows a same instance from TMLU (AST physics) and TMMLU-plus (physics test), where symbols are formatted in latex in TMLU but are not formatted in TMMLU-plus, effecting the clarity. Figure 6b shows instances from TMMLUplus (junior chemistry test and junior social studies test) which are in Simplified Chinese or unanswerable since it requires information from an image/table that is not accompanied in the question.  
 
 Appendix: https://github.com/MiuLab/TMLUhttps://yamol.tw/The source of data collection is not disclosed in the TMMLU-plus paper.The sourced website links are provided in Table4.ExperimentsWe conduct evaluation on an array of advanced LLMs on TMLU, establishing a baseline for research and community reference. Following we describe the experimental details and results.https://mathpix.com/https://www.ehanlin.com.tw/app/index.htmlWe adopt gpt-4-0125-preview and gpt-3.5-turbo-1106. 


Decision: This paper introduces a benchmark to evaluate the performance of language models in understanding Taiwanese Mandarin, which is an important task, as all reviewers (including me) found, to improve inclusivity with the rapid development of large language model techniques, as well as evaluating a number of models on the proposed benchmark.
I resonate with Reviewer 8Aze that, in principle, it would be good to include some diagnostic test cases that better represent the linguistic and/or cultural differences between Chinese and Taiwanese Mandarin, especially when adapting a Simplified Chinese model.
That being said, the benchmark is still very valuable for localizing language models into Taiwanese Mandarin.

Please also follow the suggestions by Reviewers u2Ve and 8Aze to correct the Chinese Mandarin terms with misunderstanding.

Reviews: review: paper_summary: paper_summary: This paper proposes a benchmark for traditional Chinese, TMLU, which targets the Taiwan society by construction. This benchmark is like the MMLU format but with traditional Chinese language. Compared with previous benchmarks for Taiwanese Mandarin such as TC-Eval and TMMLU, the authors argue that TMLU differs in that it originates from traditional Chinese and it is sourced from PDF or Word documents that suffer from less contamination. The paper benchmarks many LLMs on the proposed TMLU benchmark.
strength_weakness: strength_weakness: 1. While the authors try to distinguish TMLU from TMMLU-plus, it is better to have more quantitative evidence given that the two benchmarks are similar. For example, Table 1 notes that the robustness, transparency of TMMLU-plus are low, what do they mean? In Section 2 the authors mention that “In our preliminary investigation, we found that most questions in TMMLU-plus could be found on one single website.”, can you quantify this? How many can be found?  
2. In figure 3, I wonder what the absolute values of the y-axis mean, for example, does a min-k% prob of 4.0 indicate that the pretraining data is absolutely likely to present in the pretraining data? Currently the comparison in Figure 3 is relative, and I am not sure whether the gap between TMLU and TMMLU-plus is significant or has meaningful differences given that the gap looks small – maybe TMLU and TMMLU-plus both suffer from data contamination already.
questions: questions: ```
Understanding that TMMLU-Plus is a concurrent work within one month of the submission, I increase my score from 4 to 6 given that I ignore the existence of TMMLU-Plus
```
paper_summary: paper_summary: This paper introduces TMLU, a new evaluation benchmark designed to assess large language models (LLMs) specifically in the context of Taiwanese Mandarin. TMLU addresses the unique linguistic nuances and cultural intricacies of Taiwanese Mandarin, differentiating it from other varieties of Chinese. The benchmark encompasses 37 subjects, spanning middle school to professional levels, and includes approximately 3,000 multiple-choice questions across diverse disciplines. The authors evaluate 24 LLMs on TMLU and include Chain-of-Thought (CoT) examples to assess complex reasoning skills.
strength_weakness: strength_weakness: 1. Limited Linguistic and Cultural Differentiation: While the benchmark is written in Traditional Chinese, the questions themselves do not sufficiently capture the unique linguistic and cultural aspects of Taiwanese Mandarin. Many STEM questions, for example, use terminology common to Mainland China. Areas where language usage differences are more pronounced, like social media, are not included. This limitation may explain why Simplified Chinese models perform well on the benchmark.
2. Unclear Discussion and Unconvincing Conclusions: The paper's discussion lacks clarity, and some conclusions are not well-supported. For instance, the authors claim advantages over TMMLU plus, but only provide a few examples without statistical evidence.
3. Limited Innovation: The benchmark construction and analysis are similar to previous work, and the exclusive use of multiple-choice questions limits innovation. The paper would be stronger with a more novel approach to dataset construction and the inclusion of diverse test types.
4. Data Contamination Concerns: While the authors strive to avoid data contamination, their use of materials from major tests in Taiwan raises concerns. These materials may already be available online in digital form on test preparation websites, potentially accessible to LLMs during training. The observed decline in model performance on older questions (pre-2017) could indicate data contamination, as older materials are less likely to have been digitized and disseminated online.
questions: questions: Figure 2 contains an error in the answer key (the correct answer for the last question is "C"). 
The example using "Tongzhi" to illustrate language differences is not the most effective, as its primary meaning is consistent across Mainland China and Taiwan (referencing Zhang et al. 2022).
Zhang, K., Lu, C., & Zhang, J. (2022). Chinese media representations of tongzhi (2009–2019). Discourse & Communication, 16(3), 305-325.
paper_summary: paper_summary: This paper describes TMLU, a new MMLU-style dataset for LLM evaluation, targeted specifically at Taiwanese Mandarin (and the traditional Chinese script). As with MMLU and other datasets fashioned around it, the source of the data for TMLU is exam questions from middle and high school, and professional topics (e.g. traditional Chinese medicine or driving exams), with a strong focus on scraping data from PDFs and MSWord documents in the hope that it is less likely to be included in the pre-training data for LLMs. In addition to evaluating a range of open-weight and proprietary LLMs over TMLU, the paper includes some nice analysis of data contamination, direct answering vs. COT prompting, and a breakdown of results across time (based on the timestamps of the documents the questions were sourced from).
strength_weakness: strength_weakness: - not a strong reason to reject, but I would have liked to have seen more comparison of the coverage of topics in the dataset with related datasets such as TMMLU-plus and CMMLU
- I would also have liked to have seen more discussion/analysis of results over "Taiwan specific" questions (see below), and also between similar topics in CMMLU and other datasets for Mandarin Chinese
- the writing is overall good, but there were some places where things got tangled up, particularly the first sentence of Section 4.1 which should be changed  to "We perform comprehensive evaluation over TMLU using 24 different LLMs ..." (you are not evaluating TMLU itself, but rather using TMLU to evaluate LLMs); also "wide range of field*s*" (Section 3.2); "Tradition*al* Chinese (Section 2)
questions: questions: - overall, models appear to achieve slightly lower results over TMLU than CMMLU, e.g., but how much of this is attributable to script differences (traditional vs. simplified) vs. the Taiwanese/Mandarin Chinese distinction vs. "local knowledge"? For example, if you convert TMLU into the simplified script, how much do the results change? How do you explain the fact that results over "Taiwan specific" questions are actually higher than all other question types?
- in what sense is TMLU "holistic" (page 8)? I get that it is broad coverage, but is there a way to quantify the degree of comprehensiveness of the dataset (e.g. relative to MMLU or CMMLU)?
- more of a comment than a question, but I found myself getting confused at one point about what TMLU was as compared to TMMLU-plus (i.e. given the similarity between "TMLU" and "TMMLU", I assumed that TMMLU-plus was somehow an extended version of TMLU and tried to go back and find mention of how it had been expanded); perhaps remind the reader in Section 5.1 that they are completely different datasets, despite the similarity in name
- what is the basis for you saying that "the applicability of these benchmarks has decreased as the ever larger models are demonstrating human-level performance"? yes, models are improving, but there is a bit of circularity in the slightly awkward wording here, as "human level performance" is being claimed in large part due to improvement over benchmarks, but at the same time, data contamination is becoming a bigger and bigger issue; possibly nuance the claim a bit
paper_summary: paper_summary: The paper presents a new evaluation dataset (TMLU) for traditional Mandarin Chinese (Taiwanese). 
It contains 37 subjects (social science, STEM, humanities and Taiwan specific). The contributions of this eval dataset and this paper are:

It aims at better evaluating LLMs on traditional Chinese, which has unique characteristics and challenges compared to simplified Chinese 
Compared to existing traditional Chinese evaluation datasets (TC-eval and TMMLU-plus), TMLU has the following advantages: 
1. not from a single website, using pdfs or words, not easily scrapable so avoiding **data contaminations**;
2. more answerable from the question context;
3. including CoT instructions and Latex expressions
strength_weakness: strength_weakness: Although authors show the dataset has advantages compared to existing benchmarks, I hope the dataset can do even better. In particular, I hope the eval dataset can cover the following capabilities: 
1. Chatability: being able to understand a dialogue context 
2. Understanding nuances from languages 
3. Other use cases in addition to exams 

Please see my questions below.
questions: questions: 1. From my understanding “同志” can mean “comrade” and “homosexual” depending on the context even in simplified Chinese. I still agree with the point that words carry different meanings in simplified and traditional Chinese, and the challenge do exist for LLM evaluations, but this may not be the best example to showcase in the Introduction section? 
2. If I understand correctly, the data comes exclusively from examinations (high school, college or professional), that may impose some bias into this dataset in a way that we are encouraging LLMs to be good at answering those “non-daily” questions, but does not necessarily make models more “usable” at answering “daily” questions. So it mainly measures knowledge, not necessarily other aspects of language understanding, such as understanding the nuances in languages and LLMs’ usability and “chattability”. Think about people’s daily usage of LLMs, they may ask questions about trip planning, shopping ideas, financial knowledge, legal advice, emotional support, relationship questions, career advice, business knowledge, etc. I would like to see those covered as well. This could be out of the scope of this paper, though. I hope we will have more such kind of evaluation data in the future. 
3. Figure 3 shows that TMLU is less likely to be included in training data of these benchmarked LLMs, but not by a big margin. What are your takeaways on these results?

Comments: comment: Thank you for the insightful comments! Regarding localization — the Taiwanese Trivia question set (TTQA) included in TMMLU-plus is proposed by another prior work [1]. We are also glad to augment TMLU with TTQA in future revision, along with more questions such as Hokkien knowledge to improve the localization of TMLU. Thanks again for your suggestions.

[1] Extending the pre-training of bloom for improved support of traditional chinese: Models, Methods and Results
Ennen et al. arXiv:2303.04715, 2023
Thanks for the constructive feedback! We will strengthen our objective evidence by providing more relevant samples in the final version.
Thank you for pointing out that TMMLU-plus is a concurrent work; I have updated the score accordingly.
However, I am still not entirely convinced by the claim that "TMLU is better for the community to objectively evaluate the model capability due to its better transparency, localization, and robustness." While I admit that TMLU has better data quality, as the authors stated, I am not persuaded regarding its localization and robustness. I noticed that TMMLU-plus includes tests on Taiwanese Trivia and Taiwanese Hokkien, which could cover more local language use, particularly informal language.
Thank you for the clarifications! Understanding that TMMLU-plus is a concurrent work, I think the contribution of this work is more significant. However, I do suggest that when discussing some drawbacks of TMMLU-plus, it is more convincing to use objective evidence than subjective judgements, for example, if we talk that TMMLU-plus contains questions that require additional images/tables, it is advised to report a percentage maybe within only 50 examples, otherwise this point is not very meaningful. 
I increase my score to 6 given that I ignore the existence of TMMLU-plus.