Measuring Taiwanese Mandarin Language Understanding
Authors: Po-Heng Chen
Sijia Cheng
Wei-Lin Chen
Yen-Ting Lin
Yun-Nung Chen
Venue: COLM (2024)
Keywords: Large Language Models, Taiwanese Mandarin, Traditional Chinese, Benchmark
TL;DR: A comprehensive benchmark for evaluating advanced knowledge and reasoning capability in LLMs under the context of Taiwanese Mandarin.

Abstract: The evaluation of large language models (LLMs) has drawn substantial attention in the field recently.
This work focuses on evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks.
We present TMLU, a comprehensive evaluation suit tailored for assessing the advanced knowledge and reasoning capability in LLMs, under the context of Taiwanese Mandarin.
TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels.
In addition, we curate chain-of-thought-like few-shot explanations for each subject to facilitate the evaluation of complex reasoning skills.
To establish a comprehensive baseline, we conduct extensive experiments and analysis on 24 advanced LLMs.
The results suggest that Chinese open-weight models demonstrate inferior performance comparing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chinese counterparts.
The findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Taiwanese-Mandarin LLMs.
We release the benchmark and evaluation scripts for the community to promote future research.

Published as a conference paper at COLM 2024
Measuring Taiwanese Mandarin Language Understanding
Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, Yun-Nung Chen National Taiwan University, Taipei, Taiwan {ytl, yychenJ@iese...
Abstract
The evaluation of large language models (LLMs) has drawn substantial attention in the field recently. This work focuses evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks. We present TMLU, compre- hensive evaluation suit tailored for assessing the advanced knowledge and reasoning capability LLMs, under the context of Taiwanese Mandarin. TMLU consists of an array of 37 subjects across social science, STEM, hu- manities, laiwan-specifie content, and others, ranging from middle school to professional levels. In addition, we curate ofain-of-thought explanations for each subject to facilitate the evaluation of complex reason- ing skills. To establish comprehensive baseline, conduct extensive experiments and analysis on 24 advanced LLMs. The results suggest that Chinese open-weight models demonstrate inferior performance com compar- ing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chines findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Tainanise-Mandarrn LLMs. We release the benchmark and evaluation scripts for the community to promote future research.
few-shot
counterparts. The
1 Introduction
The emergence of large language models (LLMs) has revolutionized the field of natural language processing (NLP). As the core that drives the trajectory of Al development, evalu- ation stands crucial role in the era of LLMs. Conventional NLP benchmarks (Wang et al., 2018; 2019) have been widely adopted to evaluate natural language understanding (NLU) abilities in LMs. However, the applicability of these benchmarks has decreased as the ever larger models are demonstrating human-level performance, leaving little headroom for research developments (Hendrycks et al., 2020; Goyal et al., 2022; Liu et al., 2023).
Towards appropriately benchmarking LLMs, probing advanced world knowledge and measuring complex reasoning capabilities are the main focus LLM evalution in recent days (Clark et al., 2018; Hendrycks et al., 2021; Cobbe al., 2021; Wang et al., 2023). Addi- tionally, evaluation benchmarks for languages beyond English have also been introduced (Li et al., 2023; Huang et al., 2023), in parallel to the rise of multilingual LLMs (Le Scao et al., 2022; Muennighoff et al., 2022; Wei et al., 2023) and LLMs that optimized for different regional languages (Lee et al.; Nguyen et al., 2023).
Specifically, number of benchmarks (Huang et al., 2023; Li et al., 2023; Xu et al., 2023) have been proposed for assessing Chinese LLMs (Zeng et al., 2022; Yang et al., 2023; Bai et al., 2023; Al et al., 2024). However, these benchmarks and models are all developed under the context of Simplified Chinese used in Mainland China. On the other hand, Traditional Chinese, often referred as Traditional Mandarin, has been notably underrepresented. Predominantly used in Taiwan, Hong Kong, and Macao, Traditional Mandarin carries different language usage between countries locations. Specifically, Taiwanese Mandarin
https://githhb. http://ggchbb..co///aaaaa/////

---

Published as a conference paper at COLM 2024
Humanities
STEM
Tour Guide / 153
Leador
High School Biology/ / 88
Lawyer Quellfiaation 18
High Chamitry/ 184-1222
High School Chinese 18
High School Matheratioss 784-882
High School History/ 78FF-85
High School Physics / 784433322
Middle School Chiness / (S XC
High School Earth Science/ /8 18:333388
Middle School History/ 88+P-55 92
Middle School Blology/ E9
Middlle School Chemistry/ (1
Middle School Matheratiiss MI +
Middle Physics/
Middle School Earth Science/ 40
Taiwan Specific
TMLU
Driving Rule (6
Teacher Qualification R20778
Tourist Resources/ A
Social Science
Accountant 92:39
Clinical Psychologis: / BLFEC:335
Other
High School 88
High School Geography 16
Basic Traditional Chinese Medicine / + M1
Middle School Civice M
Clinical Traditional Chinese Medicine 8
Midale School Geography/ (3:
Nutritorist/ / A 98
Figure 1: An overview of our proposed TMLU benchmark. TMLU consists of 37 subjects across middle school, high school and professional levels. In addition, TMLU includes set of Taiwan specific questions in which answers are unique to Taiwanese culture.
posses linguistic nuances, cultural intricacies, and different written form which diverge from Chinese used in China and present unique challenges (Wang & Tang, 2021; Lin & Chen, 2023). For example, the word " (Tudou)" typically means "potato" in Mainland China, while refers "peanut" in Taiwan (Zhou & Zhou, 2019). Consequently, the need of building benchmarks organically in Taiwanese Mandarin is necessitated to facilitate the development localized, Taiwanese-Mandarin LLMss
In this work, we bridge the gap and present TMLU - comprehensive evaluation suit tai- lored for assessing advanced knowledge and reasoning capabílity in LLMs under Taiwanese- Mandarin context. TMLU consists of a broad spectrum of 37 subjects, spanning social science, STEM, humanities, Taiwan-specific content, and others, from middle school to professional level. Including high school and college entrance exams, civil service exams, and other localized knowledge in Taiwan. Furthermore, TMLU includes manually curated, Chain- of T--Thought-inspiredd (Wei et al., 2022) explanations to facilitate the evaluation of reasoning capability in LLMss. See Figure for an overview of TMLU.
To establish comprehensive baseline, open-weight and proprietary models, on TMLU. The experimental results indicate that, in general, proprietary black-box LLMs with multilingual capability outperform open- weight models developed by Chinese communities. Also, open-weight models tailored for Taiwanese Mandarin demonstrate suboptimal scores comparing to models build pre- dominantly in Simplified-Chiness background. Overall, the findings suggest that there still exist ample rooms for improvement, underscoring our goal facilitate the progress of Taiwanese-Mandard LLMs. In addition, we present a number of analysis and conduct data contamination test to further validate the reliability and robustness of TMLU.
evaluate 24 advanced LLMs, including both
2 Related Work
Prior to the bloom of LLMs, standard NLP benchmarks such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) have been widely adopted to evaluate linguistic proficiency and natural language understanding (NLU) abilities in LMs. However, the practicality of these benchmarks in the era of L.Mss have saturated new models are demonstrating
2

---

Published as a conference paper at COLM 2024
TC-Eval TMMLU-plus
TMLU (ours)
Robustness
medium low high
Localization
low high
high
Coverage
low
high
medium
Transparency
high
low
high
Standardization
high
medium
high
Table 1: A comparison of existing Taiwanese Mandarin evaluation benchmarks and our proposed TMLU. Notably, TMLU is superior to TC-Eval in the degree of localization, and is more robust to dataset contamination compared to TMMLU-plus. We present a detailed discussion in Section 2.
superhuman performance (Goyal et al., 2022; Liu et al., 2023). In light of this, benchmarks which focus assessing the world knowledge and complex reasoning abilities in LLMs have been introduced recently(Hendrrcks et al., 2020; Srivastava et al., 2022; Hendrycks et al., 2021; Suzgun et al., 2022; Liang et al., 2022). The MMLU benchmark (Hendrycks et al., 2020) covers an array of subjects across STEM and social sciences, from elementary to expert levels, for comprehensive multitask evaluation. The BIG-bench benchmark (Srivastava et al., 2022) contains a large collection of diverse tasks that are designed to be beyond the current ability of LLMs. However, these benchmarks are primarily constructed in English, which limit their usage in assessing LLMs developed in the context of other languages.
To this end, several benchmarks have been proposed facilitate evaluation of Chinese LLMs. C-Eval (Huang et al., 2023) presents one of the first comprehensive evaluation suits built assess LLMs' advanced knowledge and reasoning capabilities in Chinese context, with subjects across social science, STEM, and humanities. Another concurrent benchmark, CMMLU (Li et al., 2023), shares similar vision and includes questions relevant to Chinese users' culture and daily life, in addition to standardized examinations. However, these existing benchmarks are designed in Simplified Chinese, thus, unsuitable for evaluating LLMs developed in the context of Traditional Chinese.
To the best of our knowledge, only two benchmarks-TC-EVv (Hsu et al., 2023) and TMMLU-plus (Tam et al., 2024)-hayy been developed towards the goal of evaluating LLMs' capabilities for Traditional Chinese. TC-Eval consists of a collection of existing datasets, mainly from conventional NLP tasks, e.g. reading comprehension, summariza- tion, and sentiment classification (Shao et al., 2018; STPI, 2020; Narayan et al., 2018; Maas et al., 2011), and a newly introduced, MMLU-like multiple-choice dataset, termed TMMLU, Recently, IMMLU-plus is proposed to focus on the assessment of advance knowledge in Haiwanese-Mandari LLMs. Compared TMMLU, TMMLU-plus is six times larger and contains questions spanning broader range of subjects.
Our proposed TMLU differs from previous benchmarks in the following ways. Firstly, all questions within TMLU are originally in Traditional Chinese. On the other hand, sizeable portion of TC-Eval is translated from English datasets (Narayan et al., 2018; Maas et al., 2011; Srivastava et al., 2023), which misaligns the goal of evaluating LLMs in the Taiwanese- Mandarin environment due to potential western geographical biases. For example, the XSum dataset (Narayan et al., 2018) adopted by TC-Eval is sourced from BBC articles, which are likely grounded in western economic and social context. Although TMMLU-plus is tailored specifically for Taiwanese Mandarin, we still observe instances in Simplified Chinese (Figure 6b).
Secondly, compared to TMLU, TMMLU-plus exhibits several potential issues that could compromise the reliability and robustness of the benchmark. In our preliminary investiga- tion, we find that most questions in TMMLU-plus could be found on one single website.? In addition, we observe the presence of the website's content in mC4 (Xue et al., 2021), widely adopted corpus for pre-training LMs. More quantitatively, we randomly sample 100 instances from TMML.U-plus and discover 91 of them are on the website. We also examine the full mC4 corpus and find 28,685 entries have URLs under the website. Overall, this
2nttps://yamo.. tu/
3

---

Published as a conference paper at COLM 2024
Subjects
# Instance (%)
group by discipline
Social Science
589 (19.76%)
STEM
14
468 (15.70%)
Humanities
1,009 (33.85%)
Taiwan Specific
557 (18.69%)
Others
358 (12.01%) %)
group by level
Middle School
434 (14.56%)
High School Professional
17
875 (29.35%)
11
1,672 (56.09%)
Total
37
2,981 (100.00%)
Table 2: Statistics of our proposed TMLU benchmark grouped by different categories of discipline and difficulty levels.
greatly exposes its robustness to test data contamination (Sainz et al., 2023b;a; Golchin Surdeanu, 2023; Shi et al., 2023). To minimize such risks, we follow Huang et al. (2023) by collecting data sourced in PDF and Microsoft Word documents from the internet, instead of plain texts directly from the web. Moreover, TMMLU-plus contains unanswerable questions that require information (e.g,, images or tables) not accompanied in the dataset (Figure 6b).
Lastly, TMLU provides two distinct features not available in TC-Eval and TMMLU-plus: (1)
few-shot CoT demonstrations to elicit reasoning; (2) standardized mathematical expressions (e.g., symbols and equations) in 18TEX format for better clarity (Figure 6a). In sum, our proposed TMLU better for the community to objectively evaluate the model capability due to its better transparency", localization, and robustness.
TMLU Benchmark
3.1
Overview
The TMLU benchmark is an evaluation suit tailored for assessing advanced knowledge and reasoning capability in LLMs under Taiwanese Mandarin, in the format of multiple- choice questions. TMLU contains a wide range of subjects spanning social science, STEM, humanities, Taiwan-specific content, and others, across middle school to professional levels. The goal of TMLU to provide an evaluation suit, which is easy-to-use and accessible, for developers to gauge how their models would likely perform in the real-world Taiwanese Mandarin context.
3.2
Data Source
One of the main concerns of NLP evaluation in the age of LLMs is test data contamina- tion (Sainz al., 2023b;a; Golchin & Surdeanu, 2023; Shi al., 2023), sometimes referred to as data leakage. That is, the models are (pre-)trained on data that belongs to the test split of a benchmark and subsequently evaluate on the same test split, making the evaluation results questionable. As these is generally no guarantee of building contamination-fia evaluation benchmark, especially given how little information about the training data are disclosed by proprietary models, mitigating the risk of contamination is equally important.
To this end, we follow the design principle of Huang et al. 2023 and collect data that is sourced in the format of PDF or Microsoft Word documents, which are files downloaded from the website instead of readily available within website plain text. Specifically, we source our data from the following five websites from T
'The source data collection is not disclosed in the TMMLU-plus paper.
4The sourced website links provided in Table
4

---

Published as a conference paper at COLM 2024
1. Comprehensite Assessment Program for Junior High School Students (CAP) - the stan- dardized exam for Taiwanese 9-th grade students before they go to high schools or vocational schools.
2. General Scholastic Ability Test (GSAT) the Taiwanese university entrance exam, focusing on fundamental knowledge and skills from the required materials in the first two years of students' high school studies.
3. Advanced Subjects Test (AST) - the other part of the Taiwanese university entrance exam, focusing advanced knowledge of individual subjects to assess students' comprehension, reasoning, and analysis abilities. The scope covers all materials in the high school studies.
4. Ministry of Examination Exams (MOEX) the exams from Ministry of Examination of Taiwan consists of two main categories, civil service exams and professional and technical exams, spanning wide range of fields such as law, finance, medical, and pharmacy.
5. Highway Bureau Questions Bank (HB) the theory test questions from the High- way Bureau of Taiwan, including questions such as car and motorcycle driving regulations for the driving license test.
3.3 Data Processing
All of our data is in the format of PDF or Microsoft Word document initially and processed into text format subsequently, by manual annotation with the aided of OCR toolkits. 5 Mathematical expressions are standardized into ITEX format for STEM subjects that involve substantial symbols and equations, following prior works (Hendrycks et al., 2021, Huang et al., 2023). The produced EXTEX formulas are compiled and verified by annotators to ensure their correctness. In addition, all instances are validated by human for quality assurance to exclude unsuitable instances (e.g., unanswerable questions that require accompanied images or tables which are not presented in the text body).
Our final resulting benchmark, TMLU, consists of collection of 2,981 multiple-choice questions across 37 subjects, covering difficulty levels from middle school, high school, to professional. We categorize the subjects into five disciplines social science, STEM, humanities, Taiwan specific, and other. Each subject is further divided into development set consisting of five instances and a test set consisting of the remaining instances. Statistics of TMLU grouped by difficulty disciplines and levels are provided in Table 2.
3.4 Explanation Curation
To facilitate the evaluation of reasoning capability in Taiwanese-Mandarin LLMs, we man- explanations for each development set instances.
ually construct Dhain-of-Thought-lle Specifically, the main source of our explanations (55.68%) are curated by leveraging the textbook website. 6 It provides corresponding textbook-level explanations for CAP, GSAT, and AST exam questions included in TMLU, in the format of PDF documents. Other sources for explanation curation include (1) question-and-answer websites or forum (26.49%); (2) GPT-4-preview (3.78%), where explanations are adopted only if the predicted answer is correct; (3) fully handcrafted (4.86%), which is composed by thoroughly searching the Web; (4) others (9.19%), where the exact source is either ambiguous or untraceable.
4 Experiments
We conduct evaluation on an array of advanced LLMs on TMLU, establishing a baseline for research and community reference. Following we describe the experimental details and results.
"nttps://nathpix.co com/ htttps://ww.eehr www.wwwwweaannn..m... com. tw/app/inder. html
5

---

Published as a conference paper at COLM 2024
# 24:
Following are muleiple-choioe questions exams Toiwan, antwwer one
1E 1 Ac. 10 BR
Question:
A tait controlled by poir of alleles the If boch porents have the genotype Au for this trait, without considering mutations, how mony different phenotypes could their children potentially have for this trait?
with A being dominant and being recessive.
(A) (D) 2 (C) (D)
TEAE # Answert
-... demonstrations
-.]
fo 4EE
RUUE
*
Question:
Among the following the titles of Four books, each ritle indicates the content it intends to introduce, where some plants are listed with detailed descriptions of their charocteristics. Which book is the least likelly use cycods the main example of these plants?
(A) A (B) a OF (C) + SUM $ (D) < #R
(A) TTe Spread Seedso (8) s.Anolysis the Structure Conifer Conesa (C) <An Introduction to Voscular Plantso: (D) aA Pictorial Guide to Flowerst)
IEA #2 Answer
Figure An example prompt for few-shot direct answer evaluation TMLU.
4.1 Experimental Setups
Models. We comprehensively benchmark 24 LLMs on TMLU. The adopted models are capable of processing Traditional-Chinss content and vary in size and developer organiza- tions. Concretely, we conduct experiment on 6 doured, proprictary models via APTs, including models from OpenAi, Anthropic, and Google. For open-weight models, we focus models developed for Chinese, English, and multilingual purposes. This included Taiwan-LLM (Lin Chen, 2023) and Breeze (Hsu et al., 2024) for Taiwanese Mandarin, as well as Yi (AI et al., 2024) and Qwen (Bai et al., 2023) for Simplified Chinese. For complete list of the models evaluated, please refer to Table
Few-shot Evaluation. The evaluation is conducted in few-shot with setups: direct answer and COT. For each subject, the five instances from the development split are utilized as demonstrations (i.e., five shots). The adoption of few-shot evaluation is consistent with benchmarks such as C-Eval (Huang et al., 2023) and MMLU (Hendrycks al., 2020), and the convention in prior works (Touvron et al., 2023; Team et al., 2023; Achiam et al., 2023) for measuring LLM performance. The setting is believe capture the inherent, underlying potential of LLMs more robustly and extrapolate to downstream task adaptation (Huang et al., 2023).
The evaluation prompt consists of the following components: the problem definition, set of few-shot examples, and the input instance to query. For instruction-tuned (chat-oriented) models, we apply the accompanied chat templates if available. Compared with direct answer, CoT prompting includes an explanation (i.e., rationale) triggered with "Let's think
We adopt gpt-4-0125-previeee and gpt-3.5-turbo-1106.

---

Published as a conference paper at COLM 2024
step by step", "opp",be before the final answer prediction. Example prompts with TMLU instances provided in Figure (direct answer) and Figure 7 (CoT).
Answer Extraction. As show by Fourrier et al., 2023, different flavors of implementation in extracting predictions from models bring nontrivial impact to the evaluation results. To ensure a fair evaluation and address variations in model accessibility and prompting settings, we adopt two methods for answer extraction: (1) Likelihood-brae bassdd method. The method aligns with the original implementation
MMLU (Hendrycks al., 2021), which involves examining a set of candidate answer symbols (i.e., {"A", "B", "C", "D"]) and selecting the symbol with the highest assigned token probability as the model's prediction. (2) Generation--ussd method. When direct determination of candidate token probabilities is impractical, we opt for the first token generated greedily that can be interpreted as an option code to derive the answer prediction.
For proprietary models, as access to the probability of each candidate answer symbol is restricted, utilize the generation-based method for both direct and CoT prompting scenarios. In the case of open-source models, we employ the likelihood-basse direct prompting. However, when employing CoT prompting, where the model must first generate an explanation for the queried instance, we are constrained to using the generation- based method. In instances of CoT prompting, we adjust the number of shots based on the model context length to avoid failure to produce the answer.
method for
4.2 Results
The experimental results presented in Table 3a and Table 3b for direct answer and CoT prompting, respectively. In general, flagship proprietary models demonstrate performance superior to open-weight counterparts, and models trained Chinese data (i.e., Simplified Chinese, Taiwanese Mandarin, and Multilingual models) exhibit substantial improvement over models focusing on English context. For direct answer prompting, the Breeze model tailored for Taiwanese Mandarin with only 7 billions parameters - outperforms GP'1-3.5. The top-performing open-weight model Yi, with 34 billions parameters, outperforms proprietary models such as Gemini-pro and Claude-instant, and is comparable to GPT-4.
In CoT prompting, proprietary models outperform the best open-source model by signifi- cant margin of 27%. The results suggest an ample opportunity for enhancing reasoning capabilities via CoT prompting open-source models. We provide more analysis between direct answer and CoT prompting in Section 5.
In sum, the established baseline suggests that TMLU offers a proper ground, with head- room for improvements, for evaluating LLMs in the context of Taiwanese Mandarin. Fur- thermore, the exhibited performance of Taiwanese- Mandarin LLMs still lack behind the Simplified-Chines counterparts, substantiating the goal of TMLU to foster the development of localized, hhiwanese-Mandary LLMs in the future.
5 Analysis
5.1
Robustness to Test Data Contamination
To further investigate the possibility of test data contamination mentioned in Section and 3.2, we employ MIN-K% PROB (Shi et al., 2023), reference-free method for detecting pre-training data from LLMs. Based on the hypothesis that an example seen by the model before (i.e., during pre-training) is less likely to include words with low probability (iee, high negative log-likelihood), MIN-K% PROB selects a set of k% of tokens from the input text with minimum token probability and average their log- likeliidd as an indicator of whether the input text is in the pre-training data. Specifically, given an input example x = (x X2, X2,,., Xn),
MIN-K% PROB(X) MMnnK((x) xEMin-K(N)
Σ
- log -ogg((x;/X1 X2,
7

---

Published as a conference paper at COLM 2024
Model
Social Sci.
STEM
Humanities
Taiwan Spe.
Other
Average
Claude-3-Opus"
83.06
64.32
75.93
90.41
54.23
73.59
Llama-3- Jawan-20B-Instru
82.61
61.17
78.87
85.01
5164
72.46
GPT-4-turbo"
72.30
86.90
55.95
Yi-34B-Chat
73.77
47.99
73.55
83.95
62.97
68.45
Llama-3-70B-Istruuel
74.39
56.41
74.60
79.77
52.04
66.26
Gemma--2227-it
68.84
50.95
64.04
76.72
52.52
62.61
Qwen-14B-Chat
64.85
45.98
64.83
79.15
54.23
61.81
Gemini-Pro"
69.22
41.96
65.87
81.55
48.40
61.40
59.89
39.21
64.85
69.75
43.62
55.46
Qwen-77B-Chon
56.54
73.99
Claude-Instant-122
60.84
36.43
56.33
75,46
42.86
54.38
Breeze-7B-Instrust
57.19
36.68
49.79
73.80
39.07
51.31
GPT--3.5*
54.64
35.43
46.37
72.51
37.90
49.37
1ixttrrl-677B-Instrrett
51.55
33.17
46.47
66.05
34.99
46.44
chatglm3-6b
51.55
33.17
46.47
66
34.99
46.44
Claude-2.0"
40.62
30.90
41.08
73.06
37.32
44.60
Mistral-7B-Instrucct
42.43
67.71
34.40
Taiwann-LLM-13B-hall
44.08
27 89
41.80
64.39
36.15
42.86
Buichuan2-138-CC
46.08
31.1
42.84
56.64
28.57
41.21
Maiwan-LLM-7B-CCN
35.15
21.86
32.99
64.02
32.94
37.39
falcon-4bbiinstruct
36.07
21.36
36.41
51.48
32.07
35.48
OLMo-7B-Instruuc
35.70
25.38
29.88
50.37
31.78
34.62
Llama-2-13b-Chat
36.07
27.14
35.27
47.05
26.24
34.35
Yi-6B-Chat
35.52
26.38
37.66
34.50
29.74
32.76
Owen-0.58-Chat
30.78
24.12
29.15
45.57
28.57
31.64
Gemma-2-99-it
21.93
18.21
29.74
48.68
24.06
28.52
llama-2-7bccat
29.33
23.37
27.28
32.84
27.41
28.04
TAIDE-LX-7B-Chat
24.41
25.13
23.86
33.21
22.16
25.75
Falson-7b-instruct
22.59
23.76
34.32
24.78
25.66
Gemma-7b-it
27.32
20.10
24.07
32,66
23.91
25.61
Meta-Llama-3-SB
17.75
16.41
19.81
22.66
24.33
20.19
(a) The five-shot direct-answer accuracy results. (%)
Model
Social Sci.
STEM
Humanities
Taiwan Spe.
Other
Average
Claude-3-Opus"
82.15
74.37
73.44
89.67
53.06
74.54
GPT-4-turbo"
70.12
86.35
Gemini-Proo
64.66
47.24
59.54
82.66
45.77
59.97
Claude-2.0"
66.67
54.02
58.51
80.81
39.65
59,93
Claude-Instant-l.22
61.57
44.47
56.43
75.83
37.03
55.07
GPT-3.5
59
45.48
49.90
75.65
41.69
54.38
Llama-3-BB
58.14
51.38
51.88
Yi-6B-Chat
46.45
34.17
46.27
70.85
39.07
47.36
Taiwan-1 1ama-3-8--
Yi-34B-Chat
49.54
18.84
30.39
79,34
33.53
42.33
Breeze-7B-Instruct
48.63
30.19
74.35
27
40.58
Mixtral-7B-Instructt
42.81
23.87
37.14
59.23
30.61
38.73
Llama-3-Talwan-S--Be
38.62
25.36
38.00
60.89
26.88
37.99
TAIDE-LX-7B-Chaa.
38.25
67.34
37.76
Gemma-2-9b-it
37.56
32.91
34.95
50.01
30.20
37.13
Qwen-7BBChat
39.71
chatgim3-6b
37.34
21.11
32.47
63.10
23.91
35,58
Llamm-3-7UB-Instueet
40.43
24.08
42.38
27.36
32.54
Taiwan-LLM--7BChaa
26.05
23.62
19.81
56.27
24.49
30.05
Gemma-2227t--it
29.26
23.67
27.04
39.48
18.26
27.54
Baichuan2-138-C1
Taiwan-I MM-aa-IIMM13B-Chhii
26.41
12.56
17.63
48.15
14.29
23,81
Meta-Llama-3-81
25.04
20.23
23.62
23.48
23.51
(b) The five-shot Col accuracy results. (%)
Table 3: The five-shot accuracy (%) results on TMLU for direct-answer and CoT prompting. The results are sorted in descending order. The colors indicate the language for which the models are tailored. Yellow denotes Multilingual; Blue denotes Simplified Chinese; Red denotes Taiwanese Mandarin; Green denotes English. In addition, proprietary models are marked with π. Detailed comparison results are provided at Table
where Min-K(x) is the set of k% of tokens in x with the minimum token probability. The lower the MIN-K% PROB is, the more likely that the input example has been seen during pre-training.
8

---

Performance difference (0) between direct answer and CoT prompting on stem subjects. Only models exhibiting improvements Published as a conference paper at COLM 2024
TMMLU-plus
Min-K% Prob
3.5
3.0
2.5
2.0
Talwan-LUA-7B-base Breeze-78-Base
Y-6B
Qwen1.5-78
Liama-2-78
Mistral-78
Figure 3: The MIN-K% PROB (Shi al., 2023) of six base models on IMMLU-plus (Tam et al., 2024) and our dataset TMLU. The lower the MIN-K% PROB is, the more likely the input instances of the datasets are in the model's pre-training data.
We apply MIN-K% PROB on 2000 sampled instances from TMLU and TMMLU-plus with 6 different base models. We present results in Table 3 and discuss more implementation details in Appendix B. As shown, TMLU achieves higher MIN-K% PROB than IMMLU-plus across all tested models, which implies that TMLU is less likely to contain data that has been seen during pre-trained. In addition, the finding also aligns with our hypothesis that TMMLU-plu might be collected by crawling plain text from the web, and further validates the effectiveness of our design principle (Section 3.2) that sourcing data from downloaded documents could reduce the risk of data contamination.
5.2 Comparison Between Direct Answer and CoT Prompting
To further investigate the potential capability of LLMs, we leverage our curated explanations and employ CoT' prompting. CoT prompting elicits step-by-step reasoning chains towards answer derivations, which has been shown to improve tasks that require complex, multi- hop reasoning significantly. We present results comparing CoT with standard answer-only prompting in Figure for STEM subjects. Full results are reported in Table 9. As shown, models that are able to benefit from CoT are mainly large, proprietary ones, exemplified by GPT-4 (15%) and Claude-2 (23%). The finding may be attribute to the emergent nature of reasoning capability in LLMs. Interestingly, Taiwan-LLMM77B-Chä is the only model exhibiting improvement with CoT at its parametric scale.
TDiwar-LLM-13B-CC
M:34B Chat
Y168 Chat
Y168
OLMo-78
chatalm3
(96)
10-
Accuracy (%)
Powan--LM-75 Chat
Comini-Prr
Chat-Chat
Claude:33 3- Opus
GPT-3.5
GPT-4-ture ooT-4-turbo
Cleude-220
Claude Claude-Instand in--nnstannt....
Figure 4:
Figure 5: Average accuracy of models on questions of different years. The accuracy is calculated by averaging across the number of questions. Full results are provided Table 6.
(8 > 0) are presented.

---

Published as a conference paper at COLM 2024
5.3 Comparison of Model Performance Across the Temporal Dimension
Here we investigate the performance of models on the temporal dimension. We group questions from different years subsets and evaluate the average accuracy, as presented in Figure 5, Questions from before 2017 (2013~2016) are group with 2017 as one subset. The results show that, for most models, no consistent trends behaviors are exhibited. Yet, for GPT-4-turbe and Yi-34B-Chat, the scores imply rising trend where models perform better on questions from more recent years. Future endeavors could offer a more precise implication and rigorous explanation for the results.
Conclusion
In this work, we introduce TMLU- - an evaluation suite designed to assess the advanced knowledge and reasoning abilities of LLMs the context of Taiwanese Mandarin. Experi- mental results and analysis suggest great opportunities for future developers, in particular, for open-source models. Besides addressing the scarcity of benchmark for Traditional Chi- nese community, we envision TMLU to serve grounded testbad, fostering development of localized Taiwanese-Mandarin LLMs.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xrv preprint ar xrX::23033.08774 2023.
01. :I, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn
Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi
Ren, Xinyao Niu, P'egcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu
Gu, Zhiyuan Liu, and Zonghong Dai. Yr: Open foundation models by 0l.ai, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arxiv preprint arXiv: 800.05457, 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. Xiv preprint Xiv:21100 1416,, 2021.
Clémentine Fourrier, Nathan Habib, Julien Launay, and Thomas Wolf. What's going on with the open Ilm leaderboard. Hugging Face Blog (June 2023). URL: https://huggingfuce hojllo//oulluiingnmii-eeeeeeeree wawprtiinggmmmullllerrrrrrrrr 2023.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. Xrviv preprint ar Xiv:2330 088493, 2023.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3. Xiv preprint arXiv:2209.11 222.2266,,,,6, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar arviv preprint ar XXi::2009.03300 2020.
10

---

Published as a conference paper at COLM 2024
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problee solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Chan-Jan Hsu, Chang-Le Liu, Feng- ling Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-shan Shiu. Advancing the evaluation of traditional chinese language models: Towards comprehensive benchmark suite. arXiv preprint arXiiv2309.0844: 2023.
Chan-Jan Hsu, Chang-Le Liu, Feng- Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. Breeze-7b technical report, 2024.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint ar arXie:2305.08322, 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2022.
Meng Lee, Fujiki Nakamura, Makoto Shing, Paul McCann, Takuya Akiba, and Naoki Orii. Japanese stablelm base alpha 7b. URL (https://hugginggfaal httpp////uuggnngacc...cccaaaaaaa http::////ugginaacccco/////eee
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. ar preprint ar Xit::2306.09212222 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Xiv preprint ar XrXi:2211..0911 109110,, 2022.
Yen-Ting Lin and Yun-Nung Chen. Taiwan Ilm: Bridging the linguistic divide with culturally aligned language model. arxio preprint ar xxfo:2231.11487,,22 2023.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 vv// 2011//22.eemmm 1et.33mmmllmaaae m5.n. URL https:
//aclanthology. org/2023. emnlp-main 153.
Andrew Maas, Raymond E Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp- 142-150, 2011.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning, ar Xrviv preprint arXiv:2211.01786 2022.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, et al. Seallms-large language models for southeast asia. arXiv preprint orXt0:2312.00738, 2023.
11

---

Published as a conference paper at COLM 2024
Oscar Sainz, Jon Campos, Iker García-Ferrent Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamina- tion for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787 Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653 /203. findings-emnlp.77 .22.. URL
Siidinggsemmin 722.
Oscar Sainz, Jon Ander Campos, Iker Carcía-Ferrero Julen Etxaniz, and Eneko Agirre, Did chatgpt cheat on your test?, Jun 2023b. URL https: htttp::/hhitz-zentrro github.... 1o/ 1m- www.coonmmmmmmtt.//oloo.........
Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Iseng, and Sam Tsai. Dred: A chinese machine reading comprehension dataset. ar preprint ar 0rXv:18866000990 2018.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In NeurlPS 2023 Workshop on Regulatable ML, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xiv preprint ar Xiv:22006.0400 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
STPI. 2020 &a grandchallenge2022 2020.
# https: //tps://scidm..... nch.. org. tw/dataset/
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. ar preprint arXiv:2210.09261, 2022.
Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, and Hong-Han Shuai. An improved traditional chinese evaluation suite for foundation model. ar preprint arXiv:2403.01855 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, (ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arv preprint arXiv:2312.11805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXio preprint ar 2337111302.13971, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint ar xrXi::1804.07411, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Adounces in neural information processing systems, 32, 2019.
Shan Wang and Lei Tang. Comparison of changes between mainland china and taiwan. In Chinese Lexical Semantics: 21st Workshop, CLSW 2020, Hong Kong, China, May 28-30, 2020, Revised Selected Papers 21, pp. 686-710 Springer, 2021.
12

---

Published as a conference paper at COLM 2024
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arviv preprint ar arXiv:2307.1033 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, al. Polylm: An open source polyglot large language model. Xiv preprint ar arXiv:2307.06016 2023.
Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. Superclue: A comprehensive chinese large language model benchmark. ar Xiv preprint ar arXii:2307.1500 2023.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, 2021.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang. Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan Open large-scale language models, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint ar XxX1:2210.002414 2022.
Jianjiao Zhou and Shu Zhou. A study on differences between taiwanese mandarin and mainland mandarin in vocabulary. 3rd International Conference on Culture, Education and Economic Development of Modern Society (ICCESE 2019), pp. 212-215. Atlantis Press, 2019.
A Limitations and Future Work
While TMLU provides a comprehensive evaluation of language models' understanding capabilities in Taiwanese Mandarin, it is important to acknowledge its limitations, particu- larly in terms of its generalizability to generation tasks. Our benchmark, which focuses on multiple-choice question answering, may not fully capture the performance of language models in generation scenarios such as text completion, summarization, and dialogue systems.
To investigate this issue, we conduct a preliminary experiment by calculating the perplexity (PPL) of generated answers for a few models with a fixed vocabulary size of 32,000 tokens. The resulting PPL values were: Bhiwan-LLLM-7B-Chee
16.5, Breeze-7B-instruct 8836.1, 1, Yi-6B-Chat: 20887.0, ., and Llama-7B-Chat: 2038633.8. Interestingly, models trained exclu- sively Taiwanese Mandarin, such as Taiwan-LLM, have a smaller entropy in their logits compared to models trained on multilingual data, suggesting more consistent performance in generating fluent Taiwanese Mandarin text.
Therefore, while TMLU provides valuable insights into the understanding capabilities of language models in Taiwanese Mandarin, it should not be considered sole indicator of their performance in generation tasks. Future research should focus on developing generation- oriented benchmarks and exploring the relationship between understanding and generation
13

---

Published as a conference paper at COLM 2024
Data source
Website
CAP
pateaa//cap.repeet adu. wtppprpe......t///aaaaaaaaaoooo. wwwp:aaninatiionn.nnt.
GSAT
attps://wuv.cett wwwp..cce.cedee. http:////ttttaaaaaaaaaaa///00000
AST
https: http://///////ooooooooooo//////0
adu. tw/xnf
MOEX
httpa www.////.....x...e/////////
HB
hitps // /www ..b. gov. tw/Newe, Dww.oowdddwwoodddd_eeeomao wwwp:/aaaapaaaa22aa-1a220
Table 4: The sourced websites of our data collection.
Supercategory
Subjects
Questions
Choices
Social Science
AST civics (
57
AST geography ( (
58
CAP civics (fr :t)
73
CAP geugraphy ( (
45
GSAT civics (# R))
73
GSAT geography ((1-)
MOEX Accountant (#1+-R0)
117
MOEX Clinical CB
117
STEM
AST biolugy (0)
40
AST chemistry (5--4480/60::
34
mathematics (1
25
AST physics (2) (13162201
43
CAP biology (fr (5:666))
27
CAP chemistry (#: (2249)
27
CAP earth science (17 18:
15
CAP mathematics (1roaaattics--999900 (0)
115
CAP physics ( (17))
15
GSAT biology (#:10/4:65)
21
GSAT chemistry (22
29
GSAT earth science (1
24
GSAT mathematics (5)
GSAT physics (5611))))
24
Humanities
AST Chinese (3:-945815)
AST history 16--4-3358
56
CAP Chinese (5 (X)
61
history (1
GSAT Chinese (4A33 10
97
GSAT history (4P-1) (6
85
Tour guide (19:12)
99
MOEX Tour leader (3218)
145
MOEX Lawyer qualification (1)))
279
Taiwan Specific
HB Driving Rule (fe W (0))
432
MOEX Teacher qualification (8X0)11)
75
Taiwan tourist resources (LAEE)))))
50
Others
MOEX Basic Traditional Chinese Medicine (# 10
159
Clinical Traditional Chinese Medicine (#))))
79
MOEX Nutritionist (#t (kB0)
120
Table 5: Summary of all 37 subjects and corresponding numbers of choices.
abilities to gain a more holistic view of the performance of language models the context of Taiwanese Mandarin.
B Implementation Details of Data Contamination Analysis
We construct the input example for MIN-K% PROB by concatenating the question and the corresponding choices consistent to the actual scenario where the model would likely be queried for each sampled instances from TMLU and TMMLU-plus. To mitigate the effect of input example length when computing MIN-K% PROB, as discussed in the MIN-K% PROB paper, we filtered out the longest examples within each subjet subset of our dataset, as the average text length of TMLU is significantly longer than that of TMMLU-plus. This step is taken to ensure fair and comparable analysis.
14

---

Published as a conference paper at COLM 2024
[TMLU] The instarcee from "AST physics" test split.
E T 2
(2) mmathrm[eV]s H Sa ZE PM
2> $12.3 mathrm[eV/s BY # ->t-ff
It is known that the energy level formula for hydrogen atom is [E_[n===1366 [2 |machrm[eVJ5 where $nS is the principal quantum number. An electron with kinetic energy of $12.3 \mathrm{eV]s collides with hydrogen atom in its ground state. Which of the following could be the energy of a photon emitted by the hydrogen atom after excitation?
IMMLU-plus. The instance from "physics" test split.
12.52 V FE
M 6 F 3
f- HE
known that the energy formula for hydrogen atom 13.6/n where is
principal quantum number. electron with kinetic energy of 12.3 eV collides with hydrogen atom in ground Which of the following could be the energy of the photon emitted by the hydrogen after excitation?
(a)
[TMMLU-plus] A Simplified Simplifie-CChineee instance from the "physics" test split.
i m YaA E H M
Which of the following factors, if increased, can cause the intensity of light increase?
[TMMLU-phe] plus] Two unanswerable instances from the "junior social studies" test split.
# (++>))
Based on Figure 16, which of the following descriptions most uppropriately represents the meaning conveyed by this promotional slogan?
# ハ MER N ?
According information Table 9, which did Xiaoting and the people with less physical strength and experience choose?
(b)
Figure 6: Example instances from TMLU and TMMLU-plus datastes. Here choices are omitted for brevity. Figure 6a shows a same instance from TMLU TMMLU-plus (physics_test), where symbols are formatted in latex in TMLU but are not formatted in IMMLU-plus, effecting the clarity. Figure 6b shows instances from TMMLU- plus (junior chemistry test and junior social studies test) which are in Simplified Chinese or unanswerable since requires information from an image/ table that not accompanied in the question.
and
15

---

Published as a conference paper at COLM 2024
Fallowing are multiple-choice questions from exams in Taiwan. The answer one of the choices.
Question:
Which of the following statements about plant tissues correct?
(A) A sul
(B) TLL Nu T
(C) A #2 F
(D) Ae
(A) The mesophyll cells, which perform photosynthesis, are type of sclerenchyma cells.
(B) Stomata are formed by intercellular spaces.
(C) The cell walls of meristematic tissue are thicker than those of mesophyll cells.
The cork cambium is differentiated from epidermal cells.
0 パ -ー 3y EL 3E ft H3 PE
(A (B) Kts
Ji (D
< (B)
Let's think step by step. The mesophyll cells with photosynthesis belong parenchyma cells, so (A) is incorrect. (B) The description is correct, choose (B). The cell walls of the meristematic tissue are thicker than those of the mesophyll cells, (C) incorrect. Cells outside of the cambium can all turn into cork cambium, except for the epidermal cells, so (D) is incorrect. Answer: (B)
[ ... other four demonstrations ...]
F MA
Question:
Among the following genetic screening technologies, which most capable of diagnosing fetal Down syndrome?
(A) A (B) M (C M (D) #
(A) Ultrasound examination (B) Chromosomal examination (C) Genetic testing (C) Biochemical testing
Let's think step by step.
Figure 7: An example prompt for few-shot CoT evaluation on TMLU.
16

---

Published as a conference paper at COLM 2024
Model
~~2017
2018
2019
2020
2021
2022
2023
Avg.
# of questions
157
228
GPT-4-tubo
64.41
68.48
73.25
69.43
74.12
81.73
72.06
72.35
Yi-34B-Chat
60.17
65.22
70.06
68.91
66.67
76.14
71.78
70.67
Gemini-pro
56.78
65.22
63.06
61.14
61.40
67.51
64.77
64.02
Qwen-7B-Chat
48.31
56.52
59.87
51.30
54.39
37.36
57.43
36.47
Claude-instant-1.2
49.15
56.52
55.41
48.70
52.19
36.85
58.31
56.44
Yi-6B
52.23
52.33
GPT-3.5
51.69
44.57
52
44.04
41.67
35.33
51.79
50.46
Mixxtrl-&&77-lnnsttutt
43.22
51, 00
52.23
43.01
42.98
34.31
50.19
49.25
chatglm3-6b
43.22
47
49.04
46.11
44.74
52.28
48.32
47.96
Mistral-7B-Instruue
39.83
47 83
43.31
42.49
28.95
38.58
48.92
45.39
Claude-2.0
38.98
46.74
40.76
38.34
41.23
42.64
47.34
45.28
Talwaan-LLM-13B-haatt
44.07
40.22
38.85
36.79
37.72
43.65
46.16
43.96
Baichuan2-13--Chut
42.39
50.96
Taiwan-LLM-7B-Chat
27.97
29.35
32.48
30.57
27 .3
33.50
41.11
37.84
falcon-I0b-instruct
30.51
33.70
34.39
38.23
31.58
36.55
38.10
36.59
laam-2-13b-Chal
40.68
34.78
32.48
29.53
32.89
40.61
35.78
35.44
Yi-6B-Chat
33.90
46.74
35.67
34.20
38.60
32.49
32.85
34.05
Owve--05B--Chtt
35.59
27.17
26.11
27.98
26.75
33.37
Llama-2-7b-Chut
32.20
32.61
28.03
22.80
30.26
24.87
28. 44
28 22
gemma-7b-it
20.34
17.
24.84
21.76
25.00
26.93
26.84
25.79
falcom-7b-instrust
14.41
20.65
23.57
20.21
26.75
30.96
26.56
25.57
Total
41.34
44.09
44.34
41.37
41.73
46.81
46.70
45.45
Table 6: Performance comparison on questions from different years. The accuracy calcu- lated by averaging across the number of questions.
Year
#of questions
2013
10
2014
15
2016
27
2017
2018
2019
157
2020
193
2021
228
2023
1811
Table 7: Number of questions from different years.
Social Science
STEM
Humanities
Taiwan Specific
Other
Avg. Chat
Model
Base
Chat
Chat
Chat
Chat
Chat
Yi-34B
77.96
73.77
51.51
47.99
73.76
73.55
86.35
62.97
68.45
Qwen-14B
66.12
64.85
47.49
45.98
65.77
64.83
78.97
79.15
57.14
54.23
63.10
61.81
Yi-6B
60.29
35.52
40.20
26.38
60.27
37.66
78.04
34.50
50.44
29.74
57.85
32.76
Qwen-7B
60.47
57.74
39.45
39.20
60.37
56.54
75.46
73.99
52.19
46.65
57.59
54.82
Baichuan2-138
55.56
46.08
31.41
31.91
53.63
42.84
72.69
56.61
46.36
28.57
51.93
41.21
Mistral-7B
51.91
46.08
31.41
30.65
43.46
42.43
68.08
67.71
35.28
34.40
46.03
44.26
Taiwan-LLM-13B
42.08
44.08
25.38
27 89
44.09
41.80
67.90
64.399
35.28
36.15
42.94
42.86
Llama-2-13b
4444
36.07
31.66
27.14
39.32
35.27
62.36
47.05
32.36
26.24
42.03
34.35
Qwen-0.5B
36.61
30.78
26.88
24.12
35.58
29.15
52.58
45.57
26.82
28.57
35.70
31.64
Taiwan-LLM-77
34.79
35.15
24.62
21 86
29.67
32. .9
57.56
64.02
27.11
32,94
34.75
37.39
Llama-2-7b
31.69
29.33
26.88
23.37
29.98
55.54
32.84
23.32
27.41
33.48
28.04
falcon-7b
23.50
22.59
25.13
22. 82
28.53
23.76
32.29
34.32
25.36
24.78
26.96
25.66
Oo.Mo-7B
23.50
35.70
23.37
25.38
26.66
29. 88
34.50
50.37
24.78
31.78
26.56
34.62
gemma-7b
25.14
27.32
23.62
20.10
24.07
34
32.66
23.91
25.39
25.61
Table 8: Performance comparison between base and instruction-tuned models.
17

---

Published as a conference paper at COLM 2024
Social
Humanitics
Specific Col
Avg.
Model
Direct
Col
Direct
Col
Direct
Col
Direct
Direct
Coll
Direct
Col
Claude-3-Opus
83.06
82.15
64.32
74.37
75.93
73:44
90.41
89.67
54.23
53.06
73.59
74.54
GPT-4-turbo
78.14
75.77
58.79
74.12
72.30
70.12
86.90
86.35
55.98
51.31
70.42
71.54
Yi-34B-Chat
Qwen-1-BB-Cat
64.85
55.19
45.98
25.63
64.83
45.85
79.15
78.04
54.23
48.98
61.81
50.74
Gemini-Pro
69.22
64.66
41.
47.24
65.87
59.54
81.35
82.66
48.40
45.77
61.40
59.97
Qwen-7B Chat
57.74
39.71
39.20
18.09
56.54
29.
73.99
64.94
46.65
30.03
54,82
36.41
Claude-Instante2
60.84
61.57
36.43
44.47
56.33
56.43
75.46
75.83
42,86
37.03
54.38
55.07
Breeze-7B-Instruct
57.19
48.63
36.
22.61
49.79
30.19
73.80
74.35
39.07
11
51.31
40.58
GPT-3.5
54.64
59.
35,
45.48
46.37
49.99 90
72.51
75.65
37.
41.69
49.37
54.38
Claude-2.0
40.62
66.67
31,900
54.02
41.08
58.51
73.06
80.81
37.32
39.65
44.60
59.93
Mistral-7B-Instruct
39.23
Talvan-LLLM-3BB-haall
44.08
26.41
27. 27.89
12.56
41.80
17.63
64.39
48.15
36.15
14.29
42.86
23.81
Ssichuanz-138-Chat
46.06
29.51
31.91
19.85
42.84
27.07
56.64
30.26
28.57
27.11
41.21
26.76
BalvannLLMM-7--Chat
35.15
21.86
23.62
32.99
19.81
64.02
36.27
32.94
24.49
37.39
30.05
133-a-2-3v--haat
36.07
23.13
27.14
20.10
35.27
28.53
47.05
39.30
26.24
26.24
34.35
27.46
Qwen-U,5B-Chat
30.78
22.04
24
18.59
29. 15
20.02
45.57
34.69
28.57
18.08
31.64
22.68
Llama-2-7B-Chat
7.68
27
28.04
Table 9: Performance comparison between direct and CoT prompting.
18


Reviews:
[{"review_id": "GlzVPysnPZ", "reviewer": "Reviewer_1eqm", "summary": "This paper proposes a benchmark for traditional Chinese, TMLU, which targets the Taiwan society by construction. This benchmark is like the MMLU format but with traditional Chinese language. Compared with previous benchmarks for Taiwanese Mandarin such as TC-Eval and TMMLU, the authors argue that TMLU differs in that it originates from traditional Chinese and it is sourced from PDF or Word documents that suffer from less contamination. The paper benchmarks many LLMs on the proposed TMLU benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review! First, we would like to note that [TMMLU-plus](https://arxiv.org/abs/2403.01858) is a concurrent work to ours and hasn’t been published in a peer-reviewed conference.\n\n**Response to RR1**\n\nThe robustness of TMMLU-plus is considered relatively low comparing to TMLU because of the following points discussed in Section 2:\n1. Most questions in TMMLU-plus could be found on one single website, greatly exposing its risk to test data contamination if the website is scrapped for pre-training.\n2. TMMLU-plus contains unanswerable questions that require information (e.g., images or tables) not accompanied in the dataset (Figure 6b).\n\nTable 1 is a concise comparison which summarizes the discussion in Section 2. We will strengthen the description in Section 2 to outline a clear connection to Table 1.\n\nRegarding our claim – “we found that most questions in TMMLU-plus could be found on one single website.”, we conduct additional evaluation to provide more quantified evidence.\n1. We randomly sample 100 instances from TMMLU-plus and discover 91 of them are on the website: https://app.yamol.tw/ which we suspect to be the source of TMMLU-plus (TMMLU-plus does not offer clear information about their source — they simply mention “standardized test resources and PDF documents available online” in Section 2.2 in their preprint).\n2. We examine the full mC4 dataset and find 28,685 entries have URLs from the suspected website: https://app.yamol.tw/.\n\n**Response to RR2**\n\nThanks for the insightful question! The comparison metric in Figure 3, Min-K%, is an indicator of how likely the given text is in the pre-training data of the examined LM. Min-K% is by nature relative, where its absolute value varies under different settings (e.g., model size, vocabulary size).\nNote that it requires oracle labels of the tested dataset to determine a threshold that reflects whether contamination occurs (Please refer to Section 5.1 in the Min-K% paper).\n\nWe acknowledge the possibility that both TMLU and TMMLU-plus suffer from data contamination. Nonetheless, we can observe a systematic and unifying behavior in Figure 3, which shows that TMLU is less likely to be contaminated compared to TMMLU-plus in every model tested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Creating an MMLU-like benchmark for Taiwanese Mandarin is a focused contribution and could benefit the deployment and development of LLMs to serve Taiwan users.  \n2. Empirical experiments are comprehensive, there are COT and data contamination analysis included besides the main results.", "reasons_to_reject": "1. While the authors try to distinguish TMLU from TMMLU-plus, it is better to have more quantitative evidence given that the two benchmarks are similar. For example, Table 1 notes that the robustness, transparency of TMMLU-plus are low, what do they mean? In Section 2 the authors mention that “In our preliminary investigation, we found that most questions in TMMLU-plus could be found on one single website.”, can you quantify this? How many can be found?  \n2. In figure 3, I wonder what the absolute values of the y-axis mean, for example, does a min-k% prob of 4.0 indicate that the pretraining data is absolutely likely to present in the pretraining data? Currently the comparison in Figure 3 is relative, and I am not sure whether the gap between TMLU and TMMLU-plus is significant or has meaningful differences given that the gap looks small – maybe TMLU and TMMLU-plus both suffer from data contamination already.", "questions_to_authors": "```\nUnderstanding that TMMLU-Plus is a concurrent work within one month of the submission, I increase my score from 4 to 6 given that I ignore the existence of TMMLU-Plus\n```", "extra_scores": {}}, {"review_id": "TqqjRSVENu", "reviewer": "Reviewer_8Aze", "summary": "This paper introduces TMLU, a new evaluation benchmark designed to assess large language models (LLMs) specifically in the context of Taiwanese Mandarin. TMLU addresses the unique linguistic nuances and cultural intricacies of Taiwanese Mandarin, differentiating it from other varieties of Chinese. The benchmark encompasses 37 subjects, spanning middle school to professional levels, and includes approximately 3,000 multiple-choice questions across diverse disciplines. The authors evaluate 24 LLMs on TMLU and include Chain-of-Thought (CoT) examples to assess complex reasoning skills.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for the insightful comment! First, we would like to note that [TMMLU-plus](https://arxiv.org/abs/2403.01858) is a concurrent work to ours and hasn’t been published in a peer-reviewed conference.\n\n**Response to RR1**\n\nWe would like to emphasize that TMLU’s goal is to foster the development of localized Taiwanese Mandarin LMs. Though academic terminology may be similar to Mainland China, subtle nuances in the way questions are described still capture linguistic variations, which could affect language understanding of localized LMs that are trained only on Traditional Chinese data.\n\nWe hypothesize Simplified Chinese LMs have been trained on Traditional Mandarin content, thus, perform sufficiently well. However, at the same parametric scale, Taiwanese Mandarin model Breeze-7B-Instruct greatly outperforms Simplified Chinese model Qwen-7B-Chat (74.35% vs 64.94%) on the Taiwan-specific subset of TMLU (Table 3b).\n\n**Response to RR2**\n\nTMLU offers following distinct features not available in TMMLU-plus:\n1. We provide clear attribution of the data source (Section 3.2). Also, we suspect TMMLU-plus collects most questions from a single website, which greatly increases the risk of data contamination. To validate this assumption, we conduct additional analysis: Please refer to our response to point 1 for Review 1eqm.\n2. We manually curated few-shot CoT demonstrations, which is crucial to facilitate the evaluation of reasoning capabilities in Traditional-Mandarin LLMs.\n3. We standardized all instances containing symbols and equations in LaTEX format for better clarity (Figure 6a).\n\nCollectively, these features make TMLU a more ideal benchmark for the community compared to TMMLU-plus.\n\n**Response to RR3**\n\nIndeed our data construction pipeline is relatively standard, but except the concurrent work TMMLU-plus, there are no MMLU-style benchmarks that are in Traditional Mandarin by construct. We will continue exploring innovative ways to augment TMLU in the future.\n\n**Response to RR4**\n\nWe acknowledge that some questions available online might be a part of the training data. Yet, we offer support that TMLU is less likely to suffer from contamination in Section 5., showing that TMMLU-plus perform worse in every tested LMs.\n\n**Response to Q**\n\nThanks for pointing this out! We will correct the error and replace “Tongzhi” with terms such as “Tudou” (which typically means “potato” in Mainland China and “peanut” in Taiwan).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a novel benchmark, TMLU, designed specifically for evaluating the performance of large language models (LLMs) in Taiwanese Mandarin. This benchmark encompasses a broad spectrum of subjects and has been meticulously curated through manual review. Additionally, the authors attempt to mitigate data contamination by sourcing questions from PDF and Microsoft Word documents.", "reasons_to_reject": "1. Limited Linguistic and Cultural Differentiation: While the benchmark is written in Traditional Chinese, the questions themselves do not sufficiently capture the unique linguistic and cultural aspects of Taiwanese Mandarin. Many STEM questions, for example, use terminology common to Mainland China. Areas where language usage differences are more pronounced, like social media, are not included. This limitation may explain why Simplified Chinese models perform well on the benchmark.\n2. Unclear Discussion and Unconvincing Conclusions: The paper's discussion lacks clarity, and some conclusions are not well-supported. For instance, the authors claim advantages over TMMLU plus, but only provide a few examples without statistical evidence.\n3. Limited Innovation: The benchmark construction and analysis are similar to previous work, and the exclusive use of multiple-choice questions limits innovation. The paper would be stronger with a more novel approach to dataset construction and the inclusion of diverse test types.\n4. Data Contamination Concerns: While the authors strive to avoid data contamination, their use of materials from major tests in Taiwan raises concerns. These materials may already be available online in digital form on test preparation websites, potentially accessible to LLMs during training. The observed decline in model performance on older questions (pre-2017) could indicate data contamination, as older materials are less likely to have been digitized and disseminated online.", "questions_to_authors": "Figure 2 contains an error in the answer key (the correct answer for the last question is \"C\"). \nThe example using \"Tongzhi\" to illustrate language differences is not the most effective, as its primary meaning is consistent across Mainland China and Taiwan (referencing Zhang et al. 2022).\nZhang, K., Lu, C., & Zhang, J. (2022). Chinese media representations of tongzhi (2009–2019). Discourse & Communication, 16(3), 305-325.", "extra_scores": {}}, {"review_id": "acHbqFkw1H", "reviewer": "Reviewer_L2nz", "summary": "This paper describes TMLU, a new MMLU-style dataset for LLM evaluation, targeted specifically at Taiwanese Mandarin (and the traditional Chinese script). As with MMLU and other datasets fashioned around it, the source of the data for TMLU is exam questions from middle and high school, and professional topics (e.g. traditional Chinese medicine or driving exams), with a strong focus on scraping data from PDFs and MSWord documents in the hope that it is less likely to be included in the pre-training data for LLMs. In addition to evaluating a range of open-weight and proprietary LLMs over TMLU, the paper includes some nice analysis of data contamination, direct answering vs. COT prompting, and a breakdown of results across time (based on the timestamps of the documents the questions were sourced from).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback!\n\n**Response to RR1 & RR2**\n\nThe topic coverage and composition structure of CMMLU, TMMLU-plus and TMLU are similar, generally divided into STEM, social sciences, humanities, local-specific, and others. CMMLU and TMMLU-plus includes more professional-level domain-specific subjects, such as computer science and medical knowledge. Compared to TMMLU-plus, TMLU has a more complete coverage of elementary and high school -level questions.\n\nIn the local-specific section, CMMLU includes many subjects related to Chinese literature; TMLU includes subjects related to local tourism in Taiwan, such as tour guide exams and tourism resources, to evaluate the model’s Taiwan-specific knowledge, and TMMLU-plus does not have an explicit subset for localized questions.\n\nWe will incorporate the above content and a more in-depth discussion/analysis in our revised version.\n\n**Response to RR3**\n\nThanks for pointing this out! We will correct semantic errors and refine all necessary parts to improve clarity in our final version.\n\n**Response to Q1**\n\nWe hypothesize Simplified Chinese LMs perform sufficiently well because they might have already been trained on extensive Traditional Mandarin contents (in addition to more Simplified Chinese corpus that contain Taiwan’s local knowledge). Also, a potential explanation is that the Taowan-specific questions are relatively easy compared to advanced STEM subjects, resulting in higher scores compared to other academic subjects. However, we can still observed in Table 3b – at the same parametric scale, Taiwanese-Mandarin model Breeze-7B-Instruct greatly outperms Simplified-Chinese model Qwen-7B-Chat (74.35% vs 64.94%) on the Taiwan-specific subset of TMLU.\n\n**Response to Q2**\n\nInitially we use the term “holistic” in the sense that our dataset covers a wide range of topics for assessing the models’ parametric knowledge. We will refine our description of “holistic” to be precise and try to provide a more quantified comparison of the degree of comprehensiveness.\n\n**Response to Q3 & Q4**\n\nSorry for the confusion! Indeed TMMLU-plus is an extension of TMMLU, and both factors (data contamination issue and the improvement of model itself such as scaling) tangle together. We will reiterate the difference in Section 5.1 as suggested and nuance the description regarding the applicability of the benchmark in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- it is important for LLM evaluation resources to be developed for a broad range of languages, and while there are many resources for Mandarin Chinese, there are considerably fewer for Taiwanese Mandarin, and the traditional Chinese script; this combined with the scale of the dataset and attention to detail in the paper makes it worthy of publication\n- beyond just presenting results for different LLMs over TMLU, the paper includes additional analysis of data contamination (showing that it is lower than a competitor dataset), direct answering vs. COT prompting, and LLM accuracy of questions sourced from different years", "reasons_to_reject": "- not a strong reason to reject, but I would have liked to have seen more comparison of the coverage of topics in the dataset with related datasets such as TMMLU-plus and CMMLU\n- I would also have liked to have seen more discussion/analysis of results over \"Taiwan specific\" questions (see below), and also between similar topics in CMMLU and other datasets for Mandarin Chinese\n- the writing is overall good, but there were some places where things got tangled up, particularly the first sentence of Section 4.1 which should be changed  to \"We perform comprehensive evaluation over TMLU using 24 different LLMs ...\" (you are not evaluating TMLU itself, but rather using TMLU to evaluate LLMs); also \"wide range of field*s*\" (Section 3.2); \"Tradition*al* Chinese (Section 2)", "questions_to_authors": "- overall, models appear to achieve slightly lower results over TMLU than CMMLU, e.g., but how much of this is attributable to script differences (traditional vs. simplified) vs. the Taiwanese/Mandarin Chinese distinction vs. \"local knowledge\"? For example, if you convert TMLU into the simplified script, how much do the results change? How do you explain the fact that results over \"Taiwan specific\" questions are actually higher than all other question types?\n- in what sense is TMLU \"holistic\" (page 8)? I get that it is broad coverage, but is there a way to quantify the degree of comprehensiveness of the dataset (e.g. relative to MMLU or CMMLU)?\n- more of a comment than a question, but I found myself getting confused at one point about what TMLU was as compared to TMMLU-plus (i.e. given the similarity between \"TMLU\" and \"TMMLU\", I assumed that TMMLU-plus was somehow an extended version of TMLU and tried to go back and find mention of how it had been expanded); perhaps remind the reader in Section 5.1 that they are completely different datasets, despite the similarity in name\n- what is the basis for you saying that \"the applicability of these benchmarks has decreased as the ever larger models are demonstrating human-level performance\"? yes, models are improving, but there is a bit of circularity in the slightly awkward wording here, as \"human level performance\" is being claimed in large part due to improvement over benchmarks, but at the same time, data contamination is becoming a bigger and bigger issue; possibly nuance the claim a bit", "extra_scores": {}}, {"review_id": "d1yM0wkb9H", "reviewer": "Reviewer_u2Ve", "summary": "The paper presents a new evaluation dataset (TMLU) for traditional Mandarin Chinese (Taiwanese). \nIt contains 37 subjects (social science, STEM, humanities and Taiwan specific). The contributions of this eval dataset and this paper are:\n\nIt aims at better evaluating LLMs on traditional Chinese, which has unique characteristics and challenges compared to simplified Chinese \nCompared to existing traditional Chinese evaluation datasets (TC-eval and TMMLU-plus), TMLU has the following advantages: \n1. not from a single website, using pdfs or words, not easily scrapable so avoiding **data contaminations**;\n2. more answerable from the question context;\n3. including CoT instructions and Latex expressions", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive comment!\n\n**Response to Q1**\n\nThank you for pointing this out! We apologize for not fully comprehending all the meaning of the term \"同志\" in Simplified Chinese. We will revise the illustrative example with other terms such as \"土豆\" (which typically means “potato” in Mainland China and “peanut” in Taiwan).\n\n**Response to Q2**\n\nWe acknowledge your concern regarding the limited scope of our dataset, which currently only evaluates knowledge without assessing other communicative skills. Although the primary focus of our dataset is to evaluate the LLM's proficiency in Taiwanese-Mandarin knowledge, we are committed to augment TMLU in the future to include more diverse data related to “non-daily” tasks and interactive conversation in Taiwanese Mandarin. Thanks for your constructive suggestion!\n\n**Response to Q3**\n\nThanks for the insightful question! The metric adopted in Figure 3, Min-K%, is an indicator of how likely the given text is in the pre-training data of the examined LM. The absolute number of Min-K% varies under different settings (e.g., model size, vocabulary size), and requires oracle labels of the tested dataset to determine a threshold that reflects whether contamination occurs (Please refer to Section 5.1 in the Min-K% paper).\n\nNonetheless, we believe a systematic and unifying behavior in Figure 3 (where TMMLU-plus records a worse number in every model tested) could imply that TMLU is less likely to be contaminated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is great to see a new dataset introduced to address the evaluation needs of traditional Chinese (Taiwan) languages. \n2. Authors demonstrate TMLU has advantages as an evaluation dataset compared to existing traditional Chinese eval benchmarks, including better handling of data contamination, less unanswerable questions.", "reasons_to_reject": "Although authors show the dataset has advantages compared to existing benchmarks, I hope the dataset can do even better. In particular, I hope the eval dataset can cover the following capabilities: \n1. Chatability: being able to understand a dialogue context \n2. Understanding nuances from languages \n3. Other use cases in addition to exams \n\nPlease see my questions below.", "questions_to_authors": "1. From my understanding “同志” can mean “comrade” and “homosexual” depending on the context even in simplified Chinese. I still agree with the point that words carry different meanings in simplified and traditional Chinese, and the challenge do exist for LLM evaluations, but this may not be the best example to showcase in the Introduction section? \n2. If I understand correctly, the data comes exclusively from examinations (high school, college or professional), that may impose some bias into this dataset in a way that we are encouraging LLMs to be good at answering those “non-daily” questions, but does not necessarily make models more “usable” at answering “daily” questions. So it mainly measures knowledge, not necessarily other aspects of language understanding, such as understanding the nuances in languages and LLMs’ usability and “chattability”. Think about people’s daily usage of LLMs, they may ask questions about trip planning, shopping ideas, financial knowledge, legal advice, emotional support, relationship questions, career advice, business knowledge, etc. I would like to see those covered as well. This could be out of the scope of this paper, though. I hope we will have more such kind of evaluation data in the future. \n3. Figure 3 shows that TMLU is less likely to be included in training data of these benchmarked LLMs, but not by a big margin. What are your takeaways on these results?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a benchmark to evaluate the performance of language models in understanding Taiwanese Mandarin, which is an important task, as all reviewers (including me) found, to improve inclusivity with the rapid development of large language model techniques, as well as evaluating a number of models on the proposed benchmark.
I resonate with Reviewer 8Aze that, in principle, it would be good to include some diagnostic test cases that better represent the linguistic and/or cultural differences between Chinese and Taiwanese Mandarin, especially when adapting a Simplified Chinese model.
That being said, the benchmark is still very valuable for localizing language models into Taiwanese Mandarin.

Please also follow the suggestions by Reviewers u2Ve and 8Aze to correct the Chinese Mandarin terms with misunderstanding.
Decision: Accept