TMMLU+: An Improved Traditional Chinese Evaluation Suite for Foundation Models
Authors: Zhi Rui Tam
Ya Ting Pai
Yen-Wei Lee
Hong-Han Shuai
Jun-Da Chen
Wei Min Chu
Sega Cheng
Venue: COLM (2024)
Keywords: Traditional Chinese evaluation, multi-choice question answering
TL;DR: We introduce TMMLU+, a comprehensive Traditional Chinese dataset covering over 66 subjects, with insights into tokenization effects and the impacts of Simplified versus Traditional Chinese on LLM performance.

Abstract: We present TMMLU+, a new benchmark designed for Traditional Chinese language understanding. TMMLU+ is a multi-choice question-answering dataset with 66 subjects from elementary to professional level. It is six times larger and boasts a more balanced subject distribution than its predecessor, Taiwan Massive Multitask Language Understanding (TMMLU). We also benchmark closed-source models and 26 open-weight Chinese large language models (LLMs) of parameters ranging from 1.8B to 72B on the proposed TMMLU+. Our findings reveal that (1.) Traditional Chinese models still trail behind their Simplified Chinese counterparts, highlighting a need for more focused advancements in LLMs catering to Traditional Chinese. (2.) Current LLMs still fall short of human performance in average scores, indicating a potential need for future research to delve deeper into social science and humanities subjects. (3.) Among all the tokenization compression metrics examined, we identify that only the fertility score uniquely demonstrates strong correlations with our benchmark results. We foresee that TMMLU+ will pinpoint areas for future model improvement, thereby narrowing the gap between machine and human linguistic capabilities and supporting researchers in developing Traditional Chinese LLMs. Our dataset, along with the benchmark source code, is accessible at huggingface.co/datasets/ikala/tmmluplus.

Published as a conference paper at COLM 2024
TMMLU+; An Improved Traditional Chinese Evaluation Suite for Foundation Models
Zhi Rui Tam*; Ya-Ting Pai Yen-Wei Lee, Jun-Da Chen, Wei-Min Chu, Sega Cheng iKala Lab
(avery- pai, patrick. @ikala.
Hong-Han Shuai
National Yang Ming Chiao Tung University
Abstract
We present TMMLU+, new benchmark designed for Traditional Chinese language understanding. TMMLU+ a multi-choice question-answering dataset with 66 subjects from elementary to professional level. is six times larger and boasts a more balanced subject distribution than its predeces- sor, Taiwan Massive Multitask Language Understanding (TMMLU). We also benchmark closed-source models and 23 open-weight Chinese large language models (LLMs) of parameters ranging from 1.8B to 72B on the proposed TMMLU+. Our findings reveal that (1.) Traditional Chinese models still trail behind their Simplified Chinese counterparts, highlighting a need for more focused ad advancements in LLMs catering to Traditional Chinese. (2.) Current LLMs still fall short of human performance in average scores, indicating potential need for future research to delve deeper into social science and humanities subjects. (3.) Among all the tokenization compression metrics examined, we identify that only the fertility score uniquely demonstrates strong correlations with our benchmark results. We foresee that TMMLU+ will pinpoint areas for future model improve- ment, thereby narrowing the gap between machine and human linguistic capabilities and supporting researchers in developing Traditional Chinese LLMs. Our dataset, along with the benchmark source code, is accessible at datttassts//tttstss/sssss
huggingface.co, datasets/ iikaa/ tmmluplus
1 Introduction
LLMs are a group of transformer models, typically trained in the autoregressive method. They can be used to solve a wide range of tasks in Natural Language Understanding (NLU) (Rajpurkar et al. 2016 Wang et al., 2019), vision processing (Goyal et al. 2017 Fu et al., 2023), and robotics (Padalkar et al. 2023). As the abilities of LLMs continue improve, existing benchmarks start to saturate and supersede the human baseline. For example, Massive Multitask Language Understanding (MMLUU (Hendrycks et al.) 2021a) is created to test knowledge understanding across wide range of topics related to English and US culture. Since then, many subsequent works (Dac Lai et al. 2023 Son et al. 2024 Yin & Kawahara 2024 have tried to extend the same multi-choice question-answering format with a wide variety of topics into other languages.
Recent efforts have sought to adapt this approach to the Chinese language. For example, as outlined in Huang et al. (2023) Li et al. (2023), the primary focus is on Simplified Chinese and the cultural milieu of mainland China. However, the Traditional Chinese domain, which is used by 30M population around the world, nowhere to be found. Given that there are stark differences between these 2 languages despite sharing the same root. A primary distinction between Simplified and Traditional Chinese lies in their divergent
"Equal contribution
Work performed while at iKala AT Lab

---

Published as a conference paper at COLM 2024
vocabularies. For instance, the concept of "computer memory" is expressed "[A]# in Simplified Chinese, while in Traditional Chinese it is "Mu"s"" A direct character-by- character conversion of the Traditional term to Simplified Chinese would yield "iž f4", which occupies an entirely different semantic space. This lexical divergence is further complicated by cases where identical characters carry disparate meanings across the two writing systems. A salient example is the character " "H..", which in Simplified Chinese can denote either "ugly" or the period from AM to 3 AM, depending on context. In contrast, Traditional Chinese employs "N'. to convey "ugly," while reserving "H." exclusively for the aforementioned time designation.
Hsu et al. 2023) proposes TC-Eval, which features TMMLU, Traditional Chinese multi- choice question-answerig dataset. TMMLU covers 55 subjects and includes a total of 3,300 questions. Despite its advancements, it presents several limitations: Firstly, its scope is relatively narrow, with only 55 subjects and some featuring less than 100 questions, leading to potential gaps in subject diversity and coverage. Secondly, there are inconsistencies in the question formats, with 5% of questions deviating from the multi-choice format and 1% including more than four options. Lastly, the absence of a dedicated development set restricts the exploration of few-shot prompting techniques. Addressing these issues is crucial for advancing LLMs' understanding and applicability in linguistically and culturally diverse contexts.
and
We propose TMMLU+ an enhanced version of TMMLU, which aims to address the is- sues highlighted above. TMMLU+ contains 22,690 questions across 66 subjects, ranging from primary, secondary, undergraduate, and professional levels of education. Similar to Hendrycks et al. (2021a); [ et al. (2023), we include STEM subjects: physics, chemistry, and computer science with mentions of Traditional Chinese culture. TMMLU+ bench- mark specifically addresses the terminology differences between Traditional and Simplified Chinese, ensuring that models can accurately understand and respond to questions in the context of Taiwan. The terminology used in Traditional Chinese often differs from that in Simplified Chinese, reflecting variations in character forms, vocabulary, and usage. All subjects contain at least 110 questions, with 5 development questions for few-shot prompting.
We conduct experiments on 23 open-weight language models ranging from 1.8B to 72B from 5 different architectures of different pretraining datasets. To our surprise, Simplified Chinese language models outperform Traditional Chinese models in the same parameter level by at least 19%. The nop-performing open-weight models are Qwen-72B and Qwen-14B, followed by Yi-34B-Chat model, with scores of 64.3%, 50.5%, and 49.6%, respectively. These models are predominantly used for Simplified Chinese. Despite the impressive results of these models, there is still a gap between their performance and that of human test-takers. On average, exam takers can score around 68% the compared subjects. We have found that LLMs struggle in math-heavy subjects such as engineering math and accounting.
To dive deeper into the differences between Simplified Chinese language models and Traditional Chinese language models, we conduct translation experiments on TMMLU+ where we convert the Chinese characters (Traditional Chinese to Simplified Chinese) and repeat the evaluation on the newly converted version. We found that most Chinese language models perform better in simplified versions, with some showing a smaller margin than others. We also conduct experiments on few-shot prompting and chain-of-thought (CoT) prompting (Wei al. 2022). We have found only few-shot prompting increases performance in most evaluated models while CoT prompting degrades. We also study wein-t Chinese language models from various architecture with different vocabulary sizes. Our analysis has found strong correlation between the fertility scores and benchmark results.
2 Datasets
We introduce TMMLU+, benchmark dataset to evaluate Traditional Chinese LLMs'' multi- task language understanding capabilities.
2

---

Published as a conference paper at COLM 2024
2.1 Task overview
The dataset comprises 22,690 multi-choice questions from 66 subjects ranging from primary to professional level. These questions cover a wide range of subjects, including humanities, social science, and STEM fields. It includes universally applicable topics like physics, chemistry, and macroeconomics. Furthermore, some questions will require test-takers to choose the correct combination of options, such as 'ABCD, their answer. In addition, we include questions where test-takers must provide comparison results, as shown in Figure
TH ?
HH Pi 2A P. li -
(Which of the following is the correct order of electromagnetic radiation fre- quencies?)
*
d # -
(Taitung experiences extremely high temperatures in Taiwan, primarily due to the foehn wind generated as ty- phoons pass by. At what location is the typhoon's s center when Taitung is most likely to undergo foehn winds, leading to temperature increase?)
I: microwave
II: Y -rays
III: visible
IV IR
V: UV
X T * (Pacific Ocean)
A. 1> IV V
B. ii (Bashi Channel)
B. II > V > IV>
C. lf (Taiwan Strait)
C. V>>I TV>> IV>
D. (South China Sea)
D. V III IV I
Figure 2: Taiwanese geography found in the Geography subject
Figure Electromagnetic quency Order
Radiation Fre-
Additionally, the dataset features questions requiring specific knowledge of Taiwan, sourced from standardized tests in subjects such as Taiwanese Hokkien, the culture of the aboriginal peoples of Taiwan, and the geography of Taiwan shown in Figure 2 Questions from the Taiwanese Trivia Question Answering (TTQA) (Ennen et al., 2023) dataset are also included. TTQA is multi-choice question evaluation about culture, geography, and history specific to Taiwan. To correctly answer these questions, a model needs to have an understanding of Taiwan culture. To view the list of included subjects, please refer to Appendix
2.2 Data collection
We have gathered previous exams between 1998 and 2023 from various public exams such as certificate exams, comprehensive assessment programs for junior students, and college entrance examinations. This results in 962 exams from different standardized test resources and PDF documents available online. In addition, we have also included subjects from TMMLU (Hsu et al.) 2023) and TTQA (Ennen et al., 2023). Subjects related to culinary arts have been consolidated into unified culinary category, and questions about dentistry have been incorporated into our preliminary set of questions. To ensure that the testing set contains more than 100 questions, we have manually added 29 more questions to the 103 questions from TTQA. We have named this new subject TTQAv2, as it is distinct set of questions from TTQA.
We then eliminate questions containing images and transform mathematical and chemical equations into LaTeX format using GPT-4-vision (OpenAI 2023) capabilities. Next, we use n-gram filtering (Broder 1997| to remove similar questions within the same subject and eliminate redundancy. Lastly, we manually check all questions and remove any that require figures for answers depend on the context of preceding questions. Once the
3

---

Published as a conference paper at COLM 2024
questions are finalized, we follow the same practice from HemdHycdkssss et al. (2021a); Huang et al. (023); Li et al. (2023) by randomly shuffling the choices of each question to ensure uniformly distributed. This results in random guess accuracy
all subjects answers
of 25%. After refining the question pool, we establish human baseline scores for 44 of the 66 subjects. These baselines are instrumental for conducting nuanced ablation study human performance benchmarks in subsequent phases of our research.
2.3 Format
Each subject then split into test, dev, and validation. We ensure that each subject contains at least 110 questions, with 5 development questions for few-shot prompting, and at least minimum of 5 validation questions. Every question has four choices: A, B, C, or D, and the model is prompted to select one answer.
All 66 subjects can be grouped under 4 categories: STEM, Humanities, Social Science, and Others. For example, STEM subjects would test the arithmetic ability of LLMs, while humanities and social science focus more on the knowledge brevity of the models. Table shows the statistic results for all categories.
Category
Test
Dev
Validation
STEM
3,458
70
385
Social Sciences
5,958
90
665
Humanities
1,763
35
197
Other (Business, Health, Misc.)
8,939
135
995
Total
20,118
330
2,242
Table 1: Data Categories and Counts
2.4
Evaluation
For all subjects, we calculate the average accuracy score first followed by averaging all subjects by their subject category. Finally, an average score is calculated based on the categories' average scores.
2.5 Check for pretraining data leakage
Given the age of our questions, we suspect they may be present in the LLMs' training data, potentially compromising our goal to benchmark the L.Ms'' abilities. We Min-K%++ (Zhang et al. 2024) which is the current SOTA in pretraining data detection for LLMss. We use Qwen-7B the model and find that only 0.48% questions exist in its pretraining data.
Experiments
We conduct evaluations 23 distinct LLMs, focusing multilingual and Chinese-specific models. Each model varies aspects such as tokenization, pretraining datasets, size of parameters, and architectural design. Furthermore, we compare the performance of these LLMs against datasets comprising human-generated answers to assess their relative effec- tiveness. Additionally, we investigate several factors that could influence the performances of LLMs on these datasets.
3.1 Prompting method
In our approach, we formulate the prompts for the LLMs in both zero-shot and few-shot settings. For the few-shot setting, utilize the development splits, specifically employing the provided set of five examples as context examples. The complete prompts can found in Appendix
4

---

Published as a conference paper at COLM 2024
STEM
Soc. Sci.
Humanities
Other
Average
Model
Lang
0-shot
5-shot
O-shot
5-shot
0-shot
5-shot
0-shot
5-shot
0-shot
5-shot
Random
25.0
25.0
25.0
Qwen-728
SC
61.1
60.8
71.7
72.3
63.0
63.4
61.3
62.1
64.3
64.7
GPT-4
multi
60.4
66.0
67.4
74.9
56.0
63.6
57.6
65.3
60.3
67.5
Qwen-148
SC
46.9
46.4
56.7
58.1
49.4
51.5
48.8
49.4
50.5
51,4
Gemini-pro
multi
45.4
45.6
57.3
59.1
48.8
49.3
48.2
50.4
49,9
51.1
Yi-34B-Chat
SC
35.8
42.8
54.0
37.2
39.9
49.6
38.9
Qwen1.5-14B-Chat
SC
39.7
48.1
52.8
59.0
43.9
51.4
45.0
50.9
45.3
52.4
Claude-133
multi
GPT-3.5-turbo
multi
41.6
42.8
46.7
51.2
36.7
41.2
42.0
46.3
41.8
45.3
Claude-3.0 Opus
multi
43.0
61.3
45.5
69.6
35.8
63.0
40.2
63.0
41.1
64.2
Qwen-7B
SC
37.5
38.4
45.5
48.3
38.1
42.3
39.0
42.0
40.0
52.0
Yi-9B
SC
28.7
44.7
44.6
59.0
48.5
49.7
36.8
49.7
39.6
59,8
Qwen-1.8B
SC
32.6
31.3
38.9
37.0
38.3
32.4
35,3
34.3
36.3
33,8
Claude-2.0
multi
39.7
11.7
39.1
35.6
28.6
20.8
37.5
19.5
36.2
21.9
1rneze-7H-Instructer...0
TC
26.7
21.1
43.0
32.5
39.8
32.4
34.8
28.3
36.1
28.6
ChatGLM3-6B
SC
31.0
32.8
39.3
38.3
35.6
34.2
35.6
35.0
35.4
35.1
Taiwan-LLM-13B
TC
18.5
25.3
17.8
20.5
21.3
Taiwan-LLMM778
TC
15.0
7.7
16.2
4.5
15.0
3.2
16.2
5.1
15.6
5.1
Table 2: Zero-shot/Fiveeshot average accuracy (%) in answer-only setting. The average
accuracy over the subjects within each category. SC: Simplified Chinese, TC: Traditional Chinese
3.2 Setup
For open-weight models, we use the chat template provided by each model if available and use the same prompting format (Huang al. 2023) for non-conversation LLMs. Responses are parsed using modified regular expression by Huang et al. (2023) with Traditional Chinese characters added. For close-source models, we obtain the response via APIs and parse using the same set of parsers used by open- weight models.
3.3 Models
To provide extensive overview of the current landscape of Chinese LLMs, conduct benchmarks not only on Traditional Chinese-specife models (Lin & Chen 2023 but also Simplified Chinese LLMs (Bai et al. 2023 Du et al. 2022). Our evaluation encompasses a range of well-regarded closed -ource LLMs, including those from Open GPT series (Brown et al. 2020), Gemini (Team 2023), and Claude (Anthropic 2023 2024). Within the OpenAl GPT suite, our tests incorporate GPT-4 from the gpt-4-0613 model and GPT--355- turbo from the gpt-3.5-turbo-6113 model. For Gemini models, use the Gemini-pro from the gemini-1.0-pro-000 model, the only publicly accessible Gemini model at the current point of writing. Regarding Claude models, we include Claude-1.3, Claude-2.0, and Claude-3.0 (Anthropic) 2024) in our evaluation where Claude-1.3 is solely trained in the supervised fine-tuned (SFT) (Lowe & Leike, 2022), whereas Claude-2.0 undergoes training in the SFT stage, followed by reinforcement learning from human feedback (Bai et al.. 2022 The full models' detail description can be found in Appendix
3.4 Results
3.4.1
Comparative Analysis of Diverse Model Architectures
In our benchmark, we examine a spectrum of open-weight models, with parameter counts ranging from 1.8B to 72B (Bai al. 2023 Lin Chen 2023 Ennen et al. 2023 Du al.
2022). as well as proprietary models that are evaluated via API calls (OpenAI) leam
2023 Anthropic 2023 2024.. Table presents our findings, which indicate a general trend where models with larger numbers of parameters tend to show improved performance across various domains. It is noteworthy that the open-source model, Qwen-72B, achieves superior zero-shot learning scores in comparison to the closed-source GPT-4 model, albeit GPT-4 exhibits enhanced performance few-shot scenarios, surpassing Qwen-72B by considerable margin.

---

Published as a conference paper at COLM 2024
Model
Lang
STEM
Soc. Sci.
Humanities
Other
Average
Human
62.9
72.1
69.5
68.5
68.2
Qwen-72B
SC
63.9
69.9
52.1
60.6
62.1
GPT-4
multi
61.7
65.7
49.2
57.0
58.4
Qwen-14B
SC
48.3
54.8
40.3
46.4
47.5
Gemini-pro
multi
47.7
54.7
42.2
45.9
47.6
GPT-3.5-turbo
multi
43.6
43.8
31.5
40.3
29.8
Table 3: Zero-shot average accuracy (%) in 44 subjects with human answers data
Disparities in performance are noted even among models with comparable parameter sizes, which can be attributed to differing developmental approaches. Such discrepancies highlight the importance of model architecture and training regimen over the mere scale of parameters. We intend to further investigate these differences, focusing on how factors such as model tokenization contribute to the final accuracy in Section 3.5.4 Detailed comparisons across all models are available in Appendix
3.4.2 Human baseline
In this analysis, we compare human test-takers and range of LLMs, including Qwen-72B, Qwen-14B, and the proprietary GPT-4, GPT-3.5-turbo, Gemini-Pro. The human baseline scores, as represented in Table 3 have been revised to reflect the passing rate based on the proportion of correct responses total responses for each question. To illustrate, if question offered four possible answers and the majority of human respondents selected the correct one, the passing rate for that question would be calculated by dividing the number of correct answers by the total number of responses.
Our result indicates that human test-takers maintain a lead over LLMs in zero-shot learning tasks, with average accuracy of 68.2% across all domains. Notably, humans exhibit the highest proficiency in Social Sciences and Humanities, areas where nuanced understanding and contextual knowledge play significant roles. In comparison, the highest-performing model, Qwen-72B, shows an average accuracy of 62.1%, with particular strength in Social Science and STEM-related tasks.
3.4.3 Comparative Results Across Different Subjects
Based on results from Tables 2 we find subjects that necessitate mathematical reasoning, such as those in the STEM category, consistently show lower performance metrics compared to subjects that are more reliant on background knowledge and linguistic comprehension, such as Social Sciences, Humanities, and Other categories. This trend is evident in both prompting strategies, where models like GPT-4 and Qwen-72B while performing admirably across all categories, exhibit noticeable dip in their performance in the STEM field.
3.5 Analysis and discussion
3.5.1
Comparative Analysis of Traditional and Simplified Chinese Characters
This study aims investigate the impact of the linguistic differences between Traditional and Simplified Chinese characters on the performance of LLMs. Traditional and Simplified Chinese characters share only about 30% of their vocabulary, suggesting that LLMss trained in Traditional Chinese may encounter difficulties when processing Simplified Chinese text. Understanding the linguistic flexibility and limitations of LLMs within the context of these distinct yet interconnected writing systems is crucial. We utilized the widely recognized opence (Kuo) 2023) tool to convert TMMLU+ questions and prompts from Traditional to Simplified Chinese. As detailed in Table 4 Traditional Chinese LLMs tend to perform better in the Traditional Chinese version of TMMLU+ in Social Sciences, which generally require deep understanding of cultural context, and also show slight advantage in other domains

---

Published as a conference paper at COLM 2024
Model
Lang
STEM
Soc.
Sci.
Humanities
Other
Average
SC
TC
SC
TC
SC
TC
SC
TC
SC
TC
GPT-3.5-turbo
multi
42.1
41.6
46.3
46.7
37.6
36.7
41.7
42.0
41.9
41.8
Gemini-pro
multi
46.7
45.4
59.1
57.3
51.2
48.8
48.8
48.2
51.5
49.9
Qwen-14B
SC
48.1
46.9
58.0
56.7
49.8
49.4
49.3
48.8
51.3
50.5
Qwen-7B
SC
39.8
37.5
45.5
40.6
38.1
39.0
42.1
40.0
Breeze-7B-Imstructte/
TC
29.2
26.7
40.7
43.0
38.2
39.8
34.9
34.8
35.7
36.1
Taiwan-LLaMa-13B
IC
15.5
18.5
19.9
27.6
13.5
17.8
15.0
21.5
16.0
21.3
Taiwan-LLaMa-7B
IC
14.3
15.0
16.4
16.2
16.6
15.0
16.2
16.2
15.9
15.6
Table 4: Comparative Results on TMMLU+ in zero-shot settings: Simplified vs. Traditional Chinese
such as Humanities and STEM. Our findings indicate that using Traditional Chinese prompts can enhance the performance of LLMs trained specifically Traditional Chinese, while Simplified Chinese prompts significantly improve the performance of multilingual and Simplified Chinese LLMs.
3.5.2 Language Models Performance Simplified Chinese Benchmarks
In this section, we compare our dataset against established Simplified Chinese benchmarks, namely CMMLU (Li et al. 2023) and C-Eval (Huang et al. 2023), using selection of language models: Qwen series, both Traditional Chinnes L.MSs Breeze and Taiwan LLaMA as well as GPT-4. The comparison in Figure revealed that the models tended to show consistent level of accuracy, answering questions from both CMMLU and C-Eval with similar accuracy. The figure highlights that all chinese language models generally performed better on the Simplified Chinese benchmarks, with trend of results moving toward the upper left corner. Comprehensive outcomes for these models can be found in the Appendix
CMNUU
TMMLL C-Eval
Breaze-76-Inie 0
Qwen-1.88
Owen-140
Owen-728
0.6
Owen-70
0.6
T=-wan-LL-118-222 O-chat
0.5
Wai 70-22.-t2.
GPT-4
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.6
0.8
0
0.2
0.4
0.0
0.8
IMMLUH
TNMUU+
Figure 3: Comparison Model Performance: TMMLU+ vs. C-Eval and CMMLU five shot prompting
3.5.3 Does CoT improve performance on TMMLU++
As observed in GPT-4 (OpenAI 2023) and Gemini-Pro Team 2023), we use the fhw-shot CoT improve the model reasoning ability in STEM subjects found in MMLU (Hendrycks et al. 2021a). We modify the prompt from a direct answer to the Chinese version of the Col similar to Li et al. (2023) but using Traditional Chinese reasoning. All the thought processes are first generated by GPT-4 (OpenAl 2023) and reviewed by humans. Table shows performance degradation in the five-shot Col evaluation setting which coincides with previous Chinese multi-choice question (Huang et al. 2023
Liet al. 2023) results.
7

---

Published as a conference paper at COLM 2024
Model
STEM
Soc.
Sci.
Humanities
Other
Average
w/o
CoT
w/o
CoT
w/o
CoT
w/o
CoT
w/o
CoT
Gemini-pro
45.6
46.4
59.1
52.2
49.3
41.4
50.4
45.2
51.1
46.3
Qwen-14B
46.4
40.1
58.1
47.2
51.5
39.7
49.4
40.5
51.1
41.9
GPT-3.5-turbo
42.8
30.2
51.2
33.0
41.2
27.2
46.3
29.6
45.3
30.0
Table 5: Average accuracy (%) in a five-shot scenario with with/without CoT setting. Scores highlighted in bold indicate enhanced performance relative to five-shot without CoT
Model
Renyi Entropy
Average Token
Fertility
TMMLU+ Average
Yi-6B-Chat
0.516
90.32
1.49
44.64
Qwen-7B
0.459
85.92
1.40
40.01
Owen-7B-Chat
0.459
85.92
1.40
39.53
BlueLL--B--Base
0.519
92.37
1.56
37.90
Breeze-7B-Instruct-1..
0,453
79.10
1.30
36.10
M3a6G
0.494
82.78
1.38
35.40
DeepSeek-I1M-77BCCh..
0.532
121.10
1.99
35.17
BI.0S50M-+331mmsstraa-7
0.519
135.61
2.25
34.15
Yayi-7b
0.453
73.09
1.12
28.95
DeepSeck-LLMM77BA Base
0.532
121.10
1.99
26.31
Mistral-7B-80k
0.460
135.00
2.23
23.79
Chinese-Llama-2-76
0.459
135.61
2.84
16.01
Yayi-7B-Llama2
0.459
171.43
2.84
15.91
Taivar-LLMe7B--211-hhha
0.459
135.61
2.84
15.61
Atom-7B
0.478
119.20
1.97
9.68
Pearson Correlation
0.366
-0.685
-0.742
Table 6: All 7B model zero-shot results for fertility score ablation study
We attribute this drop to insufficient relevance among bridging objects, detailed in Section 5 (Wang et al. 2023). As we go through each reasoning examples we found most errors contributed by false understanding of the world which leads to incorrect answer.
LLaM
2.75
GPT-neo
Bloom
Rate
2.25
1.50
10
15
20
25
35
40
Accuracy (%)
Figure 4: Fertility score and average TMMLU+ accuracy from 15 different LLMs around 7B parameters
3.5.4 Impact of 'ookenizer Vocabulary
In our study, we identify a notable correlation between the size of a model's vocabulary and its performance in TMMLU+ tasks. Despite the broad range of pretraining corpora sizes, from 10 billion to 2 trillion tokens, which typically remain undisclosed, our findings highlight the significant impact of tokenizer design model effectiveness. Specifically, in
8

---

Published as a conference paper at COLM 2024
the context of machine translation, reference the study by (Zouhar et al. 2023), which demonstrates a strong correlation between Rényi entropy-a measure reflecting the diversity and distribution of tokens-an and the BLEU score, widely recognized metric for evaluating translation accuracy. Another work (Ali et al. 2023) has found that a low fertility score for tokenizers across different languages crucial for multilingual performance. A tokenizer with a low fertility score, when presented with the same sentence, generates a shorter sequence of tokens compared Taiwan-LLaMA, 59.5% of Traditional Chinese characters in TMMLU+ are encoded with 256 bytes. This poses a challenge for the transformer module as it increases the input sequence while operating from approximately 15 LLMs around 7B parameters LLaMA (Touvron al. 2023a b Jiang et al. 2023), Qwen (Bai et al., 2023), Bloom (Scao et al. 2022) and GPT-NNe (Black et ata,, architecture, and plot the correlation between their tokenizer fertility scores and average accuracy. Our result, shown in Figure reveals significant Pearson correlation coefficient of -0.742 between fertility score and average accuracy, indicating that high fertility
tokenizer with higher fertility score. In the case of
a smaller vocab subset. To validate our hypothesis, we aggregate data
negatively impacts downstream performances.
3.5.5 Comparative Analysis of Error Patterns in LLMs and Human Test-Takers Takers
In this section, we undertake thorough examination of the error patterns exhibited by various LLMs in comparison to those made by human test-takers. Our goal is discern whether the mistakes made by these advanced language models are akin to those typically encountered in human responses.
We partition the questions into three distinct categories based their difficulty levels: easy, medium, and hard. The difficulty level of each question is determined through an empirical approach. First, compute average threshold for the correct response rates of humans across all questions. This threshold is used as a benchmark to categorize each question. Specifically, question is classified as 'hard' if its correct response rate is lower than equal to 25%, 'medium' if the rate is above 25% but lower than or equal to the average threshold, and 'easy' the rate is higher than the average threshold. Second, the categorized data is then used to populate arrays corresponding to each difficulty level, thereby enabling comparative analysis of error patterns across the different categories. Finally, we analyze the distribution of errors in various categories and find that Qwen-72B outperforms humans in difficult questions across all major categories. However, most questions are dominated by easy difficulty, resulting in an average Qwen-72B score that falls short compared to humans.
STEM
Human Rlls
Social Science
Other
6.8
0.7
0.7-
Qwen-726
human
0.6
0.6-
Average Score
0.5-
0.5
0.5
0.5
0.4-
0.4
0.4
0.4-
0.3
0.3
0.2
0.1
Mway
mediium
hard
maay
mnddum
waay
medism
maay
Figure 5: Human level difficulty comparison with Qwen-72B zero-shot settings.
3.5.6 Comparison of Scoring Methods
Existing evaluation libraries for multi-choice question answering, such as Im-eval- (Gao et al. 2023), typically measure the negative log-likelihood of choice terms and select the option with the highest log probability as the final answer. In contrast, our evaluation relies on string matching to extract the final answer, more closely mimicking real-world user interactions. However, this method can yield scores lower than random choice if the parser fails to extract final answer. We compared these two methods across three evaluated

---

Published as a conference paper at COLM 2024
Our findings indicate that the likelihood method generally produces higher scores. Given this discrepancy, we recommend that future comparisons of newer LLMs carefully consider and explicitly state the scoring method used to ensure fair and accurate benchmarking.
models, shown in Table 7
Model
STEM
Soc. Sci.
Humanities
Other
Average
Likelthood
String
Likelhood
String
Likethhood
String
Likelibood
String
Likelilood
String
Dreeze-7B-nnttrut--1..
26.7
43.0
38.57
40.73
36.1
Qwen-7B
35.29
37.5
43.27
45.5
34.49
38.1
39.12
39.0
39.28
40.0
Taiwan-LLM-7B 26.37
15.0
30,26
16.2
26.21
15.0
28.88
16.2
28.62
15.6
Table 7: Comparison of Models Across Different Fields with Likelihood (Gao et al. 2023 and String Parsing (ours) results
4 Related Works
Many benchmark datasets have been created to evaluate the abilities of models in NLU tasks. Some of these benchmarks focus on specific subjects. For instance, the GLUE benchmark (Wang et al. 2019) is a collection of NLU tasks that mainly concentrate on language skills, A12 Reasoning Challenge (Clark et al. 2018) is a dataset comprising multi-choice questions from science exams at various grade levels. On the other hand, MATH (Hendrycks et al. 2021b) is designed to evaluate LLMs' ability to solve mathematical problems. MMLU (Hendrycks et al. 2021a) is benchmark used to assess models' knowledge comprehension in different domains. It provides broad and in-depth coverage of subjects, making it widely used for evaluating LLMs' abilities in knowledge understanding. The proliferation of benchmarks is increasing and expanding; however, it is noteworthy that most of these benchmarks have been developed in English.
With Chinese being the language having the largest user base worldwide, there also has been significant increase in the number of benchmark datasets available in Chinese recently. For example, bilingual benchmark including both English and Chinese has been released Zyonh et al. (2023) which focuses on standardized tests. Furthermore, some datasets are tailored to specific fields, such as CMATH (Wei et al. 2023), which evaluates the ability of
LLMs to solve mathematical problems at the elementary level. Additionally, Zeng (2023 proposed MMCU, which covers four key areas: medicine, law, psychology, and education.
Researchers have also created benchmarks that cover a wide range of subjects and difficulty
levels from elementary school to college, such as M3KE (Liu et al. 2023), CMMLU (Li et al..
2023
and C-Eval (Huang et al. 2023).
However, considering the discrepancies in idiomatic usage and cultural nuances between Traditional Chinese and Simplified Chinese users, the benchmark datasets in Simplified Chinese might not be entirely suitable for accurately assessing LLMs' NLU capabilities in
Traditional Chinese, Despite this scarcity, there are a few existing benchmarks specifically designed for Traditional Chinese. These include DRCD (Shao et al. 2019) focusing on read- ing comprehension, TTQA (Ennen et al. 2023) emphasizing laiwanese-related knowledge extracted from Wikipedia pages. Also, |Ssu al. (2023) released benchmark encompassing a diverse range of knowledge in 55 subjects from laiwanese standardized examinations.
Conclusion
In this work, we introduce TMMLU+, an improved benchmark for evaluating large language models understanding in Traditional Chinese through multi-choice question answering. Our preliminary results have found that existing Traditional Chinese LLMs still fall behind their Simplified Chinese counterparts in terms of understanding Traditional Chinese. Fur- thermore, our analysis indicates that none of the current LLMs achieve an average score surpassing human performance, highlighting the challenging nature of our dataset and its potential to drive forward the development of more sophisticated language understanding systems.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
We would like to thank Cheng-Kuang Wu for his review and valuable feedback. We also thank Wen-Zhong Fu for adding our dataset to EleutherAI's Im-evaluation-harnes. tool.
References
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Jo- hannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Schulze Buschhoff, et al. Tokenizer choice for Ilm training: Negligible or crucial? arXiv preprint arXio:2310.08754, 2023.
Anthropic. Model card and evaluations for claude models. https://wux-files.ce awt..ooppss...h.ii....... htap:p/rodacciioo//mgggeellllloo -arp:a/aeeeeeeeleeearaaaaa-aaaaa Claddd- pdf 2023.
Anthropic. The claude model family: Opus, sonnet, haiku. co11666555..07772241610000010110
http:////www..ccnaaannooppppppp anthropic
2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar XxXo:22399.1669, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXio preprint #rXiv:224.005862 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open- source autoregressive language model. Xiv preprint ar arXiv:2204.06745 2022.
Andrei Z. Broder. On the resemblance and containment of documents. Proceedings. Com- pression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pp. 21-29, 1997. URL
https::/api.. www.atiisccooooooooooo. Sow.ammmiccci.ll..oooooooopoooaa
wtwp/oopps 11748509
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few- shot learners. ArXiv, abs/ 200.. 14165, 2020. httppttss///rrtt http://xxxi...///bbbaaa/a wttg://a///00..1111111
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. ar e-prints, pp. arXiv-2307, 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin-Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, and Wei-Yum Ma. Extending the pre-training of bloom for improved support of traditional chinese: Models, methods and results, 2023.
11

---

Published as a conference paper at COLM 2024
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint ar XXix:2306.1339, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopí, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo wwtp:oooo//oeeeeeeeeeeeeeeeeeeee regg//rrrdd//2253333
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://arxiv. wwwpariiiv..orraaaee
03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL URL httpt:t//aaxii...trrrsrere http://awwvv.orr/a//////////// 03874
Chan-Jan Hsu, Cha Liu, Veng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da shan Shiu. Advancing the evaluation of traditional chinese language models: Towards comprehensive benchmark suite, 2023.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arviv preprint ar xrXi::231000688 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:22199- 22211,,, 2022.
Chia-Pao Kuo. Open chinese convert. httpe://ggthubb.ca/////ddd///
2023.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023.
Yen-Ting Lin and Yun-Nung Chen. Taiwan 11m: Bridging the linguistic divide with culturally aligned language model. arxiv preprint ar xxX1::2311.1747,, 2023.
Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, Xiaowen Su, Qun Liu, and Deyi Xiong. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models, 2023.
Ryan Lowe and Jan Leike. Aligning language models to follow instructions. OpenAl Blog. January, 27, 2022.
OpenAl. Gpt-4 technical report. ArXio, abbs/2303.08774,, 2023.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open embodiment: Robotic learning datasets and rt-x models. arXio preprint 2rx10:3310.0886, 2023.
12

---

Published as a conference paper at COLM 2024
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arxiv preprint (rXo::1606.05250 2016.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. Bloom: A 176b-parameter open-access multilingual language model. arviv preprint arXiv:2211.05100, 2022.
Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. Dred: chinese machine reading comprehension dataset, 2019.
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask language understanding in korean. arviv preprint ar 0atp24022.1148,, 2024.
Gemini Team. Gemini: A family of highly capable multimodal models. Technical report, Technical report, Google, 12 2023. URL hhttps://storagss googleapis. com ...., 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXi::2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. ar preprint 7XXi:2307.09288 2023b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In International Conference on Learning Representations, 2019, URL https://openreviee
net./forum7.derrA4m225555
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Ammual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2717-2739, 2023. doi: 10.18653 21006653/1/2222accloooo 153. URL https //aclanthology- org. acl acl-long.....o..... 153
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24877 2022.
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023. URL /ttttt:/s//aaiiiii org/ /abs//
2306. 16636
Ziqi Yin and Daisuke Kawahara. Japanese massive multitask language understanding benchmark. httpps://github wom/nlp-aaeed//d 2024. Accessed: August 1, 2024.
Hui Zeng. Measuring massive multitask chinese understanding, 2023.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXio:2404.02936, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
Vilém Zouhar, Clara Meister, Juan Luis Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. Tokenization and the noiseless channel. In Anmual Meeting of the Association for Computational Linguistics, 2023. URL https:://// / /api.. semanticscholar. www.oolla..roooooooooooo org/CorpusiD
259286948
13

---

Published as a conference paper at COLM 2024
A TMMLU+ Overview
Data Source
1. Data was collected from different standardized test resources and PDF documents available online (90.88 %)
2. Taiwanese Trivia Question Answering (8.45 %)
3. Taiwan Massive Multitask Language Understanding (0.67 %)
Collection Method(s) Used
1. Artificially Generated
2. Scraped or Crawled
3. Manually parsed
Data Processing
1. Eliminate questions containing images and transform mathematical and chemical equations into LaTeX format using GPT-4-vision
2. Use n-gram filtering to remove similar questions within the same subject and eliminate redundancy
3. Manually check all questions and remove any that require figures for answers or depend the context of preceding questions
Data Format
Field Name
Field Value
Description
Integer
The order number of the data point
question
String
The multi-choice question
String
Choice for the question
B
String
Choice B for the question
C
String
Choice for the question
String
Choice for the question The ground-truth answer of the question
answer
String
Table 8: Data Format
Motivations
1. Purpose: Research
2. Domain(s) of Application: Machine Learning, Natural Language Processing, Large Language Models (LLMs)
3. Motivating Factor(s): Narrow the gap between Traditional Chinese LLMs and Simplified Chinese LLMs development
B Details of TMMLU+
12 shows each of the subjects found in TMMLU+ with their broad category, well total number of questions.
Subject details Table 9 10 11
14

---

Published as a conference paper at COLM 2024
Subject Name
Chinese Name
Questions
Engineering Math
11
T
119
Organic Chemistry
126
Advance Chemistry
K2
142
Physics 14
113
Secondary Physics 15
T 40
130
Pharmacy 16
440
Computer Science
17
#
198
Basic Medical Science
18
1065
statistics and Machine Learning
19
254
Junior Science Exam
20
0 + T :
242
Junior Math Exam 21
E 23 #
199
TVE Natural Sciences
22
t A #
476
Junior Chemistry
23
237
24 TVE Mathematics
172
Table Overview of subjects in STEM
Subject Name
Chinese Name
Questions
Clinical Psychology
144
Taiwanese Trivia Question Answering v2 (ttqav2) 26
f +
131
Human Behavior 27
348
National Protection 28
#
240
Politic Science 29
1111
Educational Psychology 30
201
Education (Profession Level)
545
Economics 32
442
Psychology Disorders
33
608
Geography of Taiwan
34
858
Physical Education 35
#
204
Macroeconomics 36
#
462
Chinese Language and Literature 37
M x
226
Junior Chinese Exam 38
200
TVE Chinese Language 39
x
542
Education 40
#
143
Three Principles of People 41
-
160
Taiwanese Hokkien 42
148
Table 10: Overview of subjects in Social Science
15

---

Published as a conference paper at COLM 2024
Subject Name
Chinese Name
Questions
General Principles of Law 43
*
123
Anti Money Laundering
154
Jce Humanities 45
105
Introduction to Law 46
268
Taxation 47
BU
422
Trust Practice 48
f fTiEKT
451
Administrative Law
49
472
Table 11: Overview of subjects in Humanities
Subject Name
Chinese Name
Questions
Dentistry 50
T
448
Traditional Chinese Medicine Clinical Medicine 51
314
Technical Skills
52
*
452
Culinary Skills 53
J
330
Mechanical 54
136
Logic Reasoning 55
160
Real Estate 56
107
Music 57
314
Junior Social Studies 58
145
TVE Design 59
538
60
53
563
Auditing 61
#1
616
Veterinary Pharmacology
605
Nautical Science 63
617
Veterinary Pathology 64
320
Accounting 65
217
Fire Science 66
143
Optometry 67
E
1027
Insurance Studies
68
:
850
Pharmacology 69
646
Management Accounting 70
244
Agriculture
#
173
Official Document Management 72
KE x
252
Financial Analysis 73
429
Marketing Management 74
1H
108
Business Management 75
160
Finance Banking 76
155
Table 12: Overview of subjects in Other
16

---

Published as a conference paper at COLM 2024
Creation of year Due to concern of memorization, try to many questions from 2023 as shown in Figure 6 Our earliest questions can be found as early as 1998.
TMMLU+ Questions Distribution by Year
15.0
Percentage
12.5
10.0
0.0
1998
1999
2000
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
Year
Figure 6: Year of exam questions used in TMMLU+
Compared CMMLU and TMMLU, TMMLU+ has slightly longer tail of the distribution in question length than the prior.
Character Distribution in Datasets
TMMLU
Frequency
0.008
0.006
0.004
0.002
250
500
750
1000
1250
1500
1750
2000
Character Count
Figure 7: Character distribution comparison between TMMLU+ TMMLU and CMMLU. TMMLU+ distribution is slightly more long tail than TMMLU and CMMLU
17

---

Published as a conference paper at COLM 2024
C Percentage Breakdown of TMMLU+ Option Distribution
provide a detailed breakdown of how answer options are distributed across wide range of subjects in TMMLU+. Each value in the table represents percentage that corresponds to the portion of each option (A, B, C, D) within the specific subject.
Tables 13
Subject
A
B
C
D
engineering math
27.18
25.24
23.30
24.27
dentistry
25.56
25.56
24.06
24.81
traditional chinese medicine
25.90
24.10
24.82
25.18
clinical psychology
27.20
25.60
24.80
22.40
technical
25.12
24.63
25.62
24.63
culinary skills
22.60
26.03
25.34
26.03
mechanical
24.58
26.27
24.58
24.58
logic reasoning
23.02
25.90
23.74
27.34
real estate
23.91
28.26
22.83
25.00
general principles of law
27.36
21.70
25.47
25.47
finance banking
26.67
24.44
24.44
24.44
anti money laundering
20.90
26.12
25.37
27.61
ttqav2
23.89
27.43
24.78
23.89
marketing management
25.81
24.73
23.66
25,81
business management
23.02
24.46
26.62
25.90
organic chemistry
23.85
26.61
22.02
27.52
advance chemistry
23.58
24.39
26.02
26.02
physics
23.71
21.65
27.84
26.80
secondary physics
27.68
25.89
24.11
22.32
human behavior
25.57
25.89
24.27
24.27
national protection
24.64
24.17
25.12
26.07
jee humanities
26.67
24.44
24.44
24.44
politic science
24.12
25.03
25.63
25.23
agriculture
27.15
25.83
23.84
23.18
official document management
25.23
25.23
24.32
25.23
financial analysis
24.61
25.13
24.87
25.39
pharmacy
25.32
24.55
25.83
24.30
educational psychology
26.14
23.30
27.84
22.73
statistics and machine learning
24.55
25.00
25.00
25.45
management accounting
26.05
25.58
24.65
23.72
introduction to law
27.00
23.63
23.21
26.16
computer science
26.44
24.14
25.29
24.14
veterinary pathology
23.67
26.15
26.15
24.03
accounting
25.65
25.13
24.61
24.61
fire science
24.19
26.61
24.19
25.00
optometry
25.54
25.33
25.11
24.02
insurance studies
25.53
25.53
24.34
24.61
pharmacology
25.82
24.96
25.13
24.09
taxation
24.00
27.20
23.47
25.33
education (profession level)
24.90
25.51
24.28
25.31
economics
25.70
24.94
24.68
24.68
veterinary pharmacology
25.19
25.00
25.00
24.81
nautical science
24.86
24.68
24.86
25.59
psychology disorders
24.49
24.31
25.97
25.23
trust practice
24.44
24.69
24.69
26.18
geography of taiwan physical education
25.26
25.00
24.48
25.26
25.14
25.14
24.58
25.14
auditing administrative law basic medical science
25.09
24.36
26.73
23.82
25.48
25.24
24.52
24.76
24.95
23.90
25.37
25.79
18

---

Published as a conference paper at COLM 2024
Subject
A
B
C
D
macroeconomics
26.03
25.30
24.82
23.84
trade
25.30
25.10
25.70
23.90
chinese language and literature
25.13
25.63
24.62
24.62
tve design
23.96
26.67
23.96
25.42
junior science exam
26.29
23.47
25.35
24.88
junior math exam
27.43
23.43
25.14
24.00
junior chinese exam
25.71
24.57
24.00
25.71
junior social studies
24.60
26.19
23.02
26.19
tve mathematics
24.67
26.67
24.00
24.67
tve chinese language
26.09
23.81
25.26
24.84
tve natural sciences
24.53
25,47
24.53
25.47
junior chemistry
25.84
25.84
22.49
25.84
music
24.46
23.38
25.18
26.98
education
25.81
23.39
23.39
27.42
three principles of the people
25.90
23.74
25.90
24.46
taiwanese hokkien
25.58
23.26
26.36
24.81
Table 13: Breakdown of TMMLU+ option distribution
D Prompts used in evaluation process
Our prompts for zero-shot prompting can be found in Figure 8 The prompt for five shot and chain thought prompt can be found in Figure Φ and Figure 10 respectively. The part where LLMs response is marked in red. In CoT prompting, specifically ended the sentence in "Let's think step by step" such that the model would first reason about all four choices.
DA 8 E #E
Here are the multi-choice questions about the Taiwan regional linguistic exam, please choose the correct answer.
H EX T III an * E +
Question: Which of the following phrases came from Taiwan?
A. tu [ म
B.
C. T 2333333
D. 14 #
PU ?
# :
Answer: D
Figure 8: An example of zero-shot evaluation in answer-only scenarios. The English
translation is given beneath the corresponding Chinese text and is highlighted in blue.
19

---

Published as a conference paper at COLM 2024
D T 1
Here are the multi-choice questions about the Trust Practices Exam, please choose the correct answer.
- ?
Regarding the handling of corporate employee stock ownership trust business by the trust industry, which of the following statements is incorrect? Question:
A. X K W
A. The trustee's compensation can be paid from the trust property.
B. # f
B. They entitled to trust benefits.
C. i
C. They must not violate the terms of the trust agreement.
D.
D. should be conducted by personnel with specialized knowledge or experience.
A
Answer: C
...I Four more examples here J...
I 7 ?
Question: For members of an employee stock ownership plan (ESOP), if they wish to withdraw from the corporate employee stock trust early due to personal special reasons, which of the following must approve the withdrawal in principle?
A. Á
A. The person in charge of the company
B. A
B. The trustee
C. AIR
C. The representative of the ESOP
D. A
D. All members of the ESOP
Answer: C
Figure 9: An example of few-shot evaluation in answer-only scenarios. The English transla- tion is given beneath the corresponding Chinese text and is highlighted in blue.
20

---

Published as a conference paper at COLM 2024
D
Here are the multi-choice questions about the trade exam, please choose the correct answer.
...[ Five chain-of-thoughh examples here I... F iF
Question: Regarding the trade process, which of the following statements is correct?
A. 7
A. Both imported and exported goods must undergo inspection and customs declaration.
B. /
B. Credit investigation is generally conducted after quoting price.
C. +
C. The export-Impont Bank of China provides free foreign credit investigation services for manufacturers.
D.
D. is not always necessary to pay anti-dumping duties when declaring imports.
* 1
if AAA T IE Wt
I
A t0 30
M A
E NY
HX
C BBN
(1
IT
DI
C
66
2E
#
7
D
Answer: Let's think step by step. Firstly, Option A is not entirely correct, as not all import and export goods need to undergo inspection and customs declaration, which depends on the type of goods and the regulations of the destination country. Secondly, the statement in Option B is also incorrect; generally, credit investigations should be conducted before quoting to ensure that the buyer has sufficient credit to complete the transaction. Furthermore, the statement in Option C that the Export-Import Bank of China handles overseas credit information for manufacturers for free is not accurate, as credit investigation usually requires fee. Therefore, Option D is correct; paying anti-dumping duties is not always necessary when declaring imports, depending on the type of goods and the regulations of the destination country. Anti-dumping duties are meant to protect domestic industries from the impact of cheap foreign goods being dumped, but not all imported goods are required pay them. Therefore, the answer is D
An example of CoT in answer-only scenarios. The English translation is given beneath the corresponding Chinese text. The English translation appears below the corre- sponding Chinese text. The text highlighted in blue serves as our prompt, while the text in red was generated by GPT-4.
Figure 10:
E Detailed Description of Models
This section outlines the specifics of the models utilized our investigation.
GPT-4 and GPT-3.5-turbo represent OpenAl's advancements in conversational models. Initially, these models undergo supervised fine-tuning leveraging a dataset comprising user- assistant dialogues. Subsequently, reinforcement learning from human feedback (RLHF) further refines their capabilities. For this study, we employ versions of these models that were current of June 13th, marking the latest updates available to
21

---

Published as a conference paper at COLM 2024
LLaMA and LLaMA epitomize Meta's efforts in enhancing large, decoder-only language models. Both iterations employ Rotary Positional Embedding (RoPE) for positional encoding and Root Mean Square Layer Normalization (RMSNorm) for model normalization. LLaMA 2 distinguishes itself by undergoing an extended pretraining phase on diversified and more comprehensive dataset.
encodding
Qwen is pretrained on up to trillion tokens, including Chinese, English, multilingual texts, code, and mathematics, covering general and professional fields.
Qwen1.5-14B- --1-..hh44 incorporate SwiGLU activation, RoPE, and multi-head attention mecha- nisms. It offers enhanced ca capabilities for managing multiple languages and facilitating chat interactions, capable of supporting up to 32K tokens in context length.
Atom-7B is developed from the LLaMA 7B foundation, augmenting its vocabulary with an additional 32,000 Simplified Chinese tokens. This model benefits from further pretraining on diverse and unspecified corpus, with focus on sequences up 4,000 tokens in length.
DeepSeek-LLM-7BBBBl is a fresh instantiation of the LLaMA architecture, pretrained from the ground up on an extensive trillion token dataset, encompassing both English and Chinese languages.
DeepSek-LLM-77-CChll evolves from the Deepseek-llm-77-basee undergoing fine-tuning on proprietary instructional dataset, the specifics of which remain undisclosed.
Yayi-7b builds upon the BLOOMZ 7B model, engaging in continuous pretraining on selectively curated corpus of Simplified Chinese and English texts.
Yayi-7b-llama2 mirrors the approach of Yayi-7b, albeit starting from the LLaMA 7B model, to undertake continuous pretraining on similarly filtered corpus of Simplified Chinese and English texts.
miossom-v3.1-mistral- is developed from Mistral-7B model and only further fine-tuned on Chinese instruction dataset.
Breeze-7B-instruc-vv.1 enhances the original Mistral-7B model by incorporating an ad- ditional 30,000 Traditional Chinese tokens into its vocabulary. This expanded model then undergoes further pretraining on substantial, though unspecified, volume of Traditional Chinese data.
Mistral-7b-8Ok extends the Mistral 7B framework, focusing its additional pretraining efforts on Simplified Chinese datasets.
BlueLM-7B-Base is GPT-Neo architecture language model pre-trained high-quality data with 2.6 trillion tokens in English and Simplified Chinese.
Claude-3.0 Opus developed by Anthropic, is trained using combination of unsupervised learning and technique called Constitutional AL It involves pretraining the model on massive dataset of text and image data to acquire language and vision capabilities. Subse- quently, Constitutional Al guides the model's alignment with human values by explicitly specifying ethical and behavioral principles.
Gemini-pro is a multimodal model developed by Google AI, trained on a massive and versatile dataset seamlessly process a combination of text, audio, and visual inputs. It is capable of generating outputs that integrate both images and text.
Yi-6B-Chat and Yi-34B-Chat are pre-trained on trillion tokens, with the training data updated through June 2023. Both of them are trained with 4K sequence length and can be increased to 32K tokens during inference.
Yi-9B undergoes continuous training, building upon the Yi-6B model with 0.8T tokens.
ChatGLM3-6B retains the foundational structure of its predecessors, introducing an newly designed prompt format to support function calls, code interpretation, and the handling of intricate scenarios such as agent tasks.
22

---

Published as a conference paper at COLM 2024
Taiwar-LLM--7422.--hhh is designed specifically for Traditional Chinese, emphasizing the unique linguistic and cultural aspects of Taiwan. Originating from an extensive foundational model, it incorporates wide range of Taiwanese text sources and is enhanced through supervised fine-tuning.
Chinese-Llama-2-17 enhances the original Llama 2 by undergoing additional pretraining with Chinese text data around 120G and incorporating a more extensive vocabulary of size 55,296 for broadening coverage. It accommodates 4K context size, with the capability to extend beyond 18K through the application of the NTK method.
F Detailed Evaluation Results for the Top 4 Performing Models
Table 14 presents comprehensive accuracy breakdown across numerous subjects for Qwen-
72B, GPT-4, Qwen-14B, and Gemini-Pro.
Subject
GPT-4
Qwen-14B
Qwen-72B
Gemini-pro
engineering math
47.57
27.18
50.49
24.27
dentistry
66.67
50.13
59.65
52.88
traditional chinese medicine
43.53
51.44
68.71
43.53
clinical psychology
77.60
52.00
71.20
66.40
technical
66.92
59.70
66.42
61.97
culinary skills
60.62
56.51
66.10
71.43
mechanical skills
66.10
59.32
77.12
63.36
logic reasoning
34.53
32.37
40.29
35.25
real estate
43.48
45.65
48,91
40.18
general principles of law
61.32
51.89
67.92
50.94
Finance Banking
61.48
48.89
52.59
47.41
anti money laundering
70.90
73.88
79.85
61.94
ttqav2
78.76
72.57
86.73
71.68
marketing management
77.42
76.34
80.65
70.54
business management
70.50
59.71
69.78
60.43
organic chemistry
66.06
46.79
66.97
47.93
advance chemistry
53.66
47.15
52.85
50.41
physics
60.82
43.30
64.95
44.33
secondary physics
63,39
52.68
61.61
42.86
human behavior
73,46
66.02
78.64
63.11
national protection
63.51
62.56
74.88
64.93
jee humanities
63.33
58.89
76.67
60.00
politic science
75.18
50.85
73.67
59.30
agriculture
47.68
36.42
56.95
39.07
official document management
48.65
50.45
65.77
40.09
financial analysis
56.28
37.17
52.88
37.96
pharmacy
49.62
34.53
49.87
38.50
educational psychology
76.14
60.23
77.27
56.57
statistics and machine learning
71.88
56.70
65.18
54.27
management accounting
47.44
35.35
48.84
37.22
introduction to law computer science veterinary pathology
57.81
47.26
62.45
50.63
79.89
66.67
81.03
62.64
63.60
43.11
63.60
42.05
accounting
34.55
29.32
33.51
29.84
fire science
33.87
45.16
49.19
36.29
optometry
53.26
40.76
51.30
36.52
insurance studies
56.97
47.24
63.03
43,95
pharmacology
77.82
50.09
68.28
53.14
taxation
40.27
35.20
41.60
33.87
education (profession level)
56.38
39.51
57.41
43.00
economics
68.45
40.97
59.29
44.27
23

---

Published as a conference paper at COLM 2024
Subject
GPT-4
Qwen-14B
Qwen-72B
Gemini-pro
veterinary pharmacology
83.89
56.67
77.96
62.96
nautical science
52.45
38.66
51.18
41.85
psychology disorders trust practice
75.69
64.09
78.82
66.36
47.38
43.39
55.61
40.65
geography of taiwan
69.92
58.20
73,96
65.10
physical education auditing
61.45
41.90
68.72
45.81
56.55
42.18
61.45
42.55
administrative law basic medical science
51.19
35.48
56.90
43.57
83.54
47.69
77.04
56.81
macroeconomics
63.99
45.50
62.53
42.86
trade
47.01
31.08
54.38
35.79
chinese language and literature
46.73
51.76
67.34
45.96
tve design
80.42
67.08
79.58
65.21
junior science exam
71.36
59.15
76.06
57.01
junior math exam
40.00
40.00
49.14
29.71
junior chinese exam
80.57
82.29
85.71
72.57
junior social studies
65.87
74.60
79.37
58.73
tve mathematics
33.33
33.33
36.67
30.00
tve chinese language
73.71
73.91
85.09
66.05
tve natural sciences
69.81
58.49
74.53
59.29
junior chemistry
54.07
43.54
49.28
37.32
music
58.27
52.52
67.99
51.58
education
66.13
55.65
65.32
54.03
three principles of the people
77.70
69.06
80.58
68.35
taiwanese hokkien
27.13
33.33
42.64
34.88
Table 14: Breakdown of model performance in the top 4 models in zero-shot learning
G Detailed results of various models on different benchmarks
Table 15
presents evaluation results Simplified Chinese benchmarks and TMMLU+ to assess whether models exhibit superior performance simplified benchmarks. "It should be noted that the performance results of GPT-4 on C-Eval and CMMLU were individually reported in Huang et al. (202)) and Li et al. (2023), respectively.
Model
C-Eval
CMMLU
TMMLU+
Breeze-7B-Instructvv1..
0.468
0.460
0.286
Qwen-1.8B
0.548
0.516
0.338
Q4B
0.708
0.713
0.514
Qwen-72B
0.825
0.844
0.647
Qwen-7B
0.626
0.625
0.520
Taiwan--LMM-13B-2220cha
0.386
0.401
0.223
Taivan--LLM-7Bxv211chhll
0.368
0.363
0.051
GPT-4*
0.687
0.710
0.675
Table 15: Five-shot results of various models on different benchmarks
24

---

Published as a conference paper at COLM 2024
H
Comparison between rule-based parser and prompt-based parser
Previous work (Kojima et al. 2022 Wei et al. 2022) that prompt-based parsers often achieve higher scores compared to rule-based parsers. While prompt-based parser is much more versatile than rule-based parser as prompt response format may differ significantly between LLMs, due to cost consideration, this former method is not widely adopted as the default answer extraction method. In this section, we compare the difference between these two as shown in Table 16 For the prompt-based parser, we utilize gemini-1.5-flash to extract the final answer.
Model
STEM
Sci. String
Humanities
Other
Average
LLM
String
LLM
LLM
String
LLM
String
LLM
String
GPT-4
62.59
60.4
70.29
67.4 4
59.35
56.0
61.11
57.6
63.33
60.3
Gemini-pro
44.29
45.4
57.92
57.3
49.85
48.8
48.50
48.2
50.14
49.9
Yi-34B-Chat
50.16
40.2
64.83
57.05
54.0
47.6
57.19
49.6
GPT-3.5-turbo
43.68
41.6
49.00
39.82
36.7
42.0
44.43
Qwen-7B-Chat
35.36
37.5
44.96
45.5
40.64
38.1
40.25
39.0
40.30
40.0
Yi-9B
28.69
28.7
44.41
44.6
48.53
48.5
36.70
36.8
39.58
39.6
Breeze--B-Instruceeee
33.92
21.7
45.11
25.2
41.18
24.5
38.04
23.0
39.56
23.6
Taiwan-LLM-138
27.01
18.5
32.45
27.6
27.78
17.8
28.53
21.5
28.94
21.3
Taiwan-LLLM-BB
27.42
15.0
26.01
16.2
25.95
15.0
26.34
16.2
26.43
15.6
Table 16: Comparison of Models Across Different Fields with prompt-based parser (LLM) and rule-based parser (String) answer extraction method
I Data Examples for Each Subject
: -= (112 12 - -11 13 ?
Question: Given two vectors options incorrect?
2 and v [2 - 1 which of the following
A.
. (cross product)
(-3 - 3
The cross product of these two vectors [-3 3 3]
B. te 2 /3
B. The angle between these two vectors æ /3
C. It | म (inner product) //3
C. The inner product these two vectors is 3
D. #
(norm)
#
D. The product of the norms of these two vectors is 6
* :
Answer: A
Figure 11: An "engineering math" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
25

---

Published as a conference paper at COLM 2024
(propagation event)
Question: In the UV-irradiated reaction between chlorine gas (C12) and ethane, which of the following steps is a propagation event?
I] CHH
II + - CH
+
M) CI CCH--CH
CH3-CH2-CI
IV) Cl2 + CH3-H2C + CHI-CH,-CI++I
Cl2 light - CC++CI-
A. IV
A. and IV
B. I BEV
B. Land V
C. II, III IVV
C. I, III and IV
IVV
D. Tand
3 A
Answer: A
Figure 12: An "organic chemistry" example. The English translation given beneath the corresponding Chinese text and is highlighted in blue.
: A
Question: Which of the following substances in the work environment are more likely to cause cancer?
A. E
Pentachlorophenol
B. #
B. Toluene
C. -
C. Xylene
D. 1
D. Carbon Monoxide
* A
Answer: A
Figure 13: An "advance chemistry" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
26

---

Published as a conference paper at COLM 2024
1 : #
Question: There is AC generator with a coil area of 0.03 square meters, a total of 20 turns of the coil, and rotates at fixed speed of 600 revolutions per minute in a uniform magnetic field of 0.2 Tesla. What is the approximate maximum induced electromotive force of this generator in volts?
A. 15
A. 15
B. 7.5
B. 7.5
C. 30
C. 30
D. 2.5
2.5
Answer: B
Figure 14: A "physics" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
1 : 110 2 T
Question: In nuclear reaction, how much heat can be transformed from a mass loss of 10 grams?
10 #
A. x 109 cal
B. 9 1010 +
B. x 10 cal
C. 2.14 x 1010 #
C. 2.14 x 10 cal
D. 2.14 10 +
D. 2.14 x 10 cal
:C
Answer: C
Figure 15: A "secondary physics" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
27

---

Published as a conference paper at COLM 2024
Question: To measure the viscosity of fluid using the same capillary viscometer, addition to monitoring the flow time of the fluid, which of the following data is also required?
A. * 1
Osmotic pressure
B. К
B. Differential liquid level
C. :
C. Liquid density
D. #
D. Capillary radius
:C
Answer: C
Figure 16: A "pharmacy" example. The English translation given beneath the correspond- ing Chinese text and is highlighted in blue.
D CIDR(Classless Inter-Domain Routing) E u 1E TH Question: Which the following statements about CIDR(Classless Inter-
Domain Routing), a method of IP address allocation, is incorrect?
A. CIDR 111.19.166..11 / 25 #D 255 25..255...559999
A. The subnet mask of CIDR notation 192.168. / 25 is 255.255.255.192.
B. CIDR 1#1...2.16.000 / 12 IPvv Class B #1 IP #
B. The CIDR notation 172.16.0.0 12 is a private IP range of IPv4 Class B.
C. CIDR ي BR
C. CIDR can improve the routing efficiency on the Internet.
D. CIDR aj IP f
D. CIDR can reduce the waste of IP addresses.
; A
Answer: A
Figure 17: A "computer science" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
28

---

Published as a conference paper at COLM 2024
Question: Which of the following factors stimulates the secretion of aldosterone?
A. A $1
A. High sodium in the blood
B. F
B. Blood volume decrease
C. 1
C. High osmolarity
D. 1
D. Low potassium in blood
* : B
Answer:
Figure 18: A "basic medical science" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
64 -1 N 995% f4 E ?
Question: If random sample of 300 residents is taken from community, and the average age of this sample is 55 years old, and it is known that its average age is significantly different from 64 years old, and assuming that the age of the community residents is normally distributed, which of the following is more likely to be the 95% confidence interval for the average age?
(54, 74)
B.
(45, 65)
C.
(57, 71)
D.
(48, 62)
* D
Answer: D
Figure 19; A "statistics and machine learning" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
29

---

Published as a conference paper at COLM 2024
- 3 cm 3 - kg 1O , CAA
d> ?
Question: Take a 10 kg uniform alloy, divide into two pieces, one is made into solid cube with side length of 10 cm, and the other is made into solid sphere with mass of 2 kg. What should be the volume of this solid sphere?
A. 4000 cm3
A. 4000 cm3
B. 5000 cm3
B. 5000 cm3 3
C. 250 cm3
C. 250 cm3
D. 200 cm3
D. 200 cm3
; Answer: C
Figure 20: A "junior science exam" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
E R T 2220 Question: There are nine people in Zhuzhu's family. is known that the mode, average, median, and quartile distance of their age this year are all 20. Which of the following statements about the statistical age of these nine people three years later is incorrect?
If fi LU
A. 23
A. The average is
B. 22
B. The quartile distance is
C. #
C. The median 23
D. 22
D. The mode is 23
: Answer:
Figure 21: "junior math exam" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
30

---

Published as a conference paper at COLM 2024
T : A PE
The human body has a variety of food sources. The nutrients in food include organic and inorganic substances. Sometimes specific food additives are added during food processing. Our government has established Food Safety and Health Management Laws and implementing rules for the safety of our public's food. Which the following statements is most correct?
Question:
A. #K ti & AR ILA T
Ketogenic diet refers to consuming lot of carbohydrates, moderate protein, and a small amount of fat.
B.
B. Only carbohydrates in nutrients can provide the energy required for human survival. П
C. b
| 
C. Food additives such sweeteners, preservatives, and antioxidants should be labeled with their functional names.
D. A :
+ - -
D. Lipids organic substances and be important components such as biological membranes, enzymes, hormones, antibodies, and hemoglobin.
:
Answer: C
Figure 22: A "tve natural sciences" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
: : tCCs 1 i
4F
11120 4
क
133 म.
13 AUi
Question: The half-life of radioactive 131 atoms is 30 years. If about 3 grams remain
after 120 years, what is the closest initial mass of 131
A. 50 お
A. 50 grams
B. 60 ø
B. 60 grams
C. 30 म.
C. 30 grams
40
D. 40 grams
* : A
Answer: A
Figure 23: A "junior chemistry" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
31

---

Published as a conference paper at COLM 2024
5x + # H (x $²) T ?
Question: a, B are the roots of the equation x² 5x 6 0, what the polynomial
(x (x-4 (x-
A. x² - 11x +
A. x² -1x 30
B. x² -3x +6
B. xx2--10 -3x--33- 36
C. x² +3x + 36
C. x² 13x + 36
D. x2 + 11x 30
D. x² 1 30
* B
Answer: B
Figure 24: A "tve mathematics" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
1 ET (PET) : (fMRI) -
Question: What the difference between Positron Emission Tomography (PET) and Functional Magnetic Resonance Imaging (fMRI)?
A. #
A. The former more clearly identify the activation status of the brain when individual is listening to music or performing mathematical operations.
B. # NE
B. The former requires the injection of specific chemicals into the subject, while the latter does not.
C. T N
C. The former has three-dimensional anatomical scan images of the horizontal, sagittal, and transverse planes, while the latter only has horizontal plane scan images.
D. 1
Ef (
The former has clearer images that can evaluate the location of cerebral blood clots.
+ B
Answer:
Figure 25: A "clinical psychology" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
32

---

Published as a conference paper at COLM 2024
3 DAR
3 -
Question: kind of pancake made from spring onions is common on the streets of
Chinese societies such as Taiwan, Hong Kong, mainland China, as well as Malaysia
and Singapore. It is often eaten as breakfast or snack. The name this food is:
A.
Scallion pancake
B. ff
B. Egg pancake
C. X
C. Shui Jian Bao
D. k
D. Stinky tofu
: Answer: A
Figure 26: A "TTQAv2" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue
Question: What is the general sequence of the process known "coming out" for homosexuals, from recognizing their sexual orientation to publicly acknowledging their homosexuality?
-
Self-acknowledgment Confessing friends and family Knowing people inside the circle Coming out
B. -
B. Selfaccknowledgmeet Knowing people inside the circle - Confessing to friends and family Coming out
C.
-
C. Knowing people inside the circle Confessing to friends and family Self- acknowledgment- Coming out
D.
-
D. Knowing people inside the circle Selfaacknowledgmet Confessing to friends and family Coming out
* B
Answer:
Figure 27: A "human behavior" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
33

---

Published as a conference paper at COLM 2024
Question: What are the responsibilities of the command center?
A. A
A. Personnel appeasement and psychological counseling
B. : A 3
B. Report to the competent authority the number of people evacuated, the place of refuge, and the disaster
C.
C. Assist the people who take refuge in schools to meet emergency needs
D. f 21
D. Adjust mutual support between organizations according to the situation
x :
D
Answer: D
Figure 28: A "national protection" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
Question: Which of the following is not an important claim of the research approach to political behavior?
A. A
A. Non-interdisciillnaryy integrated research
B. A @ #
B. Using political role behavior as the basic data for research
C. 1
C. Academic research must be value-neutral
D. Using the methods of natural science
Answer: A
Figure 29: A "politic science" example. The English translation is given beneath the corre- sponding Chinese text and highlighted in blue.
34

---

Published as a conference paper at COLM 2024
: T JJN
H Kohlberg)ffj
H PA IINB [K] ?
Question: Xiao Fang bought a train ticket and sat in the designated position, but encountered an elder asking for seat. Xiao Ming said: "If you bought a ticket, you have the right use it. No one has any reason to ask you give up your seat." Explained by Kohlberg's ' stage theory of moral reasoning, which orientation does Xiao Ming belong to?
A.
Social contract orientation
B. I
B. Legal order orientation
C. t [
C. Universal ethics orientation
D. 7
D. Relative utilitarian orientation
:
:
Answer: B
Figure 30: An "educational psychology" example. The English translation is given beneath the corresponding Chinese text and is highlighted blue.
: SH
Question: In a certain textbook, before the end of the unit, few exploration topics related to the unit concept are provided. Students can explore according to their own interests and abilities. What is the form of exploration that provides exploration topics and allows students to determine the methods and results of exploration?
A. "i
A. Structured exploration
B. НК
B. Guided exploration
C. |
C. Open-ended exploration
D. t
D. Recipe-based exploration
$ :
Answer: B
Figure 31: An "education (profession level)" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
35

---

Published as a conference paper at COLM 2024
Question: What is the biggest difference between oligopolistic market structure and the other three types of market structure from the following?
When making decisions, firms will consider how competitors make decisions.
B. [ R th
B. Each firm is price determinant.
C. Ti PT
C. Different firms produce heterogeneous products.
D. NE 2
D. Different firms produce homogeneous products.
; :
Answer: A
Figure 32: An "economics" example. The English translation is given beneath the corre- sponding Chinese text and is highlighted in blue.
H Tenhula Bellack (2008) Bo (Social Skills Model)
F
(social cognition) "
Question: According to the Social Skills Model by Tenhula & Bellack (2008), which of the following descriptions falls under the category of social cognition?
A. ( (ill T (
A. Correctly detecting emotional clues (e.g., facial and voice expressions)
B. # 3 T m
B. The ability listen and understand language messages
C. T
C. Understand and effectively analyse social stimuli from interactive situations
D. tE (1) A fo #7
D. The ability to use appropriate non-verbal behaviour when speaking
T C
Answer: C
Figure 33: An "occupational therapy for psychological disorders" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
36

---

Published as a conference paper at COLM 2024
1
2 _ Question: What's the most likely reason for the "Black Ditch" name of the Penghu
Strait the Taiwan Strait?
A. 1 10
A. The variable water flow direction makes the silt muddy
B. / [ A 1
B. Few sediments allow light to reach deep
C. M m 1
C. Both sides of the strait provide abundant suspended particles
D. * th TW
D. Most of the seawater contains deep-colored sediments
Answer: B
Figure 34: A "geography of taiwan" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
1 + 100 AR 30: A
: ?
Question: In a 100-meter race, what should one do to increase speed during the 30 to 70-meter stage after start?
A. + #
A. Increase both stride length and frequency
B. (
10l
B. Keep the frequency constant, increase stride length
C. Mi H
C. Reduce stride length, increase frequency
#R #
D. Keep the stride length constant, increase frequency
* :
Answer: D
Figure 35: A "physical education" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
37

---

Published as a conference paper at COLM 2024
1 a ~
Question: Among the reasons why the aggregate demand curve is negative sloping, which one is not?
A. #
A. Substitution Effect
B. *
B. Interest Rate Effect
C. :
C. Real Balance Effect
D. 1
D. Rational Expectations Effect
; :
Answer: D
Figure 36: A "macroeconomics" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
: oC ( A -
FHE
Question: Which of the following chapter and location combinations from Lan Dingyuan's "Eastern Expedition Collection", which recorded many ancient Taiwanese place names or aliases, is incorrect?
A. ( #
A. (Record of Hebaoyu): Jiayi Puzi
B. ($
B. (Record of Shuishalian): Sun Moon Lake
C. (# (
C. (Record of Volcano): Datun Volcano Group
D. #)) #if
D. (Record of Zhuxianpu); Hsinchu
:C
Answer: C
Figure 37: A "chinese language and literature" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
38

---

Published as a conference paper at COLM 2024
- #
T : F
X C'
X
3 YI M MO -E
1
- T
Question: Lu Yu states in "The Tea Classic" that "mountain water on top, followed by stone spring, with river water next and well water at the bottom." Also "river water is drawn from afar, and well water is drawn more. Hence, it ends here, he never tasted all the waters in the world. In Zhang Youxin's "Record of Boiling Tea Water", it says Liu Bochao classified water into seven categories, ranking the Yangtze River as the first, Huishan Stone Spring as the second, Tiger Hill Well as the third, and Huai River the last. Also, it says, Lu Yu discussed with Li Jiqing that there were twenty types of water, with river water towering over mountain water, and well water standing above river water. These two theories contradict The Tea Classic. The taste of water is just good or bad, and absurd to rank all the waters in the world one by one. Lu Yu's discussion of water dislikes cloudy and stagnant water and prefers spring sources, wells are drawn more. Though the river is long, is a gathering of multiple waters, hence it is next to mountain water. This view is closer to physical reason. -Rewritten from Ouyang Xiu's "Record of Daming Water" According to the text, which of the following statements is most suitable?
X -
A. Zhang Youxin categorizes all the waters in the world into twenty kinds according
to water quality.
B.
B. Ouyang Xiu evaluates the water quality based on the distance and height of the water source.
C. fK (TEA)) *
#7
C. According the standards of "The Tea Classic", Yangtze River is better than Tiger Hill Well.
D. E < +
D. In "Record of Boiling Tea Water", Zhang Youxin refutes Liu Bochao's view.
:C
Answer: C
Figure 38: A "junior chinese exam" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
39

---

Published as a conference paper at COLM 2024
:
BAT AL BA
# 8 R - 30 . 1
-
2
BETT (E. XII
ATER
HE BUR
HO
17 10
1
1
- ) FREX A
14 ?
Question: The main function of reading is to help readers understand the origin and meaning of a fact, not just make readers know a fact. Reading is learning because there is no teacher around you to point you out, so the learning benefit depends entirely on the reader's analysis and interpretation. In other words, the subject of reading is you, and everything is up to you, so its connotation is proactive. Moreover, because it is for oneself, it also needs "How to Read Book" liken reading to baseball, readers should read book with the attitude of the catcher. To win game, the tacit understanding between the pitcher and the catcher is indispensable basis. The author of the book is like pitcher, the most important thing to catch the ball thrown by the pitcher is that the catcher should actively understand the pitcher's 's skills and style. Only in this way, we catcher will not miss the flying ball and can achieve the purpose of reading. similar to this pitching relationship. Reading can enhance observation, and observation will make life more meaningful. Reading will also increase memory because memory is the result of multiple connections of brain cells. Reading will also increase your imagination through other people's experiences and imagination, and imagination is the basis of many knowledge discoveries. Even, reading will enhance your safety because knowing other people's 's experiences will also improve your alertness. Reading will also increase your analytical power because analysis is an indispensable part of reading. These abilities will mature your judgment, then sharpen, this will benefit you all your life, can also benefit others, and contribute to society. (Rewritten from Huang Kunkyin's 's "Reading is the Only Way to Lifelong Learning") According to the above, which of the following options is the most suitable for underlined?
have self-discipline. Adler and Van Doren in their
is very
A. E
A. Insights and abilities to create
B. {
B. Interactions between teachers and students
C. A
C. Opportunity and skill to attack
D. 7
D. Relationship between the author and reader.
:D
Answer: D
Figure 39: A "tve chinese language" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
40

---

Published as a conference paper at COLM 2024
FETMEER RI WfffiWAttt. A1 / Á
:
Question: Which of the following theories of intelligence is most consistent with the saying "Everyone is born with talent that must be useful" and "Everyone can be the top scholar"?
A.
-
Two-Factor Theory of Intelligence
B. =
B. Triarchic Theory of Intelligence
C. Tτ
C. Multiple Intelligences Theory
D. #
D. Structural Theory of Intelligence
:
Answer: C
Figure 40: An "education" example. The English translation is given beneath the corre- sponding Chinese text and is highlighted in blue.
D A ?
Question: Due to mankind's ' pursuit of economic development, the incessant exploita- tion has led to nature's backlash, causing numerous disasters, and seriously endan- gering people's lives and property. What concept is this due to gradually straying from?
A. #
A. Industrial upgrading
B. [
B. Social welfare
C. *
C. Privatization of state-owned enterprises
D. 14
D. Sustainable development
Answer: D
Figure 41: "three principles of the people" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
41

---

Published as a conference paper at COLM 2024
1
+ - 3 ?
Question: Fill the blanks in the proverb "Never before learning to fly, never want to pull the melon." What should be used in the blanks?
A. +
A. learning walk, amm planting
B. fT ARE
B. walking, watering the fertilizer
C.
-
C. learning crawl, plowing the field
D. f Н
D. walking, planting melons
* A
Answer: A
Figure 42: A "taiwanese hokkien" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
A + Z.Z P & # ?
Question: How much money can ask to return according the Civil Law when the contract of sale cannot be performed due to B's fault after A promised to buy motorcycle from B and paid a deposit of NTD 10,000?
A. Æ
A. 10,000 NTD
B. 3 MTC
B. 30,000 NTD
C. 2 7
C. 20,000 NTD
D. FT
D. 5,000 NTD
* C
Answer: C
Figure 43: A "general principles of law" example. The English translation given beneath the corresponding Chinese text and is highlighted in blue.
42

---

Published as a conference paper at COLM 2024
Question: Which of the following statements about counter-terrorism financing is incorrect?
A. #
A. The sanction list designated by our country's counter-terrorism financing law is limited to individuals, corporations, or groups within the territory of the Republic of China.
B.
B. The supervisory authority of our country's counter-terrorism financing law is the Ministry of Justice.
C. A E T 111 1 xi -
C. The purpose of counter-terrorism financing is prevent and curb the financing of terrorist activities, acciooissaatvvvtttes,,,o organizations, and individuals.
D. D *
2001 911 #
D. The Financial Action Task Force (FATF) has made combating financing of terrorism priority since the US 911 incident in 2001.
+ A
Answer: A
Figure 44: An "anti money laundering" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
: T ii 1
Question: According to the letter Kang Pei wrote to Wang Jinhui, which of the following descriptions is most appropriate?
A. + A4
A. The purpose of Kang Pei's letter is to ask Wang Jinhui to take leave on his behalf.
B. 1 $
B. Kang Pei indicated that the delay in seeking medical advice led to the worsening of the condition and increase in caruncle.
C. #
- A
C. can be inferred from the whole text that the two are bosses and subordinates of similar ages.
D. B
T
D. The ending greeting to Wang Jinhui can also be "fu an".
: Answer: C
Figure 45: "jce humanities" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
43

---

Published as a conference paper at COLM 2024
1 ( 1 " ?
Question: According to the interpretation of the Grand Justices of the Judicial Yuan, which of the following statements is incorrect regarding the pre-review of cosmetics advertisements?
A.
A. It must be for the especially important public interest purpose of preventing peo- ple's lives, bodies, and health from direct, immediate, and irreversible harm.
B. E E 2 | -
B. It is a significant intervention in freedom of speech, and should in principle be unconstitutional.
C. :
C. The achievement of its goal should have substantial connection.
D. A
D. It should provide people with the opportunity for immediate judicial relief.
:C
Answer: C
Figure 46: An "introduction to law" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
Question: Which the following options is the correct order for the various deductions that taxpayers can enjoy in the consolidated income tax, arranged from high to low by deductible amount?
Standard deduction for those with spouses 2. Special deduction for children's 's education expenses 3. Special deduction for physical and mental disability 4. Special deduction for long-term care
A. 1342
A. 1342
B. 3142
B. 3142
C. 3412
C. 3412
D. 1432
D. 1432
* : A
Answer: A
Figure 47: A "taxation" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
44

---

Published as a conference paper at COLM 2024
Question: Which of the following statements about the valuation of property gift tax is incorrect?
A. 51 1
A. Foreign currency deposits are converted into New Taiwan dollars using the buying exchange rate of the Taiwanese Bank on the day of the gift.
B. 2 lh LI i2 #
B. Funds are calculated based on the net value the fund on the day of the gift.
C.
C. Listed and OTC stocks are calculated based on the closing price on the day of the
gift.
D. H.
D. Unlisted stocks are calculated based on the net value of the stock.
# :
Answer: C
Figure 48: A "trust practice" example. The English translation is given beneath the corre- sponding Chinese text and is highlighted in blue.
Question: Which of the following statements about the principle of legal application is incorrect?
A. -
A. The law supersedes regulatory orders
B. A
B. The law superior to emergency orders
C. t i
C.
The later law is superior to the former law
D. A
D. Special law is superior to ordinary law
* :
Answer:
Figure 49: An "administrative law" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
45

---

Published as a conference paper at COLM 2024
Question: Which of the following teeth cannot be easily distinguished from the left or right by the crown?
A. ,
A. Maxillary deciduous canine
B. $L
B. Maxillary central deciduous incisors
C. F
C. Mandibular deciduous canine
D. +
D. Mandibular central deciduous incisors
:D Answer: D
Figure 50: A "dentistry" example. The English translation is given beneath the correspond- ing Chinese text and is highlighted in blue.
1 iK <
F 2
?
Question: According to "Acuuuncture Science", which of the following statements
about seminal emission is wrong?
A. - ボズ
First, needle the Sanyinjiao and Yinlingquan
B. # 4 #
& /
B. Ejaculation in sleep is named as spermatorrhea
C. #
के
-
C. Continue to needle Qihai, Guanyuan, Zhongji, Shenshu, Zhishi, and moxibustion at Shenshu, Guanyuan and Zhishi
D. E HA 2
*
$
X
D. The cause of seminal emission can be nervous weakness, mental fatigue, excessive drinking, inflammation of nearby organs, and it can also be caused by excessive masturbation, lustful thoughts and indulgence
Answer: B
Figure 51: "traditional chinese medicine clinical medicine" example. The English transla- tion is given beneath the corresponding Chinese text and is highlighted in blue.
46

---

Published as a conference paper at COLM 2024
Question: What is the treatment method for the pre-embedded pipe of the power distribution place user?
-
A. Do not dealing with it
B. [o
B. Implement unilateral waterproofing
C. 6
C. Just do the embedding pipe is enough
D. B
D.
The ends of the embedding pipe should be sealed for waterproofing
Answer: D
Figure 52: A "technical" example. The English translation is given beneath the correspond- ing Chinese text and is highlighted in blue.
Question: Which of the following is the correct statement regarding things that should paid attention to when receiving ingredients?
A. 1 Ti
A. There is no need to check the transport conditions.
B.
B. Frozen food packages should not be accepted there are water stains or ice crystals them.
C. K
#
C. Those who pass on-site should be received, no record is necessary.
D. H -
D. The same person should handle both purchasing and receiving.
: : B
Answer: B
Figure 53: A "culinary skills" example. The English translation is given beneath the corre- sponding Chinese text and is highlighted in blue.
47

---

Published as a conference paper at COLM 2024
Question: What is the best method to control insect pests?
A. $HHT
A. Patting
B. (КЛ
B. Using insecticides
C. A T
C. Using net to catch
D. #1
D. Clearing the source of infestation
X :
Answer: D
Figure 54: "mechanical" example. The English translation is given beneath the corre- sponding Chinese text and is highlighted in blue.
7 # & ;
Question: The words in the sentence are not arranged order, and there are extra
words, please find out the extra words: T Y E
A.
B.
C.
Øaj
D.
t
* :
Answer: C
Figure 55: A "logic reasoning" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
48

---

Published as a conference paper at COLM 2024
28 1200 7C 1 WF # SL
Question: For three-story detached new building, the registered area of the building is pings, the registered area of the base is pings, the normal price of the house and land is 12 million yuan. Considering the local market survey data, the building value ratio calculated using the valuation method accounts for 30% of the house price. Please, find the unit price of this base.
A. 42 /
A. 420,000 NTD/ sqftt
B. 60 /
B. 600,000 NTD/sqft
C. 18 TC
C. 180,000 NTD/sqft
30 /
D. 300,000 NTD/sqft
A Answer: A
Figure 56: A "real estate" example. The English translation is given beneath the correspond- ing Chinese text and highlighted in blue.
#i TH _
Question: Which of the following statements about "Chegu Diao" false?
A. T #
A. "Drum" refers to the bell drum.
B. A
B. It's a type of tune in Peking Opera.
C. dil <
C. The song "Peach Blossoms Across the River" is one of the representatives of this musical form.
D. [ 1
D. "Che" refers to pair of long bamboo pieces.
* :
Answer: B
Figure 57: A "music" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
49

---

Published as a conference paper at COLM 2024
:
# N4
J XI
17
# H8X HI aM TIE
# B4
1.
62a69d5962675.jpp X
Question: Rum one of Cuba's famous products and is popular because of its strong flavor. Rum was made from a by-product of a certain cash crop when the local area developed tropical cultivation. This cash crop was also widely planted in the southern plains of Taiwan. Tracing back the development of this economic crop in Cuba, it is related to Spanish colonization. At that time, Spain needed lot of labor cultivate economic crops and mine minerals, but the aboriginals died in large numbers due infectious diseases from abroad, so Africans were introduced
fill the labor gap. Today, the population of Cuba is predominantly white, with those of black ancestry accounting for about third of the national population, while aborigines are rare. In the twentieth century, someone mixed rum with cola originated from the United States to make "Free Cuba", as shown in Figure 13. The name of this cocktail has a historical legend, symbolizing the United States' support for Cuba's 's independence movement in the 19th century. At the time, the US political arena made a comment on Cuba's independence movement: "Regarding any European country's 's invasion of the Americas, the United States has the right to intervene and withdraw European forces. The Spanish government losing control of Cuba and is unable to protect the lives and property of American citizens there, so the US government should intervene to restore peace and ensure Cuba's independence. " 62a69d5962675.jp Regarding the example mentioned in Tainan City, which of the following statements is most appropriate?
A. | B Asd d ITT
A. The legislature enacts laws to administer administrative penalties on violators.
B. # s
B. Local legislative bodies establish regulations to impose criminal penalties on viola- tors.
C. +
C. Local legislative bodies establish regulations to administer administrative penalties on violators.
D. A #
D. The legislature enacts laws to impose criminal penalties on violators.
:
:
Answer: C
Figure 58: A "junior social studies" example. The English translation is given beneath the corresponding Chinese text and is highlighted blue.
50

---

Published as a conference paper at COLM 2024
aesthetics ) 1 1E ?
Question: Which of the following statements about aesthetics is correct?
T A. 1 Immanuel Kant J R
A A
Immanuel Kant believed that only God, not humans, can play role explaining art theory and aesthetics.
B. ( David Hume )
H taste JI
B. David Hume used "taste" to refer to the human ability to feel and judge the aesthet-
ics of an artwork.
C. - # Tassthesis
A
C. The term "aasthetics" comes from the Greek word "aesthesis", originally referring
to human's ability of logical reasoning.
D. - ( Aristotle - -U
D. Aristotle was the first to use the term "aesthetics" to refer to the study of aesthetic experience.
x : B
Answer: B
Figure 59: A "tve design" example. The English translation is given beneath the correspond- ing Chinese text and highlighted in blue.
1 M. Mfrfaiting
Question: Which of the following statements about forfaiting is correct?
A. E
A. The related letter of credit must be confirmed.
B. # R Ai <
B. The related bill must be accepted by the issuing bank.
C. # AI J / H *
C. the relevant note is not honored at maturity, the exporter should unconditionally return the ticket money.
D. A
D. The bill under the buyer's term letter of credit is the most suitable for handling this.
* B.
Answer: B.
Figure 60: A "trade" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
51

---

Published as a conference paper at COLM 2024
[ ?
Question: Which of the following statements about analytic procedures is incorrect?
A. /
FJ 2
A. More suitable for large and predictable transactions
B. A 1
B. Can assist auditors in making overall conclusions
C. K A [4
C. More suitable for cases where the relationship between data will continue to exist
D. К
D. Must be combined with detailed tests to verify individual statements
X D
Answer: D
Figure 61: An "auditing" example. The English translation is given beneath the correspond- ing Chinese text and is highlighted in blue.
T : du
Question: Which of the following statements about distribution volume wrong?
A. T # (E
The redistribution phenomenon injectable anesthetics will prolong the anesthetic effect
B. 1
B. The degree of plasma protein binding with drugs is one of the factors affecting the distribution volume
C. t (loading dose)
C. The distribution volume determines the initial dose (loading dose)
D. P 12 A Eht ii # HA
D. is the ratio of the total amount of drugs in the body to the concentration of drugs in plasma.
# :
Answer: A
Figure 62: A "veterinary pharmacology" example. The English translation is given beneath the corresponding Chinese text and is highlighted blue.
52

---

Published as a conference paper at COLM 2024
Question: To check a quadrant soft iron ball has semi-permanent magnetism, this soft iron ball should be moved to the vicinity of magnetic compass. Which of the following inspection results can indicate that this soft iron ball has retained magnetism?
A.
()45
Rotate 45", the compass needle deviates more than 2
B. (E90 Mi 1
B. Rotate 90", the compass needle deviates more than 1
C. #145 1 F.
C. Rotate 45", the com pass needle deviates more than 1
D. (E90 1
D. Rotate 90", the compass needle deviates more than 2
:D
Answer: D
Figure 63: A "nautical science" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
?
Question: Which of the following is irrelevant to the mineralization of the endocardium in bovine hearts?
#
A. Overeating of vitamin D
B. П
B. Ketosis
C. A
C. Poisoning caused by overeating calcium-rich plants
D. :
D. Uremia
* : B
Answer:
Figure 64: A "veterinary pathology" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
53

---

Published as a conference paper at COLM 2024
X1 // 2 000 01 Xx के ö
iu f 1it 3
2 $000000000 HE
-
2 10%
Question: At the beginning of year X1, company A bought polluting device for $3,000,000, and paid another $200,000 000 for installation. The device expected to last for years. Company A estimates that it will cost $500,000 to dismantle and dispose of the device after 3 years. The company's ' capital rate 10%, and there are no regulatory requirements for the disposal of the device. What should be the cost of the device?
A. $3,000,000
B. $3,200,000
C. $3,700,000
D. $3,575,657
Answer: B
Figure 65: An "accounting" example. The English translation given beneath the corre- sponding Chinese text and is highlighted in blue.
1
+
(Flashover) म
a
Question: Which of the following statements is incorrect about the flashover phe- nomenon in buildings?
A. X A
A. Delaying the arrival of building fire to the flashover stage is the most important evacuation strategy.
B. : te
B. Flashover phenomena usually occur during the growth phase of a building fire. A A /
C. *
fft
C. When flashover occurs, the chances of survival for people in the room are low.
D.
# (F.O.T.) Æ
D. In the same type of decorative material with equal area, the wall surface has the greatest influence the Flashover Time (FOO.T.).
* D
Answer: D
Figure 66: "fire science" example. The English translation is given beneath the corre- sponding Chinese text and highlighted in blue.
54

---

Published as a conference paper at COLM 2024
B (red lens test H
> ( ドラ A
Question: If the patient has red lens placed in front of the right eye (red lens test), and sees red light point in the lower right, the patient has:
A. # F F
A. Right eye's lower internal latent strabismus
B. H
B. Right eye's upper external latent strabismus
C. 1
C. Left eye's upper internal latent strabismus
D. = +
D. Right eye's upper internal latent strabismus : Answer: D
Figure 67: An "optometry" example. The English translation given beneath the corre- sponding Chinese text and is highlighted in blue.
E A AARAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ( > Df (A
PE #5500 J de
A PEE
116 H E
tH
Question: After ensuring himself as the insurer and the insured with A Life Insurance Company (hereinafter referred to as A Company), and underwriting medical insurance for coronavirus disease, with hospitalization daily claim amount of NTD 5,000 and prepaying the first premium, the patient was diagnosed with COVID-19 and hospital- ized for treatment before A Company underwrote the insurance, and was discharged from the hospital 6 days later. Which of the following statements is true?
A. A A H B MARE 2
A. When A Company agrees to underwrite the insurance, the insurance period is retroactive from the time the first insurance premium amount prepaid.
B. (
B. Company can refuse to underwrite the insurance.
C. A n i ボे
C. A Company should compensate him NTD 30,000.
D. A A ナ
D. A Company can claim that the contract is not yet effective without underwriting.
* C
Answer: C
Figure 68: An "insurance studies" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
55

---

Published as a conference paper at COLM 2024
1
Question: Which of the following commonly used oral adrenocorticoids has the greatest lipid solubility?
A. prednisolone
B. triamcinolone
C. dexamethasone
D. hydrocortisone
# : Answer: C
Figure 69: A "pharmacology" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
| 22 HA B BE
WA A SA548 A 200 A
iz 22 dE EA ty Ht fU$26 _ M ? Question: Company A produces two products, A and B, through joint process with a joint cost of $48,000. Product A produced in 5,000 units, and the cost allocated from the joint cost is $31,200. Product A can be sold at $24 per unit at the separation point, or it can be processed further with an additional cost of $15,000 and sold at $26 per unit. Which of the following statements about product A is correct?
カ1 $15,
A. AM Mi HI B $3$260000
A. The profit from further processing and selling is less, $36,200 less
B. I ste ii n iI
$1115,000
B. The profit from further processing and selling is more, $115,000 more
C. # L 1 Ht 6
$5$5,000
C. The profit from further processing and selling less, $5,000 less
D. Ai #3 *
$26,200
The profit from further processing and selling is more, $26,200 more
; C
Answer: C
Figure 70: A "management accounting" example. The English translation is given beneath the corresponding Chinese text and is highlighted blue.
56

---

Published as a conference paper at COLM 2024
Question: Which of the following statements about wood preservation methods is incorrect?
A. 1
A. Hot water immersion prevent termite damage.
B. X122 7 H
B. Surface treatment by carbonization can achieve anti-corrosion effect.
C. A
A
C. Pressure injection of coal tar preservatives, has the best anti-corrosion effect.
D. #
D. The normal pressure hot and cold immersion agent anti-corrosion method can achieve the anti-corrosion effect.
A Answer: A
Figure 71: An "agriculture" example. The English translation is given beneath the corre- sponding Chinese text and highlighted in blue.
# #
# #O
Question: According to the "Regulations on the Use of ODF Format for Electronic Document Attachments", which of the following is not a ODF format electronic file extension?
A. doc
B. docx
C. pptx
D. E
D. None of the above.
*
Answer: D
Figure 72: An "official document management" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
57

---

Published as a conference paper at COLM 2024
T : S F. F.
ZK Æ
Question: Which ratio often considered a fundamental factor affecting stock return
rate? Price-Farnings Ratio of A; Net Asset Value Ratio of B; Company C Size
A. T A
A. Only C,C
B. T Z
B. Only A, B
C. 2 [
C. Only B,C
D. Z
D. A,
* D
Answer: D
Figure 73: A "financial analysis" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
T E # ?
Question: The consumer market includes many different participants. Which role
referred as the one who gives opinion and influences the purchase decision during the consumption process?
A. DE
Proposer
B. #
B. Decider
C. 3
C. Influencer
D. #
D. Buyer
; C
Answer: C
Figure 74: A "marketing management" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
58

---

Published as a conference paper at COLM 2024
1 1i ?
Question: Which of the following is not an external driving force for promoting change?
A. x :
A. Technological innovation
B. 3 Wa A
B. Change in economic environment
C. 3
C. Change in customer demands
D. AI
D. Change in employee performance evaluation method
X :
Answer: C
Figure 75: "business management" example. The English translation given beneath the corresponding Chinese text and is highlighted in blue.
: or ?
Question: How should the insurer reimburse the salvage expenses necessary for the policyholder or the insured to avoid or mitigate damage?
A.
7 T
Except as otherwise specified the contract, the insurer reimburses the rescue costs according to the value of the insured subject.
B.
E 4
B. The insurer must reimburse the amount and compensation amount, but should be limited to the insurance amount.
C. #
C. The insurer reimburses the amount and compensation amount, even if the total exceeds the insurance amount, the rescue costs should still be reimbursed.
D. BR サ #
D. Except as otherwise specified the contract, the insurer does not need to reimburse the rescue costs.
:C
Answer: C
Figure 76: A "finance banking" example. The English translation is given beneath the corresponding Chinese text and is highlighted in blue.
J Evaluation Environment
Our evaluation environment differentiates between open-weight and closed-source models. Text generation inference is cond conducted on a workstation equipped with eight A6000 GPUs for ween-weight models running bfloat16. In the case of closed-source models, we utilize the Python package provided by their vendor.
59


Reviews:
[{"review_id": "Nf0faRr52t", "reviewer": "Reviewer_8bdQ", "summary": "This paper introduces TMMLU+, an expanded and enhanced benchmark suite designed to assess language model performance on Traditional Chinese language understanding tasks. Compared to its predecessor, TMMLU, it offers a more comprehensive array of multiple-choice questions across a wider range of subjects, with improved balance in subject distribution. The authors conduct extensive analyses examining various model features, such as performance differences between simplified and traditional characters, the use of Chain-of-Thought (CoT), tokenizer vocabulary size, models and human performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks 8bdQ for the detailed overview and concern.\n\n> The benchmark primarily focuses on knowledge-based questions, leading to significant overlap in terminology between Mainland Mandarin and Taiwanese Mandarin. While some questions reflect real language usage differences (e.g., Figure 8 in Appendix D and the Taiwanese Hokkien questions), they constitute a small portion of the overall dataset.\n\nAlthough the underlying knowledge (rules of physics and parts of history and culture) overlaps, our questions also focus on regional knowledge, including legal laws, specialized agricultural practices unique to Taiwan, insurance, accounting rules, finance acts, humanities, and geography, which are key aspects of daily life in Taiwan. Additionally, we included questions related to indigenous people in Taiwan. If any large language models are to be deployed in this region, having a benchmark to ensure the model has a good understanding of the society is important.\n \nOur benchmark specifically addresses the terminology differences between Traditional and Simplified Chinese, ensuring that models can accurately understand and respond to questions in the context of Taiwan. The terminology used in Traditional Chinese often differs from that in Simplified Chinese, reflecting variations in character forms, vocabulary, and usage. For instance, consider the computer science subject in TMMLU+ and the computer architecture section in C-Eval. \n\nBoth questions inquire about the \"four important stages of computer development\" but use different terms for \"computer\" (`電腦` and `电子计算机`).\n\n- The 100th question from the computer science subject in TMMLU+ asks:\n\n`電腦的組成元件歷經四個重要階段，下列的發展順序（由先至後）何者正確？`\n\n- The 81st question from the computer architecture section in C-Eval asks:\n\n`电子计算机的发展已经经历了4代，这4代计算机的主要元件分别是____。`\n\nHence, our dataset could serve as a future work to compare the differences between Taiwanese Mandarin and Mainland Mandarin evaluations (C-Eval) when pre-trained on a different mix of character sources for pretraining.\n\n> In Figure 8, the English prompt is not correctly translated.\n\nNoted, we thank the reviewer for spotting this error and will include the corrected translation prompt in our latest version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. TMMLU+ significantly broadens the scope of its predecessor. It also includes questions specific to Taiwanese culture, like Taiwanese Hokkien and aboriginal cultures.\n2. The authors provide a thorough analysis of different model features, aiming to explain the reasons behind varying model performances.\n3. The paper offers interesting and inspiring findings, such as the observation that models perform better than human test-takers on difficult questions.", "reasons_to_reject": "The benchmark primarily focuses on knowledge-based questions, leading to significant overlap in terminology between Mainland Mandarin and Taiwanese Mandarin. While some questions reflect real language usage differences (e.g., Figure 8 in Appendix D and the Taiwanese Hokkien questions), they constitute a small portion of the overall dataset.", "questions_to_authors": "In Figure 8, the English prompt is not correctly translated.", "extra_scores": {}}, {"review_id": "3ikDBHlLZA", "reviewer": "Reviewer_bAwS", "summary": "The paper introduces TMMLU+, an enhanced Traditional Chinese Evaluation Suite designed to assess the capabilities of LLMs in traditional Chinese. The benchmark is divided into four categories with a total of 22,690 multi-choice questions. The paper conducts a thorough evaluation of several representative LLMs, such as close-source LLMs LLMs (e.g., chatGPT, GPT4) and many open-weight LLMs, including those tailored for simplified and traditional Chinese.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Reply to Rejection:\n\n1. Notation: TC = Traditional Chinese, SC = Simplified Chinese\n\nWe cover regional topics such as taxes, laws, insurance, accounting, finance, humanities, geography, TTQA and Taiwan-specific agricultural practices. Moreover, TTQA explores Taiwanese culture, geography, and history. While overlapping with C-Eval in many STEM fields, our terminology differs. For instance, key computer science terms use different characters. Translation tools like OpenCC may map traditional characters to simplified ones, but they don't accurately represent those used in SC.\n|TC|SC|1-to-1 Character Translation|Meaning|\n|-|-|-|-|\n|記憶體|内存|记忆体|memory|\n|計程車|出租车|计程车|taxi|\n\nNuances also appear at the character level in both languages.\n|Char|TC Usage|SC Usage|\n|-|-|-|\n|丑|醜(ugly)|丑(ugly/1am-3am)|\n||丑(1am-3am)||\n|云|雲(cloud)|云(cloud/say)|\n||云(say)||\n\n2. MIT license\n\n* Reply to Questions:\n\n1. We included only some LLMs in Table 2 to avoid overwhelming the reader. We found a miscount by -3 models and will correct this in the revision.\n\n2. We shuffle question sets for each subject to ensure a balanced mix, choosing the first 5 questions for our development set and dividing the rest between validation and testing (1:9 ratio).\n\nIn Table 2, 10 out of 17 LLMs excel with 5-shot prompting, while only 5 models excel with 0-shot. The lower effectiveness of 5-shot prompting in some models may be due to a limited context size, complicating focus on relevant questions, especially in smaller or less optimized models, as shown by declines in C-Eval[1]. We also review the development set to ensure that the topics are covered in the testing sets.\n\n3. We reviewed GPT-3.5-turbo's reasoning, finding it capable of generating steps with the final answers successfully parsed. However, some CoT answers were incorrect, resulting in a performance drop. This decline is evident in C-Eval[1], where GPT-3.5-turbo's average score decreased from 54.4 to 50.0.\n\nWe attribute this drop to insufficient relevance among bridging objects, as discussed in Section 5[2]. As in  CoT example[3], GPT-3.5-turbo incorrectly assumed that fewer sediments result in lighter water, leading to the selection of the wrong answer. \n\n4. USD 1341.63\n\n[1] Huang, Yuzhen et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. NeurIPS 2023\n\n[2] Wang, Boshi et al. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL 2023\n\n[3] https://pastebin.com/ypdXPtMv", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a traditional Chinese benchmark, TMMLU+, covering 66 subjects with a total of 22,690 questions.\n\n2. The paper provides an in-depth evaluation of many representative LLMs, offering valuable insights into their traditional Chinese performance.", "reasons_to_reject": "1. The benchmark in this work has not well distinguished itself among the many existing benchmarks for Chinese. It shares similarities with benchmarks tailored for simplified Chinese, such as C-Eval.\n\n2. The dataset's license should be explicitly specified in the paper.", "questions_to_authors": "1. Appendix E lists many LLMs. Why Table 2 doesnot report the performance of all LLMs? Additionally, Figure 4 includes additional LLMs not featured in Table 2. In abstract it says 26 open-weight Chinese LLMs, it seems not ture.\n\n2. How are examples chosen for the development and validation sets? Table 2 observes decline in performance for many LLMs with few-shot prompting. Is there a potential correlation with the examples in the development set?\n\n3. LLMs exhibit significantly reduced performance under the CoT setting compared to without. What are the reasons behind?\n\n4. How much money you spent on GPT family models?", "extra_scores": {}}, {"review_id": "z0i4fNpv42", "reviewer": "Reviewer_Zs62", "summary": "This paper introduces TMMLU+, an improved benchmark dataset for evaluating the language understanding capabilities of large language models (LLMs) in Traditional Chinese. TMMLU+ consists of 22,690 multi-choice questions across 66 subjects, ranging from elementary to professional levels. The authors conduct experiments on 26 open-weight language models and several closed-source models, comparing their performance against human test-takers. The findings reveal that Traditional Chinese models still lag behind their Simplified Chinese counterparts, and current LLMs fall short of human performance in average scores, particularly in social science and humanities subjects.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Here are some potential reasons for further investigation:\n\nNotation: TC = Traditional Chinese, SC = Simplified Chinese\n\n**One-to-Many:** Simplification in SC has led to certain characters having multiple meanings, distinct from TC. The issue is exacerbated by shared tokens between the two writing systems, impacting interpretation and vocabulary, and potentially leading to confusion.\n\n| Common Token | TC Usage | SC Usage |\n|---|---|---|\n| 云 | 雲(cloud) | 云(cloud/say) |\n|  | 云(say) |  |\n| 丑 | 醜(ugly) | 丑(ugly/1am-3am) |\n|  | 丑(1am-3am) |  |\n\n**Terminology Difference:** Transitioning between TC and SC requires nuanced translation beyond character substitution, involving differences in character forms, vocabulary, and usage, and necessitates an understanding of cultural and historical contexts, as shown in examples from TMMLU+ and C-Eval.\n\n| TC | SC | 1 to 1 Character Translation | Meaning |\n|---|---|---|---|\n| 記憶體 | 内存 | 记忆体 | memory |\n| 計程車 | 出租车 | 计程车 | taxi |\n\n**Regional Linguistic Variations:** Language evolution, influenced by popular culture and regional traits, results in unique characteristics and vocabulary reflecting geographical and cultural differences. For instance, TC incorporates elements from Taiwanese Hokkien, introducing new terms like the informal expression \"母湯\" (It's not okay!) and \"囡仔\" (little kids), commonly used in casual conversations.\n\n2. Our CoT reasonings were created using the methodology from C-Eval Appendix C, which utilizes GPT-4 for initial reasoning followed by human review. \n\nWe also reviewed reasoning from gpt-3.5-turbo, finding it capable of generating steps, with our parser parsing the final answer successfully. However, some CoT answers were incorrect resulting in a performance drop. This decline is evident in Table 5 of C-Eval [1], with GPT-3.5-turbo's average score decreasing from 54.4 to 50.0.\n\nWe attribute this drop to insufficient relevance among bridging objects, detailed in Section 5 [2]. As shown in the CoT example [3], GPT-3.5-turbo incorrectly assumed fewer sediments result in lighter water. Instead, the correct reasoning is that fewer sediments enable light to penetrate deeper, resulting in darker water due to its depth.\n\n[1] Huang, Yuzhen et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. NeurIPS 2023\n\n[2] Wang, Boshi et al. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL 2023\n\n[3] https://pastebin.com/ypdXPtMv", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces TMMLU+, improving the dataset size and subject diversity of its predecessor.\n- The paper provides a comprehensive evaluation of a wide range of language models, including both open-weight and closed-source models, allowing for a thorough comparison of their performance on Traditional Chinese language understanding tasks.", "reasons_to_reject": "- While the paper compares the performance of Traditional and Simplified Chinese language models, it does not provide the underlying reasons for the performance gap between the two. Further investigation into the linguistic and cultural differences between Traditional and Simplified Chinese could provide more insights. (especially for readers who are not familiar with the Chinese.)\n- The process of creating the CoT demonstration has been overly simplified, and there are concerns about whether the quality has been adequately ensured. Additionally, there is no explanation for why the performance of CoT is lower compared to direct methods.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a traditional Chinese benchmark, TMMLU+, covering 66 subjects with a total of 22,690 questions, along with an in-depth evaluation of many representative LLMs, offering valuable insights into their traditional Chinese performance. The reviewers were all supportive; this will be a very valuable benchmark.
Decision: Accept