E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models (2024)

Abstract: The rapid development of Large Language Models (LLMs) has led to their increasing utilization in Chinese K-12 education. Despite the growing integration of LLMs and education, the absence of a dedicated benchmark for evaluating LLMs within this domain presents a pressing concern. Consequently, there is an urgent need for a comprehensive natural language processing benchmark to precisely assess the capabilities of various LLMs in Chinese K-12 education. In response, we introduce E-EVAL, the first comprehensive evaluation benchmark specifically tailored for Chinese K-12 education. E-EVAL comprises 4,351 multiple-choice questions spanning primary, middle, and high school levels, covering a diverse array of subjects. Through meticulous evaluation, we find that Chinese-dominant models often outperform English-dominant ones, with many exceeding GPT 4.0. However, most struggle with complex subjects like mathematics. Additionally, our analysis indicates that most Chinesedominant LLMs do not achieve higher scores at the primary school level compared to the middle school level, highlighting the nuanced relationship between proficiency in higher-order and lower-order knowledge domains. Furthermore, experimental results highlight the effectiveness of the Chain of Thought (CoT) technique in scientific subjects and Few-shot prompting in liberal arts. Through E-EVAL, we aim to conduct a rigorous analysis delineating the strengths and limitations of LLMs in educational applications, thereby contributing significantly to the advancement of Chinese K-12 education and LLMs.12

heading: 1 Introduction
text: Large Language Models have made significant advancements in the field of natural language processing and artificial intelligence. The evaluation
∗Equal Contribution. †Corresponding author.
of the knowledge and reasoning capabilities embedded in these models has become progressively more difficult, leading to the development of multiple testing benchmarks. Novel benchmarks such as MMLU (Hendrycks et al., 2021), BIG-bench (Srivastava et al., 2022), and HELM (Liang et al., 2022) span multiple domains and tasks, encompassing real-world examinations and textbook knowledge. These benchmarks evaluate not only language comprehension but also the models’ ability in common sense reasoning, mathematical reasoning, and code generation. Concurrently, with the rapid development of Chinese LLMs, an increasing number of Chinese benchmarks have begun to surface. MMCU (Zeng, 2023) focuses on professional domains, AGIEval (Zhong et al., 2023) targets China’s standardized tests, C-EVAL (Huang et al., 2023) encompasses knowledge from middle school to professional fields, CMMLU (Li et al., 2023) concentrates on Chinese culture and CMB (Wang et al., 2023) focuses on the field of Chinese medicine. While these benchmarks primarily focus on the models’ advanced ability, certain specific fields and topics crucial to the models might not receive adequate attention. Currently, there is no comprehensive assessment benchmark in Chinese K-12 education that is important for assessing and analyzing the specifics of LLMs’ learning of human knowledge at all stages.
In this paper, we introduce E-EVAL, the first comprehensive evaluation suite focusing on Chinese K-12, aimed at evaluating basic models’ knowledge and reasoning ability within the context of K-12. E-EVAL comprises 4,351 multiplechoice questions across primary, middle, and high school stages, as depicted in Figure 1 covering 23 subjects including Primary School Chinese, Primary School Mathematics, Primary School English, Primary School Science, Primary School Ethics, Middle School Chinese, Middle School Mathemat-
7753
ics, Middle School English, Middle School Physics, Middle School Chemistry, Middle School Biology, Middle School Politics, Middle School History, Middle School Geography, High School Chinese, High School Mathematics, High School English, High School Physics, High School Chemistry, High School Biology, High School Politics, High School History, High School Geography. We further categorize the questions into two types: arts and science, with science encompassing disciplines like Mathematics, Physics, Chemistry, and arts including Chinese, English, History, etc., noting that the difficulty of arts subjects is generally lower than that of science.
Our evaluation of multiple open-source and commercial models on E-EVAL reveals that Chinesedominant models outperform English-dominant ones in Chinese K-12 performance, with several models even surpassing GPT 4.0. However, performance in complex subjects like Mathematics remains subpar. Additionally, we observed that almost all advanced Chinese models struggle to achieve higher performance at lower educational levels compared to higher ones. This underscores the need for improvement in LLMs within the Chinese K-12 educational domain and highlights the potential value of E-EVAL as an important evaluation suite.
1https://github.com/AI-EDU-LAB/E-EVAL

heading: 2 The E-EVAL Evaluation Benchmark

heading: 2.1 Design Principle
text: E-EVAL is a meticulously crafted benchmark designed to evaluate the performance of LLMs within the diverse educational environments of Chinese K12 education. It encompasses a comprehensive coverage of various subjects, categorizing them into arts and science to provide an all-encompassing evaluation across all educational stages. Overall statistics of E-EVAL are presented in Table 1, with detailed subject-wise data in Appendix Table 5.
The benchmark employs a multiple-choice question format, akin to (Hendrycks et al., 2021), offering a clear and effective method for evaluating the precision and reasoning ability of LLMs. The questions, carefully selected and primarily sourced from homework and local small-scale exams, reflect the real educational setting while maintaining content originality and regional specificity. Special attention has been paid to the manual collection and fine processing of data, especially for subjects involving complex equations, to ensure high data integrity.
E-EVAL deliberately avoids using questions from national exams like the Gaokao to minimize the risk of data contamination, instead opting for mock tests and specific high school online exams. The choice to use PDF and Word documents as primary sources of information, rather than plain
text or structured questions, further reduces the risk of data leakage. E-EVAL is extracted from thousands of test papers from different regions, schools, grades and subjects. This benchmark is particularly aimed at aiding developers in rapidly understanding and enhancing the capability of Large Language Models in processing subject-specific knowledge and content with cultural uniqueness in the context of Chinese education. Thus, E-EVAL is a comprehensive, effective, and culturally benchmark.

heading: 2.2 Data Collection
text: Subjects: E-EVAL encompasses a range of subjects pivotal to Chinese K-12 education, covering key disciplines at primary, middle, and high school levels to cater to the educational needs of different age groups. This comprehensive coverage ensures E-EVAL’s high applicability and representativeness across various educational stages. For better organization of these subjects, they have been divided into two main categories: arts and science, to more aptly reflect the nature and characteristics of each discipline. In the arts category, subjects that study unique aspects of human society, such as politics, economics, and culture, are included. These subjects not only focus on the transmission of knowledge but also emphasize the cultivation of thinking ability and humanistic literacy. This category encompasses subjects like Chinese, English, Politics, History, and Geography, covering the fields of social science and humanity. On the other hand, the science category includes natural science, applied science, and mathematical logic, emphasizing the cultivation of scientific methods and experimental skills, as well as understanding of the natural world and technological domains. This category covers subjects such as Mathematics, Physics, Chemistry, and Biology, encompassing the STEM (Science, Technology, Engineering, and Mathematics) fields. In total, E-EVAL covers 23 different subjects, as illustrated in Figure 1. Sources: Our data primarily come from free, regional homework, practice questions, and mock exams available on the internet3. These are typically provided by schools, educational institutions, or teachers to assist students in consolidating knowledge. Unlike public exams like the Gaokao or Zhongkao, these local homework and practice questions tend not to circulate widely, thereby posing
2https://huggingface.co/datasets/ E-EVAL/E-EVAL
a lower risk of contamination. This choice of data source ensures better originality and quality of data, enhancing the credibility of the assessment. Moreover, these regional questions better reflect the real educational environment and academic requirements, as they are closer to what students encounter in their daily learning.

heading: In terms of difficulty

heading: In terms of arts/science

heading: In terms of split
text: Processing: The collected data are in various formats, primarily PDFs and Microsoft Word documents. For arts subjects like Chinese and English, we employ scripts to automatically parse PDF and Word documents for structured data. However, for science subjects with complex formulas, manual parsing is necessary, converting formulas into standard LaTeX format. Most of the collected questions follow a format of one question with four options, and questions with fewer than four options are discarded. For questions with more than four options, one incorrect option is removed. After format conversion, we conduct three rounds of manual checks: the first ensures no data duplication, the second verifies the correctness and completeness of formulas, and the third confirms the accuracy of answers. After checks, the order of options is deliberately adjusted to achieve a more balanced distribution of correct answers among options A, B, C, and D, with the aim of minimizing the potential impact of option bias within the model. The distribution of correct answer is shown in Table 2. A total of 4351 questions were collected, categorized into development, validation, and test sets across
3Our raw data comes from https://www.zxxk.com/ and https://zujuan.xkw.com/.
the 23 subjects. Additionally, we selected five representative questions with explanations to support Few-shot assessments. A representative example with explanations is illustrated in Appendix Figure 2. The final development set contains 115 questions, the validation set contains 424 questions, and the test set contains 3812 questions, as shown in Table 1.

heading: 2.3 E-EVAL Arts and Science
text: We divided E-EVAL into two separate benchmarks by subject, E-EVAL Arts and E-EVAL Science. EEVAL Arts includes 13 subjects: primary school Chinese, primary school English, primary school ethics, middle school Chinese, middle school English, middle school politics, middle school history, middle school geography, high school Chinese, high school English, high school politics, high school history and high school geography. E-EVAL Science consists of 10 subjects: Primary school Mathematics, Primary school Science, Middle School Mathematics, Middle School Physics, Middle School Chemistry, Middle School Biology, High School Mathematics, High School Physics, High School Chemistry, and High School Biology.

heading: 2.4 Evaluation
text: We use accuracy as the metric for evaluation. To ensure fairness, only the answers to the development and validation sets were disclosed, keeping the test set answers private. This approach prevents the incorporation of E-EVAL data in pre-training datasets. Users are invited to submit their predictions for the test set on our website4 to ascertain their accuracy. The site maintains a public leaderboard, where users have the discretion to publish their model’s results.

heading: 3 Experiment
text: Here we detail our experiments. We tested 15 advanced models on this benchmark, encompassing
4https://eevalbenchmark.com
various sizes, language orientations, and stages (pretrained or fine-tuned). This in-depth analysis of their performance offers a reliable reference point for future research in this field.

heading: 3.1 Setup
text: The experimental design of E-EVAL aims to evaluate the performance of LLMs on E-EVAL. Both open-source and proprietary advanced LLMs were tested. These models were prompted to select the correct choice from a set of questions with four options (ABCD), and regular expressions were used to extract the model’s selected response. Three evaluation methods were developed: zero-shot (Kojima et al., 2022), few-shot-answer-only (few-shotao), and few-shot-chain-of-thought (few-shot-cot) (Wei et al., 2022), to deeply analyze the models’ knowledge and reasoning ability.

heading: 3.2 Prompt
text: We introduced the following phrase before each question: "以下是中国关于[subject]考试的单 项选择题，请选出其中的正确答案(Here is a multiple-choice question from China’s [subject] examination. Please select the correct answer)" In the zero-shot evaluation, the question and options were presented directly after the prompt, without any prior examples or additional information, requiring the model to rely solely on its existing knowledge and understanding to respond.
For the few-shot-ao evaluation, we appended five related questions without explanation from the development set. This method helps the model better understand and adapt to the current question by leveraging these prior examples, enhancing the model’s ability to adapt to new tasks using a minimal number of examples. In the fewshot-cot evaluation, we further included explanations and the prompt "让我们一步一步思考(Let’s think step by step)" building on the few-shot-ao approach. This is designed to encourage the model to demonstrate its step-by-step reasoning process in solving the question, rather than just providing the answer. This evaluation mode emphasizes the model’s reasoning ability, making its approach to problem-solving more akin to human thought processes. At the end of each question, we added "答案是：(Answer:)" to present the model’s final conclusion. The examples of few-shot-ao and few-shot-cot are shown in Appendix Figure 3 and Appendix Figure 4, respectively.

heading: 3.3 Models
text: We assessed 15 models from different countries, organizations, and sizes, as shown in Appendix Table 6. For commercial models, we assessed ChatGPT (OpenAI, 2022), GPT 4.0 (OpenAI, 2023), ERNIE-Bot and ERNIE-Bot 4.0 (Zhang et al., 2019). For open-source models, we tested Qwen-72B/7B (Bai et al., 2023), Yi-34B/6B-Chat (01.AI, 2023), ChatGLM3-6B (Du et al., 2022; Zeng et al., 2022), Baichuan2-13B/7B-Chat (Yang et al., 2023), Chinese-LLaMA-2-13B and ChineseAlpaca-2-13B (Cui et al., 2023), among other general models. Additionally, we evaluated the EduChat series (Dan et al., 2023), focusing on China’s education field, including Educhat-sft-00213B-Baichuan, Educhat-sft-002-7B. Please refer to Appendix F for details on each model.

heading: 3.4 Main Results
text: The testing results of various models are presented in Table 3, where we report the average accuracy for three types of prompts across different categories. The accuracy for each of the three prompts each subject are provided in Appendix C. Among the large-scale models, Alibaba’s Qwen-72B-Chat model achieves the highest accuracy rate, averaging 88.8%, attributed to its extensive parameter size and high-quality Chinese corpus. Baidu’s ERNIE-Bot 4.0 follows closely in second place, trailing Qwen-72B-Chat by only 3.3 percentage points. Yi-34B-Chat and ERNIE-Bot demonstrate comparable overall performance, with a mere 1 percentage point difference in average accuracy. Notably, GPT 4.0 and ChatGPT exhibit poor performance, ranking 5th and 11th in accuracy, respectively. The underperformance of the GPTs may be attributed to the training corpus, where the Chinese corpus constitutes a low percentage. Among models with parameters less than 10B, Yi-6B-Chat performs the best, ranking 6th, approaching the accuracy of GPT 4.0 but still trailing behind Yi-34BChat. This suggests that models with larger parameter sizes demonstrate enhanced knowledge and inference, aligning with expectations. Qwen-7BChat, Baichuan2-13B-Chat, ChatGLM3-6B, and Baichaun2-7B-Chat closely follow, outperforming ChatGPT due to their rich and high-quality Chinese corpus, despite their smaller size. ChineseLLaMA exhibits subpar performance, primarily attributed to the inadequacy of Chinese training data. Although EduChat is primarily trained on an
educational corpus, it underperformed in the evaluation, potentially due to its focus on reinforcing psychological and pedagogical theories.
Overall, Chinese-dominant models that have been trained on a wide range of Chinese corpora show excellent performance in this evaluation. In the same series, the large model outperformed the small model. In addition, all models performed much better in arts subjects than in science subjects. This is in line with our expectations, as text comprehension is a strong point of LLMs, while logical reasoning has been a weak point of LLMs.

heading: 3.5 Insight and Analysis
text: Are large language models better at arts or sciences? Observations from Table 3 regarding model performance across different subject categories reveal that all models perform better in arts subjects than in science subjects. Arts subjects emphasize memory, interpretation, and language understanding, aligning well with the basic construction of the models, which includes language processing and pattern recognition ability. Science subjects, on the other hand, involve logical reasoning, quantitative analysis, and solving complex problems, requiring strong logical reasoning and mathematical processing ability from the models. Therefore, it is logical that models exhibit better performance in arts subjects. Does the simpler the question, the better the model performs? Further observations of model performance at different educational levels in Table 3 show that although the difficulty of the problems gradually increases from primary school to high school, the top-ranked models generally perform better at the middle school level than at the primary school level. This phenomenon is highly counter-intuitive because elementary school questions are far easier than middle school questions from a human cognitive perspective. As shown in Appendix Figure 5, a very simple elementary school math question was answered incorrectly by the top three models on the E-EVAL leaderboard. " Four students ran a race, Ding Ding took 106 seconds, Qiang Qiang took 1 minute 15 seconds, Ming Ming took 92 seconds and Qi Qi took 1 minute 50 seconds. ( ) ran the fastest.". The correct answer is B: Qiang Qiang, but all three models predict C: Ming Ming. The Top-3 LLMs all thought that 92 seconds is faster than 75 seconds. Ernie-Bot 4.0 model surprisingly generated such incredible re-

heading: Model Arts Science Primary Middle High Average
text: sults as 92 < 106 < 110 < 75. This result seems to indicate that LLMs are not good at comparing sizes, yet we find that LLMs are capable of solving similarly complex high school math problems. As shown in the example in Figure 6, the LLMs can accurately determine the magnitude relationship between a = log0.3 0.4, b = log3 4, c = 4 1/3 as log0.3 0.4 < log3 4 < 4 1/3.
We speculate that the pre-training data for these models is probable to use only middle and high school and college level knowledge and ignore the very simple elementary school level knowledge. It is possible that the developers believe that the primary school level is too simple, and that once more complex knowledge is mastered, the simpler knowledge will be automatically mastered. As a result, the model is trained with a bias toward solving higher stage topics and may perform poorly on simple knowledge that occurs less frequently in the training data.
Does Few-shot prompting help? Appendix Table 8 and Appendix Figure 7 illustrate the performance of the models under zero-shot and five-shot conditions. In general, the accuracy of most models is slightly higher under five-shot conditions compared to zero-shot, which is expected. However, there is a decrease in performance for the top two models, Qwen-72B-Chat and ERNIE-Bot 4.0. This phenomenon of performance degradation with a limited number of examples has also been noted
in other studies (Liu et al., 2023a; Zeng, 2023; Li et al., 2023). We believe that they can perform zero-shot reasoning without relying on Few-shot exemplars. Upon further observation from Table 4 and Appendix Figure 7, it is evident that the improvement in arts subjects, especially in Chinese language, is more significant than that in science subjects. We speculate that there are two main reasons for this phenomenon: the quality of the sample and the richness of prior knowledge. The Arts test questions, which primarily required language comprehension and knowledge retention, were highly similar among the samples and were mostly helpful questions. On the contrary, science test questions are diverse and often involve formulas and calculations, so it is highly unlikely that the five samples will contain a sufficient number of useful samples. In addition, during the pre-training phase, the model is exposed to a large amount of textual data, and the arts test questions are formally more similar to the tasks that the model handles during this phase. As a result, the model has more prior knowledge related to the arts, and a small number of art samples can activate this relevant knowledge in time.
Does Chain-of-Thought prompting help? As observed in Appendix Table 8 and Appendix Figure 8, compared to the five-shot-ao, nearly all models show a noticeable decline in average performance on the five-shot-cot. This observation aligns with

heading: Model Arts Science Average
text: the findings of (Huang et al., 2023; Zeng et al., 2024; Song et al., 2023), who noted a deterioration in model performance when applying CoT. We believe that many subjects in E-EVAL, especially in arts, do not require complex reasoning, and additional reasoning steps might decrease performance. Further observation from Table 4 and Appendix Figure 8 reveals that while the average performance decreased, there was a divergence between humanities and sciences, humanities showed a decline, whereas sciences, particularly high school and middle school mathematics, showed an increase. This is because science questions typically have fixed principles and a logical deduction process, hence CoT prompts can effectively guide models in structured reasoning. CoT can aid models in step-bystep construction of answers, which is advantageous for science questions. On the contrary, arts questions often involve broader and more ambiguous knowledge areas. These questions might depend more on intuition, experience, and understanding of polysemous terms, which are not suited for simple logical reasoning. Introducing CoT in arts questions could lead to models over-reasoning or developing reasoning chains in the wrong direction, as these questions might require a wider range of background knowledge and creative thinking, rather than simple step-by-step logical deduction. Therefore, the application of CoT needs to be adjusted based on the question type and complexity,
to better adapt to the characteristics of different tasks.

heading: 4 Related Work
text: With the continuous advancement of natural language processing technology, benchmarks have expanded to encompass more complex tasks, such as machine translation (Bojar et al., 2014) and summarization (Narayan et al., 2018; Hermann et al., 2015). The advent of comprehensive benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Sarlin et al., 2020) heralded a new era. These benchmarks amalgamate various natural language understanding tasks, including textual entailment, sentiment analysis, and question answering, thereby providing a standard for evaluating the holistic performance of models. Popular for their ability to assess models on both understanding and generating natural language, these benchmarks have gained prominence. Yet, the emergence of LLMs like BERT and GPT shifted the focus towards assessing performance on higher-level, more intricate tasks. Such models have even surpassed human-level performance on certain tasks, notably in text summarization (Hermann et al., 2015) and reading comprehension (Rajpurkar et al., 2018; Li et al., 2022). However, some recent work (Goyal et al., 2022; Liu et al., 2023b) have demonstrate that LLM can perform even better than human or human annotators on some tasks such as summarization,
leading to a re-evaluation of the appropriateness of using these benchmarks. To offer a more encompassing evaluation, new benchmarks like MMLU (Hendrycks et al., 2021) include a multitude of domains and tasks, ranging from real-world exams to book knowledge, assessing ability in language understanding, common sense reasoning (Clark et al., 2018; Talmor et al., 2019; Sakaguchi et al., 2021), mathematical reasoning (Hendrycks et al., 2021; Cobbe et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021). The BIG-bench (Srivastava et al., 2022) includes 204 diverse tasks, some of which are deemed beyond the current ability of LLMs. The HELM (Liang et al., 2022) benchmark comprises 42 distinct tasks, evaluating LLMs across seven metrics.
With the burgeoning development of Chinese Large Models, an increasing number of Chinese benchmarks have emerged. CLUE (Xu et al., 2020), an influential Chinese NLU benchmark, has been widely adopted in the field. Additionally, the team has recently introduced SuperCLUE (Xu et al., 2023), a benchmark tailored specifically for LLMs. Concurrently, Chinese benchmarks akin to MMLU (Hendrycks et al., 2021) have surfaced, such as MMCUL (Zeng, 2023), which emphasizes medicine and education within its four major domains. AGIEval (Zhong et al., 2023) focuses on standardized Chinese exams like the college entrance exam, while C-Eval (Huang et al., 2023) encompasses questions across four levels of difficulty from middle school to professional tests. M3KE (Liu et al., 2023a) gathers 71 tasks from the Chinese education examination system, akin to the coverage of C-Eval. CMMLU (Li et al., 2023), designed for the Chinese language and cultural context, is a fully localized Chinese benchmark. Compared to these benchmarks, E-EVAL distinguishes itself by (1) focusing on the field of K-12 Education in China, covering all subjects from primary to high school. (2) including the often-overlooked domain of elementary education. (3) sourcing data from homework and smaller-scale exams, ensuring a high degree of data privacy.

heading: 5 Discussion and Conclusion
text: Although LLMs have potential in K-12 education, their accurate assessment remains critical for practical application, and the introduction of the EEVAL benchmark, customized for K-12 education in China, provides a more accurate and comprehen-
sive evaluation benchmark. Although E-EVAL is not a competitive ranking, it serves as a key tool for tracking the progress of LLMs in Chinese K-12 education. This may pave the way for a wider and more effective utilization of large-scale language models in the field of K-12 education in China.
In this work, we obtained the following potentially insightful observations.
• Chinese-dominant LLMs have outperformed powerful Generalized models like GPT-4 in Chinese K-12 education.
• Generally, the more model parameters the better the results, but smaller models can also perform better than models of larger sizes.
• The gap between open-source and closedsource models is currently very tight, and open-source models are growing rapidly.
• Some specially trained educational LLMs still lag behind generalized models in performance, suggesting that there is still much space for improvement in the education vertical.
• Overall, models perform slightly better in Few-shot compared to Zero-shot, with a more significant improvement observed in liberal arts subjects as opposed to science subjects.
• The application of CoT has a negative impact on the model as a whole, but it is helpful for complex science subjects such as mathematics.
• LLMs are better at liberal arts than science, and especially perform poorly in highly logical mathematics.
• The poor performance of the Chinesedominant large language model on simple problems at primary school level may indicate that the model’s mastery of higher-order knowledge does not mean that it has also mastered lower-order knowledge.

heading: Limitations
text: While we have made every effort to collect questions from various disciplines as comprehensively as possible, we acknowledge that certain subjects, such as physics and geography, often include questions with graphical representations. Regrettably, our current collection does not cover these types of questions with accompanying images.

heading: Acknowledgement
text: This work was supported by National Key Research and Development Program of China (2022YFF0902100), Postdoctoral Fellowship Program of CPSF (GZC20232873), GuangDong Basic and Applied Basic Research Foundation (2023A1515110718 and 2024A1515012003), National Natural Science Foundation of China (62376262), the Natural Science Foundation of Guangdong Province of China (2024A1515030166), Shenzhen Science and Technology Innovation Program (KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039).

heading: A Statistics of E-EVAL
text: Table 5 lists all subjects of E-EVAL, their categories, and the number of questions for each subject. Table 6 lists all models evaluated in this paper.

heading: B Bias of Option
text: The distribution of correct answers is presented in Table 2. We acknowledge minor fluctuations in the proportion of options, which hover around a random 25% mark and resemble those seen in C-EVA and MMLU (Huang et al., 2023; Li et al., 2023).To assess potential option bias, we shuffled the choice order in questions and randomly selected three permutations for each. We then evaluated ChatGPT, ERNIE-Bot, and ChatGLM3-6B on these permutations of E-EVAL. The zero-shot results in answeronly setting are summarized in Table 7. Our results indicate a low variance in accuracy across permutations, suggesting minimal option bias.

heading: C Results of Evaluations
text: Table 8 shows the accuracy of multiple models in different prompt scenarios. Table 9 shows the performace of partial models on each subjects. The accuracy of zero-shot, five-shot-ao and five-shotcot is shown in Table 10, 11 and 12 respectively.

heading: D Evaluation Samples
text: Figure 2 is a development example with explanations from E-EVAL. Figure 3 is an example with five-shot in answer-only setting. Figure 4 is an example with five-shot in Chain-of-Thought setting. Figure 5 is a simple primary school math problem with the predictions of the top-3 models. Figure 6 is a hard high school math problem with the predictions of the top-3 models.

heading: E Accuracy Improvement
text: The accuracy improvement from zero-shot-ao to five-shot-ao across 23 subjects is shown in Figure 7, and the accuracy improvement from five-shot-ao to five-shot-cot is shown in Figure 8.

heading: F Models being Evaluated
text: Baichuan 2-13B and Baichuan 2-7B are the new generation of open-source large language models launched by Baichuan Intelligence. It is trained on a high-quality corpus with 2.6 trillion tokens and has achieved the best performance in authoritative Chinese and English benchmarks of the same size. Baichuan 2 comes in two model variants: Baichuan 2-7B with 70 billion parameters and Baichuan 2-13B with 130 billion parameters. Both models have undergone training on a massive 26 trillion tokens. In this paper, we evaluate the models Baichuan 2-7B-Chat and Baichuan 2-13B-Chat, specifically optimized for adhering to human instructions. These models demonstrate outstanding performance in dialogue and context comprehension.
Qwen-72B and Qwen-7B are integral components of the Qwen series of language models developed by Alibaba Cloud. Both models are built upon the Transformer architecture and have been trained on a diverse range of data sources, including internet texts, professional literature, and code. Qwen-72B, boasting a substantial 72 billion parameters, excels in multiple Chinese and English downstream tasks, particularly in areas such as reasoning and translation. It has undergone extensive pretraining on over 3 trillion tokens, encompassing a wide array of languages and domains, and can support contexts of up to 32,000 tokens in length. On the other hand, the 7-billion-parameter Qwen-7B also demonstrates remarkable data coverage and diversity. In this paper, the models evaluated are the chatbot variants fine-tuned from Qwen-72B and

heading: Subject Category #Questions
text: Qwen-7B through human alignment techniques.
ChatGLM3-6B is the latest open-source model in the ChatGLM series, excels with its base model, ChatGLM3-6B-Base, incorporating diverse training datasets, sufficient training steps, and effective strategies. This culminates in superior performance on various datasets, including those involving semantics, mathematics, reasoning, coding, and knowledge, especially notable among models with less than 10 billion parameters. The model introduces an innovative Prompt format, enhancing multi-turn dialogues, function calls, code interpretation, and agent tasks. It represents a significant advance in bilingual (Chinese and English) language processing, particularly in question answering and dialogue tasks.
Yi-34B and Yi-6B are two large language models in the Yi series developed by 01.AI. Trained from scratch on a 3T multilingual corpus, they exhibit exceptional bilingual ability, excelling in language comprehension, commonsense reasoning, and reading comprehension. Yi-34B performed outstandingly in various assessments, ranking
second only to GPT 4.0 on the AlpacaEval leaderboard in December 2023, surpassing major models like LLaMA2-Chat-70B. In the field of Chinese, it ranked second in the SuperCLUE in October 2023, again only behind GPT 4.0, and ahead of models like Baidu’s ERNIE. Although Yi-6B has fewer parameters than Yi-34B, it plays a crucial role in innovative projects and diverse applications, demonstrating strong language processing ability. In this paper, we are using the chat versions of Yi-34B and Yi-6B.
ChatGPT and GPT 4.0, developed by OpenAI, represent the latest advancements in the GPT series of AI models. While ChatGPT is based on particular GPT foundation models and focuses on smooth conversational experiences, GPT 4.0 is the newest iteration, introducing the capability to process visual inputs, enriching user interactions with the model. GPT 4.0 has demonstrated improved factual response rates and a significant reduction in responses to inappropriate content in OpenAI’s internal tests. These models are trained to better follow human instructions, offering

heading: Model Creator #Parameters Access

heading: Model Arts Science Primary Middle High Average
text: helpfulness, harmlessness, and honesty. GPT 4.0’s updates also include an increased input/output capacity, enhanced creativity and collaborative ability, and the ability to connect to third-party knowledge sources. In the paper, the version of ChatGPT is gpt-3.5-turbo-1106 and GPT 4.0 is gpt-4-0613.
ERNIE-Bot and ERNIE-Bot 4.0 are advanced large language models developed by Baidu. ERNIE-Bot is an industrial-grade, knowledgeenhanced large language model that encompasses extensive Chinese data. It possesses robust capabilities in text comprehension, dialogue question-answering, and content creation. The 4.0 version of ERNIE-Bot represents a comprehensive upgrade of the foundational model, achieving significant improvements in understanding, generation, logic, and memory capabilities compared to its predecessor. The notable feature of ERNIE-Bot 4.0 is its multimodal capability, capable of generating a range of content including text, images, and videos based on simple text prompts and image inputs. Compared to ChatGPT,
ERNIE-Bot 4.0 has an advantage in multimodal ability, though ChatGPT Plus with GPT 4.0 provides multimodality, it currently does not support video generation.
Chinese LLaMA & Alpaca LLM project is based on the LLaMA-2, released by Meta. Developers open-source Chinese LLaMA-2 (foundation model) and Alpaca-2 (instruction-following model). These models extend the original LLaMA2 structure by incorporating an additional 20,000 Chinese tokens into its vocabulary and undergoes secondary pre-training and instruction fine-tuning on Chinese data, which further improved the fundamental semantic understanding of the Chinese language, resulting in a significant performance improvement compared to the first-generation models. In this paper, we used Chinese-LLaMA-213B and Chinese-Alpaca-2-13B.
EduChat is a LLM-based chatbot system in the education domain. Its goal is to support personalized, fair, and compassionate intelligent education, serving teachers, students, and parents.

heading: Model Zero-shot Five-shot-ao Five-shot-cot Average
text: Guided by theories from psychology and education, it further strengthens educational functions such as open question answering, essay assessment, Socratic teaching, and emotional support based on the existing basic LLMs. Developers use an educational corpus for pre-training to enable the model to acquire domain-specific knowledge. They further fine-tune the model on designed system prompts and instructions to stimulate a range of tool usage skills. They proposed several versions of the model. In this paper, we evaluated two models in educhat, educhat-sft-002-13B-Baichuan and educhat-sft-002-13B.

heading: Model Arts Science Primary Middle High Average

heading: Subject Qwen-72B-Chat Ernie-Bot 4.0 Yi-34B GPT 4.0 ChatGLM3-6B

heading: Model Arts Science Primary Middle High Average

heading: Model Arts Science Primary Middle High Average

heading: Future Work
text: This in-depth analysis of their performance offers a reliable reference point for future research in this field