ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models (2024)

Abstract: This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we then evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the finegrained mathematical abilities of their models and facilitate the growth of foundation models. Code is available at https://github. com/conceptmath/conceptmath.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 6815–6839 August 11-16, 2024 ©2024 Association for Computational Linguistics
This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we then evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the finegrained mathematical abilities of their models and facilitate the growth of foundation models. Code is available at https://github. com/conceptmath/conceptmath.
1 Introduction
Mathematical reasoning is a crucial capability for Large Language Models (LLMs). Recent advancements in LLMs, including Anthropic (Anthropic, 2023), GPT-4 (OpenAI, 2023), and LLaMA (Touvron et al., 2023a), have demonstrated impressive mathematical reasoning on existing benchmarks with high average accuracies on datasets like GSM8K (Cobbe et al., 2021). Although these benchmarks are able to measure the overall mathematical reasoning capabilities of LLMs on average, they fail to probe the fine-grained failure modes of mathematical reasoning on specific
* First three authors contributed equally. # Corresponding Author: Jiaheng Liu.
mathematical concepts. For example, Fig. 1 shows that the performance of LLaMA2-13B varies significantly across different concepts and fails on simple concepts like Rational number and Cylinders. It is crucial to know these specific failure modes of the language model, especially in some practical applications where we need to focus on specific mathematical abilities. For example, for financial analysts, calculation, measurement, and statistics are the concepts of most interest while others like geometry are not as important.
Moreover, the mathematics system, by its nature, is more fine-grained than holistic. It is typically organized into distinct math concepts 1, and humans develop comprehensive mathematical capabilities through a concept-by-concept, curriculum-based learning process (Simon, 2011; Fritz et al., 2013). These issues underscore the core motivation of this paper: the need for a finegrained benchmark that evaluates concept-wise mathematical reasoning of LLMs.
Therefore, first, we introduce ConceptMath, the
1https://en.wikipedia.org/wiki/Lists_ of_mathematics_topics
6815
first bilingual (English and Chinese), concept-wise benchmark for measuring mathematical reasoning. ConceptMath gathers math concepts from four educational systems, resulting in four distinct mathematical concept systems: English Elementary, English Middle, Chinese Elementary, and Chinese Middle 2. Each of these concept systems organizes around 50 atomic math concepts under a threelevel hierarchy and each concept includes approximately 20 mathematical problems. Overall, ConceptMath comprises a total of 4011 math word problems across 214 math concepts, and Fig. 2 shows the diagram overview of ConceptMath.
Second, based on our ConceptMath, we perform extensive experiments to assess the mathematical reasoning of existing advanced LLMs, including 2 close-sourced LLMs and 17 opensourced LLMs. These evaluations were performed in zero-shot, chain-of-thought (CoT), and fewshot settings. To our surprise, even though most of the evaluated LLMs claim to achieve high average accuracies on traditional mathematical benchmarks (e.g., GSM8K (Cobbe et al., 2021)), they fail catastrophically across a wide spectrum of mathematical concepts.
Third, to make targeted improvements on underperformed math concepts, we propose an efficient fine-tuning strategy by first training a concept classifier and then crawling a set of samples from a large open-sourced math dataset (Paster et al., 2023; Wang et al., 2023b) for further LLMs finetuning. In Fig. 1, for LLaMA2-FT, we observe that the results of these weaknesses improved a lot after using the efficient fine-tuning method.
In summary, our contributions are as follows:
• We introduce ConceptMath, the first bilingual, concept-wise benchmark for measuring mathematical reasoning. ConceptMath encompasses 4 systems, approximately 214 math concepts, and 4011 math word problems, which can guide further improvements on the mathematical reasoning of existing models.
• Based on ConceptMath, we evaluate many LLMs and perform a comprehensive analysis of their results. For example, we observe that most of these LLMs (including open-sourced, closedsourced, general-purpose, or math-specialized models) show significant variations in their per-
2The four concept systems are abbreviated as Elementary-EN, Middle-EN, Elementary-ZH, and Middle-ZH.
formance results across math concepts. • We also evaluate the contamination rate of our
ConceptMath and introduce a simple and efficient fine-tuning method to improve the weaknesses of existing LLMs.
2 ConceptMath
ConceptMath is the first bilingual, concept-wise benchmark for measuring mathematical reasoning. In this section, we describe the design principle, dataset collection process, dataset statistics and an efficient fine-tuning strategy to enhance the weaknesses identified by our ConceptMath.
2.1 Design Principle
We created ConceptMath based on the following two high-level design principles:
Concept-wised Hierarchical System. The primary goal of ConceptMath is to evaluate the mathematical reasoning capacities of language models at different granularity. Therefore, ConceptMath organizes math problems within a three-level hierarchy of mathematical concepts in Fig. 2. This approach provides concept-wise evaluation for mathematical reasoning of language models and makes targeted and effective improvements possible.
Bilingualism. Most of the current mathematical benchmark focuses solely on English, leaving multi-lingual mathematical reasoning unexplored. As an early effort to explore multi-lingual mathematical reasoning, we evaluate mathematical reasoning in two languages: English and Chinese. Besides, since cultures and educational systems vary across different languages, common math concepts can differ a lot. Therefore, we carefully collect concepts in both languages, instead of merely translating from one language to another. For example, measurement metrics (e.g., money, size) are different for English and Chinese.
2.2 Data Collection
Subsequently, for data collection, we take a twostep approach to operationalize the aforementioned design principles: First, we recruit experts to delineate a hierarchy of math concepts based on different education systems. Secondly, we collect problems for each concept from various sources or design problems manually, which is succeeded by quality assessment and data cleaning.
Math Concept System Construction. Since the education systems provide a natural hierarchy of math concepts, we recruited four teachers from elementary and middle schools, specializing in either English or Chinese, to organize a hierarchy of math concepts for different education systems. This leads to four concept systems: ElementaryEN, Middle-EN, Elementary-ZH, and Middle-ZH, with each system consisting of a three-level hierarchy of around 50 atomic math concepts (Fig. 2).
Math Problem Construction. Then, we conducted a thorough data acquisition from various sources (including educational websites, textbooks, and search engines with specific concepts) to collect math word problems (including both
questions and answers) for each math concept. To guarantee a balance across all concepts, approximately 20 problems were gathered for each math concept. Following this, both GPT-4 (OpenAI, 2023) and human experts were employed to verify and rectify the categorization and the solution of each problem. However, we observed that for some concepts, the problem count was significantly below 20. To address this long-tail issue, manual efforts were undertaken to augment these categories, ensuring a consistent collection of 20 problems for each concept. Furthermore, to broaden the diversity of the dataset and minimize the risk of data contamination, all gathered problems were paraphrased using GPT-4. It is impor-
tant to note that the collection and annotation processes were carried out by a team of six members, each possessing a university degree in an engineering discipline, to maintain a high level of technical expertise in executing these tasks.
2.3 Dataset Statistics
Comparison to existing datasets. As shown in Table 1, our ConceptMath differs from related datasets in various aspects: (1) ConceptMath is the first dataset to study fine-grained mathematical concepts and encompasses 4 systems, 214 math concepts, and 4011 math word problems. (2) Problems in ConcepthMath are carefully annotated based on the mainstream education systems for English (EN) and Chinese (ZH). Details on the hierarchical system. Apart from Fig. 2, we also provide the details on the hierarchical system more clearly in Appendix D. Length distribution. Fig. 3 shows the length distribution of our ConcepthMath, where number of tokens is reported 3. The minimum, average and maximum of the tokens for these questions are 4, 41 and 309, respectively, which shows that they have lexical richness.
3We use the “cl100k_base” tokenizer from https:// github.com/openai/tiktoken
2.4 Efficient Fine-Tuning Based on our ConceptMath, we are able to identify the weaknesses in the mathematical reasoning capability of LLMs through concept-wise evaluation. In this section, we explore a straightforward approach to enhance mathematical abilities towards specific concepts by first training a concept classifier and then curating a set of samples from a large open-sourced math dataset. Specifically, first, by additionally collecting extra 10 problems per concept, we construct a classifier capable of identifying the concept class of a given question. The backbone of this classifier is a pretrained bilingual LLM, where the classification head is operated on its last hidden output feature. Then, we proceed to fine-tune LLMs using this specific dataset combined with the existing general math dataset, which aims to avoid overfitting on a relatively small dataset.
3 Experiments
In this section, we perform extensive experiments to demonstrate the effect of our ConceptMath.
3.1 Experimental Setup Evaluated Models. We assess the mathematical reasoning of existing advanced LLMs on ConceptMath, including 2 close-sourced LLMs (i.e., GPT-3.5/GPT-4 (OpenAI, 2023)) and 17 open-sourced LLMs (i.e., WizardMath13B (Luo et al., 2023), MetaMath-13B (Yu et al., 2023), MAmmoTH-13B (Yue et al., 2023), Qwen-14B/72B (Bai et al., 2023b), Baichuan2-13B (Baichuan, 2023), ChatGLM36B (Du et al., 2022), InternLM2-7B/20B (Team, 2023a), InternLM2-Math-7B/20B (Team, 2023a), LLaMA2-7B/13B/70B (Touvron et al., 2023b), Yi-6B/34B (Team, 2023b) and DeepSeekMath7B (Shao et al., 2024)). Note that WizardMath13B, MetaMath-13B, and MAmmoTH-13B are specialized math language models fine-tuned from LLaMA2. InternLM2-Math and DeepSeekMath are specialized math language models fine-tuned from corresponding language models. More details of these models can be seen in Appendix B.
Evaluation Settings. We employ three distinct evaluation settings: zero-shot, zero-shot with chain-of-thought (CoT), and few-shot promptings. The zero-shot prompting assesses the models’ intrinsic problem-solving abilities without any prior examples. The zero-shot with CoT prompting
evaluates the models’ ability to employ a logical chain of thought. In the few-shot prompting setting, the model is provided with fixed 5-shot prompts for different systems (See Appendix E), which includes five newly created examples with concise ground truth targets. This approach is designed to measure the in-context learning abilities. Besides, following MATH (Hendrycks et al., 2021c), all questions and answers in ConceptMath have been carefully curated, and each problem is evaluated based on exact matches. Moreover, greedy decoding with a temperature of 0 is used.
3.2 Results Overall Accuracy. We present the overall accuracies of different LLMs on our ConceptMath
benchmark under various prompt settings in Table 2. Subsequently, we analyzed the mathematical abilities of these LLMs in both English and Chinese in Fig. 4. Our analysis led to the following key findings: (1) GPT-3.5/4 showcases the most advanced mathematical reasoning abilities among LLMs in both English and Chinese systems, and the leading open-source Qwen72B model archives comparable performance compared with GPT-3.5. (2) The scores on Chinese systems of most existing LLMs are lower than English systems a lot. For example, accuracies on Middle-ZH and Middle-EN for GPT-4 are 63.81 and 84.26. (3) Several models (e.g., WizardMath13B or MetaMath-13B) fine-tuned from LLaMA213B achieve slight improvements on English systems, but the performance results are lower than LLaMA2-13B on Chinese systems a lot, which indicates that domain-specific fine-tuning may degrade the generalization abilities of LLMs. (4). The mathematical models (i.e., InternLM2-Math7B/20B and DeepSeekMath-7B) by continuing pretraining on the large-scale math-related dataset (>=100B tokens) show sufficient improvements when compared to models with similar size, which indicates that large-scale pertaining is effective to improve the mathematical reasoning abilities.
Mean Concept-wised Accuracy. We show the mean accuracy for each concept on the tested mod-
els using zero-shot prompting on Middle-ZH in Fig. 5 (See Appendix A for results on other concept systems). It shows that the accuracies across concepts vary a lot for existing LLMs. For example, around 18% of concepts of Middle-ZH exhibit an accuracy lower than 30%. These concepts with significant potential for improvement should be prioritized to effectively enhance the mathematical abilities of LLMs.
Concept-wised Accuracy. Fig. 6 shows that most existing LLMs, whether open-sourced, closed-sourced, general-purpose, or mathspecialized, exhibit notable differences in their concept accuracies in the zero-shot prompt setting. These disparities may stem from variations in training datasets, strategies, and model sizes, which suggests that apart from common weaknesses, each model possesses its unique areas of deficiency or shortcomings. The concept accuracies of other concept systems and results of all models can be found in the Appendix A.
3.3 Analysis
Contamination. To determine whether a text is in the pretraining data of a LLM, we provide two different contamination detection methods (i.e., Rouge-based and Prob-based methods) to analyze our ConceptMath in Table 3. Specifically, for the Rouge-based method, we just input the first 50% of the question as the input and compute the Rouge-L score between the generation results and the ground-truth label of the last 50% of the text, where a lower Rouge-L score means a lower contamination rate. For the Prob-based method, we follow (Shi et al., 2023) to use the MIN-K% probability metric, which first gets the probability for each token in the test, and selects the K% tokens with minimum probabilities and calculate their av-
erage log-likelihood. If the average log-likelihood is high, the text is likely in the pretraining data. Note that we choose K as 10 in our setting. In Table 3, we observe that the contaminate rates on our ConceptMath are very low, which means that our ConceptMath can provide a reasonable evaluation for existing LLMs.
Unmastered Concepts. We also highlight the several unmastered concepts of the LLaMA2-13B in Table 4, which shows ConceptMath is effective in guiding further refinement of existing LLMs.
Evaluation Prompting. Different from the fewshot or cot prompting evaluation that can boost closed-source models, we find that zero-shot prompting is more effective for certain opensource LLMs in Table 2. This disparity may arise either because the models are not sufficiently powerful to own mathematical CoT capabilities (Yu et al., 2023; Wei et al., 2022) or because these models have already incorporated CoT data during training (Longpre et al., 2023). Consequently, to ensure a comprehensive analysis, we have employed all three prompting methods for evaluation.
Efficient Fine-tuning. To show the effect of efficient fine-tuning, we take the LLaMA2-13B as an example in Table 5. Specifically, we first select 10 concepts with the lowest accuracies in ElementaryEN. Then, we crawl 495 samples (about 50 samples per concept) using the trained classifier as the Concept-Specific (CS) training data from OpenWebMath (Paster et al., 2023). Meanwhile, to avoid overfitting, we introduce the MetaMathQA (MMQA (Yu et al., 2023) ) data to preserve general mathematical abilities. After that, we can finetune LLaMA2-13B by only using MMQA (i.e., LLaMA2 (w/ MMQA)), or using both MMQA and CS data (i.e., LLaMA2 (w/ MMQA & CS)). In
Table 5, we observe that LLaMA2 (w/ MMQA & CS) archives significant improvements on the lowest 10 concepts and preserves well on the other 33 concepts, which shows the effect of efficient finetuning and the advantages of our ConceptMath.
Classifying questions by concept. A key consideration for ConceptMath is whether it is appropriate to arrange evaluations by concept in LLMs because it is clear to humans that a question may involve one or several concepts. We verify if LLMs have this same level of understanding by computing the Mean Reciprocal Rank (MRR) 4 and accuracy of LLMs in classifying each question by concept. We use secondary categories of our hierarchical system as an example to show the MRR and accuracy of each knowledge system. There are approximately 18 concepts in the secondary
4https://en.wikipedia.org/wiki/Mean_ reciprocal_rank
categories for each knowledge system. As shown in Table 6, open-source and proprietary models can understand concepts to some degree, and the ability to understand concepts is positively correlated with the model’s mathematical capabilities. The calculation method and metrics for each concept system are detailed in the Appendix C.
4 Related Work
Large Language Models for Mathematics. Large Language Models (LLMs) such as GPT3.5 and GPT-4 have exhibited promising capabilities in complex tasks (Du et al., 2024; Guo et al., 2023b; Liu et al., 2024a; Zhang et al., 2024; Guo et al., 2024b; Sun et al., 2024). However, the proficiency of open-source alternatives like LLaMA (Touvron et al., 2023a) and LLaMA2 (Touvron et al., 2023b) remains notably inferior on these datasets, particularly in handling
non-English problems. In contrast, models like Baichuan2 (Baichuan, 2023) and Qwen (Bai et al., 2023b) pretrained on multilingual datasets (i.e., Chinese and English) have achieved remarkable performance. Recently, many domain-specialized math language models have been proposed. For example, MetaMath (Yu et al., 2023) leverages the LLaMA2 models and finetunes on the constructed MetaMathQA dataset. MAmmoTH (Yue et al., 2023) synergizes Chain-of-Thought (CoT) and Program-of-Thought (PoT) rationales.
Mathmatical Reasoning Benchmarks. Recently, many mathematical datasets (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Lu et al., 2023; Huang et al., 2016; Miao et al., 2020; Patel et al., 2021) have been proposed. For example, SingleOp (Roy et al., 2015), expands the scope to include more complex operations like multiplication and division. GSM8K (Cobbe et al., 2021) is a widely used dataset, which requires a sequence of elementary calculations with basic arithmetic operations.
Fine-Grained Benchmarks. Traditional benchmarks focus on assessing certain abilities of models on one task (Guo et al., 2023b; Wang et al., 2023a; Liu et al., 2020; Guo et al., 2022; Chai et al., 2024; Liu et al., 2024a; Guo et al., 2024a,
2023c; Bai et al., 2023a; Liu et al., 2022; Guo et al., 2023a; Bai et al., 2024; Li et al., 2024). For example, the GLUE benchmark (Wang et al., 2019) combines a collection of tasks, and has witnessed superhuman model performance for pretraining models (Radford et al., 2019). Hendrycks et al. (2021b) introduced MMLU, a benchmark with multiple-choice questions across 57 subjects including STEM, humanities, and social sciences, for assessing performance and identifying weaknesses. Srivastava et al. (2022) proposed BIGbench with over 200 tasks.
5 Conclusion
We introduce a new bilingual concept-wise math reasoning dataset called ConceptMath to assess models across a diverse set of concepts. First, ConceptMath covers more than 200 concepts across elementary and middle schools for mainstream English and Chinese systems. Second, we extensively evaluate existing LLMs by three prompting methods, which can guide further improvements for these LLMs on mathematical abilities. Third, we analyze the contamination rates, error cases and provide a simple and efficient fine-tuning strategy to enhance the weaknesses.
Limitations
Mathematical concepts may form a network, where failure to understand a complex concept can result from deficiencies in simpler foundational concepts or issues within the complex concept itself. Thus, the current hierarchical arrangement may be insufficient, and introducing an interrelatedness network of concepts could provide more insightful explanations of the mathematical abilities of language models. Building an accurate and interpretable interrelatedness network within our hierarchical system is a promising direction for future research.
Ethics Statement
ConceptMath consists of math problems, thus very few samples contain offensive content or sensitive personal information. All annotators were fully informed about the entire annotation process.
References
Anthropic. 2023. Model card and evaluations for claude models.
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. 2024. Mt-bench101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762.
Jiaqi Bai, Hongcheng Guo, Jiaheng Liu, Jian Yang, Xinnian Liang, Zhao Yan, and Zhoujun Li. 2023a. Griprank: Bridging the gap between retrieval and generation via the generative knowledge improved passage ranking. CIKM.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023b. Qwen technical report. arXiv preprint arXiv:2309.16609.
Baichuan. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al. 2024. xcot: Cross-lingual instruction tuning for crosslingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07037.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/ opencompass.
Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, and Ge Zhang. 2024. Chinese tiny llm: Pretraining a chinese-centric large language model. Preprint, arXiv:2404.04167.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335.
Annemarie Fritz, Antje Ehlert, and Lars Balzer. 2013. Development of mathematical concepts as basis for an elaborated mathematical understanding. South African Journal of Childhood Education, 3(1):38– 67.
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, Zheng Cui, and Furu Wei. 2022. Lvp-m3: language-aware visual prompt for multilingual multimodal machine translation. EMNLP.
Hongcheng Guo, Boyang Wang, Jiaqi Bai, Jiaheng Liu, Jian Yang, and Zhoujun Li. 2023a. M2c: Towards automatic multimodal manga complement. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9876–9882.
Hongcheng Guo, Jian Yang, Jiaheng Liu, Jiaqi Bai, Boyang Wang, Zhoujun Li, Tieqiao Zheng, Bo Zhang, Qi Tian, et al. 2024a. Logformer: A pretrain and tuning pipeline for log anomaly detection. AAAI.
Hongcheng Guo, Jian Yang, Jiaheng Liu, Liqun Yang, Linzheng Chai, Jiaqi Bai, Junran Peng, Xiaorong Hu, Chao Chen, Dongfeng Zhang, et al. 2023b. Owl: A large language model for it operations. arXiv preprint arXiv:2309.09298.
Jinyang Guo, Jiaheng Liu, Zining Wang, Yuqing Ma, Ruihao Gong, Ke Xu, and Xianglong Liu. 2023c. Adaptive contrastive knowledge distillation for bert compression. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8941– 8953.
Jinyang Guo, Jianyu Wu, Zining Wang, Jiaheng Liu, Ge Yang, Yifu Ding, Ruihao Gong, Haotong Qin, and Xianglong Liu. 2024b. Compressing large language models by joint sparsification and quantization. ICML.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021c. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533, Doha, Qatar. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–896.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597.
Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, et al. 2024. Cif-bench: A chinese instruction-following benchmark for evaluating the generalizability of large language models. arXiv preprint arXiv:2402.13109.
Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, et al. 2024a. E2-llm: Efficient and extreme length extension of large language models. arXiv preprint arXiv:2401.06951.
Jiaheng Liu, Tan Yu, Hanyu Peng, Mingming Sun, and Ping Li. 2022. Cross-lingual cross-modal consolidation for effective multilingual video corpus moment retrieval. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1854–1862.
Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Ken Chen, Wanli Ouyang, and Dong Xu. 2020. Block proposal neural architecture search. IEEE Transactions on Image Processing, 30:15–25.
Jie Liu, Zhanhui Zhou, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. 2024b. Storm-7b: An empirical study of iterative direct preference optimization.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In The Eleventh International Conference on Learning Representations.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
Norman Megill and David A Wheeler. 2019. Metamath: a computer language for mathematical proofs. Lulu. com.
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984.
OpenAI. 2023. Gpt-4 technical report. PREPRINT.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023. Openwebmath: An open dataset of high-quality mathematical web text. Preprint, arXiv:2310.06786.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752.
Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789.
Martin A Simon. 2011. Studying mathematics conceptual learning: Student learning through their mathematical activity. North American Chapter of the International Group for the Psychology of Mathematics Education.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
Tao Sun, Linzheng Chai, Yuwei Yin Jian Yang, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. 2024. Unicoder: Scaling code large language model via universal code. ACL.
Minghuan Tan, Lei Wang, Lingxiao Jiang, and Jing Jiang. 2021. Investigating math word problems using pretrained multilingual language models. arXiv preprint arXiv:2105.08928.
InternLM Team. 2023a. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/ InternLM-techreport.
Yi Team. 2023b. Yi: Building the next generation of open-source and bilingual llms. https:// github.com/01-ai/Yi.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 845– 854.
Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng. 2023a. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746.
Zengzhi Wang, Rui Xia, and Liu Pengfei. 2023b. Generative ai for math: Part i – mathpile: A billiontoken-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824– 24837.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv: 2309.05653.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. 2024. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv: 2405.19327.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning
for large language models: A survey. arXiv preprint arXiv:2308.10792.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2023. Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. 2024. Weak-to-strong search: Align large language models via searching over small language models. arXiv preprint arXiv:2405.19262.
A Concept Accuracy
We provide the mean concept accuracies of Middle-EN, Elementary-EN and Elementary-ZH of the evaluated models across different concepts in Fig. 7, Fig. 8 and Fig. 9.
We illustrate the concept accuracies on MiddleZH, Elementary-EN and Elementary-ZH for some selected models in Fig. 10, Fig. 11 and Fig. 12. For the results of all models, please refer to Fig. 13, Fig. 14, Fig. 15 and Fig. 16.
B Details on the Evaluated Models
In this section, we offer a detailed overview of the Large Language Models (LLMs) and present the corresponding model links in Table 7. For enhanced safety and better conversational abilities, most LLMs undergo alignment techniques (Zhang et al., 2023; Shen et al., 2023; Zhou et al., 2023, 2024; Liu et al., 2024b) to align with human intents. We include instruction-finetuned models, math language models and chat models aligned with human intents in our evaluation.
• GPT-3.5/GPT-4 (OpenAI, 2023): The most powerful closed-model from OpenAI. We utilize its API: gpt-3.5-turbo and gpt-4.
• LLaMa2-7B/13B/70B (Touvron et al., 2023b): A set of open-source models developed by Meta.
• Qwen-14B/72B (Bai et al., 2023b): This model pre-trained on multilingual data, concentrates on Chinese and English languages. We employ both the Qwen-Base-14B, and the Qwen-Base72B.
• Baichuan2-13B (Baichuan, 2023): This model demonstrates impressive performance in both Chinese and English benchmarks.
• MetaMath-13B (Megill and Wheeler, 2019): A domain-specific language model for mathematical reasoning, fine-tuned from the LLaMA-2 model using the MetaMathQA 5 dataset.
5https://huggingface.co/datasets/ meta-math/MetaMathQA
• WizardMath-13B (Luo et al., 2023): Another domain-specific language model for mathematical reasoning, fine-tuned from the LLaMA-2 model using reinforcement learning.
• MAmmoTH-13B (Yue et al., 2023): This model is specifically designed for general math problem-solving and has been fine-tuned from the LLaMA model using the MathInstruct 6
dataset. This dataset features training data that includes both chain-of-thought (CoT) and program-of-thought (PoT) rationales.
• Yi-6B/34B (Team, 2023b): This model released by 01 shows promising performance results in both Chinese and English.
• ChatGLM3-6B (Zeng et al., 2022): a lightweight and high-performance pre-trained dialogue model released by Zhipu AI in both Chinese and English.
• InternLM-7B/20B (Team, 2023a): A Multilin6https://huggingface.co/datasets/ TIGER-Lab/MathInstruct
gual Language Model with Progressively Enhanced Capabilities released by InternLM team.
• InternLM-Math-7B/20B (Team, 2023a): Wellperformed math reasoning language models.
• DeepSeekMath-7B (Shao et al., 2024): One powerful mathematical language model released by DeepSeek.
C Classifying questions by concept
We compute the accuracy and Mean Reciprocal Rank7 (MRR) of two closed-source LLMs (i.e., GPT-3.5 and GPT-4) and five open-source LLMs (LLaMA2-7B/13B, DeepSeekMath-7B, Qwen1.57B/14B) in classifying each question by concept.
The calculation method is different for closedsource and open-source methods. Specifically, for the open-source LLMs, we follow the benchmark processing on the MMLU dataset in the OpenCompass project (Contributors, 2023) to use options (i.e., A, B, ...) for labeling each concept and then calculate the log-likelihood for each option as the basis for ranking. For the closed-source LLMs (i.e., GPT-3.5 and GPT-4), as we cannot obtain the log-likelihood for each option and these closedsource methods have strong abilities to follow instructions, we use a system prompt to make the model output the relevant concepts the question belongs to, where the rankings are obtained from most to least relevant to the corresponding question. After getting the ranked concepts using the above methods, we calculate MRR and accuracy, which is shown in Table 9 and Table 8.
D Details on the ConceptMath
As shown in Table 10, Table 11, Table 12 and Table 13, we have provided the details on the threelevel hierarchical system of our ConceptMath for better illustration.
E Details on 5-shot Prompts
We provide the 5-shot prompts for our ConceptMath in Pages 17-20.
7https://en.wikipedia.org/wiki/Mean_ reciprocal_rank
Elementary-EN (5-shot prompts)
Q: For a party, Oscar gets 4 different kinds of soda. He buys a 6-pack of each kind. At home, he divides the sodas evenly among 3 coolers. How many sodas are in each cooler? A: Oscar buys 4 different kinds of soda, and he gets a 6-pack of each kind. This means that he has 4 * 6 = 24 sodas in total. He divides these 24 sodas evenly among 3 coolers. So number of sodas per cooler is 24 / 3 = 8. Therefore, the answer is 8. Q: Amazing Animals Zoo is building a new home for its 8 zebras. Each zebra will get 2 acres of grassy land to roam around in. The zebras will have a pond that covers 1 acre. How big will the zebras’ new home be? A: The total required grassland area for the zebras is: 8 zebras * 2 acres per zebra = 16 acres. Then, we need to add the extra 1 acre for the pond: 16 acres (for the zebras) + 1 acre (for the pond) = 17 acres. Therefore, the answer is 17. Q: The state taxes personal income exceeding 3500 yuan (this portion not to exceed 3500 yuan) at a rate of 3%. If Xiaoming’s father’s monthly salary is 4700 yuan, how much personal income tax should Xiaoming’s father pay each month? A: To calculate the amount of personal income tax Xiaoming’s father should pay, we can follow this simple process: Subtract the tax-free threshold from the monthly salary. Taxable income = Monthly salary - Tax-free threshold = 4700 - 3500 = 1200 yuan. Apply the tax rate to the amount that exceeds the threshold. Tax owed = Taxable income ∗ Tax rate = 1200 * 3% = 36 yuan. Therefore, the answer is 36. Q: Find the 10th term in the arithmetic sequence where the first term is 3 and the common difference is 4. A: To find the nth term in an arithmetic sequence, we can use the formula: an = a1+(n− 1) ∗ d, where: an is the nth term, a1 is the first term, d is the common difference, n is the term number. Given that the first term (a1) is 3 and the common difference (d) is 4, we can find the 10th term (a10) by plugging these values into the formula: a10 = a1+(10−1)∗d = 3+9∗4 = 3+36 = 39. Therefore, the answer is 39. Q: A regular polygon has an interior angle of 156 degrees. How many sides does the polygon have? A: To find the number of sides of a regular polygon with a given interior angle, we can use the formula: Interior Angle = [(n - 2) * 180] / n, where n is the number of sides. Given that interior angle is 156: 156 = [(n - 2) * 180] / n. Solve for n, so we get n = 15. Therefore, the answer is 15.
Middle-EN (5-shot prompts)
Q: g(x) = x2, f(x) = 3(x− 2)2, h(x) = g(x)− f(x), y = h(23), give me the value of y. A: First, calculate g(23) and f(23): g(23) = 232 = 529. Next, calculate f(23): f(23) = 3 ∗ (23 − 2)2 = 3 ∗ (21)2 = 3 ∗ 441 = 1323. Now, we calculate h(23): h(23) = g(23)− f(23) = 529− 1323 = −794. Therefore, the answer is -794. Q: What is the slope of the line segment between the two points: (3,1), (5,2)? A: The slope of a line between two points (x1, y1) and (x2, y2) can be found using the formula: slope(m) = (y2 − y1)/(x2 − x1). Given the two points (3, 1) and (5, 2), we can substitute these values into the formula: slope(m) = (2− 1)/(5− 3) = 1/2 = 0.5. Therefore, the answer is 1/2. Q: In a party, there are 5 cupcakes remaining and 10 children. A mother inserts 10 pieces of paper into a hat, 5 with a cupcake image and 5 without. The children who draw the cupcake paper will receive a cupcake. If the first child draws a paper with a cupcake image, what is the probability that the second child will also draw a cupcake paper? A: Initially, there are 5 cupcake papers and 5 non-cupcake papers in the hat, making a total of 10 papers. After the first child draws a cupcake paper, there are 4 cupcake papers and 5 non-cupcake papers left in the hat, with a total of 9 papers. The probability that the second child will draw a cupcake paper is then the number of cupcake papers left divided by the total number of papers left: Probability = Number of cupcake papers left / Total number of papers left = 4/9. Therefore, the answer is 4/9. Q: What is the total area of an irregular polygon that consists of two squares and two rectangles where the sides of the squares are 12 cm and 16 cm respectively, and the dimensions of the rectangles are 13 cm by 20 cm and 10 cm by 7 cm respectively? A: To find the total area of an irregular polygon consisting of two squares and two rectangles, you would sum up the individual areas of each shape. The area of a square is given by the formula A = s2, where s is the length of a side of the square. For the first square with a side of 12 cm: A1 = 122 = 144 cm2. For the second square with a side of 16 cm: A2 = 16
2 = 256 cm2. The area of a rectangle is given by the formula ( A = l ∗ w ), where l is the length and w is the width of the rectangle. For the first rectangle with dimensions 13 cm by 20 cm: A3 = 13 ∗ 20 = 260 cm2. For the second rectangle with dimensions 10 cm by 7 cm: A4 = 10∗7 = 70 cm2. Total Area = A1+A2+A3+A4 = 144+256+260+70 = 730 cm2. Therefore, the answer is 730. Q: Susie is making invitations to her party this coming weekend. It takes her about 5 minutes to make each invitation. It took her 2 hours to finish all the invitations. How many people is Susie inviting? A: First, convert the 2 hours into minutes to match the unit of time used for the invitation-making process. 2 hours = 120 minutes. Then, divide the total time spent by the time it takes to make one invitation to find out how many invitations she made: 120 minutes total / 5 minutes per invitation = 24 invitations. Therefore, the answer is 24.
Elementary-ZH (5-shot prompts)
Q:小文用量角器测量一个角的度数时，误把外圈刻度看成了内圈刻度，量出的角是 65°， 这个角实际是多少度？ A:量角器一般有两个圈的刻度，内圈刻度通常是顺时针方向从 0度到 180度，外圈刻度 通常是逆时针方向从 0度到 180度。如果小文误把外圈刻度看成了内圈刻度，那么测量 出的角度其实是从 180°减去错误读数。因此，实际的角度应该是：180°− 65° = 115°。因 此，答案是 115。 Q:小明以每小时 2千米的速度上山，到了山顶后立即以每小时 3千米的速度按原路返回。 求小明上下山的平均速度是多少千米？ A:要计算小明上下山的平均速度，我们需要知道他上山和下山所走的总距离以及总时间。 首先，总距离 =上山距离 +下山距离 = d + d = 2d千米。然后，计算总时间：上山时间 =上 山距离/上山速度 = d / 2小时，下山时间 =下山距离/下山速度 = d / 3小时，因此，总时间 =上山时间 +下山时间 = d / 2 + d / 3小时。最后，计算平均速度：平均速度 =总距离 /总 时间 = 2d / (d/2 + d/3) = 2d / (5d / 6) = 12 / 5 = 2.4千米/小时。因此，答案是 2.4千米/小时。 Q:一个礼盒长 20cm、宽 15cm、高 10cm，用彩带捆扎这样的一个礼盒，至少需要多长的 彩带？（接头处 12厘米） A:要捆扎这样一个礼盒，通常情况下我们会沿着礼盒的长和宽方向各绕一圈，即绕过礼 盒的长和宽两次，然后再留一些长度作为接头。首先，计算沿着礼盒的长方向绕一圈的长 度：绕长方向长度 = (礼盒的长 +礼盒的高) * 2 = (20 + 10)* 2 = 30 * 2 = 60 cm然后，计算 沿着礼盒的宽方向绕一圈的长度：绕宽方向长度 = (礼盒的宽 +礼盒的高) * 2 = (15 + 10)* 2 = 25 * 2 = 50 cm最后，计算总需要彩带长度：总长度 =绕长方向长度 +绕宽方向长度 + 接头长度 = 60 + 50 + 12 = 122 cm。因此，答案是 122厘米。 Q:千岛湖大桥的一侧每隔 10米安装 1盏灯，共需 91盏（两端都装），现在改装 61盏（两 端都装），平均每相邻两盏灯之间的距离是多少米？ A:原来每隔 10米安装 1盏灯，共需 91盏，这意味着桥的总长度覆盖了 90个 10米的间 隔，所以桥的总长度 = 90间隔 * 10米/间隔 = 900米。现在改装 61盏灯（两端都装），也 就是说将这 900米分成 60个间隔（因为 61个灯包含了两端的灯，所以间隔数比灯的数量 少一个），每个间隔的长度 = 900米 / 60间隔 = 15米/间隔。因此，答案是 115米。 Q: a是 b的 3/4，则 a比 b少 ( )，b比 a多 ( )，b与 a的比是 ( )，如果 a、b两数的和是 210，则 a＝ ( )，b＝ ( )，括号处分别为多少？ A:已知 a是 b的 3/4，则 a = 3/4 * b。因此，a比 b少 (1-3/4)/1 = 1/4，b比 a多（1-3/4)/(3/4) = 1/3，b与 a的比为 b : (3/4 * b) = 4:3。 若已知 a、b两数的和是 210，则 a+b = 210，将 a = (3/4) * b代入方程得到：3/4 * b + b = 210，解得 b = 120，a = 3/4 * b = 3/4 * 120 = 90。 因此，答案分别是 1/4，1/3，4:3，90，120。
Middle-ZH (5-shot prompts)
Q:求不等式 −x2 − 2x+ 8 > 0的解集? A: −x2 − 2x+ 8 > 0,通过分解因式方式，得到 −(x− 2)(x+ 4) > 0因此，答案是 (-4, 2)。 Q:已知函数 y = 2x+ 3与的图象 y = −1/2x+m交 y轴于点 A、B，两函数图象的交点 为 P(n,-2)，求 m和 n的值？ A:两函数图像交点为 P(n,-2)，意味着这个点同时满足上述两个函数方程：将 P点的坐标 (n,-2)代入函数 y = 3x+3，即 −2 = 3n+3，解得 n = −5/2。将 P点坐标 (-5/2,-2)代入函 数 y = −1/2x+m，即 −2 = −1/2 ∗ (−5/2)+m，解得 m=-13/4。因此，答案是 m为-13/4， n为-5/2。 Q:已知 PA是圆 O的切线，A为切点，PC与圆 O相交于 B、C两点，PB=2cm，BC=8cm， 则 PA的长是多少厘米？ A:由于 PA是圆 O的切线，根据切割线定理，我们知道从圆外一点引圆的两条割线，这 一点到每条割线与圆的交点的两条线段长的积相等，即 PA2 = PC ∗ PB ,已知 PB=2cm、 BC=8cm，则 PC = PB+BC = 2cm+8cm = 10cm。PA2 = PC ∗PB = 10 ∗ 2 = 20，解 得 PA = 2 √ 5cm。因此，答案是 2 √ 5厘米。 Q:南宋著名数学家杨辉精研数学，著有《详解九章算法》，对数的运算进行了深入研究 与总结，类比其中的思想方法，可以解决很多数与式的计算问题已知 a，b 为实数，且 a+b = 3, ab = 1，计算可得：a2+b2 = 7，a3+b3 = 18，a4+b4 = 47，由此求 a5+b5得多少？ A:已知 a+b = 3和 ab = 1，可得：a2+b2 = (a+b)2−2ab = 32−2∗1 = 9−2 = 7a3+b3 = (a+b)(a2−ab+b2) = 3(7−1) = 3∗6 = 18a4+b4 = (a2+b2)2−2a2b2 = 72−2∗12 = 49−2 = 47a5+b5 = (a+b)(a4−a3b+a2b2−ab3+b4) = 3(a4+b4−a2b2) = 3(47−1) = 3∗46 = 138。 因此，答案是 138。 Q:某单位要招聘 1名英语翻译，甲、乙两人报名参加了听、说、读、写等项素质测试，甲 的成绩分别为：90分、80分、85分、78分；乙的成绩分别为：78分、82分、85分、88 分．如果把听、说、读、写的成绩按 2 : 2 : 3 : 3计算素质测试平均成绩，那么甲和乙的平 均成绩分别是多少？ A:甲的平均成绩为 (90∗2+80∗2+85∗3+78∗3)/(2+2+3+3) = (180+160+255+234)/10 = 829/10 = 82.9分，乙的平均成绩为 (78 ∗ 2 + 82 ∗ 2 + 85 ∗ 3 + 88 ∗ 3)/(2 + 2 + 3 + 3) = (156 + 164 + 255 + 264)/10 = 839/10 = 83.9分。因此，答案是甲的平均成绩是 82.9分， 乙的平均成绩是 83.9分。

heading: Future Work
text: Building an accurate and interpretable interrelatedness network within our hierarchical system is a promising direction for future research