It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset
Authors: Jiwoo Kim
Youngbin Kim
Ilwoong Baek
JinYeong Bak
Jongwuk Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Math Word Problem
Arithmetic Reasonong
Natural Language Processing
Dataset
Large Language Model

Abstract: The math word problem (MWP) is a complex task that requires natural language understanding and logical reasoning to extract key knowledge from natural language narratives. Previous studies have provided various MWP datasets but lack diversity in problem types, lexical usage patterns, languages, and annotations for intermediate solutions. To address these limitations, we introduce a new MWP dataset, named DMath (Diverse Math Word Problems), offering a wide range of diversity in problem types, lexical usage patterns, languages, and intermediate solutions. The problems are available in English and Korean and include an expression tree and Python code as intermediate solutions. Through extensive experiments, we demonstrate that the DMath dataset provides a new opportunity to evaluate the capability of large language models, i.e., GPT-4 only achieves about 75\% accuracy on the DMath dataset.

It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset
Jiwoo Kim
Younghin Kim Ilwoong Baek Jin Yeong Bak Jongwuk Lee Sungkyunkwan University, Republic of Korea (jindog121)) 210, andyk3603, alltun100, jy bak, Longwuklee)@skku edu
Abstract
GSMBK
ASDIV
100
100
The math word problem (MWP) is complex task that requires natural language understand- ing and logical reasoning extract key knowl- edge from natural language narratives. Previous studies have provided various MWP datasets but lack diversity in problem types, lexical us- age patterns, languages, and annotations for intermediate solutions. To address these limita- tions, we introduce new MWP dataset, named DMath (Diverse Math Word Problems), of- fering wide range of diversity problem types, lexical usage patterns, languages, and intermediate solutions. The problems are avail- able in English and Korean and include pression tree and Python code as intermedi- ate solutions. Through extensive experiments, we demonstrate that the DMath dataset pro- vides new opportunity to evaluate the capabil- ity of large language models, ie.. GPT-4 only achieves about 75% accuracy the DMath dataset.
Test accuracy (%)
80
80
70
70
60
ChatGPT
GPT-4
Figure 1: The accuracy of two LLMs on MWP datasets. The result ChatGPT (gpt-3. 5-turbo; OpenAl) and GPTT4 (OpenAI, 2023). using few-shot CoT (Wei et al., 2022) prompting method, indicate that DMath is the most challenging benchmark. GPT-4 shows about 90% accuracy three existing datasets and approxi- mately 75% on DMath.
multi-aspedt diversity regarding problem types, Iex- ical usage patterns, languages, and intermediate solution forms, as reported in Table 1. The low- diversity dataset leads to over-optimistic results for MWP models by memorizing frequent patterns.
1 Introduction
The math word problem (MWP) is challenging and intriguing task that requires a deep understand- ing of natural language narratives and logical rea- soning (Bobrow, 1964; Kushman et al., 2014). The narratives typically involve several numbers and unknown variables. Also, some problems demand commonsense reasoming. Solving these problems requires comprehensive understanding of world knowledge and the reasoning between mathemati- cal representations within given context.
With the advent of large language models (LLMs), they have shown remarkable perfor- mances in various reasoning tasks. Notably, the adoption of GPT-4 (OpenAI, 2023) has yielded an impressive 92% accuracy on the GSM8K dataset (Cobbe al., 2021), which is widely used as a representative MWP benchmark. This result has led to the belief that LLMs demonstrate ex- ceptional proficiency mathematical reasoning. However, our empirical findings show that the true capability of LLMs has not been revealed because of the limited MWP benchmarks. As illustrated in Figure 1, GPT-4 achieves more than 90% accuracy the existing three MWP datasets, but its accu- racy drops approximately 75% on our dataset with multi-aspect diversity. This significant gap motivates us to develop more diverse datasets with greater complexity and challenges in real-world problems than existing ones.
As the first step developing MWP models, is vital collect a representative subset of MWP to construct MWP benchmark datasets. Although many existing studies (Koncel-Kedziorskt et al., 2016; Roy and Roth, 2017; Miao et al., 2020) have published various MWP corpora, they mostly lack
*Conresponding author 'The dataset is available at https///ithuh.ccom/ SonooiiAA//dmatt

---

Dataset
Language
Annotation
Data size
CLD
Eq. templates
Total ops
MAWPS (Koneel-Kodziorski al. 2016)
EN
Expression tree
1.921
0.26
39
MarhQA (Amini al., 2019)
Expression tree
37,259
0.05
ASDiv-A (Miao et al., 2020)
EN
Expression tree
1,218
0.50
19
SVAMP (Putel al., 2021)
EN
Expression
1,000
0.22
26
GSM8K (Cobbe et al., 2021)
EN
Natural language
8,792
0.81
Math23K (Wang al.. 2017)
ZH
Expression tree
23,160
2,187
Ape210K (Zhao et al., 2020)
Expression tree
210,488
56,532
DMath (Ours)
EN, KO
Expression tree, Python code
10,022
0.49
2,855
50
Table 1: Statistics of DMath and other MWP datasets. We used corpus lexicon diversity (CLD) proposed Miao et al. (2020) to quantify the diversity of lexical usage patterns. The higher, the more diverse. The underlined numbers represent the top-3 values in each column. Let EN, KO, and ZH denote English, Korean, and Chinese, respectively, and 'ops' denotes operators.
In this study, we introduce a multi-aspect diverse MWP dataset, called DMath, which has the fol- lowing key features: (1) DMath fully covers prob- lem types across five categories outlined in the math education guideline of grade schools in the USA (CDE, 2006) and Korea (MOE, 2022). These categories include arithmetic (ARI), comparison (COM), correspondence (COR), geometry (GEO), and possibility (POS). Thus, various mathematical reasoning skills can be evaluated on DMath. (2) DMath consists of about 10,000 problems manu- ally created by 43 human workers covering various lexical usage patterns. This diversity in lexical us- age patterns helps evaluate the general performance MWP models. (3) DMath supports bilingual in- put languages, i.e., English and Korean, to evaluate the effect of input languages for MWP models. (4) We offer the annotation of expression trees and Python code as intermediate solutions to evaluate the effect of expression forms in training MWP models. To annotate the expression trees, adopt 50 operators, some newly designed to handle the operators for list structures. (Details are given in Appendix B.)
ARI
ARI
90
90
ChetGPT
84.45
GPT-4
67.26
COM
POS
COM
40
68.26
40 76.35
44.67
49.0
68.67
65.17
78.07
GEO
COR
GEO
COR
Figure 2: Accuracy comparison over various reasoning categories on DMath for ChatGPT (gpt-3. 3. t-turbo; OpenAT) and GPT-4 (OpenAI, 2023) are presented, re- spectively. The few-shot CoT (Wei et al., 2022) prompt- ing is used for inference in these models.
found several interesting observations. (1) Due the multi- diversity, all the MWP models on DMath yield lower performance than other datasets, with ChatGPT and GPT-4 achieving about 60% and 75% accuracy, respectively. (2) The problem types heavily affect the accuracy. As shown in Figure 2, correspondence (COR) and possibility (POS) show relatively low accuracy, i.e., 44-49% and 62-65% for ChatGPT and GPT-4, respectively. We conjec- ture that math examples are biased for training LLMs. (3) For different languages, English- prompting methods perform better than Korean- based ones. As reported in OpenAl (2023), we also confirm that LLMs show more favorable perfor- mance on English corpora. (4) For different ex- pression forms, Python code is more effective than other forms. As also observed in Gao et al. (2022), Python code is more robust for addressing logical expressions using external tools.
To analyze the performance of various MWP models on our dataset, we employ fine-tuning and prompting approaches. For the fine-tuning method, we employ three representative MWP models with pre-trained language models, i.e., RoBERTa (Liu et al., 2019), GPT-2 (Radford et al., 2019), and CodeGPT (Lu et al., 2021). For the prompting method, we adopt representative LLMs, i.e., GPT-3 (Brown et al., 2020), ChaGGT Egpt-3.5-turban OpenAl), and GPT-4 (OpenAI, 2023), with various reasoning prompting meth- ods, i.e., zero-shot (Brown al., 2020), zero-shot CoT (Kojima et al., 2022), few-shot CoT (Wei et al., 2022), and PAL (Gao al., 2022).
Dataset Formulation
The MWP sample comprises natural narrative (input) and an answer (output). The input includes natural sentences, symbols, and math equations.
The output represents a unique numeric or string value. Besides, each sample is associated with its
Through our empirical studies on DMath, we

---

You want to pick 2 fruits out of an apple, peach, pear and place them in different fruit haskets. What is the number of cases?
ARI
COM
COR
GEO
POS
Total
Question (English)
Train
2,476
1,338
1,656
1,417
1,056
7,943
Test
669
334
402
383
291
2,079
+ 2 Mot AEEE m
Total
3,145
1,672
2,058
1,800
1,347
10,022
Question (Korean)
- 1 - 4.
7 8 - F 2 7
Table 3: The number of samples per category on DMath.
Human solution
lenilapple, peuch, pear)) permutation
metic reasoning, comparison (COM) and corre- spondence (COR) for relationship reasoning, ge- ometry (GEO) for geometric reasoning, and possi- bility (POS) for possibility reasoning. We split the relationship reasoning into two groups due to their distinct characteristics. Please refer to Appendix A for examples both categories. Table reports the number of samples for each category.
[LIST_SOU upple peach peur [LIST_FOL]
Equation
[LIST_ BIST_LEN] [PERM]
applee b 'peach' c= 'pear' 2
li= [] (iappend((a) liappend(h) liappend(c)
Code
e= len(li) I=
for i, clem in cnumerato(ruggood): f=f* print(int(())
Answer
Diversity in lexical patterns. We use corpus lex- icon diversity (CLD) introduced by Miao et al. (2020) to quantify lexical usage patterns. Com- pared to existing MWP datasets, the CLD of our dataset is 0.49. Although it is similar to ASDiv, the number of equation templates in dataset is 2,855, much greater than ASDiv. It indicates that our problems cover various problem types with high diversity. For detail, a template represents the process of performing an operation. If the order or the type of operators different, we regard that they are different templates.
Table 2: Example of math word problem with two lan- guages (English and Korean), expression forms (an equation and its corresponding Python code). a human solution, and answer. More examples DMath for each category can be found in Appendix A.
intermediate solution, e.g., natural language, an expression tree, or Python code, to derive the an- swer. Because it is difficult to infer the answer from the narrative directly, the intermediate solution is used as a hint to perform math reasoning and to infer world knowledge from the narrative.
Diversity in language narratives. Existing datasets (Amini et al., 2019; Wang et al., 2017; Cobbe et al.. 2021) are primarily based on monolin- gual language, i.e., English or Chinese. Our dataset provides bilingual languages, i.e., English and Ko- rean. It is beneficial for validating the effect of languages on LLMs.
To construct the MWP dataset, we annotate the sample as triplet of (natural language narrative, intermediate solution, answer). In this process, we focus extending the diversity of problem types, lexical usage patterns, languages, and intermediate solutions. Table 2 shows a sample with two input languages, i.e., English and Korean, intermediate expression forms for the expression tree and Python code, and the final answer.
Diversity in solution forms. To extend the usage, annotate two expression formats, expres- sion tree and Python code, illustrated in Table 2. The math equation has been widely used as an ex- pression tree Cooncel-Kedziorss et al., 2016; Xie and Sun, 2019). Some categories require us to enu- merate and select a specific number/string from list/sequence of numbers/strings. GEO also de- mands extensive knowledge of geometric shapes like triangles, rectangles, and circles.
Diversity in problem types. To cover diverse math concepts, we refer to the math education guide- lines of grade schools in the USA (CDE, 2006) and Korea (MOE, 2022). We identify four impor- tant reasoning skills in mathematics: arithmetic, relationship, geometric, and possibiliiyy. Arithmetic reasoning encompasses four fundamental math op- erations. Relationship reasoning requires the ability discemn relationships between objects in various phenomena. Geometric reasoning entails under- standing the nature and relationship of geometric shapes. Possibility reasoning requires the ability to represent uncertainty numerically.
To annotate our collected problems, we intro- duce 50 operators. (See Appendix B.) Follow- ing some rules for defining these operators. First, create operator names to match tual math symbols as closely as possible. (e.g., [ADD], [FLCOR], [GCD], [LIST_MAX]) Second, when some operators perform the same action, we use similar names. (e.g., (a)
Based on these reasoning tasks, we categorize five problem types, i.e., arithmetic (ART) for arith-
(b)

---

PRODUCTT (c) LIIST_GETDDIIIES (a), (b), and (c) all take LIST as the in- put and return a LIST as the output through some operation (PERM, PRODUCT, DIVISOR), so "LIST_GET_" " overlaps.) Third, we employ con- cise names, (e.g., [COMB], [LCM], [LIST_LEN], (LIST2NUM].) Lastly, if single operator performs multiple operations, we represent the goal for oper- ations. (e.g.,
augmented problems are 0.0 for min, 0.24 for mean, 0.23 for median, and 0.92 for max. Refer the de- tailed histogram in Appendix C.
Annotation. We request the worker to annotate its answer and the intermediate solution as an expres- sion tree using pre-defined operators. This expres- sion tree follows the postfix notation and consists of numbers, strings, and operators, as exemplified in Table 2. (See Appendix A for more examples.) While existing studies Kedziol-K et al., 2015; Wang al., 2017; Cobbe et al., 2021) solely focus on variables for single numbers, we introduce new variables and operators for list structure, which can be used to solve complex and various math problems. We implement an automatic code gener- ator to convert the expression tree into Python code. We also use the Python code to validate whether the human-annotated answer is correct.
SOLVER]. This operator takes an expression, and the string corresponding to the digits in the expression and finds the unknown numeric value (UNK) of the string.) Given a postfix expression tree, we also convert it into Python code automatically using our Python code generator. As the byproduct of our code generator, we can thus utilize Python code as another solution form.
3 Dataset Construction
While other MWP datasets are typically collected from the web and annotated with relevant equa- tions, the DMath dataset exclusively comprises human-generated problems. The linguistic diversity is higher than others regarding natural languages and expression trees. Specifically, we provide a comprehensive overview of collecting, augmenting. annotating, validating, and translating the DMath dataset.
Validation. We recruit 13 graduate students ma- joring in computer science who participate in the previous process (collection, augmentation and an- notation). They are familiar with the annotation for- mat of the MWP dataset and can solve elementary- level mathematical problems well. We perform the following process sequentially: (1) Question- answer validation, (2) Question-expression tree val- idation, (3) Expression tree validation, and (4) Con- ciseness validation of expression tree for multi- expression tree matching problems. If any error was detected during steps (1) to (3), we return to step (1) for re-validation. After that, we perform step (4).
Collection. Instead of curating existing math prob- lems, we ask human workers to create new prob- lems. We recruit 43 undergraduate and graduate students and let them develop new and diverse grade school-level math problems. Each worker contributes to creating about 100-200 samples, re- sulting in a total collection of 4,184 seed samples. These samples were distributed across five math categories, with 960 for arithmetic, 1,092 for com- parison, 943 for correspondence. 401 for geometry, and 788 for possibility.
We go through these procedures to reduce poten- tial risks to the correctness of the annotated data. For step (1) and step (2), we ensure that the ques- tions assigned to human workers do not overlap that they do not receive the same questions in each step. which can act as a cross-check. We employ code check step (3). For step (4), 2-3 human workers are assigned per question. More detailed procedures are as follows.
Augmentation. Human workers manually augment seed samples using three policies: paraphrasing, do- main word changes, and expanding equation trees (using a few more equations from the original sen- tence problem or adding other equations). We care- fully augment seed samples to minimize linguis- tic similarities and superficial differences. Human workers manually boost 1-3 samples from each seed sample using different expressions. We also verify the similarity between seed and augmented samples using BLEU (Papineni et al., 2002) scores and remove some samples with high similarities. The statistics for BLEU score between seeds and
Step (1): Question-answer validation. After the annotation process, we assign natural language questions to human workers and ask them to solve the provided questions. There are two cases where their answers differ from the annotated answers. The first is when the natural language problem is unsolvable, and the second is when the annotated answer incorrect. When a worker encounters these errors while solving a problem, the worker corrects the error manually. In the first case, the

---

worker should correct the natural language ques- tion. In the second case, the worker should correct the annotated answer. Besides, two or more work- ers cross-check the corrected questions to ensure no errors.
lems (# 7,121) from machine-translatee ones. We further inspect 29% problems (# 2,901) to en- sure no semantcclgammmatial errors. Afterward, all questions (# = 10,022) are checked for those errors. This is done by three graduate students who are not involved in the translation and are proficient in En- glish and Korean. The results show that 5.24% (# 525) of the questions had semantic/grammatical errors.
Step (2): Question-expression tree validation. Natural language questions that are solvable and correctly annotated with correct answers are given to human workers along with expression trees. The human workers check whether the given natural language question and the expression tree are cor- rectly paired. not, they modify the expression tree to match the natural language problem.
Experimental Setup
Datasets. We use three datasets to compare with DMath, which are GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and ASDiv (Miao et al., 2020). GSM8K is a commonly used dataset for evaluating math reasoning and has high qual- ity. SVAMP is variation of the previous dataset (MAWPS) (Koncel-Kedzzirski al., 2016). and allows us to evaluate the robustness of the meth- ods. ASDiv is a highly diverse dataset, showing the importance of increasing the lexical diversity of the problem. We did not run experiments on the MathQA (Amini et al., 2019) dataset because we found that it has incorrect answers, as also dis- cussed in existing studies (Austin et al., 2021; Jie et al., 2022; Miao et al., 2020). Also, did not compare DMath with MATH (Hendryeks et al., 2021) since there a discrepancy in domain knowl- edge levels. We aimed to tackle math problems at the elementary school level (grades 1-6), while MATH is designed for the high school level (grades 12). See more details in Appendix D.
Step (3): Expression tree validation. After the natural language question and the expression tree have been properly matched in the previous step, we verify that the expression tree produces the cor- rect answer. The operators used in expression trees can be converted to Python code. After converting each expression tree code, we verify if the out- come produced upon execution corresponds to the correct answer. Otherwise, return to step (1).
Step (4): Conciseness validation of expression tree for multi-expression tree matching prob- lems. There can be various potential solutions to a mathematical problem, and we maintain that se- lecting an expression tree is crucial to the model training. For example, the following problem has multiple expression trees. "There are six people. Everyone shook hands with each other once. Find the total number of handshakes." " We can make the expression tree as one 6* (6-11)/2 or 6C2. Here, both equations lead to a reasonable correct answer. We choose the expression tree with as short an equation length as possible (For this example, we choose 6C2.) We considered this direction of reduc- ing the equation length because the most concise solution process captures the essence of the prob- lem. However, even if formula is not the shortest, we choose it most workers agree.
Baselines. We evaluate the MWP solvers with the fine-tuning method and the prompting method. For the fine-tuning method, we use pre-trained lan- guage models. We use two representative models, ROBERTa (Liu et al., 2019) and GPT-2 (Radford et al., 2019) for the expression tree output and use CodeGPT (Lu et al., 2021) for the Python code out- put. They fine-tuned with DMath train data. For more information how the framework works, refer to Appendix E.
Translation. Once we build the Korean MWP dataset, we translate it into English. First, we use machine translation to translate Korean into En- glish to improve the quality and quantity of trans- lations in limited time using Google Translator. Then, ask nine English-majored students to re- vise any incorrect translations. This revision en- sures that the English and Korean problems share the same meaning. It can also improve the linguis- tie diversity of the English language. For the whole data (## = 10,022), translators change 71 % of prob-
For the prompting method, we use GPT-based models. We use babbage, curie, davinci (Brown et al., 2020), gpt-3.5-2 5-trbo (OpenAT), and gpt-4 (OpenAI, 2023) model using API provided by OpenAI. We denote babbage as GPT-3 [1B], curie as GPT-3 [6.7 davinci as GPT-3 [175B], as ChatGPT [175B], and gpt-4 4 as GPT-4 [Unknown]. Besides, we use four prompting methods: zero-shot (Brown et al., 2020), zero-shot

---

Framework
Params.
Model
Prompt method
ARI
COM
COR
GEO
POS
Total
124M
GPT-2BASE
39,46
44,31
41.46
20.45
38.03
36.92
125M
ROBERTaRASR
41.60
36.92
42,37
26.28
29.90
36.54
Fine-tuning
355M
RoBERTaL.ARGE
45.99
42.12
49.01
32.03
35.05
41.85
774M
GPT-2LARGE
50.67
51.10
57.30
31.15
44.44
47,56
1B
4.63
25.75
4.98
6.53
8.25
8.95
6.7B
GPT-3
Few-shot CoT
6.58
20.66
3.48
4.44
8.93
8.18
175B
18.68
37.13
9.20
10.44
10.31
17.12
Zero-shot
47.68
38.32
23.13
37.08
29.90
36.99
Prompting
175B
ChatGPT
Zero-shot CoT
64.87
69.76
45.77
64.23
36.77
57.91
Few-shot CoT
67.26
68.26
49.00
68.67
44.67
60.99
Zero-shot
49.93
50.00
28.11
46.21
35.74
43.05
Unknown
GPT-4
Zero-shot CoT
85.65
87.43
65.92
81.72
62.54
78.16
Few-shot CoT
84.45
76.35
65.17
78.07
62.54
75.18
Table 4: Accuracy comparison results of MWP models using expression tree solution form, GPT-2 (Radford et al.. 2019), RoBERTa (Liu et al., 2019), GPT-3 (Brown al., 2020), ChatGPT (OpenAT), and GPT-4 (OpenAT, 2023), the English-based DMath dataset. Due to space limitation, omit zero-shot (Brown al., 2020) and zero-shot CoT (Kojima et al.. 2022) results for GPT-3, which follow a similar trend to that of few-shot CoT.
COT (Kojima et al.. 2022), few-shot CoT (Wei et al.. 2022), and PAL (Gao et al., 2022) to compare the effect of different prompting methods.
GPT-3 (babbage, curie, davinci) and Chat- GPT 3 50tturbo--00011 and set 5 for GPT- 4 (gPt-4-0314) following the conventions (Wei et al., 2022; OpenAI, 2023). For GPT-3, set the maximum token length 128. Also, we few- shot examples of GSM8K, SVAMP, and ASDiv Wei al. (2022) and of DMath as Appendix We take them differently for fair comparison.
Evaluation metrics. To evaluate MWP solvers, we use accuracy (Acc.), commonly used in existing studies. The accuracy checks whether the final an- swer is correct without considering intermediate solution forms. To extract the answers from the MWP model, we use the following methods. For the fine-tuning method, we convert the expression tree solution form generated by models to Python code and take the result of the code as the answer. A model that generates Python code as the inter- mediate output also takes the result of the code as the answer. For the prompting method, we adopt the majority voting method. It selects the most fre- quent answer among the possible answers as the final answer. We treat any answer after "answer is" as a possible answer.
Empirical Results
Overall performance results. Table 4 presents the overall accuracies of fine-tuning and prompting MWP models. First, owing the impressive capa- bilities of LLMs, GPT-4 with few-shot CoT shows the highest accuracy, achieving 75.18 75.18%. For GPT- [175B] and ChatGPT with the same number of parameters, there exists a notable performance gap. We conjecture that ChatGPT has undergone more optimization for mathematical reasoning compared to GPT-3. It is also worth noting that all fine-tuning models surpass GPT-3, and even the fine-tuned GPT-2LARGE [774M] exhibits comparable accu- racy to GPT-4 with zero-shot. This suggests that the fine-tuning approach proves effective in optimizing the MWP task.
Implementation details. For the fine-tuning mod- els, we use pre-trained models from the hugging- face hub. We set the batch size to 32 and the maxi- mum token length to 50. We fine-tune models for three learning rates 1e-5, 3e-5, 5e-5) to find the best performance. We also run experiments with three different seed values for each learning rate, then select the best-performing learning rate and re- port the average value over the three seeds. We run our experiments using an NVIDIA TITAN-RTX GPU with 24GB memory.
Second, the worst problem categories differ across MWP models. Both GEO and POS rely world knowledge, impacting the performance of fine-tuning models, i.e., RoBERTai,Anc and GPT-2LARGE. differently. Fine-tuned models excel at POS due to condensed operations but struggle with expression tree- rree-bsed problems in GEO. Con- versely, prompting models, i.e., ChatGPT and GPT.
For prompting-basee methods, we set the tem- perature as 1.0 and top_p as 1.0. We also set the number of few-shot examples as 8 for

---

Expression form
Model
# Params,
Prompt method
ARI
COM
COR
GEO
POS
Total
Expression tree
RoBERTaRAsE
125M
41.60
36.92
42.37
26.28
29.90
36.54
Python code
SMdaG
124M
42.25
35.83
32.75
23.24
33.34
34.63
NL. prompt
Few-shot CoT
67.26
68.26
49.00
68.67
44.67
60.99
ChatGPT
175B
Python code prompt
PAL
74.74
52.99
52.49
61.62
46.39
60.56
NI. prompt
Few-shot CoT
84.45
76.35
65.17
78.07
62.54
75.18
GPT4
Unknown
Python code prompt
PAI.
88.94
83.83
81.72
67.35
79.99
Table 5: Accuracy comparison results of MWP models on the DMath dataset per expression forms in English. NL prompts used by CharGPT (OpenAl) and GPT-4 (OpenAl, 2023) mean natural language prompts. We set few-shot CoT (Wei al., 2022) NL prompts method and PAL (Gao et al., 2022) as Python code prompt. We choose RoBERTaşASя (Liu et al., 2019) as the comparison model for CodeGPT (Lu et al., 2021) because it has the most similar parameter sizes.
4 with few-shot CoT, trained on extensive world knowledge, perform better in GEO but falter with the Number of Cases problems in POS. In COR, the models' difficulty understanding specific problem formats contributes to performance degradation.
prompts appear similar, a closer look reveals that a specific prompt type is preferred depending on the problem. Among the 2,079 questions in the full test set, 15.39% (# = 320) of the questions are correctly solved only on the natural language prompt, 45.60% (# 948) are correctly solved on both the natural language prompt and the Python code prompt, 14.96% 3) are correctly solved on only the Python code prompt, and 24.05% (# 500) were incorrect.
Third, the prompting methods in L.Mss also in- cur distinct results. ChatGPT and GPT-4 show the lowest performance for the zero-shot prompting. They can be significantly improved by using CoT prompting. However, when model sizes are too small, the impact of few-shot CoT prompting is insignificant, implying that model capacity is an important factor in crompting-based models.
We further analyze these results by problem type. For ARI types, 8.97% of problems correct using only natural language prompts and 16.44% cor- rect using only Python code prompts. For COM, the corresponding percentages are 29.34% and 14.07% %, respectively. ARI and COM show the largest per- formance differences among the five categories be- tween natural language prompt and Python code prompt. Python code prompts perform better on ARI types because LLM often needs correction in its calculations. The higher accuracy of natural language prompts in COM types can be attributed to the concise language expression of large and small relationships between two objects, compared to Python code.
Lastly, we summarize our observations on five problem types. ARI and COM can be regarded as common math types frequently encountered in nat- ural language corpora. These properties explain the relatively higher accuracies of ARI and COM than other types both fine-tuning and prompt- ing methods. Mcanwhile, COR less frequent problem type in ordinary language corpora, but an appropriate substitution can solve it. This unique characteristic of COR lends itself to the clear ad- vantage of fine-tuning approach and COT. GEO of- ten requires world knowledge, thus, the prompting method with LLMs is more advantageous than the fine-tuning methods. Finally, POS is represented by relatively longer equation trees, indicating a chal- lenging problem type in general. The operations and mathematical expressions for POS are also domain-speeific, limiting the generalization power of LLMs and hindering their performance in this problem type.
Lastly, the fine-tuning and prompting methods show similar tendencies in model performance. This tendency indicates that various forms of ex- pression have their preferred categories. To clab- orate, when we state that form of expression A outperforms form B, it does not imply that every type within A uniformly surpasses every type within B. Rather, it suggests that certain types within A outperform B, while some types in B excel over A. Within the fine-tuning frame- work, compare the ROBERTaBASE with the CodeGPT SMMAAA ALL- The RoBERTa model uses an expression tree as the solution form, whereas the CodeGPT SM ALJ. model uses Python code. Here,
Expression form results. Table presents the result of different expression formats. Firstly, ChatGPT achieves 60.99% with natural language prompts and 60.56% with Python code prompts. While the performance of the models using natu- ral language prompts and those using Python code

---

Prompt method
Few-shot CoT Prompt
Acc. (%)
Dataset
Model
Zero-shot CoT
Few-shot CoT
Few-shot examples of (Wei et al.. 2022)
57.96
ChatGPT
78.85
80.82
Uniform category random
58.97
GSM8K
GPT4
93.25
91.81
Reverse uniform category random
58,68
ChatGPT
77.70
82.50
Random
59.31
SVAMP
GPT4
88.10
91.60
One category random
60.99
ChatGPT
79.71
83,80
ASDiv
Table 7: Accuracy for several few-shot CoT (Wei et al.,
GPT4
88.55
90.03
2022) prompting methods on the English-based DMath dataset.
ChatGFT 57.91
60.99
DMath
GPT4
78.16
75.18
where large language models falter, setting a path- way for future enhancements.
Table 6: Accuracy comparison between DMath and GSM8K (Cobbe et al.. 2021), SVAMP (Patel et al.. 2021), ASDiv (Miao et al., 2020) for different prompt methods in English.
Few-shot selection. The few-shot prompt is given DMath and other datasets (GSM8K, SVAMP, and ASDiv) differently. Other datasets used the exam- ple of Wei et al. (2022) as in previous studies. Unlike them, DMath is a mix of different mathe- matical categories. We consider evaluating DMath with the same prompt as other arithmetic math- ematical reasoning datasets to be unfair because there are several problem types in DMath. If we give few-shot examples of only arithmetic type, then the model will not do well except arithmetic. Therefore, various prompt selection methods were taken, and the one category random method was used as the final few-shot prompt examples in this paper. (See more details in Appendix F.)
the length of the output sequence emerges as the most significant impact. In detail, for ARI and POS types, CodeGPT SMMAL ALL using Python code per- forms better, and for COM, COR, and GEO types, RoBERTaBASæ model using expression tree per- forms better. The average token length and std. for COR types with expression tree are (6.61, 4. 4.13), which is lower than the POS type of (10.81, 4.51).
That of POS type with Python code is (129.54, 57.55), which is lower than COR type of (147.61,
prompt method,
205.04). In common, GEO types seem to have lower token lengths but lower performance due the lack of domain knowledge of pre-trained models.
We validate the best-performing prompt method by comparing five few-shot CoT prompts. Table 7 shows the performance for each prompt method. First, we use the few-shot example of Wei et al. (2022) as baseline. The other few-shot CoT prompt selects examples from DMath train data. The uniform category random selects examples ac- cording to the proportion of problems in each cat- egory; it selects 2 examples from ARI, COR, and GEO and I example from COM and POS each. The reverse uniform category random selects problem from ARI, COR, GEO and 2 problems from COM, POS each. The Random selects examples randomly from the whole dataset. Finally, the one category random selects few-shot examples based on the type of question, ie.,, the model gets an ARI type question, it randomly selects few-shot examples of ARI type, and if it gets POS type question, it selects few-shot examples of POS type.
Comparison on different datasets. Table 6 shows the results of two different prompting methods on four datasets, including DMath. Except for DMath, the GSM8K, SVAMP, and ASDiv datasets, all achieve the accuracy higher than 90% on GPT-4 and surpass 80% on ChatGPT in few-shot prompts. (Figure 1 showcases the few-shot prompt results per dataset.) The zero-shot CoT results show sim- ilar trend. This trend can be attributed to the fact that these datasets primarily handle arithmetic rea- soning problems.
Instead of intentionally inflating the difficulty to undermine model performance, we have noted sig- nificant performance differences between models on our dataset. Specifically, DMath exhibits aver- age deviations of 13.07% and 15.35% in zero-shot CoT and few-shot CoT prompting methods, respec- tively, the GPT-4 model. This suggests that our dataset contains problems that can be incrementally solved as model capacity improves. The problems are challenging yet reasonable and tailored to grade school levels. brief, our dataset pinpoints areas
Language-speeiffe performance of LLMs. Ta- ble 8 provides the results of different input lan- guages, English and Korean. By using ChatGPT and GPT-4, we observe that the English-based prompt method performs better than its Korean-

---

Language
2021; Hendrycks et al., 2021) targeted a broader diversity, but more enhancement is needed. Our dataset addresses this by providing diversity problem types, lexical usage patterns, languages, and expression forms.
Model
Prompt method
EN
KO
ChatGPT
60.99
46.03
Few-shot CoT
GPT-4
75.18
65.37
Table Accuracy comparison results MWP models, ChatGPT (OpenAI) and GPT-4 (OpenAT, 2023), on DMath. Few-shot CoT (Wei et al., 2022) is used as the prompting method.
Math word problem solving models. Early MWP models used symbolic approaches (Kintsch and Greeno, 1985; Fletcher, 1985), but their domain specificity limited their use. Later, template-based and statistical methods (Kushman al., 2014; Hos- seini et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018; 8; proved performance but faced challenges with di- verse datasets (Huang et al., 2016). Deep neural networks became popular, with RNN-based mod- els (Wang et al., 2017, 2019), tree generation mod- els (Xie and Sun, 2019; Zhang et al., 2020; Wu et al., 2021), and relation extraction models (Jie et al., 2022). Recent models leverage pre-trained language models like BERT (Devlin et al., 2019) to enhance performance (Tan et al., 2021; Li et al.. 2021; Kim et al., 2020). Large language models have also shown significant performance improve- ments with techniques like step-by-step reason- ing (Wei et al., 2022; Nye et al., 2021), calculation verifiers (Cobbe et al., 2021), and various prompt- ing methods (Kojima et al., 2022; Wei et al., 2022; Gao al., 2022). Recent studies (Huang al., 2022; Wang al., 2022; Madaan et al., 2023) indi- cate that large language models can improve reason- ing capabilities autonomously without supervised data, surpassing current techniques in accuracy.
based counterparts despite both English and Korean problems being identical in content but expressed in different languages. Ideally, if the language un- derstanding is independent of the mathematical reasoning, it should have produced the same ac- curacy. However, accuracy has shown more than a 13% and 24% difference between the two on GPT-4 model and ChatGPT, respectively, We con- jecture that the data imbalance issue of LLMs leads to performance bias, as pointed out in several pa- pers (Blodgett et al., 2020; Lai et al., 2023).
et al., 2015) im-
Qualitative analysis of GPT-4 failure cases. We conduct qualitative analysis on failure cases of GPT-4 with few-shot COT. (See several examples in Appendix G.) For ART, COM, and GEO exam- ples, as the length of the reasoning path increases, GPT-4 generates incorrect answers more frequently. For COR, the problems about the number of digits in number mostly incur incorrect answers. For GEO, GPT-4 failed elicit certain domain knowl- edge. e.g., the rectangle that can be made to have the largest area given perimeter is a square. For POS, we observe GPT-4 found difficulties in solv- ing problems related to the Number of Cases.
7 Conclusion
6 Related Work
This paper proposed DMath, a collection of 10K high-quality grade school-level math word prob- lems. It mainly pursued multi-aspect diversity for problem types, lexical usage patterns, languages, and expression forms. We evaluated various MWP models DMath and other datasets. We observed that the logical reasoning capability of the MWP model was mainly focused on arithmetic reasoning. Because of various features, our dataset is more beneficial for inspecting the diverse reasoning abil- ities of models. Our dataset can also help evaluate MWP models for diverse aspects.
Math word problem solving datasets. Auto- matic MWP-solving has been explored since the 1960s (Bobrow, 1964), with work primarily fo- cused on corpus collection and model develop- ment. Initially, small datasets were used (Kushman et al., 2014; Upadhyay and Chang, 2015; Koncel- Kedziorski et al., 2015; Hosseini et al., 2014; Roy et al., 2015; Roy and Roth, 2017; Seo et al., 2014; Shi et al., 2015). The MAWPS dataset (Koncel- Kedziorski et al., 2016) unified these small-scale datasets and incorporated various problem types. Following studies like (Huang et al., 2016; Ling al., 2017; Amini al., 2019; Zhao et al., 2020) enriched this further, but they were limited by their reliance on multiple choice and a lack diverse lexical usage patterns. Recent works (Cobbe et al.,
Limitations
There are several limitations to this work. (1) Trans- lating and annotating math problems require high human costs. Although we carefully and manually

---

check all the samples and their annotations, our dataset might include few samples with incorrect answers. (2) Although it is interesting to observe various model scales, evaluating pre-trained lan- guage models with different model sizes requires GPU resources and training time. Therefore, we employ three well-known pre-trained models for evaluation.
soning in a manner that aligns with the ACL's Code of Ethics. We encourage ongoing scrutiny and dis- cussion around the dataset's ethical implications within the research community. We are commit- ted to balancing research innovation with ethical considerations and stand ready to address any con- cerns raised by the ethics committee or users of the dataset.
Ethics Statement
Acknowledgments
Following the ACL Code of Ethics, we emphasize the ethical facets of data collection, use, and the potential ramifications of our research within artifi- cial intelligence. This study presents a new math- ematical reasoning dataset, and we have carefully examined and addressed all ethical considerations tied to its creation and usage.
This work was supported by Institute of In- formation communications Technology Plan- ning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00421, No. N020022-0- No. 2022-0-00680, and RS- 2023-00219919).
The construction of this mathematical reason- ing dataset involved authors and human workers in the data collection, creation, and augmentation phases. All contributors have strictly followed eth- ical guidelines, ensuring a respectful and fair en- vironment. We have prioritized maintaining the privacy and anonymity of all parties involved.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Sonnel-Kedziorskee Yejin Choi, and Hannaneh Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation- malisms. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Lanuagee Technolo- gies, (NAACL-HLT", pages 2357-2367.
for-
We have gone to great lengths to confirm that the data gathered does not pose any risk of harm to grade school students, the primary demographic of our study. The content has been conscientiously cu- rated to exclude harmful or inappropriate material, fostering safe learning environment.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. ArXiv, fas/2108.00733.
The dataset's accuracy is of utmost importance, and every attempt has been made to ensure the validity of the labels. Human reviewers have manu- ally checked all answer labels in the dataset. How- ever, despite our rigorous efforts, we acknowledge that some errors may still exist due to the inherent limitations of manual review and the complexity of the task. We encourage users of the dataset to remain mindful of this limitation and to report any errors they encounter for continuous improvement of the dataset.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 5454-544 5476, Online. Association for Computational Linguistics.
Daniel G. Bobrow. 1964. question- answering system for high school algebra word problems. In Proceed- ings of the 1964 fall joint computer conference, part (AFIPS), pages 591-614.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot leamners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
We also recognize the potential misuse of the dataset, and we ardently advocate for its ethical and responsible usage within the AT community. This dataset aims to advance the understanding of mathematical reasoning among grade school stu- dents, and any use that could compromise fairness propagate biases in AI systems is strongly dis- couraged.
In conclusion, we strive to ensure our dataset contributes to the field of AI and mathematical rea-

---

CDE. 2006. Mathematics Framework for California Public Schools: Kindergarten Through Grade Twelve. California Department of Education.
Bugeun Kim, Kyung Sco Ki, Donggeon Lee, and Gah- gene Gweon. 2020. Point to the Expression: Solv- ing Algebraic Word Problems using the Expression- Pointer Transformer Model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3768-3779 Online. Association for Computational Linguistics.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/21 14.16
Walter Kintsch and James G. Greeno. 1985. Under- standing and solving word arithmetic problems. Psy- chological review, 92 1:109-29.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesola. Association for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa, 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Rik Koncel-Kedzionsk, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Eizioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Transactions of the Association for Computa- tional Linguistics, 3:585-597.
Charles R. Fletcher. 1985. Understanding and solving arithmetic word problems: A computer simulation. Behavior Research Methods, Instruments, & Comput- ers, 17:565-577 571.
Rik Koncel-Kedziorsk, Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1 1152-1157, San Diego, California. Association for Computational Linguistics.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. ArXiv, 0tbs/2211.10433
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the MATH dataset. Proceedings of the Neural Information Processing Systems Track on Dutasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 271-281, Baltimore, Maryland. Association for Com- putational Linguistics.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categoriza- tion. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond en- glish: Towards comprehensive evaluation of large language models multilingual learning. ArXiv. abs/2304.05613.
pages 523 533..
Danqing Huang, Shuming Shi, Chin- Yew Lin, Jian Yin, and Wei-Ying Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset con- struction and evaluation. In Proceedings of the An- nual Meeting of the Association for Computational Linguistics (ACI).
Zhongli Li, Wenxuan Zhang, Chao Yan, Qingyu Zhou, Chao Li, Hongzhi Liu, and Yunbo Cao. 2021. Seek- ing patterns, not just memorizing procedures: Con- trastive learning for solving math word problems. ArXiv, abs/21 10.08464.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.116110
Chao-Chun Liang, Shih- Tsai, Ting- Yun Chang, Yi-Chung Lin, and Keh- Yih Su. 2016. A meaning- based English math word problem solver with under- standing, reasoning and explanation. In Proceedings of COLING 2016, the 26th International Conference Computational Linguisties: System Demonstra- tions, pages 151-155, Osaka, Japan. The COLING 2016 Organizing Committee.
Zhanming Jic, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 5944-595 5955, Dublin, Ireland. Association for Compu- tational Linguistics.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word

---

problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167, Vancouver, Canada, Association for Computational Linguisties.
Pennsylvania, USA. Association for Computational Linguistics.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve sim- ple math word problems? Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized hert pretraining ap- proach. ArXiv, abs/1197. 11692.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco. Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang. Ge Li, Li- dong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sun- daresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning bench- mark dataset for code understanding and generation. ArXiv, abs/2102.00664.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Subhro Roy and Dan Roth. 2017. Unit dependency graph and its application to arithmetic word problem solving. In Proceedings of the AAAI Conference Artificial Intelligence, pages 3082-3088.
Subhro Roy and Dan Roth. 2018. Mapping to declara- tive knowledge for word problem solving. Transac- tions of the Association for Computational Linguis- tics, 6:159-117 172.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri. Shrimai Prabhumoye, Yiming Yang. al. 2023. Self-refine: Iterative refinement with self-feedhack, arXiv preprint arXiv:2303. 17651.
Subhro Roy. Tim Vieira, and Dan Roth. 2015. Reason- ing about quantities in natural language. Transac- rions of the Association for Computational Linguis- tics, 3:1-13.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yihh Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online. Association for Computational Linguistics.
Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. 2014. Diagram understanding in ge- ometry questions. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 2831-28 1831-2838.
Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 2144-2153, Berlin, Germany. Association for Computational Linguistics.
Shuming Shi, Yuehui Wang, Chin- Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving numher word problems by semantic parsing and rea- soning. In Proceedings of the Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 1132-1142.
MOE. 2022. -65 2. E प 4 II A (public notice on elementary and middle school curriculum overview and specifics). Minstry of Edu- cation.
Minghuan Tan, Lei Wang, Lingxiao Jiang, and Jing Jiang. 2021. Investigating math word problems us- ing pretrained multilingual language models. ArXiv, abs/2 089288.
Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for interme- diate computation with language models. ArXiv, abs/2112.00114,
Shyam Upadhyay and Ming-Wos Chang. 2015. Draw: A challenging and diverse algebra word problem set.
Lei Wang. Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template- math word problem solvers with re- cursive neural networks. In AAAI Conference on Artificial Intelligence.
OPenAT. Model index for researchers. https:/platfoomm. openai.com/ wwwp//o////oooooeloooooooooccece index-for for-researchers Ac- cessed: June 23, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177
OpenAA. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 31 1-188, Philadelphia,
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pages 845-854,

---

Copenhagen, Denmark, Association for Computa- tional Linguistics.
Table of Contents
A. More examples of DMath
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.1 1903.
B. Operators used in expression tree solutions
C. Statistics for BLEU score
D. Characteristics of other benchmark datasets E. MWP solving framework for fine-tuning meth-
Qinzhuo Wu, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2021. Math word problem solving with ex- plicit numerical values. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ithh International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 5859-5869,, Online. Association for Computational Linguistics.
ods
Prompt examples G. Failed examples in GPT-4
Zhipeng Xie and Shichao Sun. 2019. A goal-driven uee-structured neural model for math word problems. In Proceedings of the Twenty-Eit Eighth International Joint Conference Artificial Intelligence, 19. pages 5299-5305, 5305. International Joint Conferences on Artificial Intelligence Organization.
Jipeng Zhang, Lei Wang. Roy Ka- Ka-Wei Lee. Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to- tree learning for solving math word problems. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3928- 3937, Online. Association for Computational Lin- guistics.
Wei Zhao, Mingyue Shang. Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape2 A large-scale and remplate-rich dataset math word problems. CoRR, Abs/2009.11506

---

A
More examples of DMath
CATEGORY: ARITHMETIC CALCULATION
Question in English
There are three numbers 10, 11 and 12. What is the remainder when the smallest number divided by the second smallest number?
Question in Korean
37 + 10, 11, 127 21. 5 7P7 2/2 4 2 42 42 4d/ 6 4
Equation
[LIST_SOL] 10 11 12 [LIST_FOL] [LIST_MIN] 2 [IST_MIN] [MOD]
Code
var_a = 10hn var_b= 11
var_c= 12hn
list_a= []\n
if "/" in str(var_c))il
cval(str(var_cc))n
list aag_appppdddaar_//
if "/" in b) var eval(str(var_b)))
list aaaaappppendvvr_bb b)
if in str(var_a)\nn var a e aist_a.apend(var__)
list list_a.reverne var_d= = Ann list_b=list_acoopYY list_b.sort(Inn
var c = list b[var_d-1Jjn var var_f= 2\n
list lost__=list_aee __a_lss_t..ooyOO a. list_c.sortt(In
var_g list [ilst_c[var_f_]]]] f-I]|n var h var_e % var_gln
print(int(var_h))
Answer
10

---

CATEGORY: COMPARISON
Question in English
A is 49 plus 33, B is 9 multiplicd by 9. Which of the two numbers A and B is greater?
Question in Korean
AE 4] 3 o) B4 9 말ब 4014.
Equation
CLISTSSOL A B [LIST [OL] [LIST_INDEX] [LIST_POP] [LIST_GET]
49 33 [ADD] 9 [MUL] [LIST EOL] 1 [ MAX]
Code
= An'nn var b = 'B'In list a= []
if "/" in str(var_ b)An var b eba(stt((aa( b))nn
list list_a.appendyva__b))
if in ar(var_a))n
var a a err_steval((s((vaa_aa
list_ list_a.append(va__a)
list_ an
c 49hn var d = 33'n
var var_o + var d'i
f=r_f= 9nn = 9In var har_h= = var f var var_gh gin
list b= []ln
if "/" in str(var_h)tt) var h = eval(str(var_hh)) h))In
bist_bappeedd(ar__b) b)In
if "/" in str(var_e)yln var eval(str(var_e)))n
list baspaapppeddvar_ee
list_ buvverreenr var = 1\n list_c=list_bcoopyll list_c.sort()An
var_j list_cf-var iJIn var_k list wew._iiiddxxvvarr---e. 1)
var = list af_aar_a__- k-1]
print(var_])
Answer
A

---

CATEGORY: CORRESPONDENCE
Question in English
If the six-digit number 341ABC is divisible by 7, 9, and 13, how much is the three-digit number ABC?
Question in Korean
L dHl 4 341ABCo] 7,9, 13 47 yo "I, ^ll (|22 4 ABC2 d 75/o12].
Equation
341ABC DEEN_POSSIBLE LIST] 7 [LIST_DIVISIBLE] 9 [LIST DIVISIBLE 13 [LIST_ DISVSIIII
341ABC FIND_UNK] 100 [MUL] 341ABC [LIST CSST_FIND_UNDT [MUL] 341ABC C
[ADD] [ADD]
Code
131AAB"'n ans ans_dict= = diet()nn var a= str(var a) list Π]
variable_camie set(l'A,, 'B', 'C", 'D', E,' 'F, G 'H', T, 'J, 'K LL, 'M', 'N', "P' 'R' 'S' 'T', 'U, 'V', 'W", 'X', 'Y', 'Z'I)nn
for v set(var_a) v in variable_candiie ans [ns_dict[v]] | Oln
candi =
repeat=len(ans__ict)))
for in candi:An str(clii))
temp var _an_ann for i, (k, in en temp temp.replace(k,
if len(var_a) len(str(in((eemp)))nn new_elem int(temp)\n lista..ppend(new_clat _elmmmn
var_ 7n list_b = []]n var_b= = int(var_b)li for in list_ int(i) if % var 0:An
list b.ppped((d var_ 9ln list_c= []
var_c = int(var_c)\n for i biln i= int(i)ln if % var_c=== 0:An list_caapend
var_d= = 13 list_d= []in
var_d= int(var_d))) d)in for in list_c:An i= int(i)n
i % var d = 0:An list_d.ae d.append(i)nn var '341ABC" var f= 'A'ln var_e str(var_e))
f=
unk idx var aar_einddex(aar_f)n var_g Oln for elem in list dAA
elem str(elem)nn var g = int(elemfunk_idk// var_h 100kn var_i var var_hin
var j '341ABC'ln
var k 'B'nn var_ = str(var var_k= str(var
k)ln k)nn
unk idx var jii.de kiln var | Oln for elem in list_d:An elem str(elem)\n
var | int(elem(unk_idyy
var_m 10hn var_n var
var_m'n
var_o '34"
var_p= 'C'An var str(var_o)ln var_ str(var p)nn
mnk_ids var var_ooindex(var_p)
var_q 0nn for elem in list_d:An elem str(elem)\n
var int(elem[unk_idxjjun
var_r= var_n + var_qin
var = var var van
print(int(var_s))
Answer
523

---

CATEGORY: GEOMETRY
Question in English
We are trying to stick tape in straight line. The three tapes are 36 centimeters (cm), 42 centimeters (cm), and 48 centimeters (cm), respectively, and if the same part is overlapped and attached, the total length is 97 centimeters (cm). Find the length of the overlapping part.
Question in Korean
d|O Qд उे E $ 2HH 37 ||]k 24 (6) 4 (cm), (c))o _ # F Iouu & Z 97/1E////((m)0044 전 #
o = d 2
Equation
36 42 [ADD] 48 [ADD] [SUB] [SUB] [DIV]
Code
var a = 36'n
var_c = var a + var var_bln bln var_d = 48'n var b = 421n
var_e = var_c + var_dln
var f = 97nn var = var_e var_fnn
var h 3'n var i = 1ln var var h var i'm var k var g var var_j'n j'n
print(':.28 forrmat(roundyaar k
Answer
14.50

---

CATEGORY: POSSIBILITY
Question in English
You want to form four-digit number using the four numbers 7, 2, 5, and 9 only once. How many four-digit numbers can you make? Question in Korean
4 7, 인 t of 4| 지ュय 4 usa 944. +
EF _ें 7
Equation
[LIST SOL] 7259
4 [LIST_ GET PERM] [LIST LEN]
Code
print(int(var_af)
7n var == 2
var_c= 5
var == 9ln
list_a= [7nn
if "/" in (ifvvar_d):n
var d eval(str(var_dul d))(n
list_ aisp_a..ppen d)An
if "/" in str(var_c)\nn var eval(strvvar_c))
list_ list_a.append(var_ey)
"/" in str(var_b).nn
var b = eval(str(var_b))
list jis_aaappenddvar_bb
if "/" in atrvvara)::u var a eval(st((var_a))n
list_ iis_a.append(var_a/
list aist_a.reverse(Inn var_ 4An list b- = [str(i) for in list_a]Kn
list b Iistiiertook.pmmmttaionnnniiiii var en
list b [" ["-join(num_]il _istttuuumssssaaue for num list in b/ln
list [str_numm for str str_stunn num in list if str num[O] != Jmn list b [float(i) for in list bjin
var f= len(list_b)ln
print(int(var__f)
Answer
24

---

B
Operators used in expression tree solutions
R: real number, N: Natural number, L: List, S: String, V: Variable, E Equation
Operation
Explanation
Examples
[ADD]
Get R., R2
Retur R1 + R2
1 [ADDD +
[SUB]
Get RI, R2 + Return R₁ R2
2 [SUB] ->
[DIV]
Get Ri, R2 + Return Rí / R2
62 [DIV] + 3
[MUL]
Get R1, R2
Return R. R2
7 [MUI]] - 35
[FDIV]
Get R1, R2
Return R₂ // R2
[3
[OOD
Get R1. R₂
Return R1 % R2
53 [MOD] -
[POW]
Get R1, R2 Return R₂ A R2
5 [POW] + 25
Get R. Retur the value that round
[CEIL]
1.13 [CEIL] + 1.2
up R. from Nth decimal place
Get R. / Return the value that round down to the Wit decimal place
[FLOOR]
34.72 [FLOOR] - 34.7
R, / Return the value that round to the M decimal place
[ROUND]
22.679 [ROUND] 22.68
[ABS]
Get R = Retur Absolute R
-13 [ARS]] + 13
Get N2 > Return N combination
[COMB]
3 2 [COMB]
N₂
Get N2 + Return Ni permutation N2
[PERM]
3 [PERM]
i. + Return Greatest Common
[GCD]
6 [GCD]
Divisor of N2
Get N1. N2 = Return Least Common
[LCM]
2 [LCM] +
Multiple of N1.
[LIST SOL) [LIST_BOL]
Declare start and end of LIST, Used in pairs
[LIST SOL] 2 [LIST EOL] + [12]
W N2 and Step N' Return arith- metic progression LIST
[LIST_ ARANGE)
041 [LIST] ARANGE) -> [(01234
Get N1, N₁ = Return odd number list
[LIST_ODD]
0 [ - 1131
between LIST N2 and
Ge numbers Return
[LIST EVEN]
0 Q - [024]
ber LIST between NV1 and N/
[LIST_POP
Pops the topmost LIST from the stack
Get with N's, .V1 Return possi- ble N digit natural numbers by using N's once
[ [LISTEELLL
CUST_GET_PERRS
[LIST_GET_PERR] - 23 32
C with A's W1 + Return all possible N' digit natural numbers allowing dupli- cate N's
[LIST_SOL]] [LIST_EOL] [LIST GET PRODUCT] [22 22
[LIST_ GET PRODUCT)
Get S with VL. V1 + Return all possible numbers by substituting 1-9 to V,
3A
+ 30,
(GEN_POSSIBLE_LISS LISTI
39]
Get C. => Sort C in descending order
[LISTSSOL] 11 10 12 [LIST [OL]
[LIST_MAX]
+ Returm W/th value in sorted &
[LIS_MAX] + 11
Get C. N + Sort C in ascending order =
[LITT [OLT 11 10 12 [OL]
[LIST MIN]
Return A th value in sorted C
LLST_MIN -> 10
[LIST_SOL] 11 10 12 [LISTT__
[LIST_SUM]
Get L + Return
the
[LIST_SUM] + 33
[LIST_SOL] 11 10 12 [LIST_EOL,
[LIST LEN]
C > Return length of the
[LIST_LEN] + 3
Get L, Wi Return With value of L (N1 starts from 1)
[LIST SOL] 11 10 12 [LIST_FOL]
[LIST_GET]
[LSST_GET] - 12
Get L. A,, + Find Wi in Li and Return
[LIST SOST_SSL]]1 11 10 12 [LIST_FOL] 11 [LIST_INDES] +
[LIST_INDER)
index of Nr from &. (index starts from
Get C, Retum total number of VV
[LIST_SOLA 2 [LIST_BOL, [LIST_ FUND__NNDUUUD -
[LIST SOST_FIND_NMM
L

---

Get N, Return LIST with L's ele-
[LIST_SOL] 10 12 [[IITTT_OO] 11
[LIST_MORE
bigger than
MORE] 12
Get L. N' Return LIST with L's ele-
[LIST SOLJ 11 12 [LIST BOL] 11
[LIST LESSS]
smaller Wh
[LIST LESS] 10
Get L. 1 Return LIST with L's elc-
[LIST SOL] 11 12 [LIST BOL] 11
[[IST_MORE_ MORE_EQUALL
or bigger than NN
[LIST_MORF [.AS_MMOR_FQUAII] (11
Get ments same or smaller than
Return LIST with L's ele-
[LIST_SOL] 11 12 [LISTEEOL, II
ILIST_ WIST_LESSEEOT [EQUL]
[LIST_LESS [QUAL] [11 10)
[LIST SOL] [LIST EOL]
Get Li, L2 Return LIST which union of L1, L2
[SET UNNINJ
[LIST [LIST_SOL] SOL. 2 [LIST EOL]
UNION] - [1 [1234]
LUST_SOLI]1 2.3 [LIST EOL]
[SET DIF_EEIFEEENC
Get L1, L1 = Retum L1 differnt from La
[LISTSSOL] 3 ILIST_EOL,
[SET_
-
[LIST SOL] 2 [LIST EOL]
Get L1, L2 Return LIST which is inter-
[SET INTERSTTR
[LIST SOL] 2 [LIST EOL] [SET INTERSECT] - [23]
sect L1.
Get C. N = Return L's clement which is divisible by NN1 in LIST
4 10 12 [LIST EOLJ4 [IST_DIVISIBLEE] - 1 12
[LIST_ DIVISIBLE]
Get L, Ny, N2 > Return LIST of L's element if the remainder of dividing L's element by N N2.
LUST_DIVIDE
[LIST_SOL]]
[LIST [OL] [LIST DIVIDE AND REMAIN] - [47]
REMAIN]
[LIST_GET DIVISOR]
Get.V > Return LIST of divisors of N
[LIST GET DIVISOR] - (110 [1236]
[LIST_ (A)T_SOL]AA)O) (B) (C) (D) [LIST_FOL)
Get L1. La = Return LIST which Li sorted descending order according to the condition of L2
[LIST_SOL] (B) (D) (B) (A) (C) (B)
_MIN]
(LIST_EOL [LIST]
(C) (B) (D) (A)
Get L = Return N with each number of digits in order of L's element
LIST SOL] 23 [LIST EOL] LIST2NUM 123
[LIST2NUM]
Get Return LIST of the digits of each digit of
[NUM2LIST]
123 [NUM2LIST] - [123]
Get = Return LIST of each L's ele-
ILIST_SOP SOL] 10 [LIST_ [OL] [IIST_NU22SUM 1115 14]
(LISTNNUM2SUM
ment's sum digits
L. IV. Aa = Return LIST, which has only elements of with Return Nith digit are Return 2/.
[LIST SOL) 112 14 32 [LIST EOL] [LIST SEARCH FIXED DIGIT] [112 14]
[LIST SEARCH FIXED DIGIT]
E, V = Return .V, which the value V. satisties ues that can fit in V, return LIST. V is unit digit.
there are multiple val-
A2+4B=69 A
IDIGITJUNK_SOLVER
Get L, S. V > Return N. which is the value of V. If there are multiple values that can fit in V. retum LIST. It should be used a pair with [GEN_]GOOO POSSIBLF_LIST
3A [GEN POSSIBLE LIST] 3A A [LIST FIND_ LIND_INNNN-A (O.....)) 9]
[LIST FIND FIND_UNK]
Get E, S + Return N. which is the value of S, satisfics E. If there are multiple values that can fit in S, return LIST. S is not unit digit but a string.
3/=6/A=B/21 [NUM_UNK_SOLVERI- 14
WUMLUNKKSOLVVE

---

1001
1000
600
427
400
163
90
0.0
0.2
0.6
0.8
BLBU score bellwarr seed sentence and augmented santlannces
Figure 3: The histogram of BLEU between seed sentences and augmented sentences.
Statistics for BLEU score
The histogram for BLEU score between seed sentences and augmented sentences is in Figure 3. The statistics for BLEU score between seeds and augmented problems are here: 0.0 for min, 0.24 for mean, 0.23 for median, and 0.92 for max.
D Characteristics of other benchmkark datasets
A model can only be considered capable of reasoning if capable of elementary school-level logical reasoning. The mathematical reasoning abilities are diverse, as shown Section 2; however, the current MWP task focused on arithmetical reasoning using four fundamental arithmetic operations (Koncel- Kedziorski et al., 2015; Roy and Roth, 2017; Miao et al., 2020; Cobbe et al., 2021).
Data such as MATH (Hendrycks et al., 2021) or MathQA (Amini et al., 2019) have other reasoning problems. But MATH is too domain-specifie. As mentioned in Liang al. (2016), we think the MWP task is good problem in evaluating the ability to understand and reason because natural language narratives of grade school-level math problems have less combined syntax and contain small amount of domain knowledge. Our objective was to develop an elementary school-level (grades 1-6) math dataset, whereas MATH caters to high-school-level (grades 8- 1-) mathematics. It encompasses intricate mathematical problems from AMC110, AMC12, and AIME that necessitate comprchension advanced mathematical concepts such as trigonometric functions or imaginary numbers. To ensure a fair comparison, we compared it with grade school-level datasets such as GSM8K, SVAMP, and ASDiv.
Also, MathQA has too low lexical diversity (Miao et al., 2020). It has been modified several times by Austin et al. (2021) and Jie et al. (2022) but still not enough. We inspect 20 problems that GPT- 4 fails on in modified MathQA dataset (Austin al., 2021) and find that 30% of problems have incorrect annotation, 10% of problems have vague problems, and 5% of problems are unsolvable. Miao et al. (2020) got similar results, saying that 67% problems have incorrect formulas, 23% of problems have problematic descriptions, and 10% of problems have valueless answers, randomly inspecting 30 inconsistent samples. So, we choose only GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and ASDiv (Miao et al., 2020) only.
MWP solving framework for fine-tuning methods E
We adopt general framework to evaluate MWP models. It consists of three modules: template-based conversion, MWP model, and equation-to-code conversion.
First, we convert some tokens in the narrative to pre-defined symbol tokens. Using a rule-based template method, we only transform pre-defined token values into specific symbols. For instance, the numerical tokens in the narrative are converted to [N1]. [Ng].... [Nz]. Besides, some pre-defined tokens are converted to the elements in the list.

---

Second, the MWP models are trained translate narrative with abstract symbols into a postfix equation. Although many existing studies have been proposed, (Xic and Sun, 2019; Zhang al., 2020; Jie et al., 2022) some models are limited to handling various operators. In other words, these models are designed to solve the binary operator data like +. ; x DMath has many operators that are not binary. So, in this paper, we employ MWP models using pre-trained language models, i.e., RoBERTa (Liu et al., 2019) and GPT-2 (Radford et ul., 2019).
Lastly, we convert the postfix expression to Python code. Because the operators are pre-defined, we generate the executable Python code from the postlix expression. When the postfix expression is imprecise, it may fail convert the postfix expression the Python code. We simply return NULL value as the answer for the exceptional case. When we execute the Python code, we utilize the values for the token mapping as the argument of the Python code.
F Prompt examples
In this section, we describe prompting methods used in DMath. The zero-shot (Brown et al., 2020) prompting is written to give a question and generate answer immediately, and the zero-shot CoT (Kojima al., 2022) prompting is written generate the answer after giving the question and 'Let's think step by step'. Prompt examples for zero-shot and zero-shot CoT per language can be found in Table 9.
Q: [problem]
Zero-shot
A: The answer is
E: (problem)
KO
[1 step]
Q: [problem]
Let's think step by step.
EN
Zero-shot CoT
step]
[problem]
A: Let's think step step. step output)
The answer is
[1 step]
3 (problem)
KO
[2 step] Ll : (problem)
G of of 2 Ho. {I step output) 8-2
Table Prompt examples per language for zero-shot (Brown al., 2020) and zero-shot CoT (Kojima et al., 2022).
In section 5, we use one category random method as examples of few-shot CoT (Wei et al., 2022) in DMath. The following table shows the few-shot examples of one category random. When using the babbage, curie, and davinci models, the prompting before the target question was truncated if few-shot CoT prompting exceeds 2049 tokens.

---

ARI PROMPT
Q: You sold 40 toys on Day 1. On Day 2, you sold 4 times the number of toys sold on Day 1. Find out the number of toys you sold more on Day 2 compared to Day 1.
A: On the Day 1, you sold 40 toys. On the Day 2, you sold times more toys than on the Day 1, so you sold * 40 = 160 toys. Therefore, the Day 2 sold 160 40 120 more toys than the Day Therefore the answer is 120.
Q: There banner 20 meters (m) long. I'm thinking of fixating this banner with nails. Nails were driven 1 meter (m) apart from both ends of the banner, and the remaining nails were nailed in at regular intervals. A total of 7 nails were used. Find the distance between one nail and another.
A: The interval between the first and last nails is 20 * 2 = 18. Since there are total of seven nails, there are 7 1 = 6 gaps between the nails. Therefore, the interval between nails is 18 / 6 6= 3. Therefore the answer is 3.
Q: Ycongchan's group consists of people. One person can fold two paper crancs in one minute. How many minutes will lake Yeongchan's group to fold 182 paper cranes?
A: In Yeongehan's group, one person can fold two paper cranes per minute, and Yeongchan's 's group, there are seven people, so Yeongehan's 's group can fold 2 * 7 14 paper cranes per minute. If Youngchan's group folds 182 paper cranes, takes 182/ 14 = 13 minutes. Therefore the answer is 13.
Q: Namjoon rode his bike and traveled 96.967 kilometers (km) in 1.3 hours. If Namjoon went steady speed, how many kilometers (km) did he travel in hour? A: The distance Namjoon went for an hour is 96.967 1.3 = 74.59 kilometers (km). Therefore the answer is 74.59.
Q: There are 24 female and male students in total. Among them, 50 sheets colored paper were equally distributed to female students only, and sheets were left. How many pieces of colored paper will the girls have there are twice as many boys as girls?
A: Female students have 50 2 48 colored papers. When the number of girls x, the number of boys 2x. Since there are 24 female and male students, is 2x 3x = 24. Therefore, the number of female students is 8. One female student has 48 / 6. Therefore the answer is 6.
Q: There is train that runs at 95 kilometers (km) per hour and car that runs 65 kilometers (km) per hour. If the train and the car travel for 8 hours at constant speed, by how many kilometers (km) does the train travel more than the car?
A: The train traveled 95 8 = 760 kilometers (km) in hours, and the car traveled 65 8 520 kilometers (km) 8 hours. Therefore, the train went 760 520 = 240 kilometers (km) longer than the car. Therefore the answer is 240.
Q: There are a total 12 of 10- (O-won, 50-won, and 100-won coins. checked the total amount and was 500 won. the number of 10-won and 50-won coins is the same, how many 100-won coins are there?
A: Let's say the number of 10- 10-won coins and 50-won coins is x and the number of 100-won coins is y. If there are 12 coins worth 10, 50, and 100-won, + y = 12. Since the total amount is 500-won, it is 10 # + 50 100 = 500. Solving this equation, x is 5 and y is 2. Therefore the answer is 2.
Q: There is scale that can weigh more than 2 kilograms (kg). If you have lifted 3 weights of 200 grams
(g) so far, find how many more weights you need to lift to measure the weight.
A: I've put up three 200 grams (g) weights, so there's 's an 200 = 600 grams (g) on the scale now.
* 1000 600 = 1400 grams (g) needed to weigh kilograms (kg) more. Therefore, 1400 / 200 = additional 200 grams (g) scales are needed. Therefore the answer
Q: (problem}
A:

---

6 12 COM PROMPT
Q: There are 24 balloons. Of these, there are 6 more blue balloons than red balloons, red balloons are 1/4 of the total, and the rest are all yellow balloons. What color balloons will be the most?
A: Since the red balloons are 1/4 of the total, 24 * (1/4)= 6 balloons. Blue balloons because there are 6 more blue balloons than red ones. The rest are yellow balloons, 24 6 12 6. Therefore, blue balloons are the most common. Therefore the answer is blue.
Q: How many of 0.8, 1/2, and 0.9 are greater than 0.4?
Of 0.8, 1/2, and 0.9, three are greater than 0.4. Therefore the answer is 3.
Q: Yoongi has 4 apples and Jungkook has 6 divided by 3 apples. Who has the greater number of apples?
Jungkook has 6/3 apples because he has divided by apples. Yoongi has four apples, so he has more apples than Jungkook. Therefore the answer Yoongi.
Q: Choose the 2nd smallest number among 5, 8, 4, 3, and 2.
The second smallest number of 5, 8, 4, 3, and is Therefore the answer
Q: Eight students stand in a line to borrow books. How many students are standing between the student standing 1st from the front and the student standing 4th from the back?
When total of eight students are standing in a line, there are 4 students between the first student the front and the fourth student the back. Therefore the answer
Q: Taehyung, Minju, Sangmin, Yoonjung, and Yoojeong took turns crossing the finish line. Who went in first?
Tachyung is first place. Therefore the answer is Tachyung.
Q: Find the number of numbers less than equal 0.4 among 0.8, 1/2, and 0.3.
The number less than or equal to 0.4 0.8, 1/2, and 0.3 is 0.3. Therefore the answer
Q: What is the smallest number among the numbers greater than 1.1 among the numbers 1.4, 9/10, 1.2, 0.5, and 13/10? Among 1.4, 9/10, 1.2, 0.5, and 13/10, the number greater than 1.1 is 1.4, 1.2, and 13/10. The smallest number of these is 1.2. Therefore the answer is 1.2.
Q: (problem)

---

COR PROMPT
Q: There are three natural numbers A, B. and C. When is divided by 7, the quotient and the remainder is The quotient and remainder are equal this equation. Find the largest number among A.
A: When A is divided by 7, the remaining C can be O, 2, 3, 4, 5, and 6. In order to make A the largest,
C must be 6 and B must be 6. Thus, the largest A is 6 * 7 = 48. Therefore the answer is 48.
Q: There is single digit number A. When 7A7 is rounded up to the tens place is 730. What is the right value for A?
A: The fact that 7A7 is raised from the decimal place to 730 means that A is 2. Therefore the answer is 2.
Q: Jisoo had 6,000 won and she bought ballpoint pen for 3,200 won, an eraser for 1,000 won, and candy for 500 won. Assuming that all remaining money is saved, find the amount saved.
A: The amount of money Jisoo has left after buying the product is 6000 -2000 1000 500 1300 won. Therefore the answer is 1300.
Q: The length of Siwon's 's ribbon is 8.8 meters longer than Seojun's. If Seojun cuts his own ribbon 4.3 meters (m) and gives it to Siwon, find out what the difference between the lengths of the two people's s ribbons.
A: If Siwon's ribbon length Scojun's 's ribbon length is 8.8. After Scojun gives the ribbon Siwon, the length of Seojun's 's ribbon x 8.8 4.3, and the length Siwon's ribbon + 4.3. The difference between the two x 4.3) (x 8.8 4.3 3 17.40 Therefore the answer is 17.40.
Q: 37 is 69 smaller than number. What is the number that is 55 greater than the number?
number that is 69 smaller than the number is 37, so the number is + 37 106. Therefore, a number greater than the number by 55 106 + 55 161. Therefore the answer is 161.
Q: 24 is a result of subtracting 63 from particular number by mistake when the original calculation was subtract 36 from that particular number. Find the result of the original calculation.
A: The number is 24 + 63 = 87. Correctly calculated, the value obtained by subtracting 36 from the number is 87 36 = 51. Therefore the answer 51.
Q: It 40 that should be subtracted from, but mistakenly subtracting 21 from a number yields 52, How much do you get when you calculate it correctly?
A: The number is 52 + 21 = 73, Correctly calculated, subtracting 40 from the number is 73 40 33. Therefore the answer is 33.
Q: You need to subtract 46 from a certain number, but mistakenly subtracted 64, and got 122. Find the result the correct calculation.
A: The number is 122 64 186. To calculate correctly, subtract 46 from the number is 186 46 = 140. Therefore the answer is 140.
Q: [problem)
A:

---

GEO PROMPT
Q: There is parallelogram that has base of 3.6 and height of 2.5 times the base. Find the area of this parallelogram.
The area of the parallelogram the product of the base and the height. Since the base of this parallelogram 3.6 and the height 3.6 * = 9, the area is 3.6 32.4. Therefore the answer 32.4.
Q: 12 friends, including Yoonscok, are standing as a dodecagon. Every student shook hands once with every friend except for the friend on either side of them. How many handshakes did Yoonseok shake? A: Yoon Seok shook hands 12 2 9 times because he shook hands with friends except for two both sides of him and one his own. Therefore the answer 9,
Q: There are pretty blankets whose shape the same regular tetradecagon. When the length of all corners 154 centimeters (cm). Find the length of one edge of the blanket. A: Since there are total of 14 edges of square, the length of one edge is 154/ / 14 11. Therefore the answer is 11.
Q: How many square centimeters (cm²) is the sum of the widths of two squares, every 11 centimeters (cm) and 5 centimeters (cm) long?
A: The area of a square with a side length of 11 centimeters (cm) is * = 121 square centimeters (cm²), and the area of a square with a side length of 5 centimeters (cm) is 5 25 square centimeters (cm²). The sum of the widths of the two squares is 121 + 25 = 146 square centimeters (cm²). Therefore the answer is 146.
Q: The perimeter of the square is 17.8 centimeters (cm). Find the length of one side of this square. The cireumference of square is four times the length of a side. Therefore, the length of one side of square with a circumference of 17.8 centimeters (em) is 17.8 / 4 4.45 centimeters (cm). Therefore the answer is 4.45.
Q: What is the number edges of a cube?
A: The number of edges of cube is 6 & = 12. Therefore the answer is 12.
Q: After arranging several cans of soda in square shape, there were cans left. So, I tried increase the width and length by row, but 11 cans were inadequate. If you arrange these cans in 7 rows, how many there in row?
A: Let's say that the number of cans of soda currently on one side of the square is x. If you want to increase the width and length by one row, you need 2 x + 1 cans of soda. Currently, there are 6 cans left, and 11 cans are insufficient when trying to increase it, so it can be considered x + 1 + 11. So is 8. The number of beverage cans currently in possession is x x 6= 8 8 6 = 70. If these 70 cans set in seven rows, there are 70/ 7 10 cans in the first row. Therefore the answer is 10.
Q: There is rhombus with (cm), how long is the other diagonal in centimeters (cm)?
of 64/5 square centimeters (cm²). If one diagonal is 64/9 centimeters
A: The area of the rhombus is equal to the product of the two diagonals in the rhombus multiplied by one-half. Thus, when one diagonal is 64/9 centimeters (cm), the length of the other diagonal is (64 / 5) (64 / 9) = 3.6 centimeters (cm). Therefore, the answer is 3.6.
Q: [problem]

---

POS PROMPT
Q: There are two types of Korean language workbooks and four types of math workbooks. When choosing one book among them, in how many cases can you choose Korean workbook or math workbook?
A: When there are two types of Korean workbooks and four types math workbooks, the number of cases in which one Korean workbooks or one math workbooks is selected is 2 + Therefore the answer is
Q: You want to create three-digit number by drawing three different numbers from 0, 1, 3, and 5, Find the sum the largest and smallest possible numbers. A: The largest number that can be made 531, and the smallest number is 103. The sum of the two 531 103 = 634. Therefore the answer is 634.
Q: Nine digits from I to used once form nine-digit number. If this number a multiple of 55, find the largest 9-digit number. A: The largest number of nine digits. multiple of 55, is 987642315. Therefore the answer is 987642315.
Q: How many two-digit numbers can be formed by selecting two different numbers from 1, 3, and 4?
A: The number of two digits that can be made by drawing two different numbers from 1, 3, and 4 is 3P2 =
6. Therefore the answer is 6.
Q: There are 5 marbles: red, blue, yellow, purple, and black. You want to choose two of these and give one to your younger brother and the other to your older brother gift. Find the number of cases in this situation.
A: The number of cases in which two of the five different colored beads are selected is 5P2 20. Therefore the answer is 20.
Q: You have 4 blue pencils and red pencil. How many different ways can you place 5 pencils in row? A: When there are four blue pencils and one red pencil, there are total of five ways to put these five pencils in one line. Therefore the answer is 5.
Q: How many multiples of 4 are there among the four-digit numbers made using the number cards 1, 3, 5, and 6 once?
A: The four digits that can be made using the number cards 1, 3, 5, and 6 once are 1356, 1365, 1536, 1563, 3156, 3165, 3516, 3516, 3561, 3615, 3651, 5136, 5163, 5163, 5361, 5613, 6135, 6153, 6351, 6513, 6531, 6531. Among them, the multiples of 4 are 1356, 1536, 3156, 3516, 5136, and 5316. Therefore the answer is 6.
Q: Find the smallest number that can be made by using all the number cards 0, 2, 4, 6, 8 and 9 only once.
A: The smallest number that can be made using the number cards of 0, 2, 4. 6, 8, and 9 once is 204689. Therefore the answer is 204689.
Q: (problem]

---

G
Failed examples in GPT-4
ARI
Question: How many four-digit numbers are divisible by the two numbers 3 and 5?
Reasoning: A number that is divisible by both 3 and divisible by their least common multiple (LCM). The LCM of 3 and 5 is 15. To find the number of four-digit numbers divisible by 15, we need to find the range of these numbers. The smallest four-digit number is 1000, and the largest is 9999. The smallest four-digit number divisible by 15 is 1005 (1000 + 5), and the largest is 9990 (9999 9). Now, we need to find how many numbers are in this range that are divisible by 15. To do this, we can divide the difference between the largest and smallest numbers by 15 and add 1. (9990 1005) / 15 + 5985 / 15 + 399 == 400. So, there are 400 four-digit numbers that are divisible by both and 5. Correct Answer: 600
COM
Question: Nine people stand in a line order from shortest to tallest. Hoseok stands the front. they line up again in order from the tallest to the shortest, what number will Hoseok stand from the back? Reasoning: Since Hoseok is the shortest and stands in the front when they line up from shortest to tallest, he will be the last person when they line up from tallest to shortest. So, Hoseok will stand 9th from the back. Therefore the answer is 9.
Correct Answer:
COR
Question: When you round 34AC to the nearest hundred, you get 3400. Find the sum of the possible As.
Question: When rounding 34AC to the nearest hundred, the possible values for A are 3, 3, 5, and 6. This is because if A is or 4, the number will round down to 3400, and if A is 5 or 6, the number will round up to 3400. The sum of the possible As is 3 + 5 + 6 18. Therefore, the answer is 18.
Correct Answer: 10 GEO
Question: How many sides are there in polygon which 12 diagonals can be drawn from one vertex?
If 12 diagonals can be drawn from one vertex, then there are other vertices that are not adjacent to the given vertex. Since polygon has as many sides as has vertices, there must be 12 + (the given vertex) = 13 sides in the polygon. Therefore, the polygon has 13 sides.
Reasoning:
Correct Answer: 15
POS
You want to create a three-digt number by drawing three cards out of number cards with 1, 7, 0, and 3 written on them and using them only once. Find the value of multiplying the second largest
Question:
number and the second smallest number that can be made.
Reasoning: The second largest number that can be made is 371, and the second smallest number is 130.
The value of multiplying the two is 371 x 130 = 48,230. Therefore the answer is 48,230.
Correct Answer: 78110


Reviews:
[{"review_id": "05hYWF68og", "reviewer": "Reviewer_ZcUJ", "summary": "", "questions": "Question A. Why does the finetuned model showed similar tendency to LLMs? Is there any further explanation about that? (lines 439-444)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our strengths. Below are our responses to your questions.\n\n## Questions For the Authors\n### 1. Different tendency between fine-tuned models and Large Language Models\n* Firstly, we aim to clarify the \"tendency\" in lines 439-444. **This tendency indicates that various forms of expressions have their preferred categories.** To elaborate, when we state that form of expression A outperforms form B, it does not imply that every type within A uniformly surpasses every type within B. Rather, it suggests that certain types within A outperform B, while some types in B excel over A. **This tendency is discernible in both fine-tuning and prompting methodologies.**\n\n* The fine-tuning and large language models (LLM) exhibit this inclination; however, **their reasons differ.** For the **fine-tuning model**, the length of the intermediate solution form significantly impacts preference determination. For instance, GEO—having the shortest code token length—performs more efficiently with Python code expression form, while COM and POS—with longer code token lengths—demonstrate superior performance with natural language expression forms.\n\n* Regarding **LLMs**, features intrinsic to the expression form greatly influence preference determination. Considerably lower mathematical computation errors distinguish Python code compared to natural language; conversely, natural language excels over Python code in succinctly expressing relationships between two objects on a large-small scale.\n\n* Lastly, we apologize for any ambiguity surrounding our use of 'tendency' in lines 439-444 and will incorporate a suitable explanation to clarify its meaning.\n\n## Typos Grammar Style And Presentation Improvements\n### 1. The size of the GPT-4 model\n* We have meticulously revisited the technical report \\[1\\] to ensure no details were overlooked. However, it does not provide any information regarding the parameter sizes of GPT-4. As stated in the \"2. Scope and limitations of this Technical Report\" section (p.2, lines 13-16), it omits specifics of model size, the hardware used, computational training power, dataset construction, training methodology, etc. Should information about parameter sizes become publicly accessible, we will certainly incorporate it.\n    \n### 2. In line 368, ‘across’ seems more suitable than ‘from.’\n* Your suggestion is indeed valuable. As you rightly pointed out, using \"across\" instead of \"from\" suits our intended message here. Our objective was not to draw comparisons between models and categories but to illustrate that different models exhibit varying degrees of performance across various categories. As per your recommendation, we will employ \"across\" for a clearer expression of our standpoint. We appreciate your insightful feedback on this matter.\n\n\n\n### 3. In lines 414-418, the term \"matched\" means \"are correctly solved\"?\n* You raise a valid observation. The subsequent line presents the outcome of classifying the questions resolved utilizing both Natural Language (NL) prompts and Python code prompts into four distinct scenarios:\n    1. The proportion of questions correctly addressed solely through NL prompts\n    2. The proportion of questions correctly addressed using both NL and Python code prompts\n    3. The proportion of questions correctly addressed exclusively via Python code prompts\n    4. The proportion of questions that remained incorrect despite employing both methods\n\n* We have employed the term **\"matched\"** to denote inclusion within any of the four scenarios as mentioned earlier. However, as you rightly pointed out, it would be more precise to phrase it as \"are accurately resolved\" in this context. We are grateful for your point and will implement this amendment.\n\n### References\n\\[1\\] OpenAI. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written, and stating what they did.\n- Providing a new dataset that ensures diversity on languages, linguistic patterns, and intermediate solution forms.\n- Providing a thorough analysis of current LLMs on their datasets", "reasons_to_reject": "Nothing.", "questions_to_authors": "Question A. Why does the finetuned model showed similar tendency to LLMs? Is there any further explanation about that? (lines 439-444)", "paper_topic_and_main_contributions": "This paper proposes a new diverse dataset for math word problems. They collected 10K problems in English and Korean, including equations and Pythonic represnations to solve the problem. Also, they tested several PLMs on their dataset and showed that the math word problem solving task isn't over, LLM should handle some of the cases that their dataset presents.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I think the size of GPT-4 model is stated in the GPT-4 technincal report.\n- In lines 368, did you try to say \"the worst problem categories differ across MWP modes\"? Because, it's weird to compare problem categories with MWP models.\n- In lines 414-418, the term \"matched\" means \"are correctly solved\"?"}}, {"review_id": "a9dqza7Aot", "reviewer": "Reviewer_jf9x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and the detailed review of our paper. We are committed to addressing your questions and concerns and have responded below.\n\n## Reasons to Reject\n### 1. Comparison between DMath and MATH \\[1\\]\n\n* Initially, a concise summary of the responses is as follows:\n\n    1. **\\[Discrepancy in domain knowledge levels\\]**\n        We aimed to tackle mathematics problems at the elementary school level (grade 1-6), while MATH \\[1\\] is designed for the high school level (grade 8-12). Therefore, in our paper, we compared data at equivalent domain levels.\n    2. **\\[General comparison between DMath and a subset of MATH\\]** \n        Despite differences in domain knowledge levels and problem categories, creating a subset of MATH \\[1\\] similar to DMath types is feasible. As you suggested, the comparison between DMath and this subset of MATH \\[1\\] demonstrates that DMath is a dataset capable of evaluating fundamental reasoning skills.\n\n#### **\\[Discrepancy in domain knowledge levels\\]**\n* **Our objective was to develop an elementary school-level (grade 1-6) math dataset, whereas MATH \\[1\\] caters to high-school-level (grade 8-12) mathematics. To ensure fair comparison, we compared it with elementary school-level datasets such as GSM8K \\[2\\], SVAMP \\[3\\], and ASDiv \\[4\\].**\n* The MATH \\[1\\] dataset contains problems at the level of high school education. According to “Section 3.1. The MATH Dataset” (p. 4, lines 20-22) in \\[1\\], it encompasses intricate mathematical problems from AMC10, AMC12, and AIME that necessitate comprehension of advanced mathematical concepts such as trigonometric functions or imaginary numbers.\n* Additionally, we chose elementary-level math problems owing to their effectiveness in evaluating fundamental reasoning abilities. Although there are several Math Word Problem (MWP) datasets available at this level \\[2, 3, 4\\], they mainly concentrate on arithmetic reasoning. Nonetheless, examining the curriculum for these grades exposes other areas, such as relationships or probability, which also require an appraisal of reasoning skills. Consequently, **we designed DMath to provide a more comprehensive yet refined dataset that could assess these areas with efficacy.**\n* We briefly outlined these points in Appendix D, lines 957-972 and line 296. However, we will provide further detail on this particular point.\n\n#### **\\[General comparison between DMath and a subset of MATH\\]**\n* The MATH Dataset \\[1\\], comprising high-level questionnaires on domain-specific knowledge for grades 8-12, categorizes seven fields as described in \"Section 3.1\" (p. 4, lines 31-33 in \\[1\\]): Pre-algebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus.\n* **Regardless of the data's domain knowledge level, it is accepted that there may be types similar to the five types of DMath (Arithmetic, Comparison, Correspondence, Geometry, and Possibility) \\[1\\].** Therefore, we can categorize the seven types outlined in \\[1\\] as either similar to DMath or dissimilar.\n    - **Similar question types**\n        1. **PreAlgebra**: According to \\[5\\], it covers Exponents, Divisors, Greatest Common Divisor, Fractions, etc. **It is similar to the ARI \\(Arithmetic\\) category in DMath.**\n        2. **Counting & Probability**: According to \\[6\\], it contains Combinations, Basic Probability, Pascal's Triangle, Binomial Theorem, etc. **It is similar to Possibility (POS) in DMath.**\n        3. **Geometry**: According to \\[7\\], it contains Lines, Parallel and Perpendicular Lines, Right Triangles, Geometric Mean, Vectors, etc. **It is similar to Geometry (GEO) in DMath.**\n  - **Non-similar question types**\n    1. **Algebra**: According to \\[8\\], it contains contents such as Linear functions, Linear Inequalities, Quadratic Functions, etc.\n    2. **Intermediate Algebra**: According to \\[9\\], it includes Linear Equations, Polynomials, Exponential and Logarithmic Functions, etc.\n    3. **Precalculus**: According to \\[10\\], it includes Trigonometric, Matrices, Series, etc.\n    4. **Number Theory**: According to \\[11\\], it includes Primes, Fermat's Theorem, Fibonacci Numbers, etc.\n\n* **Table 1 displays the accuracy of PreAlgebra, Counting & Probability, and Geometry from MATH \\[1\\] in comparison to ARI, POS, and GEO from DMath.** The accuracy of GSM8K, ASDiv, and SVAMP, which comprise arithmetic reasoning, is also demonstrated. The result for MATH \\[1\\] in Table 1 (below) originates from Table 1 (p. 8) in \\[12\\]. The table below presents GPT-4 few-shot CoT performance.\n\n#### \\<Table 1\\>\n\n| DMath | Acc.  | MATH \\[1\\] | Acc. | GSM8K | Acc.  | ASDiv | Acc.  | SVAMP | Acc. |\n|-------|-------|------------------------|------|-------|-------|-------|-------|-------|------|\n| ARI   | 76.53 | PreAlagebra            | 71.6 |       | 91.81 |       | 90.03 |       | 91.6 |\n| POS   | 57.04 | Counting & Probability | 53.1 | -     | -     | -     | -     | -     | -    |\n| GEO   | 75.72 | Geometry               | 36.5 | -     | -     | -     | -     | -     | -    |\n\n* From the results displayed in Table 1, it is evident that **MATH \\[1\\], which demands an advanced level of domain knowledge (grade 8-12), predictably exhibits lower performance compared to the datasets that necessitate a lower level of domain knowledge (grade 1-6).**\n* Additionally, **DMath** exhibits a notably smaller performance disparity with MATH \\[1\\] compared to \\[2, 3, 4\\], **suggesting increased complexity of datasets within the same domain knowledge level.** DMath can **provide insights** that previous datasets do not because it **provides multiple reasoning categories at lower levels of domain knowledge (grades 1-6).**\n  * For example, POS does not perform well on low-level domain knowledge (grades 1-6), so we can see that even GPT-4, the current SOTA model, does not yet perform well on basic probability reasoning.\n  * On the other hand, GEO performs relatively well on low-level domain knowledge (grades 1-6), and about half as well on high-level domain knowledge (grades 8-12). This suggests that basic geometric reasoning is recognized by GPT-4 and that a lack of domain knowledge can lead to poor performance in MATH \\[1\\].\n  * Consequently, this means that DMath is currently the only dataset that can help researchers conduct mathematical reasoning tasks to assess the most basic reasoning abilities.\n\n* Finally, given that 1) they \\[2, 3, 4\\] are at the same domain knowledge level and 2) the types of MATH \\[1\\] and DMath are different, we thought it was fair to compare only DMath with \\[2, 3, 4\\]. However, a rough comparison suggests that comparing DMath with a subset of MATH \\[1\\] could be helpful for future math reasoning tasks. We are grateful for this suggestion, and we will consider it.\n\n### References\n\\[1\\] Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. NIPS (2021)  \n\\[2\\] Cobbe, Karl et al. “Training Verifiers to Solve Math Word Problems.” ArXiv abs/2110.14168 (2021): n. pag.  \n\\[3\\] Miao, Shen-Yun et al. “A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers.” ArXiv abs/2106.15772 (2020): n. pag.  \n\\[4\\] Patel, Arkil et al. “Are NLP Models really able to Solve Simple Math Word Problems?” North American Chapter of the Association for Computational Linguistics (2021).  \n\\[5\\] Rusczyk, Richard, et al. Prealgebra. 2011.  \n\\[6\\] Patrick, David. Introduction to Counting and Probability. 2007.  \n\\[7\\] Hill, McGraw. Geometry, Student Edition. McGraw-Hill Education, 2012.  \n\\[8\\] Hill, McGraw. Algebra 1, Student Edition. McGraw-Hill Education, 2012.  \n\\[9\\] Miller, Julie, et al. Intermediate Algebra. McGrawHill Education, 2022.  \n\\[10\\] Miller, Julie, and Donna Gerken. Precalculus. 2022.  \n\\[11\\] Burton, David. Elementary Number Theory. McGraw-Hill Education, 2010.  \n\\[12\\] Zhou, Aojun et al. Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification. ArXiv abs/2308.07921 (2023): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset provides both Korean and English languages for math questions. To cover the diversity of math concepts, the dataset supports 5 problem types, including arithmetic, comparison, correspondence, geometry and possibility. It support two solution forms: express tree and python code. In expression tree, it provides 50 operators. Compared to SVAMP and ASDiv datasets, the experimental results show the difficulty of this dataset. ", "reasons_to_reject": "To show the difficulty of the dataset, it is compared with other datasets, such as GSM8K, SVAMP, ASDiv. The reason is now well analyzed, since those datasets are arithmetic types, and only contain basic arithmetic operators, but this dataset have more question types and operators. The Math Dataset contains similar question types as this dataset, but not compared.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new math word problem dataset. This purpose of this new dataset is to provide more diversity on languages, intermediate representations, and problem types, lexical usage patterns.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kocmY5fQOI", "reviewer": "Reviewer_uqoi", "summary": "", "questions": "Q1: Table 2: Is P(3,2) a more precise and comprehensive expression for human annotating (for Human solution, Equation and Python code) and understanding?\n\nQ2: Lines #163-#164: Why didn’t DMath provide the validation set?\n\nQ3: Table 6: The performance of the few-shot CoT is lower than the zero-shot Cot for the CGP-4 model on GSM-8K. Why? How to obtain the few-shot samples of SVAMP and ASDIV?\n\nQ4: Lines #253-#265: How to verify and confirm steps (1)~(3)? The validation process is essential for an MWP dataset. It is suggested to provide the validation details. For step (4), it is unclear how to “make the expression tree as efficient as possible”, and “be acceptable to most people”? It is also suggested to provide detailed information.\n\nQ5: Lines #277-#279: How do the remaining 71% ensure grammatical correctness?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Saying that our paper will help with future research is appreciated. Also, thank you for pointing out many things that will help our paper. The responses are below.\n\n## Reasons to Reject\n### 1. Lack of detailed description of the validation procedures of DMath.\n* **We agree that a validation procedure for MWP datasets is very important.** For this purpose, we recruited 13 graduate students majoring in computer science who participated in the previous process (*collection, augmentation*, and *annotation*). They are familiar with the annotation format of the MWP dataset and can solve elementary-level mathematical problems well.\n\n* **We tried to elaborate the validation process and performed four processes sequentially:**\n\n> **1. Question-answer validation,**  \n> **2. Question-expression tree validation,**  \n> **3. Expression tree validation, and**  \n> **4. Conciseness validation of expression tree for multi-expression-tree matching problems.**  \n\n* If any type of error \\(*i.e. errors in the question, in the expression tree*, or *in the answer*\\) was detected during \\(1\\)-\\(2\\)-\\(3\\), we returned to \\(1\\) for revalidation. After that, we performed (4\\). **These procedures were done to reduce potential risks to the correctness of the annotated data.**\n* Here's how the human workers were allocated for each step:\n  * Across step (1) and step (2), we ensured that the questions assigned to a human worker do not overlap so that they do not receive the same questions in each step, which can act as a cross-check.\n  * Step (3) was verified using code.\n  * For step (4), 2-3 human workers were assigned per question.\n\n* Now we elaborate the details of steps (1)~(4) on Lines \\#253-\\#265.\n#### **Step (1) - \\[Question-answer validation\\]**\n* After the *annotation* process, we gave human workers only natural language questions and asked them to solve and answer them manually. There are two cases where their answers differ from the annotated answers.\n* The first is when the natural language problem is unsolvable, and the second is when the answer is incorrect. When a worker encounters these errors while solving a problem, the worker corrects the error manually.\n* In the first case, the worker should correct the natural language question, and in the second case, the worker should correct the annotated answer. In addition, the corrected questions are cross-checked by two or more workers to ensure that there are no errors.\n\n#### **Step (2) - \\[Question-expression tree validation\\]**\n* Natural language questions that are solvable and correctly annotated with correct answers are given to human workers along with an expression tree. The human workers check whether the given natural language question and the expression tree are correctly paired. If not, they modify the expression tree to match the natural language problem.\n\n#### **Step (3) - \\[Expression tree validation\\]**\n* After the natural language question and the expression tree have been properly matched in the previous step, we verify that the expression tree produces the correct answer. In DMath, the operators used in expression trees can be converted to Python code. Therefore, after converting each expression tree to code, we check whether the result obtained when executing it is the correct answer. If the two values are different, return to step (1).\n\n\n* With these three steps, we wanted to reduce the errors in our dataset. After validating this, we wanted to make the expression tree more efficient.\n\n#### **Step (4) - \\[Conciseness validation of expression tree for multi-expression-tree matching problems\\]**\n* There can be various potential solutions to a mathematical problem, and we maintain that the selection of an expression tree is crucial to the model’s training.\n* For example, the following math word problem has multiple expression trees.\n> “There are six people. Everyone shook hands with each other once. Find the total number of handshakes.”\n* We can make the expression tree as one of\n```\n1) {( 6 - 1 ) * 6} / 2\n2) 6 C 2\n```\n* Here, both equations lead to the correct answer and they are  reasonable.\n* We wondered which of these multi-expression-trees would be better to choose. We decided to choose a problem with as short an equation length as possible. We considered this direction, reducing the equation length, because the most concise solution process captures the essence of the problem.\n* However, even if a formula is not the shortest, we choose it if most workers agree. We made sure that discussions about the most appropriate formulas involved at least three students and an odd number of students to prevent the majority of each question from being 1:1.\n* We also felt that \"efficient\" was a word that could encapsulate the process.\n\n- We believe that the validation process is important to reduce errors in the dataset, but due to the paper limitation, we have to compress the content and as a result not all of the details are revealed. We apologize for this and will add these details to the paper.\n\n### 2. Necessity of pre-defined intermediate form\n* **We had to define predefined intermediate forms to solve problems that cannot be solved using only four arithmetic operators.** Typical problems that cannot be solved by using only arithmetic operators are following:\n> 1. Comparing values between each other\n> 2. Number of Cases\n> 3. Problems involving rounding\n> 4. Problems involving unknowns\n* We have defined operators to solve these types of problems, and **we agree that they can be difficult for others to interpret/understand if not well designed.** Therefore, we set some rules for defining these operators.\n    1. Make the names of operators match the names of actual math symbols as closely as possible.\n    *e.g.* `[ADD]`, `[FLOOR]`, `[GCD]`, `[LIST_MAX]`\n    2. When performing the same action, match the name of the operators.\n    *e.g.* \\(a\\) `[LIST_GET_PERM]` \\(b\\) `[LIST_GET_PRODUCT]` \\(c\\) `[LIST_GET_DIVISOR]`\n    \\(a\\), \\(b\\), and \\(c\\) all take a LIST as input and return a LIST as output through some operation \\(PERM, PRODUCT, DIVISOR\\), so `‘LIST_GET_’` overlaps.\n    3. Make the name of the operators as concise as possible.\n    *e.g.* `[COMB]`, `[LCM]`, `[LIST_LEN]`, `[LIST2NUM]`\n    4. If a single operator performs multiple operations, give it a name that summarizes the operations.\n    *e.g.*  `[DIGIT_UNK_SOLVER] ⇒ A2+4B=69 A [DIGIT_UNK_SOLVER] → 2`\n    This operator takes an expression, string corresponding to the digits in the expression, and finds the unknown numeric value \\(UNK\\) of the string.\n* To reduce confusion, we made sure that the functions of complex operators, especially those involving LISTs \\(e.g., `[LIST_COND_MAX_MIN]`, given a list `L1` and a list `L2` with a condition containing the case of the elements of `L1`, return a new list of `L1` sorted in descending order according to the condition\\), were well explained to everyone involved in data annotation and validation. **To help with the interpretation, we created a table of “operator name - description including the form of input and output - example”.** This table can be found in Appendix B.\n\n## Questions For the Authors\n### 1. Effective annotation form for Table 2\n* Of course, **`P(3,2)` is a correct solution, and it is much simpler and clearer than `len([apple, peach, pear]) permutation 2`.** However, our objective was to create a solution that emphasizes the logical progression step-by-step.\n* Let’s consider the example problem in Table 2 again. The problem was as follows: \n\n> “You want to pick 2 fruits out of an apple, a peach, a pear and place them in different fruit baskets. What is the Number of Cases?”\n\n- In this case, **the number 3 is not directly present in the problem**, so we have included the process of obtaining 3 in the solution and represented it as `len([apple, peach, pear]) permutation 2`, instead of using permutation.\n\n\n### 2. Validation set of DMath\n* We did not separate the validation set following GSM8K, which is currently the most widely used in MWP tasks.\n\n### 3. Lower performance of the few-shot CoT than the zero-shot CoT for GPT-4 model on GSM8K\n\n* Table 6 shows the experimental results of ChatGPT and GPT-4’s zero-shot CoT and few-shot CoT for GSM8K, SVAMP, ASDiv, and DMath. Here, the performance of zero-shot CoT is higher than that of few-shot CoT only when using the GPT-4 model for GSM8K, so it is quite natural to question it.\n* \\[1\\] states that they used GSM8K data to train GPT-4. **We believe that using GSM8K data improved the zero-shot CoT performance of the GSM8K in GPT-4, and the few-shot examples used previously acted as noise in the trained model.** There are two reasons for this.\n  * The first is that for ChatGPT without GSM8K as training data, the performance of the zero-shot CoT is lower than that of the few-shot CoT.\n  * The second is that the performance varies depending on the few-shot example. This is also shown in \\[2\\], where the performance of chain-of-thought prompting on GSM8K using LaMDA 137B is varies for different few-shot examples.\n* We would like to clarify at this point **that our experimental results are not erroneous**: the results of the GSM8K GPT-4’s few-shot CoT in Table 6 are similar to the 92% (with 5-shot, chain-of-thought) reported in the GPT-4’s technical report \\[1\\], so we believe that our experiments are correct. (The results of the zero-shot CoT are not reported here.)\n\n\n\n### 4. Few-shot samples of SVAMP and ASDiv \n* **We use the same few-shot example as GSM8K for ASDiv and SVAMP**, following the experimental setup in \\[3\\], because the three datasets consist of similar arithmetic reasoning problems. We apologize for any confusion caused by our lack of clarity, and thank you for pointing out what we missed. We’ll add this to the text.\n\n\n\n### 5. Grammatical correctness of remaining 71%\n* For the 29% of the questions that did not change, we used a grammar checking tool (Grammarly) to correct grammatical errors.\n* Afterward, all questions (# = 10,022) were checked for grammatical errors. This was done by three graduate students who were not involved in the translation and were proficient in English and Korean. The results showed that 5.24% (# = 525) of the questions had grammatical errors, which were corrected.\n* Due to the page limit, we had to compress the content, so the relevant information may not be clear. We apologize for the lack of explanation, and we will clarify this in the paper.\n\n### References\n\\[1\\] OpenAI. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023): n. pag.  \n\\[2\\] Wei, Jason et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NIPS (2022)  \n\\[3\\] Gao, Luyu et al. PAL: Program-aided Language Models. ArXiv abs/2211.10435 (2022): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a new dataset. It not only focuses on the diversity of the language pattern and problem type but also provides bilingual MWPs and multiple solution reasoning paths. Observations, insights, and discussions of relevant experimental results are detailly and will be helpful for future research on MWP solving. Furthermore, most previous studies of non-LLM models are limited to binary operators, so the proposed dataset (including many non-binary operators) will help future research on MWP solving.", "reasons_to_reject": "(1) Lack of detailed description of the validation procedures of DMath (refer to Q3). Therefore, there are potential risks to the correctness of the annotated data.\n\n(2) The intermediate solutions (including Human solutions and Equations) are annotated by pre-defined intermediate form instead of mathematical languages. Such annotations may be hard to interpret and comprehend directly by the human.", "questions_to_authors": "Q1: Table 2: Is P(3,2) a more precise and comprehensive expression for human annotating (for Human solution, Equation and Python code) and understanding?\n\nQ2: Lines #163-#164: Why didn’t DMath provide the validation set?\n\nQ3: Table 6: The performance of the few-shot CoT is lower than the zero-shot Cot for the CGP-4 model on GSM-8K. Why? How to obtain the few-shot samples of SVAMP and ASDIV?\n\nQ4: Lines #253-#265: How to verify and confirm steps (1)~(3)? The validation process is essential for an MWP dataset. It is suggested to provide the validation details. For step (4), it is unclear how to “make the expression tree as efficient as possible”, and “be acceptable to most people”? It is also suggested to provide detailed information.\n\nQ5: Lines #277-#279: How do the remaining 71% ensure grammatical correctness?", "paper_topic_and_main_contributions": "This paper introduces a new MWP dataset, DMath, which is diverse in lexicon usages and problem types. It is also bilingual (English and Korean languages) and annotated two kinds of solutions (an expression tree and Python code). Experimental results illustrate that the GPT-4 LLM achieves 70% accuracy on DMath; the performance gap is worth exploring for the future research on MWP solving.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper’s major contribution is a novel math word problem dataset in English and Korean characterized by diversity at various levels: in problem types, in lexical patterns, in language narratives, in solution forms, which can help better evaluate the actual linguistic and reasoning performance of LLMs. The authors conducted experiments with various pre-trained language models and found that the math word problem-solving task remains challenging. Reviewers converged on soundness and found the work novel and reproducible. Overall, the paper does not show major weaknesses and may improve results and findings in this area. 

**Pros.** 

- The paper is well written, and argumentation is sound and clear; 

- A novel gold dataset of math word problem of higher complexity and diversity on various aspects; 

- The dataset addresses gaps in previous works and has the potential to enhance future research in the field; 

- The methodology seems transferable to other languages. 

**Cons:**

No a serious weaknesses to highlight. However, the rationale of the choices made for the comparison with other data sets could be more clearly explained/ motivated.
Decision: Accept-Main