Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
Authors: Haoyi Wu
Wenyang Hui
Yezeng Chen
Weiqi Wu
Kewei Tu
Yi Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: math problem
reasoning

Abstract: Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain a small amount of data in one specific topic, making it hard to analyse AI's behaviour with reference to different problems within a specific topic in detail. In this work, we propose Conic10K, a challenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with different reasoning depths, while only the knowledge from conic sections is required. Since the dataset only involves a narrow range of knowledge, it is easy to separately analyse the knowledge a model possesses and the reasoning ability it has. For each problem, we provide a high-quality formal representation, the reasoning steps, and the final solution. Experiments show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and reasoning. Our dataset and codes are available at https://github.com/whyNLP/Conic10K.

CONIC10K: Challenging Math Problem Understanding and Reasoning Dataset
Haoyi Wu.l2 Wenyang Hui Yezeng Chen 1336, Weiqi Wul?, Kewei Tu Yi Zhou*
School of Information Science and Technology, ShanghaiTech University
Shanghaii Engincering Research Center of Intelligent Vision and Imaging School of Information Science and Technology, University of Science and Technology of China
National Engineering Laboratory for Brain-inspired Intelligence Technology and Application
Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education "Shanghai Innovation Center for Processor Technologies
DDpaarmeent of Computer Science and Engineering, Shanghai Jiao Tong University [wuhy1, huiwy, chenyz, tukw)@shanghai tech. edu.cn; wuwq1 jangs wtw.su...tuuduuucrmo.oo cdu yi zhou@ustc. edu. cn
(Lample and Charton, 2020; Wei et al., 2022b), mathematical reasoning are far from being solved by existing methods (Lu et al., 2022), even with symbolic reasoners (Hopkins et al., 2019) and large language models (LLMs) (Lightman et al., 2023). To evaluate and analyse the mathematical ability, various datasets and benchmarks have been pro- posed in recent years (Zhao et al., 2020; Hendrycks et al., 2021; I; Mishra et al., 2022b.a). However, these datasets or benchmarks often suffer from the fol- lowing problems: (1) The problems can be solved with only few reasoning steps, so language mod- els may rely shallow heuristics to achieve high performance (Patel et al., 2021); (2) The dataset covers wide range of topics and hence there is only a small amount of data for each topic, which makes hard to distinguish whether the model fails because of a lack of background information, or due weak reasoning ability.
Abstract
Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain small amount of data in one specific topic, making it hard to analyse AI's behaviour with reference to differ- ent problems within specific topic in detail, In this work, we propose CONIC10K, 10K, a chal- lenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with differ- ent reasoning depths, while only the knowl- edge from conic sections is required. Since the datasst only involves a narrow range of knowledge, it easy separately analyse the knowledge model possesses and the reasoning ability it has. For each problem, we provide high-quality formal representation, the reason- ing steps, and the final solution. Experiments show that existing large language models, in- cluding GPT-4, exhibit weak performance complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and rea- soning. Our dataset and codes are available at https://git thub. com/whyNLP/Com OK.
To address the above issues, we propose CONIC10K, an open-ended math problem dataset on conic sections in Chinese senior high school education. This dataset contains 10,861 carefully annotated problems, each has formal repre- sentation, the corresponding text spans, the answer, and natural language rationales. Figure shows an example problem our dataset. To evaluate the mathematical understanding and reasoning ability, we perform two different tasks on existing LLMs: semantic parsing and mathematical question an- swering (mathQA). Semantic parsing assesses a language model's ability understand mathemat- ics. The model is required to translate the math problem in natural language into its formal mean- ing representations. MathQA jointly evaluates the language model's ability of mathematical under- standing and reasoning. The model needs to gen-
1 Introduction
Mathematical understanding and reasoning abil- ity is an important component of human intelli- gence. Such an ability is the foundation of data analysis, financial applications and scientific re- search. Though there have been lots of studies
Equal Contribution.
Corresponding Authors.
Work completed while the author was at ShanghaiTech University.

---

erate the answers questions. Since the topic of CONIC10K is restricted to conic sections, the knowledge required to solve different problems the same, while the only difference is the difficulty in reasoning. Therefore, the model is able to solve simple problems but not hard ones, we are assured that the failure lies in the lack of ability in mathematical reasoning.
(Wang et al., 2017) and Ape210K (Zhao al., 2020) are popular datasets about Chinese math word problems with open-domain scenarios and simple reasoning steps. Geometry33K (Lu et al., 2021) ) is a geometry problem-solving dataset that provides formal representations, but the dataset size is small and the problems do not require complex reasoning, AQuA (Ling et al., 2017), NumGLUE (Mishra et al., 2022b) and Lila (Mishra al., 2022a) are large-scale datasets of various math problems. They have been used as benchmarks in solving math word problems and mathematical rea- soning tasks, but we find that these datasets require only few reasoning steps. MATH (Hendrycks et al., 2021) is the one with the longest reason- ing steps among these datasets. It has been used as a standard benchmark in recent work of LLMs (Lewkowycz et al., 2022: Lightman al., 2023). However, while covers wide range of prob- lems, it contains limited data in each specific topic, making it hard to analyse the model behavior detail with reference to one topic. It also does not provide any formal representations. Our proposed CONIC1 OK contains problems of long reasoning steps using closed-domain knowledge and has high- quality annotations with formal representations. A detailed comparison between the aforementioned datasets and CoNIc1 is shown Table 1.
Our experiments show that current models ob- tain good performance in semantic parsing. How- ever, in mathQA, these models are far from be- ing satisfactory. When performing zero-shot chain of thought (CoT) (Wei et al., 2022b) prompting, the best model GPT-4 (OpenAI, 2023) can only achieve 15.5% accuracy using human evaluation. When finetuning is further applied, the best model M-6b (Du et al., 2022) still obtains a poor accuracy of 22.5% under human evaluation. When we translate the problems into English and apply zero-shot CoT to reason in English, the accuracy of GPT-4 is 26.0%, which is still far below the perfor- mance of human experts at 57.5% with a 3-minute time limit for each problem. This shows that the poor performance not due the language being used but to a deficiency in reasoning ability. There- fore, we believe the mathematical reasoning ability of language models is still limited despite their huge success in natural language understanding.
We conclude our contributions as follows: 1) We propose CONIC10K, a challenging math problem dataset on conic sections in Chinese senior high school education, with high-quality annotations of formal representations: 2) We perform experiments to inspect the mathematical understanding and rea- soming ability of LLMs separately; 3) We give de- tailed analysis on the model behaviour and conduct comprehensive case studies. We hope that our work could help the community to better analyse L.LMs in mathematical understanding and reasoning and inspire more advanced techniques to enhance the mathematical reasoning ability of LLMs.
3 Dataset
3.1 Formal Representation
We design formal representation that avoids am- biguity and is close natural language. Specifi- cally, our representation is built upon Assertional Logic (Zhou, 2017). Assertional Logic (AL) is a powerful knowledge representation that is more ex- pressive than first-order logic while easier to read and write for humans. In this work, we use vari- ant of AL with three components: declarations, facts and queries. Declarations define individu- als with their types (e.g. G:Ellipse). Facts are assertions that describe the conditions in the prob- lem (e.g. (G)=(()) F2)). Queries are the terms that represent the goal of the problem (e.g. (anne(Eccenticctty())))). See more details in Appendix A.
2
Related Work
There has been a wide range of datasets on math problems in the literature. MATHQA (Amini et al., 2019) and GSM8K (Cobbe et al., 2021) are math word problem datasets. They focus on open-domain understanding, where the objective is to extract a single equation based on the in- formation about quantities in the problem, rather than mathematical reasoning. Similarly, Math23K
3.2 Dataset Format
An example presented Figure| For each question, we give 1) the question text in natural language with math formulas in LTEX, 2) the ra-

---

Question:
Formal Representation:
Span:
P: Point
AP(Cy))
M.A. RAR,P 01
PointonCurve(P, G)=True
(Qu.)) y) be arbttray point cllipse
R, and foo allipse, and LFPPS, What mange valurs for cccencteity the allipas?)
Coondinate(P== yt)
P(rg))
x1,y1: Number
P(r.)
Rationale:
Ellipsee
MRG - 1
B ALl SMPPR bou b1) 2F,FF; -0 Mett H6M
Fil "7 (8>6>> vt
Ve. 18 Phe c/a. M tie alwad
Number
at 5" (z b>
B 1
ae BA <
ot 6r 1 2
(Whmn the point locatted (0,80 (0, -b), angle PF, so as muximum. this case cos PFg- afiac 4.2
-+
at 0,a v2c, Sincee
F1, F2: Point
mage repessents eccentricty dllipe,, within (7,
(G)=(1)) F2)
Answer:
Aug.:or(f degroe)
4P,PM2 90' SKAREE C.
(0, ]
Aange(Eseentricitt(())))
Figure 1: Example problem from the CONIC10K dataset.
Dataset
Size
Language
Formal Rep.
Rationale
Reasoning Steps
AQuA (Ling al., 2017)
100,000
English
X
Natural Lunguage
2.15
Math23K (Wang ct ul., 2017)
23,162
Chinese
Equation
1.59
MATHQA (Amini al., 2019)
37,297
English
X
Program
2.99
Ape210K (Zhao al., 2020)
210.488
Chinese
X
Equation
2,02
GSMBK (Cobbe et al., 2021)
8,792
English
Natural Language
2.25
Geometry3K (Lu al., 2021)
3,002
English
X
2.57
MATH (Hendrycks et al., 2021)
12,500
English
Natural Language
4.65
NumGLUE (Mishra et al., 2022b)
101,835
English
X
x
1.67
Lila (Mishra et al., 2022a)
134,000
English
X
Program
1.70
Conic10K (Ours)
10,861
Chinese
Natural Language
4.23
Table 1: Comparison of CONIC100 10K dataset with existing datasets. CONIC1OR is the largest dataset that has formal representation annotated. It is also the dataset that has the second longest average number of reasoning steps in all languages and has the longest average number of reasoning steps in Chinese.
tionale in natural language, 3) the answer to the question, 4) the formal representation and 5) the text span corresponding each sentence in the formal representation.
problem using fuzzy matching, After the above pro- cess is finished, the size of the dataset is reduced from around 20,000 to approximately 14,000.
3.3.2 Annotation
3.3
Dataset Construction
To ensure the correctness of the data and avoid ambiguities, apply strict quality control during the annotation process?. The complete process is as follows:
3.3.1
Data Collection
To construct the dataset, we first collect approxi- mately 20,000 open-ended problems about conic sections from two websites that focus on Chinese high school education in image format. Each prob- lem image contains the problem text, rationale, and answer. Then, we use mathpix to convert these images into text. Since our dataset focused on conic sections, we filter out problems that involve knowledge from other topics such as sequences and solid geometry. After that, we remove duplicated
Initiation We first build small dataset with hun- dreds of samples, write the annotation guidelines and design a rule-based Al assistant for annotation. The rule-based AI assistant is able to recognize LTTEX math expressions and complete simple for- mal representations, which greatly accelerates the annotation process and reduces annotation errors.
'htns://mattpix.com/
"Sec Appendix A.3 for more details.

---

0.25
0.20
2000
0.15
1500
0.10
500
14
2
3
Exuation
intaalA
Number of Ressoring Stips
Answwer Coragories
Figure 2: Distribution reasoning steps 50 sampled problems from CONIC10K. All numbers rounded to their nearest integers.
Figure 3: Distribution of the 7,758 training examples on answer categories.
Num. prohlems operators
10,861
94
Verification We select the annotators from group of candidates by their performance on the small dataset. These annotators are provided with annotation guidelines along with hundreds of sam- ples. Annotators with the best performance will take part in the rest of the annotation process.
Num. concepts
20
Avg. 18TEX expressions in problem
5.76
Avg. tokens in a problem
83.43
Avg. sentences in problem
3,41
Avg.
in formal rep. of problem
10.55
Avg. operators formal rep. problem
15.70
Avg. individuals in a formal rep. of problem
4.51
Annotation We ask the annotators to further fil- ter out problems about other topics, write the for- mal representation, select the corresponding text spans and fix the incorrectly recognized problem texts and answers. Each problem is annotated by two annotators, and then validated by another val- idator with an automated tool for comparison. We also randomly check 3% of the annotations. This process takes 4 months in total.
Table 2: Statistics about CONIC10K. Problems are tokenized using bast-base-chinese tokenizer' in Avg. tokens in problem.
minimum number of rules required to get enough information to obtain an answer. Since the pro- cess of applying rules is subjective, we ask two graduate students to individually annotate the rules used to solve the problems. We uniformly sam- pled 30 problems from each of the datasets listed in Table and ask the two students to annotate the reasoning steps. Results show that CONIC10K is the dataset with the second largest number of rea- soning steps. The distribution of reasoning steps in CONIC10K is depicted in Figure 2. We show additional dataset statistics in Appendix B.
Finalization After the annotation is finished, we train language model through 5-fold cross- validation, manually check the inconsistency be- tween model predictions and the annotated formal representations, and fix the errors in annotations. This helps us correct another 2% of the data. Then we randomly split the dataset into train, validation, and test sets with the ratio 7.5:1:2. The train set size is 7,758, the validation set size is 1,035, and the test set size is 2,068. We proceed to the evaluation of LLMs with this split.
To facilitate model analysis, divide the into 6 categories described in Table 3. Figure shows the distribution on these categories.
3.4 Dataset Statistics
Experiments
Table 2 presents the basic statistics about CONIC10K. The problems in our dataset tend to be long and complex. Besides these metries, also estimate the number of reasoning steps by the
This section describes experiments to evalu- the mathematical understanding and reasoning abilities of various models.
4.1 Tasks
We finetune the OPUS-mt-zh-en (Tiedemann and Thot- tingal, 2020). It is machine translation model that translates Chinese into English.
Based on data provided by CONIC10K, we intro- duce two tasks: semantic parsing and mathQA.

---

Category
Examples
Description
In zero-shot CoT inference for mathQA, we use the same prompt GAOKAO-Benchmark (Zhang et al., 2023) to instruct the models to give an an- swer together with a rationale. In MathQA, we also experiment with in-context learning (Min et al., 2022), which adds in-context demonstrations of the task in the prompt, and self-consistency (Wang et al., 2023), which conducts majority voting on the sampled results GPT-3.5-turbo. In seman- tic parsing, however, the formal representation is unknown the above models. Since it requires more than 3,000 tokens to explain the syntax and semantics of each component in the formal lan- guage, which is out of the context length limit of most models listed above, we do not evaluate the performance of zero-shot CoT in semantic parsing. In addition to the methods mentioned above, we also evaluate the following two methods in mathQA as a reference: (1) Guessing $2': Predicting the most frequent answer the train set, which is '2', (2) Human Experts: We randomly select 20 prob- lems from the test set and ask two graduate students to answer. Each problem has a 3-minute time limit. We report the average accuracy of these two stu- dents.
Simple Number
2.
Numerical values
posed of a single number
Complex Number
1/3, Vb
Numerical values com-
posed of multiple numbers
Equation
x 4 1///4-1
Equations
Coondinate
(0. (-)20))
Conrdinates of points
Interval Set
-1,1),00.11
Intervals and sets
Text
"ellipse"
Texts
Table Answer categories with examples and descrip- tion.
Semantic parsing requires a model to translate math problems in natural language into formal repre- sentations, while mathQA needs a model to give correct solutions to math problems. The semantic parsing task aims solely at assessing the model's s ability understand mathematics, and the mathQA task jointly evaluates the model's ability of mathe- matical understanding and reasoning.
4.2 Models
We evaluated the performance of several popular pretrained models on the above two tasks. The models used for evaluation as listed in Table 4.
4.4 Metries
4.3 Evaluation Details
4.4.1
Semantic Parsing
Due limited computation resources, we con- ducted full finetuning on models with size of less than 4B. For models around 7B, we performed pa- rameter efficient finetuning using LoRA (Hu et al., 2022) and 8-bit quantization (Dettmers et al., 2022). We also apply zero-shot CoT inference without fine- tuning for models with sizes between 7B and 13B. The models evaluated in zero-shot CoT setting all have undergone instruction tuning or RLHF in their respective pretraining process. When finetuning, we use instruction tuning (Wei et al., 2022a) to train the models. The instructions are architecture- specific and task-specific. as depicted in Table 5,
For semantic parsing, we evaluate the model pre- dictions by micro-F1, macro-F1 and accuracy. The accuracy is the proportion of the problems that have one-to-one match between all sentences in the prediction and the ground truth. Micro-Fi (mi-Fl) and macro-F1 (ma-F1) are defined as follows:
mi-Fl pr
(1)
++
Σ FL
ma-Fi
(2)
where n is the total number of problems, # of all matched sentences is the overall precision, of predicted sentences # of all matched sentences is the overall recall, F1, is the gold sentences
When finetuning language models, we use the following hyperparameter settings. We use AdamW as the optimizer. The learning rate is se- lected from {8e - 5, 2e - 5}. with a linear learning rate decay. For models using LoRA, we set tar- get modules to q, k v for Falcon-7b and to q, for other models. The LoRA rank is set to 16 for models with size around 7B. To ensure similar number of trainable parameters, we set the LoRA rank to 24 for Bloomz-3b and 32 for Bloomz-1b7. We use greedy decoding in all generations.
Fl score of problem i.
To compute the metrie, we need to find the num- ber of matched sentences between the prediction and ground truth. Since the formal representation is insensitive to individual naming, we enumerate all possible individual name mappings between pre- diction and ground truth and select the mapping
hhtps:/chat.oppnai.coom com/, we use GPT-3.5-turbo-0314 version.
'w use GPT-4-0314 version.

---

Model
Sizes
Architecture
Base Model Chinese-Oriented
IT & RLHF
mT5 (Xue 2021)
300M-13B
Fncoder-decoder
X
mT (Muennighoff et al., 2022)
300M-13B
decoder-decoder
mT5
X
LLaMA (Touvron el al., 2023)
7B-65B
Decoder-only
Vicuna (Chiang
2023)
Decoder-only
LLaMA
J
Ziya (Yang al., 2022)
13B
Decoder-only
LLaMA
Bloom (Scao et al., 2022)
560M- 176B
Decoder-only
X
X
Bloomz (Muennighofl al., 2022)
560M-176B
Decoder-only
Bloom
ChatGLM (Du et al,, 2022)
6B
Prefix Decnder
Falcon (Penedo et al., 2023)
7B, 40B
Decoder-only
Falcon-inst (Penedo et al., 2023)
7B, 40B
Decoder-only
Falcon
X
GPT-3.5-turbe
?
Decoder-only
GPT-4 (OpenAI, 2023)
Decoder-only
Table 4: Models used in our experiments. Chinese oriented refers to whether methods, such as increasing the portion of Chinese data and designing tokenizer for Chinese, used improve performance in Chinese tasks. IT stands for instruction tuning and RLHF stands for reinforcement learning with human feedback.
Architecture
Task
Instruction
Trainable Param.
Syntax Err.
Model
mi-F1
ma-F1
Acc.
Encoder-decnder
SP
Please translate the following problem into expressions: "prob- lem"
Finetuning PLM mTS--base
580M
93.1
93.7
66.3
Encoder-decnder
MQA
Please give an answer the fol- lowing problem: "problem" translation into expressions of "problem"" is The unswer to "problen" is
m'0-base
580M
95.8
96.2
77.2
m T--aage
1.28
95.8
96.2
77.6
Decoder-only
SP
mT0-large
1.2B
96.7
96.9
80.7
6
mTS-xl
3.7B
96.9
97.2
82.6
Decoder-only
MQA
mTT0-l
3.7B
97.4
97.5
84.6
Finetaning I.M using LoRA
Table 5:
Instructions used in finetuning. problem is replaced by the problem text when training.
Bloomz-1b7
7M
90.0
62.7
Bloomz-3b
7M
91.5
67.6
Bloomz-7b1
94.3
71.3
Falcon-7h
58.0
LLaMA-7b
94.0
71.1
that achieves the maximum number of matched sentences. We optimize the evaluation script by only considering individuals with the same type so that the evaluation time on the validation set and test set is acceptable.
ChatGLM-6b
74.7
Vicuna-7b
96.2
76.9
Table 6: Results semantic parsing in CONIC10K. The fully finetuned mT00xl achieve the highest accu- racy, while the LoRA finetuned Vicuna-7b achieves the lowest syntax error rate.
4.4.2 MathQA
In mathQA, since it is nontrivial to automatically determine whether two answers are the same (e.g., 1/V2 vs. v2/2, x - -- vs. æ = y, and 3r + 4y = 5 vs. 2x+ 0), we rely on human evaluation to determine the correctness of model answers.
The best model m mT5xxl can successfully translate 84.6% of the problems into formal representations. For the problems it fails to accurately translate, the predictions only differ from the ground truth in minor details. The F1 score and accuracy from Bloomz family and Falcon-7b are much lower than other models. The performance of finetuned in- struction tuned models consistently better than that of finetuned base models.
5 Results and Discussions
In this section, we introduce and explain the results of the experiments. The main results of semantic parsing and mathQA are shown in Table and Table 7 respectively.
Models pretrained code show strong ability in learning syntax. Models except for the mT5 family have been pretrained on code. The syntax emror rates of these models are on average lower than that the mT5 family, even though their FI
5.1 Semantic Parsing
Language models show good ability of under- standing on math problems after proper training.

---

score and accuracy may be lower than the mTss family. Since the formal representation resembles programming languages in syntax, pretraining on code may be able to help model to learn the syntax formal representations more easily.
The scaling law is less clear compared to mantic parsing. Though we observe that increas- ing the model size continuously and effectively improves model performance in semantic parsing, such phenomenon disappears in mathQA tasks. In mT5 and mTO series, large models do not nec- essarily outperform small models. Similar observa- tions have been made in MATH (Hendrycks et al., 2021) where the authors find that accuracy on math problems increases only modestly with model size.
Increasing model size effectively improves model's performance in semantic parsing. From the results of the model families mT5, mTO and Bloomz, we find that increasing the model size from the smallest to largest in our experiment can significantly improve the accuracy by at least 7.4%.
Chinese-oriented language models have better performance in mathQA in CONIC10K, In the zero-shot CoT setting, the two Chinese-oriented models, Ziya-13b and ChatGLM-6b, achieve the best performance below GPT-3.5-turbo, In the finetuning using LoRA setting, ChatGLM-6b achieve an accuracy of 22.5% and outperform other models by a large margin.
5.2 MathQA
Language models generally show poor perfor- mance on mathQA CONIC10K. Under the zero- shot COT setting, most models achieve an accuracy close to O. Even after finetuning, the accuracy of the best model is still significantly lower than that of human experts by 35.0%.
Translating problems into English does not make the performance of GPT-4 on par with human experts in mathQA. We translate the problems into English and evaluate GPT-4 zero- shot CoT setting to determine whether the poor performance is due to language or long reasoning steps. The results in Table 8 show the performance is significantly improved from 15.5% to 26.0% by translating the problems into English. However, this accuracy is still low compared to 57.5% from human experts. Therefore, the primary challenge of mathQA in CONIC10K still lies in how do mathematical reasoning correctly.
Simple problems under finetuning setting may not be simple under zero-shot CoT setting. Most models finetuned on CONIC10K have the best performance in Simple Numbers among the answer categories. However, when it comes zero- shot CoT setting, GPT-4 and GPT-3.5-turbo ob- tain best accuracy in Coordinate. One possible rea- son that after sufficient training on CONICIOK. the model can develop shallow understanding of the task (Patel et al., 2021), including the frequent answers of a specific kind of questions. Since Sim- ple Numbers are simpler in form and have fewer potential answers compared Coordinates, being familiar with the answer distribution can effectively increase the probability hit the correct answer. However, zero-shot CoT setting, the model is un- aware of these distributions, so it has no advantage in difficult problems that have simple answers.
5.3 Case Study
We inspect and analyse both success and failure cases in the experiment, which leads us to some interesting findings.
The accuracy is close to 0 in zero-shot CoT. Un- der the zero-shot CoT setting, Bloomz-7bi and Falcon-7b-inst show extremely poor performances with 0 accuracy in all problems. These models tend to generate repetitive patterns, and in most cases fail to give an answer. Other models ex- cept for GPT-4 generate text that looks like valid rationale, but the majority of reasoning steps incorrect. They often produce hallucinations in premises and rules, and derive wrong results. In Table 9, even with in-context demonstrations or majority voting, the performances are still low. We showcase some failing cases in Table 10.
LLMs have limited ability in understanding long 1TEX expressions. 9.7% of the incor- reet predictions from mTO0xl are due to er- rors in translating simple but long 1STEX ex- pressions. Common failures include missing terms, flipped signs and incorrect copies. For example, the ISTEX expression in the problem is 22t.2222srrtt2xx4Assrrtt2222eee but the translated sentence becomes -4*sqrt(2)*y +2* (2xx
In this example, we observe both a flipped sign and an incorrect con- stant. We do not observe similar errors in relatively short INTXX expressions.

---

Trainable Param.
Accuracy of Answer Category
Model
Simple Num.
Complex Num.
Expression
Coordinate
Interval Set
Text
All
Finetuning P.M
mTS-basee
580M
8.4
1.6
0.0
7.2
m'To-base
580M
22.8
13.6
7.3
2.5
4.9
0.0
13.0
m mT--large
1.3B
21.0
14.8
8.1
3.7
4.9
0.0
13.0
m0--large
1.3B
16.7
17.0
12.5
3.7
6.6
0.0
13.8
mTS5xl
19.9
11.0
14.8
m'00xl
3.7B
18.1
13.6
10.3
2.5
6.6
0.0
12.5
Finetuning LLM using LoRA
Bloomz-1b7
7M
23.2
0.0
0.0
0.0
0.0
0.0
6.3
Bloomz-3b
7M
26.1
7.6
8.1
3.7
1.6
0.0
12.0
Falcon-7b
16M
31.5
4.8
8.4
15.0
8.2
14.0
Bloomz-7bl
27.9
12.5
3.3
0.0
LLaMA-77b
8M
34.1
9.1
9.9
8.7
4.9
0.0
15.8
Vicuna-7b
8M
37.7
9.4
12.8
10.0
8.2
0.0
17.9
ChatGLM-6b
39.3
23.1
13.1
10.6
6.5
0.0
22.5
Zero-shot Col
Bloomz-7bl
0.0
0.0
0.0
Falcon-7b-inst
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Vicuna-7b
1.5
0.0
0.0
0.0
0.0
0.0
0.4
Vicuna-13b
0.9
Ziya-13b
2.8
0.9
0.7
0.0
0.0
0.0
1.1
C-a0GLM-6b
4.0
0.7
0.2
1.3
0.0
14.3
1.5
GPT-3.5-turbo
8.5
4.6
4.0
12.3
0.6
14.3
6.2
GPT-4
20.4
5.3
0.0
15.5
References Guessing '2'
0.0
0.0
4.5
Human Expert
62.5
56.3
50.0
50.0
66.7
57.5
Table 7: Results mathQA in CONIC10K ChatGLM-6B achieves the best overall accuracy after finetuning using LoRA among the models. In fully finetuning setting, mT0xxl shows strongest performance. the zero-shot CoT setting, GPT-4 has the highest overall accuracy. However, the performances of the above models significantly lower than human expert's performance. GPT-4 is evaluated on 200 randomly sampled problems. Human Expert is evaluated on 50 randomly sampled problems. Text accuracy Human Expert is empty because sampled problems do not contain answers of category Text.
Language
Overall Accuracy
Language
Overall Accuracy
GPT-3.5-turbo + CoT
6.2
Chinese 15.5
GPT-3.5-turbo CoT ICL
5.9
English
26.0
GPT-3.5-turbo + CoT + SC
6.8
Table Results Chinese problems and problems translated to English in mathQA in CONIC10K using GPT-4 with zero-shot CoT. Both are evaluated on the same 200 sampled problems.
Table 8: Results on mathQA in CONIC10K using GPT-3.5-turbo with in-context-leamiing (ICL.) or self- consistency (SC)
Models can hardly find shortcuts in reasoning in mathQA. We observe that models usually em- ploy naive approaches to solve problems and fail to find shortcut solutions, which leads to more compli- cated computation and longer reasoning steps. The additional reasoning steps and computation make the models more likely to make mistakes during reasoning. Some examples naive solutions from GPT-4 and the corresponding shortcut solutions are listed in Table and 12.
GPT-4 and GPT-3.5-turbo probably lack knowl- edge about certain concepts. When asked prob- lems about focal distance, GPT-4 and GPT-3.5- turbo keep giving incorrect answers and often give value that is half of the ground truth. Based on these observations, we suspect that these two mod- els lack knowledge about focal distance. We ask GPT-4 and GPT-3.5-turbo to explain what focal distance in both Chinese and English, and they

---

keep defining it as the distance between the center of an ellipse or hyperbola and one of its foci instead of the correct definition, the distance between the two foci. A probable reason is that 'focal distance' is not a commonly used term within the English cor- pus, making the models unlikely to obtain correct knowledge about it.
Ethics Statement
CoNIC10K is dataset that requires massive data sources and heavy annotation. We claim that our work is free of ethical risks from the following perspectives:
Data Source The problems in CONIC10K are collected from two websites that do not limit the us- age of data for education and research purpose. We strictly follow the term of use and manually check all the data to avoid inappropriate information in the annotation stage.
6 Conclusion
We present CONIC10K, a math problem under- standing and reasoning benchmark. It provides problems that require complex reasoning, while only involving knowledge about conic sections in Chinese senior high school education. We test popular LLMs on both semantic parsing and math question answering, inspecting model perfor- mance and behaviours. Results show that existing LLMs, including GPT-4, have poor performance in mathematical reasoning, while most models could achieve good performance in mathematical under- standing (but not perfect yet). We analyse the model predictions in detail and find LLMs tend hallucinate in reasoning, often fail find the shorteuts solution, and may lack the knowledge to solve problems. We hope our dataset, CONIC10K, can help to discover the weaknesses of LLMs in mathematical understanding and reasoning and in- spire more advanced techniques to enhance the mathematical reasoning ability of LLMs.
Annotation We hire a group of 14 annotators for formal representation annotation and sign a contract to prescribe the rights from both sides. We clearly state the purpose of our study and the future data use. These annotators are well-paid for their work. The authors take responsibility maintain the annotation website, provide necessary documents, answer questions from the annotators and clean up the data.
Acknowledgements
This work was supported by the National Nat- ural Science Foundation of China (61976139, 62250057), and by Shanghai Frontiers Science Center of Human-centered Artificial Intelligence and MoE Key Lab of Intelligent Perception and Haman-Machine Collaboration (ShanghaiTech University).
Limitations
CONIC10K is dataset with high-quality formal representation annotations, but there are still some limitations:
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzzirskdiiirkkii. Yejin Choi, and Hannanch Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2357- 2367, Minneapolis. Minnesota. Association for Computational Linguistics.
We design the formal representation be accurate, unambiguous and close to natural language, but such representation is not com- monly used and does not fit any existing sym- bolic reasoners. The conclusion may not ap- ply other formal representations such as propositional logic and first-order logic, or rationales like executable programs.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023, Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
In conic sections, the commonly used mathe- matical reasoning strategies could be limited. For example, our problems may require solv- ing simultaneous equations systems, but not likely mathematical inductions. Therefore, our dataset cannot evaluate some reasoning strategies such as mathematical induction.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.

---

2021. Training verifiers to solve math word prob- lems. CoRR, abs/2110.14168.
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint ar arXiv:2305.20056
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via bloek-wise quantization, In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event April 25-29, 2022. OpenReview.net.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167, Vancouver. Canada. Association for Computational Linguistics.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang, 2022. GLM: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 320-335, Dublin, Treland. Association for Computational Lin- guistics.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 6774- 6786, Online, Association for Computational Lin- guistics.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the MATH dataset. Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022. A survey of deep learn- ing for mathematical reasoning. arXiv preprint 2xxv:2212.100335
Mark Hopkins, Ronan Le Bras, Cristian Petrescu- Prahova, Gabriel Stanovsky, Hannaneh Hajishirzi, 2019. SemEval-2019
and Rik task 10: Math question answering. In Proceedings of the 13th International Workshop on Semantic Evalua- rion, pages 893-899, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809 Seattle, United States. Association for Computational Linguistics.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022a. LILA: A unified bench- mark for mathematical reasoning, In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 5807-5832, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyr. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natu- ral Language Processing, pages 1545-1556, Seattle. Washington, USA. Association for Computational Linguistics.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney. Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan, 2022b. NumGLUE: A suite of fun- damental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- 1: Long Papers), pages 3505 -323,, Dublin, Ireland. Association for Computational Linguistics.
Guillaume Lample and François Charton. 2020. Deep learning for symbolic mathematics. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak. Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual gen- eralization through multitask finetuning. CoRR, abs/2211.01775
Aitor Lewkowycz. Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone. Cem Anil. Imanol Schlag, Theo Gutman-Sole Yuhuai Wu. Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems.

---

OpenAl. 2023. GPT-4 technical report. CoRR. abs/2303.08774.
Copenhagen, Denmark. Association for Computa- tional Linguistics.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguisties.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM:: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:230.001116.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022b. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguisties.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier. Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay. Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211 1.05100..
Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin Zhang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, and Tetsuya Sakai. 2022. Zero-shot learners for natural language understanding via a unified multiple choice perspective. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Process- ing, pages 7042-7055, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, abs/2305. 12474.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-
MT building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Associa- tion for Machine Translation.
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: 10k: A large-scale and template-rich dataset of math word problems. CoRR, abs/2009. 11506.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Karie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR. abs/2302.13977
Yi Zhou. 2017. From first-order logic to assertional logic. In Artificial General Intelligence: 10th In- lernational Conference, AGI 2017, Melbourne, VIC, Australia, August 15-18, 2017, Proceedings, pages 87-97, Cham. Springer International Publishing.
A Formal Representation
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net
A.1 The Assertional Logic
Assertional Logic (AL) (Zhou, 2017) is a formal representation where all kinds of knowledge are formalized by equality assertions. It builds upon the equality properties and the set theory. AL rep- resentations are human--fiendly and it has been proved that the expressiveness of AL is stronger than first-order logic (or kth-order logic for any k $ 1).
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pages 845-854,

---

Here, we briefly introduce the syntax of AL. Given a specific domain, the syntactic structure AL is composed of three components: individu- als, concepts and operators. Individuals represent objects in the domain, concepts represent groups of objects and operators represent relationships and connections among individuals and concepts. Op- erators are similar to functions and predicates in first-order logic (FOL), but they could accept high- order constructs (concept, concept of concepts), which leads to the strong expressiveness of AL.
assertion Is(var, type) True. For simplic- ily, we allow defining multiple individuals in one sentence, with commas separating different indi- viduals.
The facts are assertions that describe the conditions in the problem. For clarity, we allow the use of syntactic sugar, which includes <, S. + -, x, +, aa. That is, sentence could be an inequality such as > b. which indicates assertion (a > b) True.
Facts
An assertion is the form a = b, where a, are two terms (individuals, either atomic or com- pound). The knowledge base of AL is just set of assertions.
The queries are the terms that represent the target of the problem. They ought to be an assertion with the left ( the query term and the right-hand Saat--aann--iddRHHBa side(RHS) an unknown individual in AL, but we use the simplest format during the annotation.
Queries
A.2 Our Representation
We apply AL as our formal representation because of its strong readability. Our principle is that the formal representation should 1) avoid ambiguity. The formal representation should resolve the ambi- guity in natural language and with the information inside the annotations, should be possible to work out the solution by hand; 2) close to natural lan- guage. It should be able to represent the problem without rephrasing it; 3) simple and clear. Design- ing a representation with thousands of operators is definitely expressive and powerful, but it sacrifices the strength of logic and fails to extract common knowledge underneath.
A.3 Annotation Quality Control
Our previous study shows that the annotation of formal language is extremely hard for humans. It is difficult for an experienced annotator to reach an accuracy above 50%. As a result, we employ multiple measures to control the dataset quality. including:
We provide rule-based Al assistant com- plete most of the annotations with high preci- sion.
Therefore, we apply only 94 operators and 20 concepts (see Table 2) to represent all the problems in the dataset. To better accommodate the natural language, we also designed pseudo operators: OneOf, WhenMin, WhenMax. These operators do not fit the semantics of AL, but greatly simplify the representation and are closer natural language. Also, it is trivial to convert these operators to terms in AL.
2. We only hire annotators with the highest per- formance the small dataset we built in ad- vance.
3. During the annotation, we ask the annotators to raise questions whenever they are not con- fident about how to annotate. We provide de- tailed documents and dedicated help ensure the correctness of the annotation.
There also has been evidence showing that rephrasing significantly impacts learning (Kwiatkowski et al., 2013). To avoid rephrasing, we write detailed documents for the annotators, ask them to raise questions when they are not confident and frequently check the data during annotation.
4.
In addition to formal representations, we ask the annotators to annotate the text spans. We find it helps increase the annotation accu- racy.
5. Each problem will be annotated by two an- notators individually, then passed to another validator. We design web UI which could automatically compare two annotations and extract the difference. The validator will de- termine which one is correct, or a third anno- tation is required.
We design our representation in three compo- nents: declarations, facts and queries.
The declarations define individuals with their types. has the format of var: type, where var is an individual and type is a concept. These sentences are a special representation of the
Declarations

---

Ellipsee e
arabola
Histogram Lengths
Two
One Losca
Pare
Or
Origin
Asymptote tote
Hoving
Coordinat
Circle
ine
Range
Nunber of Questere
Focus
purre et
Meriiaa
Eccentricity
Hype
ppeboll
moboa
Hopoint
200
Distance
Equation
Directrix
Veriex Real
100
Figure 4: Word cloud of the keywords in CONIC10K
200
250
300
Question Length
6. Every time the annotators finish 1000 annota- tions, we randomly sample problems for additional checks. After all the annotations were finished, we randomly sample 200 prob- lems for additional checks. In the additional check, we independently annotated the sam- pled problems, and then compare them with the existing annotation. We ask the annota- tors do thorough check if the accuracy is below 80%?.
Figure 5: Distribution of Question Length
in natural language, but its solution often fails find the shortcuts designed in the problems. Such solution often requires complicated calculations. Due to its limited ability in arithmetic calculation, the final answer from GPT-4 likely to be incor- rect. We provide some samples in Table 11 and 12.
7.
We provide competitive payments (> 150k CNY in total, A 20k USD) to the annotators. We allow adequate time for the annotation process.
8. After the annotation is finished, we finetune zh-en translation model for further validation. We split the whole dataset into five random splits of the same size. Then, we pick four of them to finetune the model and collect pre- dictions for the last split. We manually check all the problems whose prediction does not match the annotation. We repeat this process five times and obtain the final dataset.
B Additional Dataset Statistics
We show the frequency keywords in Figure 4, and the distribution of question length in Figure 5. In question length, we count all latex commands such as \frac, Aleftarrow as one token.
C Additional Results
We showcase some interesting outputs from models. Models including Bloomz-7b and ChatGLM-6b often generate redundant text and hallucinate when reasoning. We provide some samples in Table 10. GPT-4 shows strong ability to perform reasoning
the final check, about 99% of the 200 sampled formal representation annotations (without considering text spans) pass the check.

---

Problem
Rationale & Solution
Given that M is a point on the
Redundant solution by Bloomz-7b: |MF
= MA
parabola x²² = 4y, F is its fo-
[MF]
+
MMA
+ M F|
+ |MA] + [MF]
+ MA
+ M
cus, and point 1)1,5), what is the minimum value of [M + |MA|?
|MA]
MF
+
MA + |MF] + |MA] +
MF
+ MA
[MF]
MA
+ MF
+ |MA] + |MF]
+ |MA]
+ [MF]
|M
[MF]
|MMA
|MF] + [M
If the eccentricity of an ellipse and one of its foci coincides with the focus of the parabola
Hallucination in reasoning by ChatGLM-6h: According to the problem, we know that the eccentricity of the ellipse 1.2 and one of its foci coincides with the focus of the parabola y² - 8x. First, let's find the coordinates of the focus of the parabola y² 8x. According to the definition of a parabola, the X-coordinate of the focus is 4. where is the focal length. Since the standard equation of the parabola is y² 4pm, can determine that p 2. Therefore, the x-coordinate of the focus is 2 1...
y² 8x, then what is the stan- dard equation of the ellipse?
Table 10: Translated failing cases. The red text is the reasoning step where hallucination takes place.
Problem
Rationale & Solution
Given point P on the parabola with equation y² 4z, let's de- note the distance from point P to the y-axis as d. For given point A(1, 5), what the mini- mum value of [P] + d?
Naive solution by GPT-4: First, we can write the equation of the parabola as y² 4r. Let's assume the coordinates of point P as (ro, vo),
have y/ 4zg. The distance from point P to the y- axis is denoted as d. which is equal to d zg. We need find the minimum value of PA| + d. which is equivalent to finding the min- imum value of [PA| + 20- Using the distance formula between two points, we have PA = / (To 4)2 + (yo 5)2. Substituting
vē = 4ro, we get [PA| = V - 4) (42g- 5) Now we need to find the minimum value of v(= 4)2 (4m. + To simplify the calculation, we can first find the minimum value of 2 5) 2 +2 Expanding (xo (zo-4))+(42n) 5) +20 we get 17x 10xo 11. We can find the extreme points by taking the derivative. Differentiating 17xz 40m- 41, we get 34.0 40.
Setting the derivative equal O, we find xg = 17- Substituting z0 = - into 17x - 40zo + 41, we get the minimum value as
144 17
144 Therefore, the minimum value of PA + dis 17
Reference shortcut solution: Since the distance between a point on a parabola and its focus is the same as the distance between the point and directrix of the parabola, PA| + d can be rewritten into [PA| + d1 d2 = [PA] + [FP) | - d2, where di is the distance between P and the directrix, d2 the distance between the directrix and the y-axis, F is the focues. By the definition of parabola,
de = 1, F = (1, Since |PA| + FPI de 2 APP d2 v 1) 55 0 = v 31 1. Therefore, the minimum
value of [PA| + dis V44-1
Table 11: Translated example of solutions from GPT-4 and shorteut solutions. The red text is the reasoning step where the solution goes wrong.

---

Problem
Rationale & Solution
The equation of the line that
Naive solution by GPT-4: First, we need to find two symmetric points on the hyperbola.
passes through the point
Let these points be Afr v1) and B(rg, v2), which are symmetric with respect to the point (3, -1), According to the properties of symmetric points, we have:
(3, -1) and bisects the chord
ot of the hyperhola 4 y² is?
2
81 + 1/2 - (-1)
Since points A and B lie the hyperbola, they satisfy the equation of the hyperbola:
z
v
4
1/
4
Substituting g1
32 into equation, we get:
( 92)²
Substituting x1 6 22 into the above equation, we get:
(6 x2)²
v2)
This is system of linear equations in terms of 02 and y2. By solving for z2 and ya. we can obtain the coordinates of points A and B. Snlving, lind i2 6 y2 FV2
Therefore,
[tty /2 1)
B [--/ /2 1/2-))
Next, we need to find the equation of the line passing through points A and B. Let the equation of the line be kz + b. Substituting the coordinates of points A and B into the equation, we obtain two equations:
v2 1
2
6 v2
-2
2
By suhtracting two equations to eliminate we gel:
-2v2 = k
Solving k we find 22, Substitating this value Therefore, equation line passing through point (3, 1) and bisecting of the hyperbola y -2
equation, get
Reference shorteut solution: Let the points intersected with the chord be A(mi, and B( w... Then, they satisfy the equation of the hyperbola:
2
Vi
4
V/
4
Subtract two equations:
(uf
(x]
[ar
x9)
(y1
22)(11 12)
Since M(3 11) is the midpoint of the chord, .r + mg into the above equation:
6, + 82 -2 Substitute this
66
x2)
2(21 w2)
4
Then, have the slope the chord:
21 x2
21
1/2
Since (333-1) is the chord, 3r
0 is the line equation.
Table 12: Translated example of solutions from GPT-4 and shortcut solutions. The red text is the reasoning step where the solution goes wrong.


Reviews:
[{"review_id": "IOz9HZVpg8", "reviewer": "Reviewer_WDny", "summary": "", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! Glad to have your helpful feedback!\n\n\n> 1. I was looking for more insights on data quality analysis (beyond section 3.4), specially because the dataset is not available for us to look at some samples and get more idea about the data quality.\n\nSorry, but we are confused about what you mean by 'data quality':\n1. If it means the annotation quality, then this is described in section 3.3.2 and Appendix A.3. We apply multiple methods to ensure the annotation quality (this is what we are proud of).\n2. If it means the data source quality, then this is described in section 3.3.1. All the questions come from real-world exams or exercises. We also ask annotators to manually check all the question texts and rule out incomplete or repeated questions.\n3. If it means the diversity of the dataset, then it is related to the data source. Since we collect data from real-world scenarios instead of using templates/model generation, our data distribution is consistent with real-world applications. We looked into the references you provided and found that some analysis methods like n-grams may not suit our work. This is because our dataset is not an MWP dataset. Instead, it uses a closed vocabulary and all questions are expressed in the mathematical language. So if you do n-gram, you will find that many natural language phrases are quite similar (we did this, only to find over 75% of the questions start with '则'). The actual difference, or diversity, lies in the information provided by the question (or the reasoning path required).  \nAfter looking into the references you provided, we decide to add the following statistics in our next version: 1) a word cloud that has been translated to English; 2) a question length distribution. We count each latex command as one single character. The figures are shown in the links below:  \nhttps://i.imgur.com/Gy38zI5.png  \nhttps://i.imgur.com/JlfjuuI.png\n4. It is possible that our guesses above are all wrong. If so, please point it out and we are happy to discuss this further.\n\nWe provide a few samples in Figure 1 and Table 9-11. We will add more samples in the Appendix in the next version. Meanwhile, the readers can always visit our GitHub link and check the entire dataset after the publication of this paper.\n\n> 2. how reliable is the evaluation? Is there a human study to gauge efficacy of the evaluation protocol of this benchmark?\n\nWe perform 2 tasks in our experiments: semantic parsing and mathQA.\n- For mathQA, we simply do human evaluation (section 4.4.2). We manually compare the answers and reference solutions. Since the reference solution is just a number, math expression or natural language phrase, the evaluation is totally objective and accurate.\n- For semantic parsing, we develop a script to automatically evaluate the results (section 4.4.1). When the project is initiated (line 194), we find that it is very difficult for humans to recognize the mistakes in formal representation, especially when the prediction is very close to the reference. Script evaluation is much faster and more accurate than other methods, including human evaluation. During the annotation stage, we use the same script to help the validator compare two annotations (line 213), and the feedback shows that the evaluation is accurate and reliable. This is the best evaluation method we could come up with except for spending lots of time to build a logic reasoner and compare the final result (which might also introduce false positive results, where the formal representation is incorrect while the final result accidentally hits the reference solution).\n\n> 3. L443 \"We inspect and analyse both success and failure cases in the experiment\": did you select a subset of the benchmark? can you describe more how you did this evaluation?\n\nThis is just a case study. We follow these steps:\n1. We inspect all failure cases (both the semantic parsing and the mathQA tasks), manually checking them one by one;\n2. We try to conclude some common features from these failure cases;\n3. We confirm these features by inspecting the success cases, ensuring that these features do not appear in the success cases.\n\n\n> 4. L276: instruction tuning: did you try detailed instructions than the ones in Table 5? Since this is a hard benchmark requiring specialized knowledge, detailed instructions in the style of https://aclanthology.org/2022.acl-long.244.pdf might be helpful to improve models.\n\nThank you for the advice. After reading feedback from all reviewers, we decide to add the results of instruction tuning with detailed instructions in our next version. The detailed instruction consists of task name, task definition, and examples. The results are consistent with our previous observations, and they indeed better support our conclusions (line 493).\n\nThe results are shown below:\n\n| Method                                       | MathQA Overall Accuracy |\n| -------------------------------------------- | ----------------------- |\n| Bloomz-7b1+LoRA+simple instruction (Table 7) | 15.4                    |\n| Bloomz-7b1+LoRA+detailed instruction         | 15.7                    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting benchmark highlighting weakness of LLMs like GPT4\n- inclusion of formal representation and reasoning steps has merit\n- senior high school education content acts as default quality checker\n- analysis in section 5 is interesting and useful", "reasons_to_reject": "- I was looking for more insights on data quality analysis (beyond section 3.4), specially because the dataset is not available for us to look at some samples and get more idea about the data quality.\n\n- how reliable is the evaluation? Is there a human study to gauge efficacy of the evaluation protocol of this benchmark? \n\n- L443 \"We inspect and analyse both success and failure cases in the experiment\": did you select a subset of the benchmark? can you describe more how you did this evaluation?\n\n- L276: instruction tuning: did you try detailed instructions than the ones in Table 5? Since this is a hard benchmark requiring specialized knowledge, detailed instructions in the style of https://aclanthology.org/2022.acl-long.244.pdf might be helpful to improve models.", "questions_to_authors": "See above", "paper_topic_and_main_contributions": "The authors propose CONIC10K, a challenging math problem dataset on conic sections in Chinese senior high school education. This contains various problems with different reasoning depths, while only the knowledge from conic sections is required. For each problem, authors provide a formal representation, the reasoning steps, and the final solution. Experiments show that this benchmark is hard for GPT4.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "See above\nalso \nhttps://arxiv.org/abs/2209.09513\nhttps://aclanthology.org/2020.acl-main.92/\nhttps://aclanthology.org/N19-1246.pdf\nhttps://aclanthology.org/2023.eacl-main.30/\nhttps://aclanthology.org/K19-1033/"}}, {"review_id": "UKzyet3waB", "reviewer": "Reviewer_As6G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\n> 1. Lack of experiments on the relationship between semantic parsing and mathQA. For example, does the semantic parsing help mathQA?\n\nWe would like to clarify that these two tasks, semantic parsing and mathQA, are not in a 'pipeline' relationship. Instead, they are two separate tasks that evaluate different abilities of LLMs (line 64). Our purpose is to figure out how well LLMs understand math problems and do mathematical reasoning respectively. Semantic parsing, in our work, is not designed to help mathQA.\n\nWhile it is beyond the scope of this paper, we have tried multi-task training to see if the performance could be improved. The ratio of training data from semantic parsing and mathQA was set to 3:2. Experimental results show that simple multi-task training hurts the performance of mathQA.\n\n| Method                       | Semantic Parsing Acc | MathQA Acc |\n| ---------------------------- | -------------------- | ---------- |\n| Bloomz-7b1+single-task | 71.3*                 | 15.4       |\n| Bloomz-7b1+multi-task        | 71.6                 | 12.3       |\n\n_*This result is different from that reported in Table 6. This is because we fixed a bug about tailing spaces due to the special tokenizer of Bloomz. We will update Bloomz-related results in Table 6 in our next version._\n\n\n> 2. Experimental analysis reveals little insight. For example, how can this dataset be used to help models improve their reasoning? What are possible future research directions?\n\nWe would like to point out that our dataset is designed as a benchmark to evaluate models' ability of mathematical understanding and reasoning, just like the famous MATH dataset [1] (which serves as a benchmark for mathematical problem solving).\n\nOur results show that existing large language models have poor performance in mathematical reasoning, while most models could achieve good performance in mathematical understanding (but not perfect yet).\n\nWe call for new LLMs (or any other methods, e.g., neural-symbolic methods) in the future to tackle the challenge of accurate mathematical understanding (math problems require absolute accurate understanding) and complex reasoning. As discussed in the introduction, this challenge has seldom been emphasized before because of a lack of dedicated data sources, and we present this challenge to the community.\n\n[1]: Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This dataset labeled with formal representations is useful for LLMs reasoning research.\n2. Paper is well-written and the idea is intuitive.", "reasons_to_reject": "1. Lack of experiments on the relationship between semantic parsing and mathQA. For example, does the semantic parsing help mathQA?\n2. Experimental analysis reveals little insight. For example, how can this dataset be used to help models improve their reasoning? What are possible future research directions?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This papers proposes CONIC10K, a challenging math problem dataset on conic sections in Chinese senior high school education, with high-quality annotations of formal representations.\nResults show that existing LLMs have poor performance in mathematical reasoning, while most models could achieve good performance in mathematical understanding (semantic parsing).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iH7XHjbVeb", "reviewer": "Reviewer_4LaT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\n> 1. Existing models have strong performance on semantic parsing, making this data less useful as a semantic parsing benchmark dataset.\n\nIn Table 6, the mi-F1 and ma-F1 are indeed nearly perfect. However, to correctly understand a math problem, we must correctly understand every sentence in the problem -- it is almost impossible to solve a problem based on imperfect understanding. This indicates that the main metric of mathematical understanding should be accuracy instead of F1 scores. And in Table 6, we can find that the highest accuracy is 84%, which is far from being perfect yet (line 497). Our case study (line 446) also provides new insights on improving LLMs' ability to mathematical understanding.\n\nFurthermore, we would like to note that resources for math problem semantic parsing are very limited. Existing works containing formal language data are either not open-source [1], or too simple such that a simple string processing script is sufficient [2]. We believe that good performance of semantic parsing must rely on sufficient high-quality data, and that is what we create in this work.\n\n\n> 2. This work only explores zero-shot chain-of-thought without adding demonstrations. It will be good to know if adding more demonstrations can work.\n> 3. This paper lack of results of self-consistency, which sample multiple reasoning paths and conduct majority vote on the predictions.\n\nThank you for the advice. After reading feedback from all reviewers, we decide to add experimental results of in-context learning and majority voting in our next version. These results are consistent with our previous observations, and they indeed better support our conclusions (line 493).\n\nThe results are shown below:\n\n| Method                        | MathQA Acc |\n| ----------------------------- | ---------- |\n| GPT-3.5-turbo (Table 7)       | 6.2        |\n| GPT-3.5-turbo+in-context-learning        | 5.9        |\n| GPT-3.5-turbo+majority-voting | 6.8        |\n\nWe find that for in-context learning, randomly adding samples from the train set slightly hurts the performance. By looking into the failure cases, we find the model is likely to copy the reasoning steps or numerical values from the few-shot demonstrations, which are often incorrect. Though it is possible that better in-context learning methods can improve performance marginally, we think the evidence is enough to support our conclusions. \n\n\n\n[1]: Sun, R., Zhao, Y., Zhang, Q., Ding, K., Wang, S., & Wei, C. (2019). A neural semantic parser for math problems incorporating multi-sentence information. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(4), 1-16.  \n[2]: Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., & Zhu, S. C. (2021, August). Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 6774-6786).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This work proposes a 10k math problem dataset on conic sections including various problems with different reasoning depths.\n\n(2) Detailed performance analysis of multiple state-of-the-art language models on subcategories of 10K datasets.\n", "reasons_to_reject": "(1) Existing models have strong performance on semantic parsing, making this data less useful as a semantic parsing benchmark dataset.\n\n(2) This work only explores zero-shot chain-of-thought without adding demonstrations. It will be good to know if adding more demonstrations can work.\n\n(3) This paper lack of results of self-consistency [1], which sample multiple reasoning paths and conduct majority vote on the predictions.\n\nReference:\n[1] Wang, Xuezhi, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai-hsin Chi and Denny Zhou. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” ArXiv abs/2203.11171 (2022): n. pag. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes CONIC10K, a challenging math problem dataset on conic sections in Chinese senior high school education. The dataset contains various problems with different reasoning depths and only requires conic sections knowledge. Experiments on multiple models show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
The authors introduce CONIC10K, a challenging dataset consisting of mathematics problems focused on conic sections in Chinese senior high school education. The dataset encompasses a set of problems having different levels of reasoning complexity, all requiring a basic knowledge limited to conic sections. For each problem, the authors give a formal representation, outline the reasoning steps, and present the final solution. Experimental results across multiple models demonstrate that even state-of-the-art large language models like GPT-4 exhibit mediocre performance when confronted with intricate reasoning tasks.

**Strengths:**
The reviewers are in consensus on the paper's strengths, which can be summarized as follows:
1. The paper introduces a 10,000-problem dataset on conic sections, encompassing a wide range of problems with different levels of reasoning complexity. This dataset serves as an interesting benchmark, shedding light on the limitations of Large Language Models (LLMs) such as GPT-4.
2. The paper includes a comprehensive performance analysis of several state-of-the-art language models across subcategories of the 10,000-problem dataset.
3. An additional noteworthy aspect is that the senior high school education content serves as a quality checker.

**Weaknesses:**
Reviewers are in consensus that the paper lacks insights into data quality analysis, especially given the unavailability of the dataset for sample examination. Questions also arise regarding the reliability of the benchmark's evaluation protocol. Additionally, the study lacks experiments investigating the relationship between semantic parsing and mathQA, as existing models excel in semantic parsing, diminishing the dataset's utility as a semantic parsing benchmark. Furthermore, it would be beneficial to explore the effectiveness of adding more demonstrations. Lastly, the paper lacks results regarding self-consistency, which involve sampling multiple reasoning paths and conducting a majority vote on predictions.

**Author-Reviewer discussion and acknowledgment:**
Reviewers presented comments and questions, all of which the authors addressed in their rebuttal response. All reviewers acknowledged the authors' rebuttal.

**Conclusion:**
The paper is well-written and the idea is intuitive. However, the reviewers recommend that the authors make further improvements to enhance the paper and incorporate missing references into the bibliography.
Decision: Accept-Findings