Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning
Authors: An-Zi Yen
Wei-Ling Hsu
Venue: EMNLP 2023 Findings (2023)
Keywords: Mathematics Learning
Adaptive Feedback
Large Language Models
TL;DR: This work formulates three research questions investigating the pedagogical ability of large language models to help students learn mathematics.

Abstract: Due to the remarkable language understanding and generation abilities of large language models (LLMs), their use in educational applications has been explored. However, little work has been done on investigating the pedagogical ability of LLMs in helping students to learn mathematics. In this position paper, we discuss the challenges associated with employing LLMs to enhance students' mathematical problem-solving skills by providing adaptive feedback. Apart from generating the wrong reasoning processes, LLMs can misinterpret the meaning of the question, and also exhibit difficulty in understanding the given questions' rationales when attempting to correct students' answers. Three research questions are formulated.

Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learning
An-Zi Yen' and Wei-Ling Hsu
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan azyen@nycu.c edu. tw, weiling hsu cs 1@nycu. 1@nycu.edu. edu.tw
Abstract
of LLMs into education. Since each subject has its own issues that must be addressed, we explore the scenario LLM use in mathematics education and the challenges thereof. Many studies address math word problem solving (Shen et al., 2021; Yu et al., 2021; Jie et al., 2022), but the human utility of the mathematics reasoning processes and the natural language explanations generated by mod- els within educational settings is rarely discussed. Given the extraordinary capability of LLMs gen- erate free-text rationalization, we investigate their mathematical srovingg competence and as- sess whether the generated step-by-step explana- tions constitute a useful educational resource. In particular, we analyze the ability of LLMs to pro- vide adaptive feedback by identifying and explain- ing errors within a student's math problem-solving process.
Due the remarkable language understand- ing and generation abilities of large language models (LLMs), their use in educational ap- plications has been explored. However, little work has been done on investigating the peda- gogical ability of LLMSs in helping students to learn mathematics. In this position paper. we discuss challenges associated with employ- ing LLMs to enhance students' mathematical problem-solving skills by providing adaptive feedback. Apart from generating the wrong reasoning processes, LLMs can misinterpret the meaning of the question, and also exhibit difficulty in understanding the given questions' rationales when atlempting to correct students' answers. Three research questions are formu- lated.
1 Introduction
We address these issues using the MathQA dataset (Amini et al., 2019), which consists of GRÉ-level questions that require advanced math- ematical knowledge. The questions cover arith- metic, algebra, geometry, and data analysis. Appendix we explain in detail our reasoning be- hind the dataset selection. For all experiments, we use GPT-3.5 (Ouyang et al., 2022). We raise and discuss a total of three research questions. In this pilot study, conduct both quantitative and quali- tative evaluations answer the research questions. The contributions of this work are threefold.
After the pandemic, e-lcarning has become part of mainstream education (Alqahtani and Rajkhan, 2020; Jafar et al., 2022). However, for students, online learning is not without its problems (Ab- dur Rehman et al., 2021). Apart from the difficulty in maintaining focus in online classes, the lack of real-time communication and timely feedback are also serious problems. In particular, in online as- sessments, students may fail to understand their mistakes, even after reviewing the provided an- swers; such failure to immediately clarify points of confusion yields poor learning outcomes. Syn- chronous communication between teachers and students in an e-learning setting necessary, but teachers find that promptly responding to students' questions a significant challenge.
We explore the application of LLMs for in- structing students solving math word prob- lems.
2. Drawing on the results of LLMs to solve math word problems, we comprehensively identify the existing problems of current models.
As large language models (LLMs) offer a wide range of applications, several studies (Tack and Piech, 2022; Kasneci et al., 2023; Zhang et al., 2023) address the use of LLMs in education. In this work, seek investigate the integration
3. We discuss the pedagogical suitability of the generated equations and the corresponding free-text explanations the context of mathe- matics education.
"Corresponding author.

---

crving
Calegory Questions Zero-shot Few-shot
CoT
poliie Fascands Whit the lingt of true
All
1,605
66.54%
65.67%
66.11%
General
663
64.71%
63.20%
64.86%
Gain
345
71.88%
70. 72.17%
The conanike uecnne eaterppal Liaare is perretta Mllonca to mavo aidn deaees 1520030000 i
Physics
410
68.54%
65.37%
67.32%
oeencccas offer whe dengllor monee.
Geometry
100
63.00%
65.00%
60.00%
Probability
12
58.33%
83.33%
75.00%
Other
75
53.33%
57.33%
58.67% %
Quardie bast
Table 1: MathQA results
Figure I: An LLM helping a student to learn mathemat- ics
proach. Thus the ability of L.LMs to solve math word problems and to understand and explain equa- tions is critical. The following sections address these two key points.
2 Utility of LLMs in Mathematics Learning
To explore the utility of LLMs in mathematics learning, we raise the first research question (RQ1):
3 LLM Ability to Solve Math Problems
How can LLMs be utilized to assist students in learning mathematical problem-solving skills? Since exams are a common way for teachers to evaluate student learning progress, in this study, we focus on leveraging LLMs to equip students with the knowledge skills needed to solve math word problems.
Many textbook math exercises provide the calcu- lation process but lack a natural language explana- tion. Accompanying each step with an explanation would greatly enhance the student's understanding of each equation. This leads to the second research question (RQ2): What is the problem-solve solving ca- pability of LLMS in mathematical problems and their ability to explain the computational process? Automatic Evaluation: The dataset used in this work is a modified version of MathQA in which unsolvable questions were removed by Jie et al. (2022). We utilized OpenAl's API, in particu- lar the -gpt-3.5-turbo-0335 ature was set to O. Table I shows the accuracy of three commonly used prompting methods: zero- shot prompting (Zero-shot), few-shot prompting (Few-shot), and chain-of oftthuhf- (CoT) prompt- ing. Three prompting templates are shown in Ap- pendix B. "Questions" denotes the number of ques- tions of each type in the test set. We also show the results for six MathQA question types. In this ex- periment, we compared only the model-generated answers with the provided answers, without ver- ifying the correctness of the reasoning process. As shown in Table 1, the CoT performance was poorer than expected, possibly due to the higher mathematical skill demands of MathQA compared to previously-used datasets. Zero-shot prompting does not significantly outperform CoT (p < 0.8), Exploring suitable prompting methods is left as fu- ture work. Based on the results, we observe that the calculation abilities of the GPT-3.5 model remain far from satisfactory. It frequently fails at simple arithmetic operations or counting.
In terms of question types, in addition to true/false and multiple-choice questions, short- answer questions are also included in math ex- ams. Students answer the questions using problem- solving processes. Automated assessment (Moore et al., 2022) of the student's problem-solving pro- cess remains to be investigated. In education, feed- back is also crucial (Shute, 2008). Simply scoring the student answer often insufficient, as scores do not reflect the reasons for incorrect answers. To learn from their mistakes, students need the cor- responding explanation. Figure 1 illustrates the scenario of an LLM applied in mathematics edu- cation. After student responds to a question, the model determines whether the student's answer is correct not. If the answer is correct, the system informs the student of the test objective for that question, enhancing their understanding of the ex- amined skills. If the answer is incorrect, the system provides adaptive feedback (Bernius et al., 2022: Sailer et al., 2023) by indicating the location of the error and offering an explanation, assisting the student to clarify any misunderstanding.
model. The temper-
Given that there are multiple methods for solv- ing a math word problem, model must be able to understand and correct student's thought process. If the overall strategy is sound apart from minor concept errors or obstacles, the model should guide students to the next step following their chosen ap-
Human Evaluation of LLM Results: It is known that LLMs might produce the correct answer even

---

type
Percentage
and why to request augmentation. Otherwise, the improvements yielded by the augmented informa- tion would remain limited in various real-world applications. For instance, for mathematical cal- culations, Schick et al. (2023) propose having the LLM use a calculator API to solve arithmetic tasks, However, the propensity of LLMs to misinterpret question meanings and substitute incorrect num- bers remains prominent challenge in advanced mathematical reasoning. From our observation, the GPT-3.5 model behaves much like student fo- cused on formula memorization that it struggles to adapt to variations the question, particularly in probability questions. Hence, enhancing the mathe- matical reasoning capabilities of LLMs is critical research direction.
Misconception problem-solving
36.54%
Incomrec provided answer*
17.31%
Unclear question d
11.54%
Calculation error in equation
9.61%
Misinterpretation of question
7.69%
Arithmetic error
5.77%
Absence of necessary diagrams* Counting error
3.85%
3.85%
Undefined symbols in question* Incomplete problem-solving
1.92%
1.92%
Table 2: Error types annotated by an expert. indicates that the error is not from the model's response but from an error the question.
with incorrect reasoning, or give an incorrect an- swer despite correct reasoning (Laskar et al., 2023; Lightman et al., 2023). However, detailed analy- ses and statistical evaluations of model errors have been less extensively studied. To further analyze whether LLMs are able to reason through complex mathematical problems, we invited an expert who majored Mathematics to evaluate the answers generated by the GPT-3.5 model. A total of 120 questions-22 20 from each of the six question types were selected by the expert. Each selected question involved a process of reasoning and calculation of more than four steps. The error types shown in Table 2. The most common error made by the model was "misconception in problem-solving the model understood the question but used incor- rect formulas or methods to solve it. "Misinterpre- tation of the question". in turn, is a different error: the model does not understand the question and generates an unrelated result. We also find that the GPT-3.5 model is good at providing comprchensive explanations for each equation without omitting parts of the problem-sole solving process. However, it exhibits inconsistent calculations, often making simple arithmetic errors. Furthermore, its grasp of set theory and three-dimensional spatial reason- ing is limited. Examples of some error types are provided in Appendix C.
Pedagogical Ability of LLMs to Rectify Students' Answers
The most important thing when helping students to learn mathematical problem-solving is provid- ing immediate adaptive feedback. In this section, measure the pedagogical ability of LLMs in mathematics. Pedagogical ability refers the abil- ity understand and help the student (Tack and Piech, 2022). This leads the third research ques- tion (RQ3): Are LLMs able to identify errors students' answers and provide corresponding ex- planations?
Teacher-Student Framework for Quantitative Evaluation: Due to the difficulty in obtaining real-world student answers, we simulate scenario in which students answer questions and teachers correct the students' responses. Based on the ex- perimental results from Table 1, we use the re- sponses from zero-shot prompting as the student answers. These answers are then input into the GPT-3.5 model, which acts teacher, correct- ing the student's answers according the ques- tion. The GPT-3.5 model is tasked with identifying whether the student's answer is correct and explain- ing why. Specifically, given an input question z. prompt Pe, and model M, we obtain the initial result = -Mgg; Next, we input ys to M, and ask A to act as a teacher with prompt P. to correct y. based q. Finally, we obtain the feedback yı M(g, Uo, Pi), where is the rationale of q provided in MathQA. If M struggles understand its responses (with detailed processes and natural language explanations), then its potential to assist teachers in corrections be-
Research Issues of Augmented Language Mod- els: Using external tools for calculation may be one way to address LLM drawbacks. Mialon et al. (2023) refer to language models that utilize exter- nal tools (Gao et al., 2022; Liu et al., 2022), retrieve relevant information (Izacard al., 2022), use specific reasoning strategies (Wei al., 2022) as augmented language models (ALMs). We argue that for a model to solve more complex tasks, should comprehend the tasks and know when, how,

---

Type
W/o rationale
rationale
show that Pe with r is better than that without r, However, the results in Table are different, be- cause the questions selected by the expert are more challenging. If Ve is incorrect, M has a 53.85% chance of erroneously claiming that ys is correct when is not provided. When r is given, M cor- rectly identifies that y,s is incorrect. Nonetheless, M has only 10.29% chance to accurately identify ys as correct with r. In addition, according our statistic, .M has only 3.85% and 1.92% chance to accurately correct the calculation results and the problem-solving processes, respectively. This is primarily because it has difficulty exploiting and because it usually misunderstands the Ve equations, even to the point of inaccurately correcting those that are already correct. Furthermore, it often for- gets the equations and calculations in ys- To verify whether LLMs can understand and correct human groblem-solvingg processes, we also invited five col- lege students to answer three questions. The results are presented in Appendix E.
All
53.96%
73.71%
General
54.75%
73.30%
Gain
54,78 78%
73.33%
Physies
50.00%
72.20%
Geometry
60.00%
83.00%
Probability
25,00%
83.33%
Other
61.33%
73.33%
Table 3: Results of maacher-student framework
Result y/
y, is correct
Mr incorrect
PP
w/o r
v
w/o
w/
Identify ys is correct
63.24%
10.29%
53.85%
0.00%
Say other words
17.65%
67.65%
17.31%
17.31%
Correct the process
11.76%
16. 18%
11.53%
50.00%
Correct the calculation
7.35%
5.88%
17.31% 32.69%
Table 4: Error types in teacher-student framework
comes questionable. The Pt template is shown in Appendix B. We refer to this framework as the "teacher-student framework".
Table presents the results of the teacher- student framework. Accuracy is adopted as the evaluation metric: accuracy measures whether M correctly identifies the correctness of ys- As y1 could be lengthy paragraph, we simply use key- words such as "is correct", "is incorrect", "not cor- rect", "almost correct" "not correct" to identify M. We compare the results with and without the question rationales, Interestingly, if the correspond- ing rationales are not given as input, the accuracy of the GPT-3.5 model acting as a teacher to correct students' answers (53.96%) ) is lower than that when directly answering questions (66.54%). However, if the corresponding rationales given input, accuracy is only 73.71%. Thus the GPT-3.5 model has difficulty understanding the equations given in the rationales.
Utility of LLMs in Complex Reasoning Tasks for Education: The failures of the GPT-3.5 model are comprehensively analyzed by Borji (2023). this work, we find that LLMs tend be confused by human answers, especially when tasks demand advanced knowledge or reasoning skills, even if rel- evant and correct information is provided. Hence, an alternative framework is needed under which to leverage LLMs' language understanding ability in complex reasoning tasks. Other crucial research issues are how to make the models aware of what they do not know and how to produce truthful and interpretable results (Phillips et al., 2020). Besides, this work primarily focuses on the capabilities and challenges of directly using LLMs for correcting students' answers. Developing cognitive model for reasoning processes and their potential roles in rectifying student mistakes is crucial.
Human Evaluation of Correcting Model- Generated Answers: To understand why the GPT- 3.5 model exhibits these characteristics when cor- recting answers, our expert also analyzed the re- sults of the teacher-student framework based on the selected 120 questions. Examples are given in Appendix D. Table 4 presents the correction results yt for ys by human evaluation. "w/o r" and "w/ r" denote P, without and with rationales, respectively. Comparing the results of "w/o r" and "w/ r", find that providing rationales seriously confuses M, such that it determines Y/ is wrong in most cases. Furthermore, M simply rephrases the con- tent of y, (67.65%). Note that the results in Table
Conclusion
In this work we propose a research agenda for leveraging LLMs in mathematics learning. First, we explore the use of LLMs to assist students in learning math word problem-solving skills. Then, analyze the mathematical reasoning ability of LLMs. Finally, we investigate the pedagogical abil- ity of LLMs in terms of rectifying gener-te or human answers and offering adaptive feedback. We conduct experiments with the GPT-3.5 model, and conclude that there remains room for improve-

---

ment in the LLM's performance in solving com- plex mathematical problems. In addition, although it generates comprehensive explanations, the LLM is limited in accurately identifying model's and human's 's errors due to its poor ability to interpret mathematical equations. In the future, we plan to develop an advanced method by which to improve the utility of LLMs in mathematics education.
Acknowledgements
This research was partially supported by National Science and Technology Council, Taiwan, under grant NSTC 111-222-E-A44-000-MMY
References
Mohsin Abdur Rehman, Saira Hanif Soroya, Zuhair Abbas, Farhan Mirza, and Khalid Mahmood. 2021. Understanding the challenges of e-learning during the global pandemic emergency: The students' perspec- tive. Quality Assurance in Education, 29(2/3):259 276.
Limitations
Considering that LLMs are widely accessible the general public and many educational institutions are examining the challenges and benefits of their use by teachers and students, this paper primarily focuses on the application of LLMs in educational settings. However, our experiments employed only the GPT-3.5 model and did not explore other LLMs such GPT-4 (OpenAI, 2023) and LLaMA (Tou- vron et al., 2023). Furthermore, while our current work investigates the utility of LLMs in enhanc- ing students' mathematical problem-son solving skills, there are many other applications in education. For instance, LLMs could be utilized to help teachers to generate teaching content or questions. Addi- tionally, the potential issues of violating teaching ethics involved in introducing LLMs into educa- tional applications are significant topics that have not been included in this study currently. We also conducted a human evaluation on 120 questions, which may be insufficient. Although we invited the expert to select questions that cover as many types of questions as possible, there may remain worth- while examples that were not selected for analysis. Moreover, the mistakes made by humans and those made by LLMs may differ. However, at the cur- rent stage, we invited only five college students to answer three questions: larger-scale experiment would be more helpful.
Ammar Y Algahtani and Albraa A Rajkhan. 2020. E- learning critical success factors during the COVID- 19 pandemic: A comprehensive analysis of e- learning managerial perspectives. Education Sci- ences, 10(9):216.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Sencel-Kedziorski Yejin Choi, and Hannanch Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-basel for- malisms. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 2367.
Jan Philip Bernius, Stephan Krusche, and Bernd Bruegge. 2022. Machine learning based feedback on textual student answers large courses. Comput- ers and Education: Artificial Intelligence, 3:100081.
Ali Borji. 2023. A categorical archive of ChatGPT failures. arXiv preprint a3Xiv:2302.03494.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Ethics Statement
In the context of educational applications, we uti- lize students' personal data and answer records for our experiments, which raises privacy concerns. For this study, we invited college students to answer mathematical questions investigate issues that might be found in real-world applications. These participants fully understood how their data would be used, and we ensured that their personal infor- mation would not be leaked.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint XrXiv: 10.11666.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. PAL: Program-aided language models. arXiv preprint arXiv:2211,10433

---

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers. Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoulopoolls, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schotimann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuy Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2022. State-of-the-art gen- eralisation research in NLP: a taxonomy and review. CoRR.
model reasoning through simulation, arXiv preprint arXiv:2210.05355.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Railcanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, al. 2023. Augmented language models: A survey. arXiv preprint arXiv:2302.07833
Steven Moore, Huy A Nguyen, Norman Bier, Tanvi Domadia, and John Stamper. 2022. Assessing the quality of
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299
short answer questions using GPT-3. In Educating for a New Future: Mak- ing Sense of Technology Enhanced Learning Adop- tion: 17th European Conference on Technology En- hanced Learning, EC-TEL 2022, Toulouse, France, September 12-16, 2022, Proceedings, pages 243- 257. Springer.
Adi Jafar, Ramli Dollah, Nordin Sakke, Moham- mad Tahir Mapa, Ang Kean Hua, Oliver Valen- tine Eboy, Eko Prayitno Joko, Diana Hassan, and Chong Vun Hung. 2022. Assessing the challenges of e-learning in Malaysia during the pandemic of COVID-19 using the geo-spatial approach. Scientific Reports, 12(1)11316.
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroli Wainwright, Pamela Mishknn Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744.
Zhanming Jie, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5944-5955.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094.
Enkelejda Kasneci, Kathrin SeBler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Sei- del, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023. ChatGPT for good? On opportunities and challenges of large language mod- els for education. Learning and Individual Differ- ences, 103:
Jonathon Phillips, Carina A Hahn, Peter C Fontana, David A Broniatowski, and Mark A Przybocki. 2020. Four principles of explainable artificial intelligence. Gaithersburg, Maryland, page 18.
Michael Sailer, Elisabeth Bauer, Riikka Hofmann, Jan Kiesewetter, Julia Glas, Iryna Gurevych, and Frank Fischer. 2023. Adaptive feedback from artificial neu- ral networks facilitates pre-service teachers' diagnos- tic reasoning in simulation-basl based learning. Learning and Instruction, 83:101622
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan. Shafiq Joty. and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of chatgpt on bench- mark datasets. arXiv preprint arXiv:2305. 184886
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint 2.02:::330.04777
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 55(5):1-35.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang. Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word prob- lems. In Findings of the A Association for Computa- tional Linguistics: EMNLP 2021, pages 2269-2279.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Yen Wu,
Soroush Vosoughi, Claire Cui, Denny Zhou, and An- drew M Dai. 2022. Mind's Eye: Grounded language
Valerie J Shute. 2008 Focus on formative feedback. Review of Educational Research, 7851)1153-18.

---

Anaïs Tack and Chris Piech. 2022. The Al Teacher Test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues. In Proceedings of the 15th International Conference on Educational Data Mining, page 522.
Input: You are a student who is good mathematics. Apply the appropriate mathematical operations solve the math problem. Please choose an answer from the options.
Q: a student gets % one subject 65 % in other. to get an overall of 55 % how much should get third subject. Options: a ) 75 % ) 25 % ee 45 % d 55 e 65 Output: Let's assume that the third subject carries the same weight as the other two subjects.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397..
To get an overall of 55%, the average of the three subjects should be 55%.
Let x be the percentage the student needs to get in the third subject.
Then, we can set up the equation: (55% 65% + x)/3 = 55%
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pages 445-854.
Simplifying this equation, we get:
120% + x 165%
x = 45% Therefore, the student needs to get 45% in the third subject to get overall of 55%.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
The answer c) 45%.
Table 5: Zero-shot prompting
Cheng-Kuang Wu, Wei-Lin Chen, and Hsin-Hsi Chen. 2023. Large language models perform diagnostic reasoning.
B Input Formats
LLMs are known for their success in zero-shot or few-shot learning (Brown et al., 2020). We can solve various tasks (Liu et al., 2023) through prompt engineering by providing a few examples or instructions about the task descriptions. For solving math word problem, Wei et al. (2022) pro- pose thain-of-thought prompting (CoT) by pro- viding step-by-step reasoning examples to cause the model to decompose the given question and generate reasoning steps. They demonstrate that CoT achieves promising results in solving math word problems. Moreover, Wu al. (2023) demon- strate the CoT's effectiveness by incorporating it into the medical diagnostic framework. Hence, we use zero-shot, few-shot, and CoT prompting in our experiments. Tables 5, 6, and 7 contain tem- plates of zero-shot, few-shot, and CoT prompting. respectively. The template of the proposed teacher- student framework is shown in Table
Weijiang Yu, Yingpeng Wen, Fudan Zheng, and Nong Xiao. 2021. Improving math word problems with pre-trained knowledge and hierarchical reasoning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 3384-3394.
Sarah J Zhang, Samuel Florin, Ariel N Lec, Ea- mon Niknafs, Andrei Marginean, Annie Wang, Keith Tyser, 7ad Chin, Yann Hicke, Nikhil Singh, Madeleine Udell, Yoon Kim, Tonio Buonassisi, Ar- mando Solar-Lezane ploring the MIT mathematics and eecs curricu- lum using large language models. arXiv preprint 206.9930
and Iddo Drori 2023. Ex-
A Dataset Selection
Datasets commonly used to evaluate model's ability to solve math word problems include MathQA (Amini et al., 2019), SVAMP (Patel al.. 2021), Math23K (Wang et al., 2017), and GSMSK (Cobbe et al., 2021). GSM8K consists of grade-school math word problems, and SVAMP pri- marily covers up to grade 4. Math23K is designed for Chinese elementary school students, and fo- cuses on evaluating the ability of students to solve one-unknown-variae variable linear math word problems. By contrast, MathQA is composed of GRE-level math problems, which serve as a benchmark for uni- versity research institute admissions in the United States. Therefore, we use MathQA our exper- iments to examine the utility of LLMs in higher education applications.
C Examples Error Types when Solving Math Word Problems
Among the 120 selected questions, the model cor- rectly answers 68 questions and incorrectly an- swers 52 questions. Fighteen of the incorrect re- sponses due to unclear question definitions, 24 stem from errors in the model's problem-solving process, and 10 are caused by calculation errors. In this section, we will present errors made by the GPT-3.5 model to show its limitations in mathemat- ics reasoning.

---

Input: You are student who good at mathematies. Apply the appropriate mathematical operations to solve the math problem. Please choose an answer from the options. Q: the banker' s gain of certain sum due years hence at 10 & per annum Is. 36, what is the present worth? ? Options: a ) 38. 27.675 c 30, d) data inadequate e none of these
Input: You are a student who is good mathematics. Apply the appropriate mathematical operations to solve the math problem. Please choose an answer from the options.
Q: a student gets % one subject 65 % in the other. to get an overall of 55 % how much should get third subject.
Options: a ) 75 % b) 25 % 45 % d 55 e A: Let's think step by step.
A: The answer a.
Output: To get an overall of 35% the average of the three subjects should be 55%.
Q: sophia finished 2/ 3 of book... she calculated that she finished 90 more pages than she has yet to read. how long is her book? ? Options: ) 229, b) 270, 877, d 266, e 281
Let's assume the third subject has a weightage of 100%.
So. the total weightage of the three subjects is 100% + 100%
+ 100% = 300%.
A: The answer is b.
Now, we know that the student got 55% in the first subject and 65% in the second subject.
Q: a student gets 55 % in one subject, 65 % in the other. to get an overall of 55 % how much should get in third subject. Options: ) 75 % b) 25 %, e 45 % d 55 %, )) 65 %
So, the total marks obtained in the first two subjects are: 55%
65% 120%
Output: Let's assume that all three subjects carry equal
To get an overall of 55% the total marks obtained in all three subjects should be:
marks
To get overall percentage of 55%, the sum of the percent- ages obtained in all three subjects should be 3 55% = 165%. The student has already obtained 55% and 65% in the first two subjects.
55% 3= 165%
Now, we can find out how much the student should get in the third subject:
Total marks in all three subjects = Marks obtained in the first
Therefore, the percentage required in the third subject to get an overall percentage of 55% is: Percentage required in the third subject 165% (55%+ 65%) 45%
two subjects Marks obtained in the third subject I 165% = 120% Marks obtained in the third subject Marks obtained in the third subject 165% 120% 45%
Therefore, the answer option c) 45%.
llence, the answer option e) 45%.
Table CoT prompting
Table 6: Few-shot prompting
model's identification results. Figure 2 shows the confusion matrix of answer correction. We define the difficulty of the questions based on the number of equations in the given rationales. We categorize the question difficulty into three groups: those with fewer than five equations, those with 5 to 10 equa- tions, and those with more than 11 equations. "Cor- rect" and "wrong" in the true labels indicate that ys is correct and incorrect, respectively. "Correct" and "wrong" in the predicted labels label indicate that the identification of y1 is correct and incor- rect, respectively. Thus "correct" in the predicted labels indicates that the model identifics ys to be correct when ys is indeed correct. As shown in Fig- ure 2, the model's ss accuracy in correcting answers decreases as the problem becomes more difficult, especially when no rationale is provided to the model. Clear and explicit rationales aligned with the mooblem-solvingg process in ys cause the teacher model to understand correctly and thus accurately identify whether the ys is correct.
As shown in Table 2, the most common mistake is misconception in problem-solving. Moreover, as mentioned in Section 3, one weakness of the GPT- 3.5 model is in reasoning about three-dimensional spatial problems, as illustrated in Figure 3. The model directly divides the length, width, and height by three to calculate how many cubes can be placed in box, without considering the fact that cubes cannot be accommodated if the height is not evenly divisible. The question in the red box should be 3 x 3 2, not 4 x 3 x 2.
Misinterpretation of the question is another crit- ical error. Figure 4 is an example of the GPT-3.5 model misinterpreting the meaning of the question. Based on the result responded by the model, once we obtain the value of "x", the greater number is "2r". However, the model incorrectly interprets the question as asking which number is greater after adding five.
D Example Results from Teacher-Student Framework
Additionally, when Ys is correct, the GPT-3.5 model frequently identifies Ys as incorrect but its explanation is merely a paraphrase of ys- The rea- son may be that the GPT-3.5 model does not merely determine correctness based on the answer's value. Although it has stringent requirements for this pro- cess, its comprehension of equations is less than ideal, resulting in misinterpretations.
According to our observations, when acting teacher, the GPT-3.5 model achieves high accuracy in correcting simple equations when the rationale contains a single equation. However, for more com- plex problems involving two or more equations, the quality of the rationale significantly impacts the

---

You are math teacher. According to [Question] [Rationale], please indicate whether the [Student's Answer] correct or nol, and apply appropriate mathematical opera- tions to correct [Student's Answer]. [Question]: student gets
Input:
Without Rationale
4.2
151
96
52
2
one subject. 65 in other. get an overall of how much should get third )) %;
395
2417
126
12
subject. Options: a %
% 55 %
|Rationate|: explanation t= 3 years r= id= bg 100 / tr 36 100 /
With Rationale
x 10 = 12 10 rs 120 id= = pw x tr ) 100 =20= (p x 3 / 100 1200 = pw x pw 1200 / =ss 400 answer option a [Student's Answer]: Let's assume that the third subject carries the same weight as the other two subjects.
566
134
18
241
313
8B
107
11
To get an overall of 55%, the average of the three subjects should be 55%.
Let x be the percentage the student needs to get in the third subject.
Figure 2: Confusion matrix of answer correction based on question difficulty
Then, we can set up the equation:
(55% + 65% + x)/3 = 55%
Simplifying this equation, get:
120% + x 165%
= 45%
E Human Evaluation on Correcting Human's Answers
Therefore, the student needs to get to get overall of 55%.
get 45% in the third subject
The answer is <) 45%.
Since the mrblem--solving approaches of LLMs may differ from those of humans, to verify whether LLMs can understand and correct human problem- solving processes, we also invited five college stu- dents from different departments (Mathematics, Statisties, and Japanese) to answer three questions selected by the expert. Their answers were in- put into the GPT-3.5 model for correction. Subse- quently, the expert analyzed the correction results. Based on the collected answers, we find that LLM is more likely to be misled by human answers and even ignore reasoning processes written by humans. Whether the answer is correct or not, LLM tends to identify the answer as incorrect. As shown in Figure 7, the model misidentifies the answer of the human as "$8000". This may be because humans sometimes omit mathematical conversions in their answers, or because the problem-sol strategy may differ from that the LLM. This can con- fuse the LLM. For instance, in Figure 8, the model fails to comprehend the equations without natu- ral language explanations, erroneously identifies the human's problem-solving approach as incor- rect, and proceeds to solve it using its own method. Even when provided with rationale, the model usually fails make an accurate identification, of- ten leading to erroneous outcomes.
Output: The answer is correct.
Explanation:
To find how much the student should get the third subject have an overall of 55%, can the formula: (55% 65% + x)/3 55%
can simplify this equation by multiplying both sides by 3: 55% 65%
165%
Now we can solve by subtracting 55% and from both sides:
X 165% 55% 65%
45%
Therefore, the student needs to get 45% the third subject to have an overall of 55%.
Table 8: Prompting under the stachet-student frame- work
To measure the ability of LLMs to correct stu- dents' answers, utilize the problem-solving re- sults returned by the GPT-3.5 model the students' answers and then ask the model to correct the input answers. Figure 5 presents an example in which the reasoning in the answer is not finished: it stops at calculating the total volume of the drinks. The model, which acts as a teacher, accurately points out the error and provides the complete solution. However, the GPT-3.5 model may also mistakenly rectify correct answers incorrect. Taking Fig- ure 6 as an example, the result in the given answer correctly solves for the number of sheep, yet the model identifies as incorrect, further producing an erroneous reasoning process.
Type of Generalization
Table 9 shows the type of generalization we are investigating, based the taxonomy defined by Hupkes al. (2022).

---

"Problem': 'how much spece,
maximum
cubes fitted rectangular measuring
To solve this problem, first need to determine how many 3 3 cubés can ntt insice the given rectangular
length
equal
partially).
width of the box is units, which is equal to And the height of the box 6 units, equal
Therefore,
length,
height.
giving a total of x 24 cuiee.
The total volume of the cubes is 24 x x3xx 3xx 3) 648 cubic units.
The volume the rectangular box is 6 9hx 894 cubic units.
Therefore, the vacant space left the box is:
648 -54 cubic units
This meens that there is no vacant space left in box, and the cubees completely with room
Figure Misconception a diree-dimenional question
'Problem':
of both, their changes
:5,
Let's assume the numbeer the ratio 1:2 and 2x, respectively.
According to the problem, when 5 added both numberss their ratio becomes 3:5, So up following equation:
5) 5)
x cross-multiply simplify:
5)
5x + 25 6x 15
Therefore, numbers retio
greater number,
can simply add 5 to each and compare:
10 + 15
20 8 25
So the greater number 25.
Figure 4: The GPT-3.5 model misinterpreting a mathematical question

---

'Problem': 75 liter solution cool drink is made from 8%% jasmine water. liters of jesmine and liters water added to the solution wwhtt percent of the solution jasmine
Student:
"To solvve the problem, can use the concapt the amount of jasmine water in the original solution and the amount jasmine water in the final solution.
Initially, the solution contains 8% jasmine water, which means that in 76 liters of solution, there are 0.88 x 75 liters of jasmine water.
When 3 liters of jasmine water are added to the solution, the total amount of jasmine water becomes 6 3 m liters.
Similarly, when 12 liters of water are added to the solution, the total volume of the solution becomes 75 3 12 90 liters'
You are a math taachar, verify whether this student wrong, wrong. correct it and give him explanation
correct. However, conclusion mizsing.
determine percentage jasmine
amount of jasmine water as percentage of the total solution.
The total amount of jasmine water in the final solution is liters, which is 9/90 = 01 or 10% the final solution.
Therafora,
that final solution contains 10% jasmine
So, the student needs to add final step to their solution to calculate the percentage jasmine water in the final solution.
Corrected solution:
Initiellyy, the solution contains 8% Jasmiine water, which meens that liters of solution, there are 0,88 x 75 liters jesmine water.
When 3 liters jasmina water are addad the solution, the total amount of jasmina water
Similarly,
solution,
solution
becomes 75 3 12 90 liters.
Therefore, the final solution contains 10% jasmine water (9 liters of jasmine water out of 90 solution).
Figure 5: Completing the problem-sol process in the saccher-student framework

---

'Problam:; 'the ratio between the number shoap and the number of horses at the stewar farm is to if each horse fed 230 aunces of horse food per day and the farm noeds total 12880 ounces of horse food per day. what is number shesp in the form ???
Student:
"Let the number of sheep be 3x and the number of horses be 7x. Then the total number of enimals on the farmis TX 10x.
Since each horse is 230 ounces of horse per day, the amount horse food needed for 7hx horses 230 1610x ounces per day,
givan
horse needed antire
per day, So
following equation:
Solving get:
Therefore, sheep
So, 24,"
You math teacher, verify whether this student wrong, he wrong.
correct it and him explanation
CR
The student's work correct up to the point of determining the value of as à However, final incorrect.
farm, horses on the ferm.
thee
The ratio of sheap to horsas 3/7, which maans that for avary sheap, there are horses. following equation to find numbar sheap:
Sinca thera
3/7 xx/66
Cross-multiplyingg get:
7x 9156
Simplifying, get:
7x 188
Dividing both sides by we get:
x=
Therefore, the number of sheep on the farm is 3x 3-24 72.
So, the correct answer is 72 sheep.
Figure Providing the wrong reasoning process in the teacher-studee student framework

---

'Problem': person sumn money per annum simple years amounted
the
Student:
money represented
(X"0.0588 480-xX
0/4X+480-X
=>006x=480
X=800*
You math
verify whether this student wrong, he
correct and give explanation
SR
The student's answer incorrect. Let's go through the problem and solve it step by step.
Let's assume the amount of money lent is represented by XX'
According to the problem, the interest eerned in 8 years is $480 less than the sum lent So,
earned $480.
simple interest:
Simple Interest = Principal Rata Time
this the principal X, the rate 5% (0.05 decimal), and the time
Therefore, write the equetion
(X 0.05 x 8) (X (X80)80)
Simplifying the equation, we have:
0,4X S-8880
To solve for X, we need to isolate it on one side of the equation. Let's subtract X from both
044X X
both
= $480) (-06)
Simplifying side of equation,
$800
So, the correct answer that the lent $800 $8000 as the student suggested.
Figure Misidentification the human's answer

---

'Problem'' person lent certain sum money per annum simple interest years amounted sum sum
Student:
"0.05X"0=X-480
06X-480
X=800'
verify whether wrong, wrong, correct give explanation
The student made mistake in satting up the equation. Let's go through the problem and error.
money dollars.
According problem, So, the interest earned (X- $480).
years $480
The formula for simple Interest is: Interest Principal" Rate Time
this case, the rate givan as 5% per annum, which can be written as 0.05. The time 8 years.
Therefara, interest earned can be calculatad 0.05 8 0.4X.
Now, the equation based the informetion:
0-4X S-880
To solve Xor let's bring the terms involving to side the equaation:
0,4X - $480
Simplifying, wa get:
-$800
Now, we can multiply both sides of the equation by to get rid of the negative sigrt
08X
$480
Finally, dividing both sides of the equation by 0& gives us:
= $800
Tharafora, the correct answar is that the sum lant was $800.
Explanation:
The mistake the student made the initial equation setup. They subtracted $480 from the interest side of the equation, which to incorrect equation. It's important to up the equation accurately solve the problem correctly.
Figure 8: Misinterpretation the human's answer

---

Motivation
Practical
Cognitive
Intrinsic
Fairness
Generalisation type
Compositional
Structural
Cross Task
Cross Language Cross Domain
Robustness
O
Shift type
Covariate
Label
Full
Assumed
Shift locus
Train test
Finetune train test
Pretrain train
Pretrain test
Table 9: GenBench evaluation card


Reviews:
[{"review_id": "DxwSbtP0Wx", "reviewer": "Reviewer_PZHc", "summary": "", "questions": "1. What would be another question that could asked in a somewhat related context, however extending the current discussion into a deeper aspect in terms of reasoning of LLMs? could this extended discussion be also included in the current submission?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1-1: The main weakness of this paper is the answer of RQ3. The discussion on reasoning task may be a little far--fetched with just this analysis.\n\n\nA1-1: Thanks for your comment. We acknowledge the importance of developing a cognitive model for reasoning. At the current stage, we primarily focus on the capabilities and challenges of directly using LLMs for correcting students’ answers. But the weakness you mentioned is important. We'll address this limitation in our final version.\n\n\n\nQ1-2: What would be another question that could asked in a somewhat related context, however extending the current discussion into a deeper aspect in terms of reasoning of LLMs? could this extended discussion be also included in the current submission?\n\n\nA1-2: We believe that a deeper discussion on the reasoning of LLMs is feasible based on the results and observations of our experiments. We will extend this idea to conduct comprehensive research at the next stage. Thank you for your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "## Reasons\n\nThere are a few reasons to accept the work, surrounding the thoughtful nature of the analysis. The authors take time to breakup the analysis into three separate steps to understand the internals of LLMs. Specifically,\n\n1. With a deeper understanding and with an answer to RQ2, the authors able to categorize the responses that LLMs generate as explanations when solving math problems. This may better help in understanding the development of cognitive tutors with LLMs\n\n2. With a general understanding on the limitations and capacities of LLMs, it is a great way to scope the expectations that people seem to be developing surrounding the hype around LLMs. ", "reasons_to_reject": "## Reasons\n\nThe main weakness of this paper is the answer of RQ3. With discussions on the teacher-student framework for quantitative evaluation, to compare with correcting model generated answers, there is a certain possibility to understand if LLMs can act as a tools to help students correct errors. However, the discussion on reasoning task may be a little far--fetched with just this analysis. It is important to develop a cognitive model for the process of reasoning and the roles it could play in the process of correcting student errors. However, this may be a minor aspect to discuss here. \n\nquick reference Lovett, Marsha C. \"Cognitive task analysis in service of intelligent tutoring system design: A case study in statistics.\" International Conference on Intelligent Tutoring Systems. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998.", "questions_to_authors": "1. What would be another question that could asked in a somewhat related context, however extending the current discussion into a deeper aspect in terms of reasoning of LLMs? could this extended discussion be also included in the current submission?", "paper_topic_and_main_contributions": "# Summary\n\nIn this work, the authors explore answers to three simple questions -- Use of LLMs in Math Learning,the mathematical and problem-solving abilities of LLMs and the use of LLMs to identify errors in student solutions to math problems.  Even as they may be simple questions to ask, the paper does a fair job at analyzing the answers to these questions. For the first question, the paper provides literature review evidence on the use of LLMs as assessment tools. For the second question, the paper use a modified version of the MathQA dataset to test GPT models with automated and human evaluation to identify inconsistencies. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "r7Finkuv51", "reviewer": "Reviewer_RNa9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q2-1: The author considered this as a proposal of \"how to use LLM for math problem solving\". However, I only see the author shared where the current status is with GPT3.5, I am not seen much of about the \"how\" part being discussed.\n\nA2-1: I understand why you have this doubt. The primary focus of our paper is to discuss the potential issues of directly applying LLMs in an educational context like GPT-3.5. We use the math problem-solving task as a means to study the types of errors LLMs might commit. Given that LLMs are among the most accessible tools for the general public and many educational institutions are exploring the challenges and benefits of teachers and students using LLMs, our intention with this paper is to explore these related topics through some experiments. We will add this clarification in our final version.\n\nQ2-2: It seems to be known that LLM isn't for solving math problems.\n\nA2-2: Given the remarkable explanatory capabilities of LLMs, we believe they have notable potential to help students and enhance their learning processes. Thus, our work starts by analyzing the LLMs' capabilities in solving mathematical problems and then examines the weaknesses in LLM's correction and explanation.\n\nQ2-3: How were these categories developed and why they are such categories? And what we should do these categories of errors? \n\nA2-3: The categories of errors were mainly defined by an expert who majored in mathematics and graduated from a university of education. We invited the expertise of this specialist to categorize the errors made by the model. While we know the limitations of LLMs in various tasks, our primary goal was to closely examine the nature of errors that LLMs make in problem-solving and correction. We hope the insight from our experiments will facilitate future research on the deployment of LLMs within educational domains. We will add the explanation in our final version. Thanks for your comment.\n\nQ2-4: The categorization of errors were done by 1 person. How do we ensure the subjectivity not impacting the categorization or accuracy?\n\nA2-4: We'd like to note that asking an expert with both mathematical and educational expertise willing to assist in this endeavor was a challenge, and we hope you take this into consideration. But I completely understand that this is a reasonable concern.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Contribution above is interesting. It made an abstract question more concrete.\n2) Used a dataset and experiments to quantify the problems and issues. ", "reasons_to_reject": "1. The author considered this as a proposal of \"how to use LLM for math problem solving\". However, I only see the author shared where the current status is with GPT3.5, I am not seen much of about the \"how\" part being discussed.\n2. The experiments' findings aren't new. It seems to be known that LLM isn't for solving math problems. Categorizing the errors provided some new data points. How were these categories developed and why they are such categories? And what we should do these categories of errors?\n3. The categorization of errors were done by 1 person. How do we ensure the subjectivity not impacting the categorization or accuracy?\nBased on above, I am not if there are enough contributions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Aiming for understand \"how can LLM be utilized to assist students in learning mathematical problems skills\", this paper investigated two sub problems:\n1) How much LLM can solve the math problems with computational process\n2) How much LLM can check the correctness of a given answer and generate feedback.\n\nThis paper utilized GPT-3.5 as the LLM, used MathQA dataset to run experiments. \n1. They reported the correctness of how much ChatGPT can answer the questions in MathQA; \n2. They used a math expert to categorize the errors made by LLM; \n3. They used a teacher-student prompt to check if LLM can identify the correctness of a given answer then provided categorized error type for the errors.\n\nContributions:\n1. A proposal to breaking LLM for Math Education problems into two sub problems\n2. Ran experiment to show where GPT3.5 is in solving these problems, and what types of errors GPT3.5 is making\n3. The teacher-student prompting to get LLM to check if a given answer is correct is a bit interesting\n\n", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z8ToL0lo1V", "reviewer": "Reviewer_pKga", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q3-1: it would have been beneficial to present the validity of the LLM-generated answers as alternatives to students' answers.\n\nA3-1: I do agree with your perspective. We have indeed explored the LLM's capability to correct students' problem-solving processes, as detailed in Appendix E. Our choice to use LLM-generated answers was primarily driven by the need to obtain a large number of problem-solving processes to facilitate an analysis at a certain scale. Moreover, we were curious if the machine could comprehend its answers. If the LLM struggles to understand its responses (with detailed processes and natural language explanations), then its potential to assist teachers in corrections becomes questionable. The experimental results in Table 4 show that even when provided with the correct answer for a question, the LLM is unable to correct it effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study presents a research agenda for the effective use of LLMs in mathematical education. It also explores the advantages and disadvantages of utilizing LLMs in this context. This study has also explored the LLM's capability to correct students' problem-solving processes.", "reasons_to_reject": "This study carried out an experiment using an LLM both as a student for answering questions and as a teacher for correcting students' answers and providing feedback on incorrect answers. This approach is understandable as a starting point. The author(s) mentioned room for improvement in LLM performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study introduced a research agenda on the utilization of Large Language Models (LLMs) in mathematical education. It raised a set of research questions, which were qualitatively and quantitatively derived. First, this study argued for the utility of LLMs in mathematical education through reviewing the previous research on automatic evaluation methods. Then, it analyzed the LLM's errors in answers for mathematical problem-solving questions. It also evaluated the error-correction ability of LLMs. These findings indicated the potential for the research agenda regarding educational utilization of LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main conclusions of the reviews and the post-rebuttal discussions:
- 3/ 3 reviewers consider the paper sound (scores 4, 3, 3)
- 2/ 3 reviewers find the paper exciting (scores 4, 2, 3)

Front reading the rebuttal and seeing the scores above, I find that the reviewers consider strong points for soundness the following:
- In-depth exploration of 3 RQs, on the math and problem-solving abilities of LLMs and the use of LLMs to identify errors in student solutions to math problems
- especially RQ1 and RQ2 seem to contain valuable analysis, in particular for developing better tutors with LLMs
Concerns regarding the use of just 1 annotator were clarified in the rebuttal as it is challenging to find an expert in both math and education.
Decision: Accept-Findings