Detecting Problematic Questions to Support Math Word Problem Design

Authors: Mao-Siang Chen, Yu-Hang Deng, An-Zi Yen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Problematic Question Detection, Question Design Support, Self-Optimization Prompting

Abstract:
When designing math word problems, teachers must ensure the clarity and precision of the question to avoid multiple interpretations and unanswerable situations, thereby maintaining consistent grading standards and effectiveness. We address these issues to provide comprehensive support to teachers in creating clear, solvable, and formal math word problems. In this paper, we present MathError, a dataset of real-world math word problems annotated with error types to investigate the need for question correction. Our work explores how large language models (LLMs) can assist teachers in detecting problematic questions to support math word problem design in scenarios with limited data, simulating real-world conditions with minimal training samples. Preliminary results demonstrate the models' capabilities in detecting problematic questions and identify areas for further research and development in educational applications.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730691004257

thread_type:
official_review

<thread_object>
creator: reviewer_6crd

<content>
summary:
The paper presents the MathError dataset, an extension of the Math23K dataset that includes error types for each question. The goal is to improve the clarity of questions within the dataset, and to achieve this, the authors define five key factors that form the basis for these error types. After annotating the questions based on these defined error types, the paper presents various prompting strategies that focus on self-optimization/reflection to evaluate how different LLMs perform in detecting these errors. The findings indicate that combining model-generated definitions of error types with human-provided examples can enhance the LLMs' performance in identifying the defined error types.

strengths:
1. The research question studied in the paper is both interesting and important.
2. The experiments are thorough and cover various prompting scenarios and models.

weaknesses:
1. The related work section is not thoroughly covered. Many studies have proposed various methods for self-reflection/evaluation, which were not mentioned in the paper.
2. Using only one annotator per batch of 5,815 makes the annotations prone to bias toward certain error types.
3. Although the paper addresses an interesting and important problem, I am not convinced that the proposed error types for improving clarity will achieve this effectively. Specifically, three out of the five items listed in Section 3 as factors for enhancing clarity seem to have the opposite effect:
- 3.1. Multiple Interpretations: When a question allows for different approaches to reach the final answer, it can actually make solving it easier for students. This flexibility increases their chances of finding a solution.
- 3.2. Informal Wording: Formal language does not always improve clarity. In the context of elementary school math questions, which the paper focuses on, using less formal language may help students understand the questions more easily.
- 3.3. Calculation Error: In the example provided on line 215, the phrasing “6000 divided by 59, and then subtract 35 from this result” is indeed difficult to interpret, as this form is uncommon. Additionally, it is unclear whether the question refers to regular division or floor division. By contrast, the original phrasing, "the difference between 6000/59 and 35," is far clearer and more familiar. Moreover, based on the paper, the goal of this factor is to address cases where "imprecise words are used to describe a mathematical expression." However, in this example, the mathematical operation is already there, so it is unclear what specific problem this factor intends to resolve.

  Given the aforementioned limitations in how clarity is evaluated, I believe the impact of this study on real-world applications may be minimal.

4. While the proposed method (PRO) is straightforward, the presentation in Section 5 is so awkward that it is impossible to understand the method without referring to the provided algorithm/figure.
5. Since the paper used the `GPT-4o` checkpoint, the results are likely not reproducible. This is because the `GPT-4o` checkpoint continues to receive updates and is not part of the isolated checkpoints designated for research purposes. Therefore, the results obtained may not be reliably reproduced.

questions:
**Suggestion:**
1. Line 258: LLMs do not possess understanding or consciousness. Therefore, they do not have comprehension capability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714342369

thread_type:
official_review

<thread_object>
creator: reviewer_an3l

<content>
summary:
The paper proposes a new, challenging, task for LLMs: identifying errors (and especially ambiguity) in math word problems which could be detrimental to learning for human students that try to solve them.
To facilitate this tasks, the authors annotate an existing dataset of MWPs in chinese language consisting of ca. 23k problems with five different error types.
The authors benchmarks various LLMs on the task of identifying such errors and propose a new method that uses iterative prompt optimization and can improve performance.

strengths:
- The writing and presentation are clear
- The paper tackles and interesting, and challenging, new problem of identifying errors in generated math word problems

weaknesses:
- Labeling 5,815 examples seems to be a very challenging and exhausting tasks for annotators, especially since none of the examples were also annotated by another annotator, as all of them received disjoint sets. This raises questions whether the annotators were still able to produce high-quality annotations and it would be important if the authors could clarify how this is ensured.
- The authors also mention that they are “unsure whether the five error types are sufficient” (L145) which might be due to them being established by manually inspecting only 200 examples. It would likely be helpful to either analyze more examples (as shown in L208 some errors only occur very infrequently) or to consult relevant literature, e.g. from the learning sciences.
- The self-optimization algorithm needs to be put better into perspective with related works.
- The experiments are a bit limited in scope, focussing only on proprietary LLMs (GPT-3.5, GPT-4o)

questions:
- It would be good to back up statements like that ambiguity in questions needs to be reduced (which might be intuitive nevertheless) with literature from the learning sciences that supports them.
- L131: we focus extends -> our focus extends (?)
- Section 3 is titled: “From Math23k to MathError”. I would expect based on the title that it details how the authors construct MathError but the section only describes error types.
- The figure captions are rather short and could be improved, similarly Figure 2 does not have axis labels
- It would be helpful in Fig 1. a) to have a pointer where the start of the algorithm is
- The caption of Figure 2 suggests the y-axis shows the difference between refinement rounds but it actually shows the inverse (similarity)
</content>
</thread_object>

</thread>
<thread>
creation date:
1731266194188

thread_type:
official_review

<thread_object>
creator: reviewer_ulvd

<content>
summary:
This paper introduces MathError, a dataset of annotated math word problems aimed at helping teachers identify and correct issues that could lead to ambiguous or unsolvable questions. By leveraging large language models (LLMs), the study demonstrates how these models can aid teachers in designing clear and consistent math problems. To do so, the authors propose a self-optimizing framework where the model iteratevily refines the demonstration.

strengths:
- The paper is well written, has clear motivation, and has a well-explained methodology.

weaknesses:
W1: How significant is the problem of erroneous questions? When humans communicate, language often contains implicit information and inherent ambiguity, yet people generally understand and perform well despite this. The main question is: how much do expression errors in math questions impact the performance of both humans and models? An analysis comparing the performance of large language models (LLMs) and humans on ambiguous math questions versus corrected versions would be valuable. Does performance improve when questions are clarified, or can models, like humans, effectively handle the natural ambiguity in language?

W2: The primary weakness of this paper’s proposed method is that the self-refinement technique performs poorly, with overall scores not exceeding 33% and some results falling below random chance. This may stem from ambiguities in defining error types or, as noted in W1, the model’s inability to detect "errors" that are subtle nuances of human language due to its inherent implicitness. How do humans perform on this task?

W3: Given that this work aims to address error detection, it is essential to include results on the model's overall error detection performance—specifically, how accurately it identifies the presence of any error. This should be supported with precision and recall scores to provide a clear measure of effectiveness.

questions:
Q1: In the dataset construction phase, what is the background of the human annotators? Are they math experts? Please provide more details.

Q2: What is the distribution of the classes in the dataset? Are there descriptive statistics of the size of the dataset along with the number of samples per class?

Q3: Based on the results in Table 3, the authors state in Line 431 that model-generated definitions are better than humans. However, in Line 406, they state that based on Table 5, human-written definitions outperform the other prompting techniques. How are these results related? What is the setting presented in Table 3, then?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201351308

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We have noticed that there are some misunderstandings about flaws in math problem descriptions, such as "Multiple Interpretations" and "Informal Wording." 
We would like to provide additional clarification here. For example, in the Introduction (lines 31–35):
> "The original price of an apple is 2 dollars. It has been discounted twice: the first discount is 10%, and the second discount is 5%. What is the current price of the apple?" It is unclear whether the second discount is to be applied to the original price or to the price after the first discount.

In this case, the wording leads to two possible interpretations of the discount calculation: whether the second discount applies to the original price or to the price after the first discount. As a result, there could be at least two possible answers, which likely does not align with the teacher's intention. This question is primarily designed to test students' familiarity with multiplication operations, so having a single, definitive answer is crucial for quickly verifying whether students have mastered the skill, rather than allowing for open-ended responses.

We consider this issue to be significant because the dataset we use consists of real exam questions. During the process of using this dataset, we identified numerous errors in problem descriptions, indicating a high likelihood of oversight during the problem design process. 
Such oversights can lead to questions failing to effectively fulfill their intended educational or assessment purposes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732764765143

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Replying to Official Comment by Reviewer 6cRD and Reviewer AN3L

comment:
We would like to thank the reviewers once again for their valuable comments, which have enabled us to add more details to help readers better understand our work.

We have already replied the concerns regarding the dataset construction in a separate response. 
Below, we provide further clarification and additional details:

We chose to first establish a rigorous process to define error types and ensure that annotators thoroughly understood the task and reached a consensus. Following this, each annotator was assigned to label a specific portion of the data. This decision was guided by the nature of our research, which required annotators with a solid mathematical background and a willingness to engage closely with us to address any issues during the annotation process.

Given our limited budget, we adopted this annotation workflow to ensure the highest quality of labels while completing the dataset within the constraints of time and resources.

It is also worth noting that there are precedents in academic research where a single annotator has been employed for data labeling under specific circumstances. Therefore, our approach is not an uncommon practice in such cases:


* Gandhi, Nupoor, Anjalie Field, and Emma Strubell. "Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

* Mehta, Maitrey, and Vivek Srikumar. "Verifying annotation agreement without multiple experts: A case study with Gujarati SNACS." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

* Lin, Chin-Yi, et al. "Argument-Based Sentiment Analysis on Forward-Looking Statements." Findings of the Association for Computational Linguistics ACL 2024. 2024.

* Yen, An-Zi, Hen-Hsen Huang, and Hsin-Hsi Chen. "Personal knowledge base construction from text-based lifelogs." Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a dataset called MathError, which contains annotated math word problems, pointing out errors that could be an impediment for learning in human students.  

Strengths:
The paper is well written, and well motivated.  The experimentation is solid.

Weaknesses:
Overall, I found that the reviewers are not extremely enthusiastic regarding the content of the paper--they point out reproducibility issues due to the utilization proprietary LLMs and issues with annotation methodology of utilizing one annotator for very large batches of data.  In addition, Reviewer 6cRD brings up a good point that they are not convinced that the proposed error types for improving clarity will achieve the overall goal effectively.