VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency

Authors: vernon  toh
yan  han
ratish  puduppully
nancy f chen
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-05-30 10:53:32

Abstract: Large Language Models (LLMs), combined with program-based solving techniques, are increasingly demonstrating proficiency in mathematical reasoning. For example, closed-source models such as OpenAI GPT-4 and Claude show excellent results in solving math word problems. However, progress in math word problem-solving for open-source LLMs is limited, and the challenges these models face are not well-studied. In this paper, we study the performance of strong open-source LLMs, including Llama 2 (7B), Code Llama (7B), and Mistral (7B) on math word problems using program-based solving techniques. Specifically, we analyze the outputs of these models when applied to math word problems and identify a category of problems that pose a significant challenge, particularly those involving quantities spanning multiple units. To address this issue, we propose a systematic approach by defining the units for each quantity and ensuring the consistency of these units during mathematical operations. We developed Unit Consistency Programs (UCPs), an annotated dataset of math word problems, each paired with programs containing unit specifications and unit verification routines. We fine-tuned Llama 2 (7B), Code Llama (7B), and Mistral (7B) models with UCPs to produce their VerityMath variants. Our findings indicate that our approach, which incorporates unit consistency, currently slightly underperforms compared to an approach that does not. To understand the reasons behind this, we conducted an in-depth error analysis and suggested options for future improvements.

Main paper: 
 introduction: 
 The ability to reason during the process of thinking and decision-making is a fundamental aspect of human intelligence. Replicating this ability in machines has been an objective in the field of Natural Language Processing. Large language models (LLMs) (OpenAI, 2023; Anil et al., 2023) mark significant progress toward this goal, demonstrating remarkable proficiency across a range of tasks, including mathematical reasoning (Zhou et al., 2023; Zhao et al., 2023; Zheng et al., 2023) . Specifically, methods like Program Aided Language Model (PAL) (Gao et al., 2023) as well as Program of Thoughts (PoT) (Chen et al., 2023) have demonstrated improvements in LLMs' ability to solve complex mathematical problems. These methodologies empower LLMs to formulate programs as intermediate reasoning steps and delegate the execution of these steps to a Python interpreter, thereby enhancing computational accuracy.However, open-source LLMs like those referenced in (Touvron et al., 2023; Rozière et al., 2023; Jiang et al., 2023) demonstrate limited success in math reasoning tasks. For example, after fine-tuning on the GSM8K-PAL dataset provided by Jie & Lu (2023) , Mistral (7B) achieves just 70.4% accuracy on GSM8K (Cobbe et al., 2021) (Ref Table 4 ). Our analysis of the fine-tuned Llama 2 (7B), Code Llama (7B) and Mistral (7B) reveals challenges in solving math word problems with multi-unit quantities. These issues are more pronounced in multi-step reasoning, where early errors can lead to incorrect final solutions. Our study thus identifies specific challenges the model faces.We propose a methodological framework to enhance the reasoning capabilities of LLMs by introducing a unit system for quantities and enforcing unit consistency. Ensuring unit consistency is crucial for accurate solutions in the context of mathematical word problems. To achieve this, we introduce Unit Consistency Programs (UCPs) (Figure 1 ) designed to enhance LLMs' reasoning abilities by enabling them to selfverify unit consistency within equations. UCPs consist of Counter objects responsible for tracking variable units and assert statements generated following each equation involving an operation. These assert statements verify the consistency of units within the equation and can trigger an assert error when inconsistent units are detected.We have developed a dataset that pairs math word problems with unit consistency programs containing unit specifications and verification routines. Our preliminary study presents the outcomes of fine-tuning Llama 2 (7B), Code Llama (7B) , and Mistral (7B) using these programs. Although our approach, which incorporates unit consistency, currently slightly underperforms compared to a non-unitconsistent approach, we conducted an in-depth error analysis to understand the reasons behind this discrepancy and proposed several options for future improvements. 
motivating analysis: 
 Recent studies have utilized the concept of program-based prompting to generate pseudo-gold programs as an intermediary step for training smaller models (Jie & Lu, 2023; Zhu et al., 2023) . Notably, this approach has shown promising outcomes, although these results still fall short of the performance achieved by larger models like GPT-4 (OpenAI, 2023) . To better comprehend the gaps in the mathematical reasoning abilities of smaller models, we fine-tuned Llama 2 (7B) (Touvron et al., 2023) , Code Llama (7B) (Rozière et al., 2023) , and Mistral (7B) (Jiang et al., 2023) using the GSM8K-PAL dataset provided by Jie & Lu (2023) , and conducted a comprehensive analysis of the fine-tuned models. The GSM8K-PAL dataset contains approximately 6.8k word problems paired with their PAL annotations in the training dataset as shown in Table 1 .After fine-tuning these models on GSM8K-PAL, we observed that they struggle with math word problems involving multiple different units. As illustrated in Figure 1 (top), the example illustrates a unit mismatch in the model trained on the PAL-based approach. Specifically, the subtraction operation between variables shirt cost × num shirts and discount amount is incorrect. The units are incompatible: the former is in dollar, and the latter is in dollar/shirt.To support our observation that the model struggles with problems containing multiple units, we employed GPT-3.5 Turbo1 to categorize the examples from both the train and test splits into two distinct groups. The first group comprises of questions involving a single unit, while the second group comprises of questions with multiple units. This classification was achieved using few-shot prompting, with GPT-3.5 Turbo serving as the backend engine. The specifics of the few-shot prompt utilized are detailed in Appendix A.2, and the distribution of these categories is presented in Table 2 . Our analysis reveals that approximately 40% of the problems in both training and test splits involve multiple units.To further evaluate the accuracy of GPT-3.5 Turbo in identifying questions with multiple units, we conducted a smallscale human assessment, detailed in Table 3 . The first author manually annotated 100 randomly selected test examples from GSM8K and compared the annotations with the classifications made by a precision of 80.4%, indicating that GPT-3.5 Turbo generally excels in predicting questions involving multiple units.We have extended this analysis to the SVAMP (Patel et al., 2021) , as presented in Appendix A.3, to demonstrate that this phenomenon is not exclusive to GSM8K.Based on the test dataset split we collected, we divided the accuracy of the fine-tuned models into two categories: one for questions with a single unit and another for questions with multiple units. This categorization is shown in Table 4 .A detailed examination of Table 4 reveals that our observations remained consistent across all three fine-tuned models, indicating superior performance on single-unit problems compared to those with multiple units. Motivated by these findings, we developed Unit Consistency Programs (UCPs) aimed at addressing the limitations inherent in PAL-based solutions. 
unit consistency programs: 
 Unit consistency checks are essential safeguards, helping to identify and prevent errors from inconsistent units in mathematical equations. In contrast to PAL/PoT approaches that directly generate programs to solve math word problems, our method enhances these programs by integrating specialized Counter objects. These objects are responsible for tracking variable units and ensuring the correct handling of operations with differing units. Additionally, we incorporate assert statements after each equation, as illustrated in Figure 1 (bottom). These assert statements verify unit consistency within equations, triggering an error if unit mismatches are detected.Consider the example in Figure 1 (bottom), illustrating a multiplication operation between shirts count (measured in 'shirts') and cost per shirt (measured in 'dollars per shirt'). In this operation, the units of 'shirts' from x for which a matching program couldn't be obtained was discarded.We fine-tuned smaller models with our annotated dataset D ucp through standard causal language modeling techniques. The objective is to generate a corresponding Python program P for a given math word problem x. After finetuning, the model was used to generate Python programs, which were then executed using a Python interpreter to obtain the final answer. We employed strong open-source LLMs such as Llama 2 (7B), Code Llama (7B), and Mistral (7B) as our models to fine-tune.  
dataset: 
 We conducted our experiments primarily on GSM8K, employing few-shot prompting with GPT-4 for the first 1,000 examples2 and GPT-4 Turbo for the remaining 6,473 examples3 in the GSM8K train dataset. We used six manually crafted Unit Consistency Programs (UCPs) samples, as detailed in Appendix A.1. We successfully annotated 59.9% of the GSM8K train dataset, creating our annotated UCPs dataset, D ucp . Table 1 presents the statistics of our UCPs dataset.Our baseline models consist of different models such as Llama 2 (7B), Code Llama (7B), and Mistral (7B) fine-tuned on GSM8K-PAL. We use this as a direct baseline to our method as it provides a more effective comparison between our method UCPs and existing methods like PAL/POT since our UCPs serve as extensions to typical Python programs used for solving mathematical problems, as demonstrated in PAL/POT.We conducted fine-tuning experiments on GSM8K-PAL and UCPs, details of both datasets can be found in Table 1 . In our fine-tuning experiments, we utilized the QLoRA technique (Dettmers et al., 2023) for enabling efficient fine-tuning. All QLoRA hyper-parameters were set as presented in Dettmers et al. (2023) . In all our experiments we use NF4 with double quantization and bf16 computation datatype. We set LoRA The program is unable to compile. r = 64, α = 16, and add LoRA modules on all linear layers of the base model. We also use max grad norm of 0.3 and LoRA dropout of 0.1. We use AdamW optimizer and set the learning rate to 2e -4 , with a batch size of 32 and a maximum context length of 1024. We trained the model for 20 epochs using 4 A100 40 GB GPUs which took roughly 14 hours and evaluated it on the test dataset.Our model, VerityMath-Mistral (7B), fine-tuned on UCPs achieved an overall accuracy of 68.2% on the GSM8K test dataset. Specifically, it attained 71.5% accuracy for problems involving a single unit and 63.7% accuracy for those with multiple units, as detailed in Table 4 . When compared to the Mistral (7B) (PAL) baseline, VerityMath-Mistral (7B) exhibits a slight overall accuracy decrease of 2.2%. Meanwhile, VerityMath-Code Llama (7B) and VerityMath-Llama 2 (7B) experienced more significant declines in their overall accuracy, approximately 9% lower than their respective PAL counterparts. Specifically, VerityMath-Code-Llama achieved 54.2% overall accuracy, with 58.4% for single unit problems and 48.6% for multiple units, while VerityMath-Llama-2 achieved an overall accuracy of 46.2%, with 51.9% for single unit and 38.7% for multiple units.In this section, we conducted an in-depth analysis of the potential causes for the decline in overall accuracy in the GSM8K test dataset. We focused on VerityMath-Mistral (7B) for all of our analysis. Correctness of Counter and assert statements In an error analysis of VerityMath-Mistral (7B) outputs from the test dataset, we observed some challenges that led to decreased performance, specifically, the correctness of Counter and assert statements. We reran the whole evaluation but this time, when we were met with a program that raised an assertion error, we removed the Counter and assert statements and executed the programs again. If the program compiles and produces the correct answer after this modification, it indicates that the program was originally incorrect due to incorrect Counter or assert statements.Referring to Figure 2 , we observed a notable percentage of output programs that contained incorrect Counter or assert statements in VerityMath-Mistral (7B) outputs. Specifically, 2.3% of the problems with single units and 8.0% of the problems with multiple units have incorrect Counter and assert which caused correct programs that would have resulted in the correct answer to have a false assertion error resulting in the wrong answer. Examples of such cases with incorrect Counter and assert are shown in Appendix A.5.2.We further conducted a detailed analysis of code solutions categorized by the number of assert statements, as shown in Figure 3 . Each bar represents the total number of code solution that consists of a specific number of assert statements.The green segments of the bars indicate the count of code solutions that resulted in the correct answer, while the red segments represent those that resulted in an incorrect answer. The percentage of correct answers is annotated on each bar for clarity. It is evident from the plot that the percentage of correct answers generally decreases as the number of assert statements increases, from code solutions with 2 to 4 assert statements having approximately 70% accuracy to code solutions with 5, 6, and 7 assert statements having 55.7%, 62.5%, and 40.9% respectively. Highlighting a trend where more complex code solutions with more assert statements are more likely to result in incorrect answers. This aligns with the earlier observations regarding the correctness of assert statements, and suggests that with more assert statements in the code solution, it is more prone to having errors due to the incorrect assert statements which would then result in a wrong answer. In our in-depth anaysis, we identified a notable bottleneck in our current method, which is the correctness of Counter and assert statements. This issue led to a slight decrease in performance. Our method, UCPs, is a relatively more complex method for existing 7B LLMs to learn, but with a significant increase in dataset annotations, it is highly possible that our method will outshine the existing PAL method.Another approach could involve data augmentation using synthetic examples (Wu et al., 2021) . Examples showcasing the efficacy of UCPs are available in Appendix A.5.1. 
related work: 
 Our research builds upon the Program of Thoughts (PoT) approach (Chen et al., 2023) and the Program Aided Language Model (PAL) (Gao et al., 2023) methodologies, which have shown effectiveness in solving mathematical problems. These approaches have outperformed techniques like the Chain-of-Thought (CoT) (Wei et al., 2022) , which can struggle with computational inaccuracies (Lewkowycz et al., 2022) . We extend their work by focusing on the use of programs for solving math word problems and the concept of self-verification to improve LLMs' reasoning capabilities.The advancement of GPT models (Brown et al., 2020) has inspired various studies (Ho et al., 2023; Fu et al., 2023; Magister et al., 2023; Shridhar et al., 2023) on creating synthetic datasets for fine-tuning smaller models (Hinton et al., 2015) . Notably, Zhu et al. (2023) In mathematical problem-solving, ensuring solution validity is crucial due to hallucinations in LLMs (Bubeck et al., 2023) and challenges in executing multiplications (Dziri et al., 2023) . Prior research has focused on training additional verifiers for answer accuracy (Cobbe et al., 2021) , providing feedback for each intermediate reasoning step (Lightman et al., 2023) , and integrating tools to agents (Gou et al., 2024) . However, Weng et al. (2023) and Miao et al.(2023) have shown potential for LLMs to self-verify solutions. Our approach builds on these insights, incorporating programs for solving math word problems and leveraging self-verification to enhance LLM reasoning. 
conclusion and future work: 
 In this study, we analyzed open-source Large Language Models (LLMs) and pinpointed their struggle with math problems involving multiple units, highlighting a key improvement area. We introduced Unit Consistency Programs (UCPs) as a novel method to address LLMs' reasoning and verification abilities, especially in complex math problems. We identified some limitations in our current approach. Future work will focus on advancing unit check methodologies in UCPs to address these limitations. 
 
 Appendix: Recent creations of synthetic datasets for math problemsolving often rely on prompting large language models (LLMs), such as GPT-4. However, this approach can be costly, and on a large scale, the expenses escalate. Our dataset creation incurred a total cost of approximately $350 USD. Due to budget constraints, we couldn't sample multiple reasoning paths per question, as presented in Wang et al. (2023) , limiting the potential for increased annotations.This paper presents work whose goal is to advance the field of math problem-solving using LLMs. However, it is also crucial to be aware of the potential risks associated with VerityMath. Due to the current challenges of VerityMath, the units initialized by Counter and assert statements may not always be accurate. Consequently, it is strongly recommended to exercise caution when relying on VerityMath outputs for any use.Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., and Severyn, A. Teaching small language models to reason. In Rogers, A., Boyd-Graber, J., and Okazaki, N. " " " T h e r e i s a g r o u p o f 10 p e o p l e who a r e o r d e r i n g p i z z a . I f e a c h p e r s o n g e t s 2 s l i c e s and e a c h p i z z a h a s 4 s l i c e s , how many p i z z a s s h o u l d t h e y o r d e r ? " " " p e o p l e t o t a l = 10 p e o p l e t o t a l u n i t = C o u n t e r ( { " p e o p l e " : 1}) p i z z a s l i c e s p e r p e r s o n = 2 p i z z a s l i c e s p e r p e r s o n u n i t = C o u n t e r ( { " s l i c e s " : 1 , " p e o p l e " : -1}) p i z z a s l i c e s t o t a l = p e o p l e t o t a l * p i z z a s l i c e s p e r p e r s o n p i z z a s l i c e s t o t a l u n i t = C o u n t e r ( { " s l i c e s " : 1}) a s s e r t p i z z a s l i c e s t o t a l u n i t == p e o p l e t o t a l u n i t + p i z z a s l i c e s p e r p e r s o n u n i t s l i c e s p e r p i z z a = 4 s l i c e s p e r p i z z a u n i t = C o u n t e r ( { " s l i c e s " : 1 , " p i z z a " : -1}) p i z z a t o t a l = p i z z a s l i c e s t o t a l / s l i c e s p e r p i z z a p i z z a t o t a l u n i t = C o u n t e r ( { " p i z z a " : 1}) a s s e r t p i z z a t o t a l u n i t == p i z z a s l i c e s t o t a l u n i t -s l i c e s p e r p i z z a u n i t r e s u l t = p i z z a t o t a l r e t u r n r e s u l t Q u e s t i o n : Lana h a s 2 b a g s w i t h 2 m a r b l e s i n e a c h bag . x f e n c e u n i t = C o u n t e r ( { " m i n u t e s " : 1 , " d a y s " : -1}) t o t a l d a y s t o f i x f e n c e = m i n u t e s t o f i x f e n c e / m i n u t e s p e r d a y t o f i x f e n c e t o t a l d a y s t o f i x f e n c e u n i t = C o u n t e r ( { " d a y s " : 1}) a s s e r t t o t a l d a y s t o f i x f e n c e u n i t == m i n u t e s t o f i x f e n c e u n i t -m i n u t e s p e r d a y t o f i x f e n c e u n i t r e s u l t = t o t a l d a y s t o f i x f e n c e r e t u r n r e s u l t A.1.2. SYSTEM PROMPT You are an experienced professional skilled in using python programs to solve math problems and enhancing code robustness through the strategic implementation of unit counters and assert statements to ensure flawless unit-level consistency checks. You will only write code blocks. the cost of each wash (dollars). By performing a unit consistency check, we can ensure that the conversions between different units are accurate and that the final result makes sense in the context of the problem. In this case, we can calculate the total number of hair washes per bottle by dividing the total volume of the bottle by the volume used per hair wash, and then calculate the total money made by multiplying the number of hair washes by the cost of each hair wash. Answer: Yes.Question: Natalie and Liam have a total of 72 stickers. However, Natalie has half the number of stickers as compared to Liam. How many stickers does Liam have?In this problem, a unit consistency check is not required to ensure the accuracy of measurements and calculations. We are dealing with a single unit -the number of stickers. The calculations involve simple addition, subtraction, multiplication, and division of the same unit, so there is no need for unit conversions or comparisons between different units. In this case, all the quantities are given in terms of the number of stickers, and there are no conflicting units or measurements that would require a unit consistency check. Answer: No.Question: Emma makes $35,000 a year. She spends $12,000 on rent every year, $3,000 on groceries every year, and $4,500 on gas every year. If she wants to have at least $15,000 left, how much more money must she make each year?In this problem, a unit consistency check is not required to ensure the accuracy of measurements and calculations. We are dealing with a single unit -money (dollars). The calculations involve simple addition and subtraction of the same unit, so there is no need for unit conversions or comparisons between different units. In this case, all the quantities are given in terms of dollars, and there are no conflicting units or measurements that would require a unit consistency check. Answer: No.Question: Maria pays $35/hour for 4 hours per week of flute lessons and $22/hour for 6 hours a week of guitar lessons. How much more does she spend on guitar lessons than flute lessons in a year?In this problem, a unit consistency check is required to ensure the accuracy of measurements and calculations. We are dealing with different units -the cost per hour (dollars/hour) and the number of hours per week. By performing a unit consistency check, we can ensure that the conversions between different units are accurate and that the final result makes sense in the context of the problem. In this case, we can calculate the total cost of flute lessons per week by multiplying the cost per hour by the number of hours, and then calculate the total cost of guitar lessons per week by multiplying the cost per hour by the number of hours. Finally, we can calculate the difference in cost between guitar and flute lessons in a year by subtracting the total cost of flute lessons from the total cost of guitar lessons and multiplying it by the number of weeks in a year. Answer: Yes.Create a coherent explanation for the importance of performing unit consistency checks in various mathematical problems and identify whether a question requires verification of unit consistency. Be as detailed as possible and write your explanation in 1 paragraph and end with Answer: Yes / No. The SVAMP dataset comprises a total of 1000 examples, with 700 allocated to the train dataset and 300 to the test dataset.The dataset encompasses four problem types: subtraction, addition, common-division, and multiplication. However, our analysis focuses solely on multiplication and common-division, as problems involving only addition or subtraction are defined to only consist of a single unit. We can observe from 5 that 46.9% and 58% of the problems are classified as multiple units in the train and test dataset respectively. " " " K e l l y h a s 5 q u a r t e r s and 2 d i m e s . I f s h e b u y s a c a n o f pop f o r 55 c e n t s , how many c e n t s w i l l s h e h a v e l e f t ? " " " q u a r t e r s = 5 q u a r t e r s u n i t = C o u n t e r ( { " q u a r t e r s " : 1}) d i m e s = 2 d i m e s u n i t = C o u n t e r ( { " d i m e s " : 1}) c e n t s p e r q u a r t e r = 25 c e n t s p e r q u a r t e r u n i t = C o u n t e r ( { " c e n t s " : 1 , " q u a r t e r s " : -1}) c e n t s p e r d i m e = 10 c e n t s p e r d i m e u n i t = C o u n t e r ( { " c e n t s " : 1 , " d i m e s " : -1}) t o t a l c e n t s q u a r t e r s = q u a r t e r s * c e n t s p e r q u a r t e r t o t a l c e n t s q u a r t e r s u n i t = C o u n t e r ( { " c e n t s " : 1}) a s s e r t t o t a l c e n t s q u a r t e r s u n i t == q u a r t e r s u n i t + c e n t s p e r q u a r t e r u n i t t o t a l c e n t s d i m e s = d i m e s * c e n t s p e r d i m e t o t a l c e n t s d i m e s u n i t = C o u n t e r ( { " c e n t s " : 1}) a s s e r t t o t a l c e n t s d i m e s u n i t == d i m e s u n i t + c e n t s p e r d i m e u n i t t o t a l c e n t s = t o t a l c e n t s q u a r t e r s + t o t a l c e n t s d i m e s t o t a l c e n t s u n i t = C o u n t e r ( { " c e n t s " : 1}) a s s e r t t o t a l c e n t s u n i t == t o t a l c e n t s q u a r t e r s u n i t == t o t a l c e n t s d i m e s u n i t c e n t s n e e d e d f o r p o p = 55 c e n t s n e e d e d f o r p o p u n i t = C o u n t e r ( { " c e n t s " : 1}) c e n t s l e f t = t o t a l c e n t s -c e n t s n e e d e d f o r p o p c e n t s l e f t u n i t = C o u n t e r ( { " c e n t s " : 1}) a s s e r t c e n t s l e f t u n i t == t o t a l c e n t s u n i t == c e n t s n e e d e d f o r p o p u n i t r e s u l t = c e n t s l e f t r e t u r n r e s u l t => Example 287 i n t h e t e s t d a t a s e t . " " " Milo i s making a m o s a i c w i t h c h i p s o f g l a s s . I t t a k e s t w e l v e g l a s s c h i p s t o make e v e r y s q u a r e i n c h o f t h e m o s a i c . A bag o f g l a s s c h i p s h o l d s 72 c h i p s . Milo w a n t s h i s m o s a i c t o be t h r e e i n c h e s t a l l . I f he h a s two b a g s o f g l a s s c h i p s , how many i n c h e s l o n g c a n he make h i s m o s a i c ? " " " c h i p s p e r s q u a r e i n c h = 12 c h i p s p e r s q u a r e i n c h u n i t = Counter("chips": 1, "square inches": -1) c h i p s p e r b a g = 72 c h i p s p e r b a g u n i t = C o u n t e r ( { " c h i p s " : 1 , " b a g s " : -1}) b a g c o u n t = 2 b a g c o u n t u n i t = C o u n t e r ( { " b a g s " : 1}) t o t a l c h i p s = c h i p s p e r b a g * b a g c o u n t t o t a l c h i p s u n i t = C o u n t e r ( { " c h i p s " : 1}) a s s e r t t o t a l c h i p s u n i t == c h i p s p e r b a g u n i t + b a g c o u n t u n i t t o t a l s q u a r e i n c h e s = t o t a l c h i p s / c h i p s p e r s q u a r e i n c h t o t a l s q u a r e i n c h e s u n i t = Counter("square inches": 1) a s s e r t t o t a l s q u a r e i n c h e s u n i t == t o t a l c h i p s u n i t -c h i p s p e r s q u a r e i n c h u n i t m o s a i c h e i g h t i n c h e s = 3 m o s a i c h e i g h t i n c h e s u n i t = C o u n t e r ( { " i n c h e s " : 1}) m o s a i c w i d t h i n c h e s = t o t a l s q u a r e i n c h e s / m o s a i c h e i g h t i n c h e s m o s a i c w i d t h i n c h e s u n i t = C o u n t e r ( { " i n c h e s " : 1}) a s s e r t m o s a i c w i d t h i n c h e s u n i t == t o t a l s q u a r e i n c h e s u n i t -m o s a i c h e i g h t i n c h e s u n i t r e s u l t = m o s a i c w i d t h i n c h e s r e t u r n r e s u l t => Example 808 i n t h e t e s t d a t a s e t . I n t h i s c a s e t h e v a r i a b l e c h i p s p e r s q u a r e i n c h u n i t s h o u l d h a v e a C o u n t e r ( { " c h i p s " : 1 , " i n c h e s " : -2}) and t o t a l s q u a r e i n c h e s u n i t s h o u l d h a v e a C o u n t e r ( { " i n c h e s " : 2}) .Turbo annotations were obtained in Septemberannotations obtained in September 2023.Turbo annotations obtained in December 2023. 


Decision: Acceptable paper. It presents a novel self-verification mechanism of mathematical expressions through unit consistency. Despite not obtaining positive experimental results, the proposed idea is novel, and the contributed dataset is useful for further research. Reviewer et4K has made some good points about the weaknesses of the paper, but overall, I think this paper makes a significant contribution and is suitable for presentation in the workshop.

Reviews: review: paper_summary: paper_summary: This paper introduces VerityMath, an innovative approach aimed at enhancing mathematical reasoning and problem-solving capabilities by emphasizing the self-verification of mathematical expressions through unit consistency. The primary contribution of this work is the development of a framework that allows for the automatic verification of units within mathematical equations, ensuring their consistency and correctness. Additionally, a dataset has been made open source for future research.
strength_weakness: strength_weakness: 1. Unfortunately, there is no comparison with other verification baseline models, such as [1].  
2. What I am more curious about is whether this kind of verification can calibrate the confidence of the model. Some simple experiments can be carried out, such as self-consistency. Maybe this method can break through the upper bound of the voting method.


**Reference**  
[1] Weng Y, Zhu M, Xia F, et al. Large language models are better reasoners with self-verification[J]. arXiv preprint arXiv:2212.09561, 2022.  
[2] Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models[J]. arXiv preprint arXiv:2203.11171, 2022.
questions: questions: 1. Can you compare with more baseline models?  
2. Why is the accuracy of the paper method based on GPT-3.5-Turbo and GPT-4 not included in Table 4?
paper_summary: paper_summary: The paper studies the performance of open-source LLMs (Llama 2, Code Llama, Mistral) on math word problems, highlighting challenges with quantities in multiple units. The authors introduce Unit Consistency Programs (UCPs) to ensure unit consistency during operations. Fine-tuning these models with UCPs led to VerityMath variants, which slightly underperformed compared to non-unit-consistent approaches. The paper includes an error analysis and suggests future improvements.
strength_weakness: strength_weakness: - The experimental results do not support the effectiveness of the method.
questions: questions: Do the authors have any ideas to make it work? or explain why UCPs  slightly underperformed the non-unit-consistent approaches?
paper_summary: paper_summary: The work tackles the challenging math questions in the GSM8K dataset by creating a new Counter (which holds the "measure unit") and the assert statement to help the model learning the answer and the Counter. This should help the model to improve the mathematical reasoning, at least in theory.
In practice authors show that finetuning the model delivers worse results, and authors investigate the reasons.
strength_weakness: strength_weakness: W1. The analysis section (4.5) lacks of a comprehensive explanation. From the learning of section 4.5, it follows that the model is not learning the Counter/assert syntax and more assert is actually penalizing. With this in mind, I would like to see how to incorporate this information in a new training phase. 

W2. The impact of training annotations is somewhat pretty generic. It's clear that the in-domain performance is driven from more examples, but it's not clear if the model is too small or if the dataset is too small. It's also not clear how to augment this dataset, given the task is pretty specific and not AI friendly. 

W3. The experiment area is very limited. For example, it would be interesting to see why pretrained models have less gap between multiple and single (table 4) compared to finetuned models. I would also be interested in a task where the unit measure is more relevant (maybe physics?).
questions: questions: Related to W1. 

- Maybe the authors could remove some "easy" asserts from the training dataset? 

- Or did the model even learn that these tokens are somewhat important? 

- Could we build a customize loss to generate a coherent Counter and assert expression?

Related to W2. 

- Did the authors experimented with smaller models and found a "scaling law"? Did you try with, say, 100 or 500 examples only?