Venue: NAACL 2024

# Paraphrase and Solve: Exploring and Exploiting the Impact of Surface

Form on Mathematical Reasoning in Large Language Models

 Yue Zhou\({}^{1*}\) Yada Zhu\({}^{2}\) Diego Antognini\({}^{2}\) Yoon Kim\({}^{3}\) Yang Zhang\({}^{2}\)

\({}^{1}\)University of Illinois Chicago \({}^{2}\)MIT-IBM Watson AI Lab, IBM Research MIT CSAIL yzhou232@uic.edu, yzhu@us.ibm.com

yoonkim@mit.edu, {diego.antognini, yang.zhang2}@ibm.com

This work was done during the first author's internship at MIT-IBM Watson AI Lab.

###### Abstract

This paper studies the relationship between the surface form of a mathematical problem and its solvability by large language models. We find that subtle alterations in the surface form can significantly impact the answer distribution and the solve rate, exposing the language model's lack of robustness and sensitivity to the surface form in reasoning through complex problems. To improve mathematical reasoning performance, we propose Self-Consistency-over-Paraphrases (SCoP), which diversise reasoning paths from specific surface forms of the problem. We evaluate our approach on four mathematics reasoning benchmarks over three large language models and show that SCoP improves mathematical reasoning performance over vanilla self-consistency, particularly for problems initially deemed unsolvable. Finally, we provide additional experiments and discussion regarding problem difficulty and surface forms, including cross-model difficulty agreement and paraphrasing transferability, and Variance of Variations (VOV) for language model evaluation.

## 1 Introduction

Despite the impressive performance of large-scale language models (LLMs) across many tasks, their ability to reason through complex problems such as mathematics remains a bottleneck [50, 48, 6]; they can solve problems that are challenging for humans but can also struggle with seemingly simple ones. This raises the following question: what factors contribute to the difficulty of a math problem for an LLM?

Specifically, the information in a math problem can be divided into two types. The first is the _semantic information_, which involves the problem specification and knowledge involved in the math problem. The second is the _surface form_, _i.e._, how the questions, assumptions, and constraints are described in the math problem. Intuitively, the semantic information should be the primary determining factor of the difficulty of the math problem, and surface form should only have a marginal impact. This paper investigates the extent to which this is true for LLMs.

In this paper, we explore into the relationship between the problem's surface form and its solvability by an LLM. Specifically, we follow the self-consistency setting [79] to sample multiple answers to the same math problem and compute _solve rate_ as the percentage of correct answers. Our primary finding is that, counter-intuitively, subtle alterations in the surface form of a math problem can significantly impact the answer distribution and solve rate. Consider the example in Figure 1, where the left and right panels contain an identical math problem described in two different ways. Despite the no change in problem semantics, the solve rate increases from 5% to 100%, with all reasoning paths leading to the correct answer - what initially appears to be a difficult problem to the language model transforms into an easily solvable one. This phenomenon exposes the language model's lack of robustness and sensitivity to the surface form in reasoning through complex problems.

Motivated by this finding, we propose to improve the mathematical reasoning performance of the language model by diversifying reasoning paths from specific surface forms of the problem. We leverage the language model's paraphrasing ability to generate surface forms with identical semantics1 and propose **Self-Consistency-over-Paraphrases (SCOp)**, which consists of two steps: For each math problem, generate \(K\) paraphrase using an

LLM; and A Ask the LLM to generate \(N/K\) reasoning paths for each paraphrase, and then select the most consistent answer among the \(N\) answers. The intuition is that if a problem exhibits a low solve rate and ineffective reasoning paths due to its original surface form, introducing diversity in its surface forms can be beneficial. We also introduced in-context exemplars to the language model when paraphrasing, which are the paraphrases that obtain a solve rate improvement over their original problem, aiming to generate surface forms with the same semantics yet a higher solve rate through language models' in-context learning abilities Min et al. (2022); Brown et al. (2020).

We evaluate our approach on four mathematics reasoning benchmarks: GSM8K Cobbe et al. (2021), AUoA Ling et al. (2017), MATH Hendrycks et al. (2021), and MMLUMath Hendrycks et al. (2020), over three large language models: LLaMA-2-70b Touvron et al. (2023), GPT-3.5-turbo and GPT-4 OpenAI (2023). Our experiments show that SCoP improves mathematical reasoning performance over vanilla SelfConsistency, particularly for problems initially deemed unsolvable. In additional experiments, we show that the difficulty ranks across language models are positively correlated, with higher agreement within the GPT model family and simpler datasets. We propose Variance of Variations (VOV) as a metric for evaluating language model robustness against surface form variations. Finally, we explain why SCoP can be effective using a data difficulty map based on the entropy of answer distribution and the solve rate. Our code is publicly available.2

Footnote 2: [https://github.com/Yue-LUM-Pit/SCop/](https://github.com/Yue-LUM-Pit/SCop/)

## 2 Problem Difficulty and Surface Forms

In this section, we present our pilot study of the impact of surface form on LLMs' ability to solve the problem. In all our studies, we follow the selfconsistency setting Wang et al. (2022), which extends over chain-of-thought Wei et al. (2022) by using sampling to generate a variety of reasoning paths. From this setting, we quantify the _difficulty_ of a problem _w.r.t_ a language model by its **solve rate**, which is the proportion of the reasoning paths that lead to the correct answer. When the solve rate exceeds 50%, a majority vote guarantees the correct answer. Note the solve rate measures the difficulty of a single problem input and is also a model-dependent metric.

To study how surface form impacts the solve rate, we use the math word problem from the GSM8K dataset Cobbe et al. (2021). For each math problem, we generate a paraphrase using GPT-3.5-turbo3 (detailed instructions are shown in Appendix E). We then compare the solve rates of the original problem and the paraphrase solved by GPT-3.5-turbo using self-consistency with \(N=40\) and a temperature of \(0.7\).

Footnote 3: [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5)

Our finding is that the solve rate varies significantly across the surface forms. Figure 1 shows an example with the original problem on the left and the paraphrased one on the right. In the original problem, the reasoning paths result in a disarrayed answer distribution, with merely 5% achieving the correct answer "40" and the aggregated answer "1.54" (20%). In contrast, the solve rate of the paraphrase problem is 100%. W

Figure 1: Comparison of the answer distribution and solve rate between surface form variations of a math word problem from GSM8K, when prompted to GPT-3.5-turbo using Self-Consistency, with 40 sampled reasoning paths. Solve rate can vary dramatically between surface forms with equivalent semantics.

 

more such examples with drastic improvement in solve rate, presented in Table 6.

We further calculate the histogram of the solve rate changes in the paraphrased problem compared to the original one, shown in Figure 2. As can be observed, the distribution is heavy-tailed, with 11.7% of the paraphrases resulting in over 25% absolute improvement in solve rate and with 13% resulting in over 25% absolute deterioration.

This phenomenon exposes the language model's lack of robustness and sensitivity to a comprehensive problem's surface form. It suggests that the challenge of some problems may not be due to the model's limitations, but rather the ineffective generation of reasoning paths from certain surface forms. We therefore seek to take advantage of this phenomenon to improve language model reasoning through surface form modifications, mirroring the way paraphrasing aids a student's cognitive and problem-solving processes Swanson et al. (2019).

## 3 Self-Consistency over Paraphrases

Motivated by the findings in Section 2, we propose a framework called **Self-Consistency-over-Paraphrases (SCoP)**, which leverages the LLMs to generate paraphrases of math problems to improve their ability in solving them.

### Framework Overview

As shown in Figure 3, SCoP consists of two steps.

\(\bullet\)**Step 1: Paraphrase.** Prompt the LLM to generate \(K\) paraphrases of the original problem. For notational ease, denote \(p\) as the original problem, and \(\bigcup_{k=1}^{K}\{q_{k}\}\) as the \(K\) paraphrases.

\(\bullet\)**Step 2: Solve.** For each paraphrase, we ask the LLM to generate \(N/K\) reasoning paths, and thus the total number of generated answers is \(N\). We then select the most consistent answer across the \(N\) reasoning paths as the final answer.

The intuition behind SCoP is that if a problem exhibits a low solve rate and ineffective reasoning paths due to its original surface form, introducing diversity in its surface forms would be beneficial.

There are two important notes regarding SCoP. First, when we increase \(K\), the total number of reasoning paths \(N\) is held fixed, which separates the effect of increasing the diversity of reasoning paths from increasing the number of reasoning paths. This also ensures a fair comparison with other self-consistency baselines.

Second, there are two procedures in SCoP that involve an LLM, one to generate paraphrases (Step 1) and one to generate answers (Step 2). We use the _same_ LLM to perform both tasks. In this way, we can ensure that any performance improvement of SCoP is due to the diversity of paraphrasing itself, rather than cross-sharing of knowledge across different LLMs. In addition, there is no human annotation, training, fine-tuning, or auxiliary models involved in our SCoP framework.

### Paraphrase Generation

The paraphrase generation in Step 1 is crucial to the success of SCoP. In this work, we explore two paraphrase generation methods.

Naive.The naive approach instructs the language model to generate \(K\) paraphrases of the math problem. However, this could generate many paraphrases with _worse_ solve rate, because the solve rate change has high variability in both directions (as shown in Figure 2).

In-Context Learning.To increase the chance of generating 'good' paraphrases, we propose an

Figure 3: A comparison between Self-Consistency and our SCoP. SCoP splits \(N\) reasoning paths over \(K\) instructured learned paraphrases, instead of devoting all \(N\) reasoning paths to the single original problem \(P\). The final answer is selected by aggregating all reasoning paths from these paraphrases with a majority vote.

Figure 2: GSM8K - solve rate difference - from original to one of the random naïve paraphrases.

 

in-context learning approach,4 where we obtain \(N_{shot}\) 'good' paraphrases as the in-context exemplars (marked as [Exemplars] in Figure 3). The 'good' paraphrases are formally defined as paraphrases that contribute to a solve rate improvement (by a preset margin \(\delta\)) over the original problem. To obtain the 'good' paraphrases, we first generate some candidate paraphrases using the aforementioned naive approach on a small number of math problems with labeled answers. We then compute the solve rate of the original problem and the paraphrases and select those whose improvement is over the margin \(\delta\). The detailed algorithm is presented in Algorithm 1.

Footnote 4: An alternative can be automatic prompt engineering, see Appendix B.

```
1:Input: Training data \(D^{tr}\), \(N_{shot}\), margin \(\delta\). Init. Candidates list \(C\).
2:for step \(t\) in \(\{1,2,\dots,T\}\)do
3:if\(\mathrm{Length}(\mathsf{C})=N_{shot}\)then
4:break
5: Sample a problem \(p\) from \(D^{tr}\) without replacement.
6: Compute solve rate \(\mathrm{SR}(p)\)
7: Obtain \(K\) Paraphrases \(\{q_{1},\dots,q_{K}\}\) of \(p\).
8:for\(k=1\) to K do
9: Compute solve rate \(\mathrm{SR}(q_{k})\)
10:if\(\mathrm{SR}(q_{k})>=\mathrm{SR}(p)+\delta\)then
11: Add {\(p\), \(q_{k}\)} to Candidates list \(C\).
12:break
13:end
```

**Algorithm 1** Paraphrase Exemplar Search

## 4 Experiments

In this section, we will describe our experiment results evaluating the effectiveness of SCoP, as well as additional studies on how SCoP works.

### Experimental Settings

DatasetsWe evaluate our approach on the following public mathematics reasoning benchmarks: \(\bullet\)**GSM8K**Cobbe et al. (2021) contains 8.5K linguistically diverse grade school-level math questions with moderate difficulties.

\(\bullet\)**AQuA**Ling et al. (2017) consists of 100K algebraic word problems, including the questions, the possible multiple-choice options, and natural language answer rationales from GMAT and GRE. \(\bullet\)**MATH**Hendrycks et al. (2021) is a competition mathematics dataset containing 12,500 problems with challenging concepts such as Calculus, Linear Algebra, Statistics, and Number Theory.

\(\bullet\)**MMLU**Hendrycks et al. (2020) is a comprehensive dataset containing various subjects. We specifically utilized the mathematics section of the dataset, which comprises college and high-school-level mathematics, statistics, and abstract algebra.

Language ModelsWe utilize three popular LLMs trained with RLHF Ouyang et al. (2022): LLaMA-2 (70B) Touvron et al. (2023), an open-source LLM by Meta AI, GPT-3.5-turbo (version 0613), and GPT-4 OpenAI, (2023), accessed via the OpenAI API. All experiments are conducted in zero-shot or few-shot settings, without training or fine-tuning the language models. We choose the temperature \(T=0.7\) and Top-p \(=1.0\) for sampling decoding for all three language models. The total number of reasoning paths \(N\) we sample for each problem is \(40\), following Wang et al. (2022).

Implementation DetailsFor paraphrase generation (Step 1), we evaluate the two aforementioned schemes \(\blacklozenge\)**Naive**: We use the template "_Paraphrase the following math problem:_ {question}" to prompt the language model to paraphrase the original problem; \(\ 

racy over hard problems, defined as the problems whose original accuracy is below 50%. The accuracies over all problems and hard problems are reported inside and outside parentheses respectively. HPR% (Hard Problem Ratio) denotes the percentage of such hard problems.

There are three general observations. First, SCoP with the two paraphrasing schemes both outperform the vanilla self-consistency baseline. Surprisingly, even the naive paraphrasing can lead to performance improvement, despite the high chances of generating paraphrases with a worse solve rate (see Figure 2). We will discuss a hypothesis in Section 5. Between the two schemes, ICL\({}_{\text{para}}\) consistently outperforms Naive. Second, the performance improvement generally increases as \(K\) increases. Third, more significant performance gain over LLaMA-2-70B.

The results further indicate that MATH and MMLU are considerably more challenging than GSM8K and AQuA, as evidenced by their high HPR% and low overall accuracy. Moreover, significant accuracy gains are from the original "Hard Problems", suggesting that changing surface forms can solve the problems initially deemed unsolvable by self-consistency. Finally, when solving the MATH dataset with LLaMA-2-70b, ICL\({}_{\text{para}}\) underperforms Naive paraphrasing. We hypothesize that the MATH problems present a significant challenge for LLaMA-2-70b, making it difficult to effectively learn paraphrasing from in-context examples.

Four-Shot CoTOne caveat of the zero-shot CoT results is that SCoP (ICL\({}_{\text{para}}\)) has indirect access to additional ground-truth information from inconsect exemplars. There is also a question of whether the advantage of SCoP over SC will diminish as both are exposed to more examples. To ensure a fair comparison and further validate the effectiveness of SCoP, Table 2 shows results under the four-shot CoT setting, where the baselines also have access to some ground-truth answer information. Due to resource constraints, we evaluate GPT4 with 100 random samples from each dataset. The results show that while four-shot CoT can improve SC and SCoP in general (compared with zero-shot CoT), SCoP still consistently outperforms SC over all three language models. The

\begin{table}
\begin{tabular}{l l c r r r r r r} \hline \hline  & \multicolumn{3}{c}{GPT-3.5-Turbo} & \multicolumn{3}{c}{LLaMA-2-70b} \\ \cline{2-9}  & \multicolumn{2}{c}{GSM8k} & \multicolumn{1}{c}{AQuA} & \multicolumn{1}{c}{MATH} & \multicolumn{1}{c}{MMLU} & \multicolumn{1}{c}{GSMRSK} & \multicolumn{1}{c}{AQua} & \multicolumn{1}{c}{MATH} & \multicolumn{1}{c}{MMLU} \\ \cline{2-9}  & HPR (\%) & 31.3 & 42.5 & 68.0 & 64.0 & 52.0 & 76.3 & 98.2 & 81.6 \\ \hline SC & & 76.3 (24.5) & 66.9 (22.2) & 59.0 (39.7) & 52.8 (26.3) & 58.7 (20.5) & 40.5 (22.0) & 10.5 (8.9) & 32.8 (17.4) \\ \hline \multirow{4}{*}{SCop} & \(k=1\) & 72.2 (27.7) & 63.4 (28.9) & 55.0 (37.5) & 48.4 (27.5) & 51.0 (28.2) & 38.1 (26.8) & 24.6 (23.2) & 27.2 (20.2) \\  & \(k=2\) & 76.0 (34.0) & 65.8 (28.9) & 56.5 (39.7) & 52.8 (32.5) & 54.3 (26.9) & 39.5 (25.0) & 29.8 (28.6) & 29.6 (22.3) \\ (Naïve) & \(k=4\) & 77.7 (36.2) & 67.3 (29.8) & 57.5 (39.0) & 56.0 (36.3) & 55.7 (32.1) & 41.4 (25.6) & **31.6 (30.4)** & 32.0 (24.9) \\  & \(k=8\) & 79.3 (39.4) & 68.1 (33.5) & 59.5 (43.4) & 55.6 (33.8) & 60.3 (33.3) & 41.4 (25.6) & 28.1 (26.8) & 35.6 (28.0) \\ \hline \multirow{4}{*}{SCop} & \(k=1\) & 77.9 (39.0) & 66.4 (29.8) & 64.0 (36.8) & 52.5 (32.6) & 58.7 (39.9) & 42.9 (29.8) & 23.4 (22.0) & 34.6 (23.7) \\  & \(k=2\) & **80.5 (39.2)** & **65.5 (31.7)** & 57.5 (39.1) & 55.5 (41.4) & 59.3 (36.3) & 43.7 (30.4) & 24.6 (32.3) & 37.6 (26.3) \\ (ICL\({}_{\text{para}}\)) & \(k=4\) & 79.2 (38.3) & **70.5 (35.4)** & 80.4 (12.4) & **58.0 (39.5)** & 61.7 (40.5) & 44.5 (30.4) & 26.9 (25.6) & **37.8 (26.5)** \\  & \(k=8\) & 80.2 **(40.6)** & 69.7 (34.4) & **60.0 (44.1)** & 56.5 (34.9) & **63.3 (40.5)** & **46.5 (31.9)** & 25.2 (23.8) & 37.6 (25.8) \\ \hline \end{tabular}
\end{table}
Table 1: Accuracy of SCoP distributing \(N/K\) reasoning paths over \(K\) in [1, 2, 4, 8] paraphrases in Naïve and ICLR\({}_{\text{para}}\) settings, against Self-Consistency (SC). Hard Problem Ratio (HPR%) represents the percentage of problems with an original solve rate \(\leq 0.5\) by Self-Consistency (SC). Accuracy is reported for both Hard Problems (HPR% \(\leq 0.5\)) (inside parentheses) and global accuracy across the entire dataset (outside parentheses).

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline Model & & GSM8K & AQuA & MATH & MMLU \\ \hline \multirow{3}{*}{ILaMA-2-70b} & HPR (\%) & 56 & 75.2 & 95.2 & 81.6 \\ \cline{2-5}  & Self-Consistency & 61.1 (30.0) & 44.1 (25.7) & 13.4 (9.2) & 34.4 (19.6) \\  & SCoP (ICL\({}_{\text{para}}\), \(k=8\)) & **65.1 (38.6)** & **48.8 (33.5)** & **23.6 (20.2)** & **36.4 (27.0)** \\ \hline \multirow{3}{*}{GPT-3.5-Turbo} & HPR (\%) & 22 & 36 & 75 & 62 \\ \cline{2-5}  & Self-Consistency & 80.0 (9.1) & 70.0 (16.7) & 51.6 (29.8) & 54.4 (26.9) \\ \cline{1-1} \cline{2-5}  & SCOP (ICL\({}_{\text{para}}\), \(k=8\)) & **82.0 (36.4)** & **74.0 (27.8)** & **57.6 (38.2)** & **58.4 (36.5)** \\ \hline \multirow{3}{*}{GPT-4} & HPR (\%) & 4 & 18 & 58 & 38 \\ \cline{2-5}  & Self-Consistency & 98.0 (50.0) & 84.0 (11.1) & 64.0 (37.9) & 74.0 (31.6) \\ \cline{1-1}  & SCoP (ICL\({}_{\text{para}}\), \(k=8\)) & 98.0 (50.0) & **86.0 (33.3)** & **66.0 (41.4)** & **78.0 (57.9)** \\ \hline \end{tabular}
\end{table}
Table 2: A comparison of the performance (accuracy) between SC and SCoP (ICL\({}_{\text{para}}\) paraphrasing, with \(k=8\)) using 4-shot in-context chain-of-thought exemplars over three language models. Accuracy is reported for both Hard Problems (HPR% \(\leq 0.5\)) (inside parentheses) and global accuracy across the entire dataset (outside parentheses).

 

only exception is GPT4 on GSM8K, which already achieves near-perfect performance with SC, thus SCoP only achieves equivalent performance.

### Additional Studies

Searching for ExemplarsSince our in-context learning paraphrasing scheme requires access to ground-truth answers, we would like to study how many problems with ground-truth answers are needed. Figure 4 illustrates how many data points in the training set, on average, need to be sampled to obtain \(N_{shot}\) 'good' paraphrases (x-axis) with different margins. We can observe that, although satisfying a large margin requires more samples, it is relatively easy (typically every \(\pm 5\) example) to find a sample that substantially improves the solve rate after paraphrasing. This, again, indicates the sensitivity of the language model to surface form variations in mathematical reasoning.

Difficulty Beliefs Across Language ModelsAn intriguing question is how different language models rank the difficulty of the problems. We measure the agreement between language models on problem difficulty by Spearman's rank correlationof the solve rate for original problems across four datasets. As shown in Table 3, the ranks of the difficulty (by solve rate) are all positively correlated. However, the degree of correlation varies, with higher agreement observed within the GPT model family and on simpler datasets.

Paraphrase TransferWe investigate whether paraphrases from a stronger LLM can be transferred to weaker ones and improve SCoP. Table 4 demonstrates the paraphrase transfer performance of SCoP (Naive, \(k\) = 8) on 100 randomly sampled data points from MMLU and GSM8K under the zero-shot CoT setting. In general, paraphrases produced by GPT-4 can be utilized by GPT3.5-turbo or LLaMA-2-70b for further performance improvements, with an exception with LLaMA-2 on MMLU, where GPT4 and LLaMA-2 exhibit the lowest Spearman rank correlation of solve rate. We hypothesize that the benefits of transferring paraphrases across models may depend on the agreement in their beliefs of problem difficulty.

Variance of VariationsIn light of the considerable variability observed in solve rates among problem surface forms (Figure 2), we propose and advocate **Variance of Variations (VOV)** for evaluating language models on reasoning robustness. Let \(X(p)\in[0,1]\) be the random variable representing the solve rates of various paraphrases of a problem \(p\). Then the VOV value of the dataset \(D\) is then defined as:

\[\mathrm{VOV}=\mathbb{E}_{p\sim D}[\mathrm{Var}(X(p))] \tag{1}\]

where \(\mathrm{Var}(\cdot)\) is the variance. A large value of VOV indicates high variability in the language model's reasoning ability against problem surface forms. We compute VOV using the solve rate for the \(k=8\) paraphrases and the original problem as \(X(p)\) for each \(p\). As shown in Table 5, while VOV decreases when a robust model solves a more manageable dataset (_e.g._, GPT-4 on GSM8K), and ICLara generated paraphrases can generally reduce VOV, VOV remains unreasonably high over more challenging datasets and all language models.

Examples of 'Good' ParaphrasesWe provide some qualitative examples comparing the solve rates between the original problem and a paraphrased version in Table 6. It is difficult to visually tell what contributes to a good paraphrase. We will publish these data to encourage future research.

## 5 Discussion

We have an intriguing observation that even the naive scheme of generating math paraphrases can

\begin{table}
\begin{tabular}{l l c c} \hline \hline Solver & Paraphraser & MMLU & GSM8K \\ \hline GPT-3.5 & Self & 50.0 & 78.0 \\ GPT-3.5 & GPT-4 & **54.0** & **84.0** \\ \hline LLaMA-2 & Self & **37.0** & 61.0 \\ LLaMA-2 & GPT-4 & 34.0 & **69.0** \\ \hline \end{tabular}
\end{table}
Table 4: Performance of SCoP (Naïve, \(k\) = 8) on MMLU and GSM8K, with different paraphrisers.

\begin{table}
\begin{tabular}{c l c c c} \hline \hline  & \multicolumn{2}{c}{GTM3.5, GPT4 & GPT3.5, LLaMA-2} & GPT4, ILaMA-2 \\ \hline GSM8K & 0.573* & 0.649*** & 0.445* \\ AUQA & 0.543*** & 0.227*** & 0.314* \\ MATH & 0.554*** & 0.242* & 0.433* \\ MMLU & 0.313* & 0.320*** & 0.233 \\ \hline \end{tabular}
\end{table}
Table 3: Spearman’s rank correlation of original problems’ solve rate across language models.

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline  & & GSM8K & AUoA & MATH & MMLU \\ \hline \multirow{2}{*}{LLaMa2} & Naive & 20.3 & 17.5 & 12.9 & 17.5 \\  & ICLaur & 18.9 & 15.7 & 12.2 & 16.6 \\ \cline{2-5}  & Nave & 20.6 & 16.1 & 15.8 & 16.9 \\ \cline{2-5}  & ICLparn & 16.2 & 10.7 & 15.6 & 15.6 \\ \hline GPT-4 & ICLpara & 9.7 & 11.5 & 17.0 & 21.3 \\ \hline \end{tabular}
\end{table}
Table 5: VOV values across datasets and language models, shown as standard deviation.

 

improve the overall accuracy. However, the naive scheme has a significant chance of generating worse paraphrases. Why would aggregating over the mixture of better and worse paraphrases still significantly improve the performance?

To explain this, Figure 5 shows three scatter plots of the solve rate against the entropy of answer distributions. The outcome of solving each random paraphrase is represented as a black dot. As can be observed, the dots roughly form a triangular region. The top left corner represents the ideal case with high solve rates and high confidence. The bottom corners, on the other hand, represent two failure modes. The bottom right corner represents the case with low solve rates and low confidence, and the bottom left corner with low soft rates but high confidence (commonly known as over-confidence).

The blue arrows in Figure 5 visualize the cases where the paraphrases improve the solve rate, and they mostly point to the top-left corner. The arrows in Figures 5 and (c) represent the cases where the paraphrases lower the solve rate, and we can observe that the arrows pointing to the bottom right corner (yellow arrows in (b)) far outnumber those to the bottom left corner (red arrows in (c)). This indicates that while the 'good' paraphrases would sharpen the answer distribution, the 'bad' paraphrases mostly would flatten the distribution. Since the final aggregated answer distribution is predominantly influenced by the sharp distributions, the damage brought by the "bad" paraphrases is small compared to the benefit brought by the 'good' paraphrases, and thus the aggregate effect across all the paraphrases is still positive.

## 6 Related Work

Mathematical Reasoning in LLMsThe complexity of mathematics necessitates System-2 reasoning, characterized by a slow, step-by-step cognitive process Kahneman (1911). Numerous works have sought to emulate this process in solving mathematics with LLMs Wei et al. (2022); Wang et al. (2022); Kojima et al. (2023); Lightman et al. (2023); Qiao et al. (2022). As a prominent framework, chain-of-thought Wei et al. (2022); Kojima et al. (2023) prompts the language model to generate a sequence of reasoning steps instead of a direct answer; Wang et al. (2022) extended chain-of-thought by Self-Consistency, in which they replaced greedy decoding with sampling decoding to generate a

Figure 4: (a) GSM8K (b) AQuA (c) MATH (d) MMLU. The average number of data points in the training set needed for obtaining \(N_{shot}\) exemplars at different margins.

Figure 5: Data Difficulty Map for GSM8K using GPT3.5, with three types of changes from solving the original problem to one of its random paraphrases: (a) Improvement, (b) Overconfidence, and (c) Uncertainty. Arrows indicate the solve rate and entropy change from solving the original problem to its paraphrased version.

 

variety of _reasoning paths_, with multiple paths potentially leading to the same answer from different angles. Other multi-step reasoning variations with verifiers exist Lu et al. (2023); Besta et al. (2023); Yao et al. (2023); however, they are less related to our focus primarily on the language model's internal ability to solve mathematical problems.

Paraphrasing VariabilityPrevious research on the impact of paraphrasing mathematical problems on their solvability by language learning models (LLMs) is limited. The study by Gonen et al. (2022) explored how paraphrased instructions affect the performance of traditional NLP benchmarks. This sensitivity of instructive prompts has inspired further research in prompting learning Shin et al. (2020); Zhou et al. (2023); Sordoni et al. (2023) and in-context exemplar mechanisms Min et al. (2022); Brown et al. (2020); Ye and Durrett (2022). However, our work focuses on the sensitivity of the mathematical problem presentation itself instead of the instruction or in-context examples.

## 7 Conclusions

This work highlights the variability in the solve rate of large-scale language models to the surface form of mathematical problems. Leveraging this, we introduced the Self-Consistency-over-paraphrases (SCoP), which improves mathematical reasoning performance over Self-Consistency. We hope our findings will inspire the need for more robust language models that can reason effectively regardless of how a problem is presented.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Problem** & **Source** & **Label** & **SR** & **Voted (F)** \\ \hline
**Original:** Jema has 4 roommates. Each month the electricity bill is $100. How mach will each & & & 0.15 & “300” (0.8) \\ roommate pay per year for electricity, if they divide the share equally? & GSMSK & ”240” & & \\
**Paraphrased:** Jema shares an apartment with 4 other people. The electricity bill is $100 per month. & & & 0.95 & ”240” \\ If they split until heaily, bow much will each room commutate contribute towards the electricity bill in a year? & & & & \\
**Original:** Jemy goes to the frontist to buy some flowers. Rossets Cost $2 each and $15 for a dozen. & & & & \\ If she bought 15 poses and arrived with five 5 dollar bills and only have quarters for change. & & & & \\ how many quartes does be knew wife? & & & & \\
**Paraphrased:** Jerry visits the flower shop to purchase followers. She can buy roses individually for $2 each & GSMSK & ”16” & & \\ or buy a dozen rosers for $15. Jump decides to bey 15 poses in total. She pays with five $5 bills and & & & & \\ the florist can only give her me change in quarters. The question asks how many quartes Jessry receives as change. & & & & \\
**Original:** Assistant was needed to prepare for preparation. Each helper can make either 2 large sales or & & & & \\
35 small callcast. The kitchen is available for 3 hours and 20 lapt calves \& 700 small calls are needed. & & & 0.2 & A (0.25) \\ How many helpers are required? & & & & & \\
**Paraphrased:** How many helpers are needed if each helper can make either 2 large clakes or 35 small cases & AUUA & B & & \\ per hour, and the kitchen is available for 3 hours and needs 20 lapt clades and 700 small cake? & & & & 0.8 & B \\
**Options:[A)8,B]0,C12),D5),E]19]** & & & & & \\ \hline
**Original:** A starts business with Rs.40,000. After 2 months, B joined him with Rs.60,000. C joined them after some mete time with Rs.120,000. At the end of the year, out of a total profit of Rs.375,000, C gets Rs.150,000 as his share. How many months after Bjoined the business, did C join? & & & & \\
**Paraphrased:** A starts a business with Rs.40,000 and after 2months, B joins with Rs.60,000. C joins the business at some point later with Rs.120,000. At the end of the year, the total profit is Rs.375,000, and Creectives & AUUA & B & \\ Rs.150,000 as their share. How many months after Bjoined the business’did C join? & & & & \\
**Options:[A)2,B]4,C]2,D3),D24),E]** & & & & \\ \hline
**Original:** A star-polyois drawn on a clock face by drawing a chord from each number to the fifth number & & & & \\ comanded clockwise from that number. That is, chords are drawn from 12 to 5, from 5 to 10, from 10 to 3, & & & \\  & and so on, ending back at 12. What is the degree measure of the angle at each vertex in the star-polygon? & MATH & ”30” & \\
**Paraphrased:** What is the ensure of the angle at each vertex in the star-polygon formed by drawing a chord from each number on the check face to the fifth number counted clockwise from that number? & & & 0.5 & ”30” \\ \hline
**Original:** By partial fractions, \(\frac{\sqrt{x^{2}+bx^{2}+cx}}{x-\frac{-b+Q^{2}-ac}{2a}}+\frac{\frac{B}{x-b- Q^{2}-ac}}{x-\frac{b-Q^{2}-ac}{2a}}\text{Find }A+B\). & & & & \\
**Paraphrased:** Find the sum of \(A\) and \(B\) in the expression & & & & \\ \(\frac{\sqrt{x^{2}+bx^{2}+cx}}{x-\frac{-b+Q^{2}-ac}{2a}}+\frac{B}{x-b-Q^{2}-ac}\). Note the HJEcode was paraphrased from \(\backslash\)frac to \(\backslash\)frac. & & & \\ \hline
**Original:** Statement 1 I Forey every positive integer there is a cyclic group of order n. & & & & 0.05 & C (0.95) \\ Statement 2 Every finite cycle group contains an element of every order that divides the order of the group. & & & & \\
**Paraphrased:** Statement 1 says that there exists a cyclic group of any positive integer n. & MALU & A & & \\ Statement 2 ways that in any finite cyclic group, where is an element for every possible order that divides the order of the group. & & & & \\
**Options:[A)True, True, B)False, False, CSType,False,Due]False,True]** & & & & & \\ \hline
**Original:** What is the probability that a randomly selected integer in the set \(\left\{1,2,3,\ldots,100\right\}\) is divisible by 2 and not divisible by 3? Express your answer as a common fraction. & & & & \\
**Paraphrase:** What is the chance that if we randomly choose an integer from the set of numbers 1 to 100, it will be divisible by 2 but not divisible by 3? Write your answer as a fraction. & MALU & D & & 0.8 & D \\
**Options:[A)**\(\frac{\|1\|}{6}\), \(\frac{\|B\|}{6}\), \(\frac{\|C\|}{31}\), \(\frac{\|T\|}{7}\), \(\frac{\|T\|}{7}\)]]** & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 6: Qualitative examples where the original problems and corresponding surface form variations exhibit substantial solve rate difference using GPT-3.5-turbo.

 



### Limitations

While we derive thorough conclusions about the relationship between the surface form of a mathematical problem and its solvability by large-scale language models with the effectiveness of SCoP and additional studies, one limitation is the need for a mechanism for identifying or generating surface forms that are easier to solve than others. The study is solely conducted in English, while the generalizability of SCoP in other languages is unexplored. Future research could address these by exploring the rationalization of surface forms, _i.e._, determining the optimal form given the original one, and the verifiability of the framework in other languages.

## Ethics Statement

The datasets that we used in experiments are publicly available. In our work, we explore the relationship between the surface form of a mathematical problem and its solvability by large-scale language models. We do not expect any direct ethical concern from our work.

## Acknowledgements

This study was supported by MIT-IBM Watson AI Lab.

## References

* M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, L. Gianinazzi, J. Gajda, T. Lehmann, M. Podstawski, H. Niewiadomski, P. Nyczyk, and T. Hoefler (2023)Graph of thoughts: solving elaborate problems with large language models. Cited by: SS1.
* R. Bhagat and E. Hovy (2013)What is a paraphrase? computational linguistics39 (3), pp. 463-472. External Links: ISSN 1581-469X, Link, Document Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askelt, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1581-469X, Link, Document Cited by: SS1.
* K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021)Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS1.
* H. Gonen, F. Iyer, T. Blevins, N. A. Smith, and L. Zettlemoyer (2022)Demystifying prompts in language models via perplexity estimation. External Links: 2004.0042 Cited by: SS1.
* D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020)Measuring massive multitask language understanding. In International Conference on Learning Representations, Cited by: SS1.
* D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021)Measuring mathematical problem solving with the math dataset. NeurIPS. External Links: 2004.00422 Cited by: SS1.
* D. Kahneman (2011)Thinking, fast and slow. macmillan. External Links: 1106.00022 Cited by: SS1.
* T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa (2023)Large language models are zero-shot reasoners. External Links: 2305.0000 Cited by: SS1.
* P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soyltu, M. Yasunaga, et al. (2023)Holistic evaluation of language models. External Links: 2305.0000 Cited by: SS1.
* H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023)Let's verify step by step. arXiv preprint arXiv:2305.20050. Cited by: SS1.
* W. Ling, D. Yogatama, C. Dyer, and P. Blunsom (2017)Program induction by rationale generation: learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 158-167. External Links: ISBN 978-3-319-142-4 Cited by: SS1.
* P. Lu, B. Peng, H. Cheng, M. Galley, K. Chang, Y. N. Wu, S. Zhu, and J. Gao (2023)Chameleon: plug-and-play compositional reasoning with large language models. External Links: 2305.00002 Cited by: SS1.
* S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022)Rethinking the role of demonstrations: what makes in-context learning work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Chhabi, United Arab Emirates, pp. 11048-11064. External Links: Link, Document Cited by: SS1.
* O. Qi, Y. Ou, N. Zhang, X. Chen, Y. Yao, S. Deng, C. Tan, F. Huang, and H. Chen (2022)Reasoning with language model prompting: a survey. arXiv preprint arXiv:2212.09597.



Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, et al. 2022. Scaling language models: Methods, analysis and insights from training gopher.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. 2020. Auto-prompt: Eliciting knowledge from language models with automatically generated prompts.
* Sordoni et al. (2023) Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Cote, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. 2023. Joint prompt optimization of stacked llms using variational inference.
* Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abbabkar Abid, Adam Fisch, Adam R. Brown, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Swanson et al. (2019) H Lee Swanson, Jennifer E Kong, Amber S Moran, and Michael J Orsco. 2019. Paraphrasing interventions and problem-solving accuracy: Do generative procedures help english language learners with math difficulties? _Learning Disabilities Research & Practice_, 34(2):68-84.
* Touvron et al. (2022) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Akanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.
* Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. _Advances in neural information processing systems_, 35:30378-30392.
* Zhou et al. (2023) Yongchaao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers 



## Appendix A Choose Margin

We examine the effect of margin on selecting exemplars for in-context paraphrasing using GPT-3.5 and separate dev-sets from GMS8K and MMLU, each with 250 data points. The results in Table 7 show that a moderate margin outperforms a large one in SCOP, as the latter may decrease the diversity of exemplars.

## Appendix B APE Alternatives

A potential alternative to finding an optimal prompt for paraphrasing is to use the Automatic Prompt Engineering (APE) settings Zhou et al. (2023). We formulate the procedure into four steps:

1. Present a set of input-output pairs where the inputs are the original problems and the outputs are the paraphrased exemplars. Prompt the language model to generate \(C\) candidate instructions that could produce the outputs from the inputs.
2. Prompt each candidate instruction to the language model to generate paraphrases for a batch size \(B\) of problems in the development set and compare the mean solve rate change before and after paraphrasing.
3. Choose the instruction that maximizes the mean solve rate change.
4. Repeat steps 1 - 3 \(E\) times.

We implemented this procedure using GPT-3.5 on the AQUA development set to obtain the instruction (\(C=15\), \(B=30\), \(B=0\)). We tested the performance in both AQUA (in-domain) and GSM8K (out-of-domain), comparing it with ICLpara. Although the in-domain AQUA performance was similar to ICLpara, the out-of-domain performance worsened, and APE required more data than ICLpara. Therefore, this approach has yielded negative results. The performance results are presented in Table 8.

## Appendix C Temperature and Randomness

To further validate that SCoP goes beyond simply increasing randomness, we introduce two variants of SC, where we increase the temperature up to 0.9 and 1 using GPT-3.5 on 1/4 of the MMLU and AQuA datasets. The results are shown in Table 9.

As can be observed, although increasing temperature brings a slight improvement in the hard problems, the performance gain soon saturates and is not nearly comparable to that of SCoP.

## Appendix D Surface Forms with Solve Rate Degradation

As previously discussed, surface form modification by paraphrasing can lead to degradation in the solve rate. Here, we present additional qualitative examples where the solution rate worsened after paraphrasing. See Table 10.

\begin{table}
\begin{tabular}{l l l} \hline \hline Margin & MMLU (Dev, k = 8) & GSM8K (Dev, k = 8) \\ \hline SC/HPR\% & 53.2 (26.9) / 64 & 73.6 (21.4) / 33.6 \\ \hline
0.2 & 55.6 (34.4) & 75.2 (31.0) \\
0.3 & 56.8 (35.6) & 74.8 (32.1) \\
0.4 & 55.2 (33.8) & 75.6 (33.3) \\
0.5 & 53.6 (32.5) & 74.4 (35.7) \\ \hline \end{tabular}
\end{table}
Table 7: Ablation on the margin effect of exemplar selection.

\begin{table}
\begin{tabular}{l c c} \hline \hline Temperature & MMLU & AQuA \\ \hline
0.7 (baseline) & 49 (21.5) & 68 (27.3) \\
0.9 & 49 (24.6) & 68 (27.3) \\
1 & 49 (24.6) & 64 (18.2) \\ \hline \end{tabular}
\end{table}
Table 9: Increasing randomness by temperature saturates reasoning performance. The numbers inside the parenthesis are the accuracy for the hard problems.





## Appendix E Prompt Templates

We list the prompt templates used in the paper below.

``` Input: Profile wants to buy cupcakes for everyone in his class as a special treat. He's not sure if people will want vanilla or chocolate cupcakes so he decides to get one of each for everyone. If he gets the same amount of 2 cupcakes for each himself, his teacher, and his 25 classes, how many cupcakes should Howe buy?agraph**: Howie wants to purchase cupcakes for his entire class as a special treat. Since he is unsure of the flavor preference, he plans to buy both vanilla and chocolate cupcakes. Howie wauts to ensure that he has an equal amount of cupcakes for himself, his teacher, and his 25 classes. How many cupcakes should Howe purchase in total? Original_Janice hikes at 10 miles per hour, while Jennie bikes at 20. How long until they have collectively liked 1 mile? Paraphrased_Janice juriesking at different speeds. Janice bikes at a rate of 10 miles per hour, while Jennie wikes at a rate of 20 miles per hour. How much time will it take for them to collectively like a distance of 1 mile? Options(A) minute, B) minutes, C) minutes, E) minutes] 
\begin{tabular}{l l c c c} Origin: & How many primes are in the row of Pascal’s Triangle that starts with a 1 followed by a 6? Paraphrased_Starting with the numbers 1 and 6, how many prime numbers are there in the sequence of numbers in Pascal's Triangle? Original_John divided his souvenir hat pins into two piles. The two piles had an equal number of pins. He gave his brother one-half of one-third of one pile. John had 66 pins left. How many pins did John originally have? Paraphrased_John started with a certain number of souvenir hat pins. He divided them into two equal piles. He then gave his brother one-half of one-third of one of the piles. After that, John was left with 66 pins. How many pairs did John have at the beginning? Options(A) 396, B) 72, C) 66, D) 36] \\ \end{tabular} Table 10: Qualitative examples where paraphrased surface forms of the original problems can also exhibit solve rate degradation, using GPT-3.5-turbo.