RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
LLMs have shown impressive performance on various tasks (Brown et al., 2020 ; Ouyang et al., 2022 ; Touvron et al., 2023 ) , and CoT reasonings (Wei et al., 2022 ; Kojima et al., 2022 ) further boost the LLMs’ performance.
To make LLMs better learn to think step-by-step, it is common to enhance the CoT ability of LLMs by adding training samples that contain CoT reasoning in instruction tuning (Longpre et al., 2023 ) .

While step-by-step reasoning is useful, it lengthens the LLM’s output.
Longer outputs imply longer inference latency and cost more money on commercial LLMs.
Moreover, if LLMs give long answers to questions that can be easily solved without complex reasoning, this may confuse the users and make them falsely believe that the questions are difficult and must be solved step-by-step.
It is thus more desirable that LLM only triggers CoT reasoning ability when necessary, but it is unclear if LLMs know when to think step-by-step and only generate step-by-step reasoning when necessary.
Prior works on CoT mostly focus on the correctness of the CoT reasoning steps or whether the reasoning steps are faithful to the question and support the final answer (Ye and Durrett, 2022 ; Golovneva et al., 2023 ) .
However, no past study focuses on the redundancy in CoT reasonings.

As an initiative to study the redundancy of LLM outputs, we aim to understand the following research question: Does LLM generate redundant reasonings when they clearly need not do so?
To study this question, we construct a math QA dataset, GSM8K-Zero, which contains trivial questions that can be answered without any calculations and reasoning.
Using this curated dataset, we can define the redundancy of output from LLMs.
We evaluate seven LLMs trained with reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022 ) , and we find that LLMs tend to generate redundant calculations that complicate the responses and sometimes lead to the wrong answer.
To explain our observation, we show that GPT-4 (OpenAI, 2023 ) and ChatGPT (OpenAI, 2022 ) , which are widely used in gathering the preference data for training a reward model in RLHF (Guo et al., 2023 ; Anand et al., 2023 ) , show a strong preference towards long answers that contain redundant calculations, even if the long answers are incorrect.

Our contributions are summarized as follows:

To the best of our knowledge, we are the first to study the redundancy of LLM outputs.

We construct and release a dataset, GSM8K-Zero, which reveals the LLMs’ tendency to generate redundant reasonings.

We show that LLMs tend to generate redundant calculations on math questions that can be answered without any calculation.

We show that LLMs’ tendency to generate long answers may stem from the imperfect reward model that prefers longer answers regardless of their correctness.
Construction of GSM8K-Zero
To study LLMs’ tendency for redundant calculations, we created GSM8K-Zero from GSM8K (Cobbe et al., 2021 ) .
A question in GSM8K comprises (1) the known information (blue parts in Figure 1 ) and (2) a query for an un- known quantity (orange parts in Figure 1 ).
Using questions in GSM8K, we aim to create questions whose answers are directly stated in the questions and can be obtained without any calculations.

We use the following procedure to achieve this goal.
The following procedure is best read with Figure 1 (a).
Given a question in GSM8K, we remove the last sentence from the question that queries for an unknown variable and keep the known information .
Next, we generate a question that asks the value of a known variable (green parts in Figure 1 (a)) based on the known information and append the question behind the known information .
The question is generated by randomly selecting a number in the known information as the ground truth answer and using few-shot prompting to generate a question whose answer is the selected ground truth using ChatGPT.
We then use GPT-4 to answer the newly generated question.
If GPT-4’s answer deviates from the ground truth answer, the question is discarded.
We randomly select 3,500 questions from GSM8K’s training set 1 1 1 In our preliminary experiment, we find that our results also hold when we use the testing set of GSM8K to construct the questions in GSM8K-Zero and obtain 2,978 question-answer pairs after the above procedure.

Based on a manual inspection of 250 random question-answer pairs by the authors, we estimate that about 85% of question-answer pairs in GSM8K-Zero are valid.
Refer to Appendix B.2 for a detailed description of our manual inspection of GSM8K-Zero.
Evaluating Redundancy
We define redundant outputs as any superfluous information in LLM responses that are not required for accurately answering the question .
Measuring this redundancy is often challenging for existing datasets.
However, GSM8K-Zero offers an easy way to evaluate LLM output’s redundancy due to its unique nature: questions can be answered without any calculations since the answers are explicitly stated within the questions.
If an LLM’s answer includes calculations, it is deemed redundant.
We identify mathematical operators ( \times,+ , and = ) in LLM outputs by a regular expression and say that the LLM’s answer is redundant whenever mathematical operators are found.
Experiments
We test LLMs on GSM8K-Zero in zero-shot, as zero-shot inference closely mirrors most users’ practical use of LLMs-as-assistants .
Instead of leveraging advanced prompting techniques like zero-shot CoT (Kojima et al., 2022 ) or Plan-and-Solve (Wang et al., 2023 ) , we present a single question to the LLM and take its response.
For each question, we sample one response from the LLM.
In our preliminary experiments, we find the observations in our paper are robust toward the hyperparameters used for sampling outputs from LLMs.

Our evaluation encompasses proprietary LLMs, such as GPT-4, ChatGPT, Claude-2 (Anthropic, 2023 ) , and PaLM ( text-bison-001 ) (Anil et al., 2023 ) , and open-source ones like Llama-2-chat models of different sizes (Touvron et al., 2023 ) .
We assess LLMs’ performance on GSM8K-Zero using two metrics:
(1) Redundancy : Determined by the percentage of LLM answers containing numerical operators like \times,+ , and = .
(2) Accuracy : Accuracy measures how often the LLM’s answer, extracted using a regular expression, aligns with the GSM8K-Zero ground truth.
Main Results
We show the LLMs’ performance on GSM8K-Zero in Table 1 .
First, we observe almost half of the LLMs we test have an accuracy lower than 50% (second column in Table 1 ).
Recall that the answers to the question in GSM8K-Zero can be easily extracted from the question without any calculations, which makes GSM8K-Zero more like an extractive QA than a math QA.
As simple as this dataset is, some LLMs still cannot perform well on it.

Next, we turn our attention to the redundancy in the answers.
It can be seen that both proprietary and open-source LLMs generate redundant calculations and reasoning to answer the questions.
ChatGPT yields unnecessary calculations in their answers in almost half of the answers, and all Llama-2 models generate lengthy reasoning steps and redundant calculations in more than 80\% of their responses while they are not explicitly prompted to do so.

We show some answers with redundancy generated by different LLMs in Table 3 .
By inspecting the outputs from LLMs, we find that in most cases, LLMs solve all the unknown variables in the questions, which are not asked in the questions (#2 in Table 3 ).
This behavior is problematic for a helpful assistant since it complicates the responses and may falsely lead the users to think solving all the unknown variables is necessary to arrive at the final answer.
We also find that the LLMs sometimes only provide the values of the unknown variables but do not answer the value asked in the question (#1 in Table 3 ), showing that LLMs cannot follow user instructions very well in these cases.

After discussing redundancy and accuracy independently, we want to know if redundant calculation co-occurs more often with wrong answers.
We separate the model outputs into two groups: one that contains calculations and another that does not have calculations, and we calculate the accuracy for the two groups.
The results are shown in the two rightmost columns in Table 1 .
When the LLM’s answers contain calculations, the accuracy drops significantly for almost all models except for PaLM.
By randomly browsing the wrong answers that include calculations of models except PaLM, we find that sometimes LLMs hallucinate variables not specified in the questions (#4 in Table 3 ).
Sometimes, LLMs make calculation errors, leading to the wrong answer (#3 in Table 3 ).
This shows that redundant calculations not only waste time and resources but can also hurt the LLM’s performance due to calculation errors and incorrect reasoning.

For the outputs of PaLM that contain calculations, we observe that PaLM often first generates an Arabic number as the answer, followed by some calculations as the explanation.
In this case, the numeric answer of PaLM does not depend on the calculations, so even if the calculation and reasoning following the answers are wrong, they cannot affect the answer.
This makes the accuracy of answers with and without calculation similar in the case of PaLM.
Do LLMs Know When to Use CoT?
Section 3.1 reveals that LLMs can generate redundant calculations and unnecessary CoT reasoning steps.
This is possibly because, during instruction tuning, LLMs are trained to generate CoT reasoning for mathematical problems when the input instruction does not specify how to solve the question , forcing them to apply CoT on every question that looks like a mathematical question.
Hence, we are curious whether LLMs can drop the CoT reasoning and calculations when properly instructed .
To explore this possibility, we append the following instruction after the questions in GSM8K-Zero: " If the question is simple enough, you can omit the step-by-step reasoning and just give the answer. "
Here, we only test on the LLMs that generate answers with higher redundancy in Section 3.1 .

The results are shown in Table 2 .
We can see that when LLMs are allowed to omit step-by-step reasoning, the redundancy of the LLMs significantly drops compared with Table 1 while the accuracy significantly boosts for almost all models.
The decrease in output redundancy implies that LLMs do know that some questions in GSM8K-Zero are easy enough to answer directly.
However, even when they are allowed to omit step-by-step reasoning, the redundancy in these LLMs is still higher than 25%.
This means that LLMs cannot always correctly infer the difficulty and whether step-by-step reasonings are necessary for the questions.
Conclusion
In this paper, we construct GSM8K-Zero to illustrate the redundancy in the output from LLMs.
We show that LLMs tend to generate redundant calculations and unnecessary reasoning, sometimes leading to a wrong answer.
We reveal that LLMs may not differentiate questions requiring step-by-step reasoning from simpler ones, suggesting a possible direction for improving LLMs.
To explain our observation, we use proxy RMs and find that these models prefer lengthy answers even if they are wrong.
Through this paper, we hope future researchers can focus more on the redundancy of the outputs of LLMs and develop training techniques to teach LLMs when to think step-by-step.
Limitations
The main limitation of our paper is that we only study redundancy on a manually constructed dataset, GSM8K-Zero.
The reason is that it is easier to define and calculate redundancy on GSM8K-Zero; we believe this is an ample contribution since it is a phenomenon never mentioned in the literature.
While exploring redundancy on other existing datasets will be interesting, we leave it to future works.

Another limitation of our paper is that we rely on ChatGPT and GPT-4 to construct GSM8K-Zero, so noises in the constructed dataset are inevitable.
We emphasize that future researchers need to keep the noises in the dataset in mind and take special caution when interpreting the results evaluated on GSM8K-Zero.
To understand the noises in the dataset, the authors randomly selected 250 samples from GSM8K-Zero and reviewed them.
As stated in Section 2.1 , we estimate that 85% of question-answer pairs in GSM8K-Zero are valid.
We present the details about our manual review of the dataset in Appendix B.2 .
We also discuss that our results and observations in the main content still hold when considering the noises in the dataset.

Last, since our paper is a short paper, an obvious limitation is that there is still a lot to explore, but we cannot include them in our paper.
While we deem our paper’s main content to be self-contained, we include some potential questions that might be raised by curious and enthusiastic readers in Appendix A (FAQs section).
Ethical Statements
We do not see our work to have possible harmful outcomes.
We follow the ACL ethical guidelines when conducting the research in this paper.
Acknowledgements
We want to thank the reviewers for providing detailed feedback and actionable suggestions, which helped us strengthen our paper.
We also want to thank the senior committee members for monitoring the reviewing process.
Cheng-Han Chiang is supported by a Google PhD Fellowship and a Ph.D. scholarship program by Delta Electronics.
B.1 Dataset Cards
GSM8K-Zero is constructed from GSM8K (Cobbe et al., 2021 ) .
Since GSM8K does not include the dataset license, we are unsure what license to release GSM8K-Zero.
B.2 Manual Review by the Authors
The authors randomly sample 250 samples from GSM8K-Zero to understand the quality of the samples and whether using regular expression to calculate accuracy has a high precision.
The human (author) evaluation is conducted in the following steps:
First, we randomly sample 125 samples from the answers of ChatGPT that are correct together with their corresponding questions, and we sample 125 samples for the answers of ChatGPT that are incorrect together with their corresponding questions.
Recall that the accuracy is calculated using regular expressions.
We search for the first or last number that appears in the last sentence of the model’s response, and we count the model response to be accurate if the ground truth matches the number extracted by regular expressions.
While this process may falsely consider the model to be correct when the model’s answer is wrong, we find that this merely happens during our manual review of 250 answers from ChatGPT.
We separately sample questions that ChatGPT correctly answered and questions that ChatGPT got wrong because those two groups of questions might be systematically different.

Given a question, an answer from ChatGPT, and the ground truth answer, one of the authors labels the sample into four categories:

The ground truth is correct, and the answer from ChatGPT is correct

The ground truth is wrong, while the answer from ChatGPT is correct (matches the real ground truth)

The ground truth is correct, but the answer from ChatGPT is wrong

The question is invalid, including that ground truth is wrong, the question cannot be answered without calculation, or the question is ambiguous.

We find that for questions that ChatGPT is correct, 89% of questions are valid, and the ground truth answer is always correct.
However, we find that for 7% of the questions, ChatGPT’s answer is wrong, but we count it as correct due to imperfect parsing of regular expressions.
For questions that ChatGPT is inaccurate, about 70% of the questions are valid, and the ground truth is wrong in 2% of the cases.
Only in 4% of the cases, the regular expression we use considers the answer of ChatGPT to be wrong when it is correct.
Considering that ChatGPT’s accuracy is about 80%, we estimate that the proportion of invalid questions in GSM8K-Zero is 14.8%.
B.2.1 Does Invalid Questions Affect the Results?
Readers may be concerned about whether the invalid questions change the observations in the main content.
The short answer is no.
We explain as follows:
For the redundancy shown in Table 1 , if the model generates CoT reasonings and calculations for those invalid questions, then the redundancy should be around 15%.
But clearly, all model except GPT-4 has a redundancy much higher than 15%.
Thus, LLMs can still generate a lot of redundant calculations for other valid questions.
As for GPT-4, we still find that it does generate redundant calculations in some cases.
#2 in Table 3 is such a case.

Next, for accuracy, even if LLMs are wrong for all the invalid questions, their accuracy should be around 85% if they get all the valid questions correct.
However, this is clearly not the case for all LLMs except Claude-2.
Next, for the rightmost column in Table 1 , if we assume that all the invalid samples happen to be the samples that LLMs include calculations in the answer, the accuracy in this column should increase.
However, by some simple maths, the readers can easily verify that even considering this, the accuracy of answers containing calculations is still much lower than that of answers that do not include calculations.
Thus, our observation in the paper still holds.
Figure 1: (a) A question in GSM8K-Zero.
The blue parts are the known information, and the orange part is the original question in GSM8K and is removed in GSM8K-Zero.
The green part is the new question in GSM8K-Zero.
(b) The answer generated by ChatGPT.; (a) A question in GSM8K-Zero.
The blue parts are the known information, and the orange part is the original question in GSM8K and is removed in GSM8K-Zero.
The green part is the new question in GSM8K-Zero.
(b) The answer generated by ChatGPT.
Figure 2: The preference of GPT-4 and ChatGPT between longer and shorter answers.
(a) The case when the longer answers are correct.
(b) The case when the longer answers are incorrect.; The preference of GPT-4 and ChatGPT between longer and shorter answers.
(a) The case when the longer answers are correct.
(b) The case when the longer answers are incorrect.
The redundancy ( Red. ) and accuracy of LLMs’ responses.
We report the average accuracy ( Avg. ) on all questions (second column), the accuracy for answers without calculation (Cal. ✗, third column) and with calculation (Cal. ✓, fourth column). \dagger : The accuracy of GPT-4 is 100% by construction since we use GPT-4 to filter samples when constructing GSM8K-Zero.
The redundancy and accuracy of answers from LLMs when allowing LLMs not to use CoT.
Example of different types of redundant calculations from various LLMs.
The ground truth answer to the question is underlined in the question.
We replace the newline character from the LLMs’ output with space or period for easier illustration.
We highlight the wrong calculations and hallucinated variables.