Why Do LLMs Generate Redundant Calculations?
After seeing that LLMs produce excessive calculations, we seek to understand why.
We speculate that the reward models (RMs) in RLHF might favor more verbose outputs over concise ones, making RLHF-trained models prone to generate lengthy output even if it is redundant.
To test this hypothesis, we would like to compare RM’s preference between long and short answers.
However, we cannot access RMs used to train ChatGPT or Llama models.
As a workaround, we use ChatGPT and GPT-4 as the proxy of the RMs; we call these models proxy RMs in this case.
To obtain the preference of the proxy RMs, we give proxy RMs some instructions, a question in GSM8K-Zero, a pair of long and short answers, and ask the model to select a better answer.
We follow the instructions used in Zheng et al. ( 2023 ) , which asks the proxy RMs to consider the accuracy and helpfulness of the answer.
The experiment is repeated by inverting the order of the short and long answers to counteract potential position bias.
Using ChatGPT or GPT-4 as the proxy RMs is reasonable, as these models should learn the preferences of their RMs during RLHF.
Additionally, prior works have used ChatGPT and GPT-4 to generate the preference data to train the RMs (Anand et al., 2023 ) , so the preference of ChatGPT or GPT-4 can reflect the preference of RMs.

We prepare the long and short answers as follows:
To collect long answers, we collect ChatGPT’s answers to questions in GSM8K-Zero, select those with redundant calculations, and group those answers into two: correct answers and incorrect answers, with approximately 100 samples in each group.
The 100 samples in the correct-answer group are reviewed by one of the authors to ensure that the answer is correct instead of a false positive due to imperfect regular expressions when extracting the answer from the LLM’s response.
The same procedure is done for the 100 samples in the incorrect-answer group.
Next, for each long answer collected, we construct a short answer counterpart by the template, " The answer is [[ground truth]] ", where "[[ground truth]]" is filled in with the ground truth in GSM8K-Zero.

The preference of proxy RMs between long and short answers is shown in Figure 2 .
First, we observe that when both the long and short answers are correct (Figure 2 (a)), both GPT-4 and ChatGPT prefer long answers.
By scrutinizing the evaluation results, we find that GPT-4 and ChatGPT frequently complain about the shorter answer to "only answer the question without any further details," while the long answer "shows more information."
However, when reading the long answers, the authors find it hard to locate the answer to the question since the model outputs too much unnecessary information and complicates the problem, making the answer unhelpful.
Next, when the long answer is incorrect and the short answer is correct (Figure 2 (b)), we find that ChatGPT consistently prefers lengthy but wrong answers.
While GPT-4 successfully prefers the short and correct answer in 61% of the cases, GPT-4 still votes for long but wrong answers in 34% of the cases.
Overall, the results in Figure 2 show that proxy RMs strongly prefer long outputs that contain redundant calculations and unnecessary reasoning, even if the final answer is wrong!
If we use the proxy RMs’ preference data collected in this section, it is easy to think that we will obtain RMs that favor lengthy output, eventually leading to an LLM that generates redundant calculations.
We repeat the above experiment using the answers from Llama-7b-chat and observe a similar result.