Rational Metareasoning for Large Language Models

## Abstract

Abstract Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning’s cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37% fewer tokens generated across three models) while maintaining task performance across diverse datasets.

## 1 Introduction

Large language models (LLMs) rely on substantial computational power to handle complex problems (OpenAI et al., ; Chowdhery et al., ; de Vries, ). While initial studies mostly focused on the cost of training (Verdecchia et al., ), LLMs’ widespread deployment has made inference-time costs an increasingly important factor. Model compression techniques such as quantization, pruning, and knowledge distillation can lower post-training costs (Wan et al., ). However, there is a fundamental tension between inference cost and task performance: while many of these methods reduce costs at the expense of performance, others, such as chain-of-thought prompting (CoT; Wei et al., ; Kojima et al., ), do the opposite, raising inference costs to enhance task performance (Snell et al., ). It is worth noting that none of the previous approaches are *adaptive*: model compression modifications and existing CoT methods tend to raise or lower the inference cost on all queries, regardless of task complexity.

In stark contrast to this static tradeoff, humans are able to adaptively allocate computational resources based on task difficulty (Krämer, ; Russell, ; Lieder & Griffiths, ). In this work, we draw inspiration from *rational metareasoning* – literally, reasoning about reasoning – a concept originally from the artificial intelligence literature (Russell & Wefald, ) that has been used to explain how humans adaptively manage computational resources (Lieder & Griffiths, ; Lieder et al., ; Griffiths et al., ). Building on this, we develop a novel reward function based on the Value of Computation (VOC; Russell & Wefald, ), which formalizes the trade-off between inference cost and task performance. We adopt an iterative reinforcement learning process inspired by the Expert Iteration algorithm (Anthony et al., ). In each iteration, we generate multiple reasoning chains for each question. These reasoning chains are ranked using the reward function, and the dataset is filtered to retain only the best reasoning chain for each question. The model is then fine-tuned using this filtered dataset. However, unlike previous applications of Expert Iteration to LLMs (Zelikman et al., ), which filter generated examples based on correctness, our method optimizes for both the correctness *and the cost* of the reasoning process.

We evaluated the effectiveness of our solution across a diverse set of tasks, from science knowledge (ARC; Clark et al., ) to commonsense reasoning (CommonsenseQA; Talmor et al., ), mathematical problem solving (GSM8K; Cobbe et al., ), and logical deductive reasoning (ProofWriter; Tafjord et al., ). Additionally, we assess the out-of-domain generalization on MMLU (Hendrycks et al., ), a multitask benchmark. Our approach achieves a substantial reduction in generated tokens (35.6-37.4% compared to few-shot prompting and 20.1-27.3% compared to STaR; Zelikman et al. ) while retaining comparable performance. Thus, we make the following contributions:

- 1.
We introduce the idea of using rational metareasoning to optimize the tradeoff between inference cost and performance of LLMs.
- 2.
We formalize a novel reward function inspired by the Value of Computation (VOC) and integrate it into LLM training.
- 3.
We empirically demonstrate that rational metareasoning achieves the same task performance at lower inference costs (20-37% fewer tokens on average) across various datasets and reasoning tasks.

## 2 Rational Metareasoning

Unlike artificial intelligence, humans have limited time and cognitive resources (Griffiths et al., ; Griffiths, ). We face diverse challenges requiring different approaches: avoiding a sudden obstacle while driving needs quick, intuitive thinking, while selecting a retirement investment strategy requires slow, deliberate reasoning (Krämer, ). Rational metareasoning (Russell & Wefald, ) suggests agents should adapt their reasoning based on the problem at hand. Intuitively, while reasoning solves a problem, metareasoning solves the problem of *how* to solve a problem: deciding which computations to perform while problem-solving. The essence of rational metareasoning is calculating the value of computation (VOC; Russell & Wefald, ) for each potential computation. The VOC balances the benefit of computation $c$ (characterized by the expected increase in the agent’s eventual utility) against its cost (usually time or energy).

To formalize this, agents are assumed to have some internal belief state $b\in\mathcal{B}$, which determines their expectation about the value of each action $a\in\mathcal{A}$: $\mathbb{E}[U(a)|b]$. A rational agent would simply choose the highest-value action: $a^{*}=\operatorname*{arg\,max}_{a\in\mathcal{A}}{[U(a)|b]}$. In contrast, a meta-rational agent can perform computation to change their belief state before choosing an action. Each computation $c\in\mathcal{C}$ updates the agent belief to $b^{\prime}$ with probability $P(b^{\prime}|c)$, which in turn affects their beliefs about the value of actions. However, each computation has an associated cost ($\text{cost}(c)$). The VOC then quantifies the value of performing computation $c$ given a starting belief state $b$,

$$ $VOC(c,b)=\mathbb{E}_{P(b^{\prime}|c)}[\max_{a^{\prime}}\mathbb{E}[U(a^{\prime} )|b^{\prime}]-\max_{a}\mathbb{E}[U(a)|b]]-\text{cost}(c).$ (1) $$

Thus, a meta-rational agent should pursue the computation $c^{*}$ with the highest VOC: $c^{*}=\operatorname*{arg\,max}_{c\in\mathcal{C}}VOC(c,b)$. If no computation has positive VOC, the agent should stop thinking and act in the world. Rational meta-reasoning can explain how humans allocate cognitive resources in various tasks (Lieder & Griffiths, ; Lieder et al., ; Callaway et al., ; ; ; Russek et al., ).

## 3 Rational Metareasoning with Large Language Models

To achieve an optimal balance between performance and efficiency, our approach introduces a new VOC-inspired reward function (Eq. 3.1) into an Expert Iteration training loop (Anthony et al., ; Zelikman et al., ), fine-tuning a LLM to produce reasoning chains adaptively depending on task difficulty.

### 3.1 Reward modeling

Chain-of-thought prompting is a technique in which a language model is encouraged to generate an intermediate output – a “chain of thought” – prior to producing the answer to a question (Wei et al., ; Kojima et al., )
We define the reward of a chain of thought as the difference between its utility and its cost,

$$ $\mathcal{R}_{\pi}(x,y,z)=\mathcal{U}_{\pi}(z|x,y)-\mathcal{C}(z)$ (2) $$

where $x$ denotes the input for the task, $z$ represents the chain of thought, and $y$ is the target solution. The utility of the chain of thought is represented by $\mathcal{U}_{\pi}(z|x,y)$, and the cost of the intermediate computations is denoted by $\mathcal{C}(z)$. Here, utility quantifies the increase in the likelihood of generating the target sequence $y$ when the chain of thought $z$ is added to the input $x$, under the policy $\pi$:

$$ $\mathcal{U}_{\pi}(z|x,y)=\log{\pi_{\theta}(y|z,x)}-\log{\pi_{\theta}(y|x)}.$ (3) $$

Specifically, $\pi_{\theta}(y|z,x)$ indicates the probability of generating the target sequence $y$ given both the chain of thought $z$ and the input $x$, while $\pi_{\theta}(y|x)$ denotes the probability of generating $y$ with only the input $x$. The cost is directly proportional to the logarithm of the number of tokens in the chain of thought $l(z)$:

$$ $\mathcal{C}(z)=\gamma\cdot\log l(z).$ (4) $$

The hyperparameter $\gamma$ scales the cost and utility to the same magnitude. We tested both a linear and logarithmic cost function, finding that the logarithmic version was more stable and produced better results. A key benefit of this reward function is that it is parameterized by the same weights $\theta$ as the generative policy $\pi_{\theta}$, eliminating the need for an external reward model. This allows for direct estimation of the utility of a reasoning chain using the policy itself.

### 3.2 CoT Generation

We begin with a pretrained language model $\pi_{\theta}$ and an initial dataset of problems $x$ along with their corresponding correct final answers $y$: $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{D}$. Following prior work in online RL (Tang et al., ), we utilize the model itself to generate the reasoning chains, with few-shot prompting as a guide. Specifically, we concatenate a small set of examples, denoted as $\mathcal{P}$, each containing intermediate reasoning chains $z$, to each example in $\mathcal{D}$. For each task $\tau_{i}=(x_{i},y_{i})$ in the original dataset $\mathcal{D}$, we generate $K$ reasoning chains: $\hat{\tau}_{i}=\{(x_{i},z_{k,i},y_{i})\}_{k=1}^{K}$.

If none of the $K$ generated reasoning chains for a task $\tau_{i}$ leads to the correct answer, we discard all samples. To minimize the likelihood of this outcome, we adopt the rationalization approach introduced by STaR (Zelikman et al., ): if the model fails to generate the correct answer, we generate a new reasoning chain by supplying the model with the correct answer in the prompt. This allows the model to reason backward: given the correct answer, it can more easily generate a relevant reasoning chain. Finally, we assess each reasoning chain using the Rational Metareasoning reward function (Equation 2).

### 3.3 Metareasoning Training

We demonstrate the effectiveness of our reward using a variation of the Expert Iteration algorithm (EI, Anthony et al., ). EI is known for its sample efficiency and strong performance on reasoning tasks (Havrilla et al., ; Zelikman et al., ). As an example of an online reinforcement learning (RL) algorithm, EI involves both exploration and policy improvement phases, with the policy $\pi_{\theta}$ being updated using data from the exploration phase. The training process is described in Algorithm 1.

Figure: Algorithm 1 Rational Metareasoning Training

Initially, in the exploration phase, we approximate the optimal policy $\hat{\pi}^{*}$ by using rejection sampling on our student policy $\pi_{\theta}$. After generating $K$ intermediate reasoning chains $z_{1},\ldots,z_{K}$ for a given question $x$ (Section 3.2), we evaluate them using our reward function $\mathcal{R}_{\pi}$ (Section  3.1). We then construct $\mathcal{D}_{1}^{*}=\{(x_{i},z_{i},y_{i})\}_{i=1}^{N}$ by selecting the best reasoning chain for each task $i$: $z_{i}=\arg\max_{k}\{\mathcal{R}_{\pi}(x,z_{k},y)\}_{k=1}^{K}$. These rollouts are distilled into a policy $\pi_{1}$ using standard cross-entropy loss. This process can be iteratively repeated to refine the policy $\pi_{n}$ on the dataset $\mathcal{D}_{n}^{*}$.

Instead of utilizing the entire training dataset at each iteration, as is typical in standard EI algorithms (Anthony et al., ; Gulcehre et al., ), we begin with a batch of $T$ steps during the first iteration and progressively increase the number of fine-tuning training steps by $T$ with each subsequent iteration. This approach allows the model to encounter new examples gradually, resulting in slower training initially, which ultimately enhances model performance.

## 4 Experiments

We now detail the datasets (Sec. 4.1), baselines (Sec. 4.2), and training (Sec 4.3) used to evaluate our method.

### 4.1 Datasets

In our efforts to develop a general-purpose reasoning model, we applied our training process and assessed its effectiveness across a diverse range of datasets and reasoning tasks. We constructed our training set by combining the training sets from these datasets into one dataset $\mathcal{D}$ and then evaluated the model on all corresponding test sets $\mathcal{T}$. Below is a detailed description of the datasets used:

- •
ARC (Clark et al., ). The AI2 Reasoning Challenge (ARC) dataset comprises grade-school science questions, designed to evaluate a model’s capability to apply scientific knowledge.
- •
CommonsenseQA (Talmor et al., ). This dataset is centered on commonsense question answering. It leverages implicit human knowledge that is commonly known and sensible, testing the model’s ability to provide answers based on everyday reasoning.
- •
GSM8K (Cobbe et al., ). This dataset includes a variety of linguistically diverse grade school math word problems. It assesses the model’s proficiency in solving mathematical problems that require comprehension and application of arithmetic reasoning.
- •
ProofWriter (Tafjord et al., ). This dataset assesses logical deductive reasoning by asking the model to determine if a conclusion can be inferred from premises presented in natural language.

These datasets have very different train split sizes. To ensure fairness and balance between the datasets, and to manage computational costs, we composed our training mixture by sampling 1,024 random samples from each of the training sets. We then evaluated the model on the public test set of each dataset.

To further assess the generalization of our approach, we conducted out-of-distribution testing on MMLU (Hendrycks et al., ), a massive multitask benchmark consisting of multiple-choice questions from various branches of knowledge. Again, to limit computational costs, we conducted the test on the first 100 samples of each subject within the MMLU benchmark, giving us a total of 5700 questions across 57 subjects.

### 4.2 Baselines

We illustrate the advantages of our model by comparing its performance to two types of prompting strategies: Direct prompting, where the model is required to provide an immediate answer, and Chain of Thought prompting (CoT), where the model is encouraged to reason through the problem step-by-step before arriving at a solution. Since we are using pretrained models which are not specifically trained for instruction following, we provide five few-shot examples for each task from the unused portion of the training dataset. These examples are carefully chosen to ensure that the length of the reasoning chain matches the perceived difficulty of the question. In addition to these prompting methods, we adopt a finetuning baseline, comparing our method to STaR (Self-Taught Reasoner; Zelikman et al. ), which also uses the expert iteration algorithm.

### 4.3 Training Details

For our experiments, we use Microsoft Phi-2 (Javaheripi et al., ), Meta Llama-3-8B (Dubey et al., ), and Mistral-7B-v0.3 (Jiang et al., ) as the pretrained base models. We sample $K=4$ reasoning chains for each question, using a temperature $t$ of 0.5 and a $top_{p}$ value of 0.9. These parameters are chosen to balance the exploration and exploitation trade-off, allowing us to generate diverse yet relevant reasoning chains. For each iteration $n$, we sample a dataset $\mathcal{D}_{n}$ of size 512 from the union of four training datasets $\mathcal{D}$ described in 4.1. In the self-supervised fine-tuning step, we use a batch size of 16 and a learning rate of 1e-6. We believe that further improvements are possible through a more comprehensive hyperparameter search; however, due to computational constraints, we leave this for future work. Finally, we evaluate all models using greedy decoding to ensure consistent and deterministic output generation. We use pattern matching techniques to extract the answers; an exact match between the generated answer and the ground truth is considered correct.

## 5 Results

### 5.1 Performance vs Cost

We first evaluate our approach against baselines (Sec. 4.2) across several datasets (Sec. 4.1). Our key criteria are *performance* (measured by accuracy) and *cost* (measured by the number of input and output tokens).

Our experiments confirm that across all three models and four datasets, our training approach reduces cost while matching or improving performance (see Table 1 for results averaged across datasets; Tables 4 and 5 for per-dataset results; and Appendix A.1 for example reasoning chains). Fig. 1 shows these results for all three models: the left panel compares cost across different baselines and datasets, while the right compares performance. We first consider the performance of our baselines: CoT Few-Shot prompting uses a large number of input and output tokens, but yields reasonable performance. In contrast, Direct Few-Shot prompting uses fewer input (and far fewer output) tokens, but yields poor performance on reasoning-intensive datasets (GSM8K and Proofwriter, see Table 4). STaR improves on all of these approaches, using significantly fewer tokens, while achieving comparable or superior performance. Finally, Metareasoning further improves the cost-performance tradeoff by matching STaR’s accuracy while generating 20-37% fewer tokens on average.

**Table 1: Comparison of different methods based on accuracy and length metrics, averaged across datasets (means with 95% confidence intervals; bold indicates best performing approaches with overlapping 95% intervals; see Tables 4 and 5 for per-dataset results). Metareasoning achieves comparable performance while using significantly fewer input and output tokens compared to STaR or CoT Few-Shot prompting.**
| Model | Method | Accuracy (%) $\uparrow$ | # Input Tokens $\downarrow$ | # Output Tokens $\downarrow$ |
| --- | --- | --- | --- | --- |
| phi-2 | Direct Few-Shot | 50.7 ($\pm$ 1.3) | 441.6 ($\pm$ 0.9) | 0.0 ($\pm$ 0.0) |
| CoT Few-Shot | 56.6 ($\pm$ 1.3) | 1047.3 ($\pm$ 0.9) | 193.9 ($\pm$ 4.3) |  |
| STaR | 63.2 ($\pm$ 1.5) | 75.1 ($\pm$ 0.9) | 156.9 ($\pm$ 2.4) |  |
| Metareasoning | 64.7 ($\pm$ 1.4) | 75.1 ($\pm$ 0.9) | 123.1 ($\pm$ 2.1) |  |
| Meta-Llama-3-8B | Direct Few-Shot | 55.5 ($\pm$ 1.3) | 421.3 ($\pm$ 0.9) | 0.0 ($\pm$ 0.0) |
| CoT Few-Shot | 63.7 ($\pm$ 1.4) | 995.0 ($\pm$ 0.9) | 148.4 ($\pm$ 3.1) |  |
| STaR | 63.9 ($\pm$ 1.3) | 73.3 ($\pm$ 0.8) | 119.7 ($\pm$ 2.0) |  |
| Metareasoning | 64.4 ($\pm$ 1.4) | 73.3 ($\pm$ 0.9) | 95.6 ($\pm$ 2.2) |  |
| Mistral-7B-v0.3 | Direct Few-Shot | 52.7 ($\pm$ 1.4) | 498.2 ($\pm$ 1.0) | 0.0 ($\pm$ 0.0) |
| CoT Few-Shot | 57.5 ($\pm$ 1.5) | 1180.9 ($\pm$ 1.1) | 182.7 ($\pm$ 4.3) |  |
| STaR | 60.1 ($\pm$ 1.3) | 83.2 ($\pm$ 0.9) | 157.4 ($\pm$ 2.0) |  |
| Metareasoning | 60.5 ($\pm$ 1.5) | 83.2 ($\pm$ 1.0) | 114.4 ($\pm$ 2.2) |  |

Figure: Figure 1: Cost and performance. Left panel: the distribution of input and output tokens across various methods and datasets. Our method, Metareasoning, eliminates the need for few-shot prompting (reducing input tokens) and trains the model to use fewer reasoning tokens compared to STaR. Right panel: Metareasoning matches or improves accuracy relative to other methods, despite substantially reducing inference costs. Across datasets, the reduction in token count (left) is inversely related to the effectiveness of the CoT (right): the model learns that CoT is essential in GSM8K and Proofwriter, but less necessary in CommonsenseQA.
Refer to caption: extracted/5908727/plots/rm_results_phi.png

### 5.2 Adaptive computation

Section 5.1 demonstrates that our method reduces computational costs *on average*. But does it actually teach models to reason *adaptively* (by adjusting reasoning to match task complexity), or just to reason *less*?

To address this question, we first divided our test set $\mathcal{T}$ based on whether or not reasoning was needed to obtain the correct answer. We split the data based on whether Direct Few-Shot on Phi-2 obtained the correct answer. This yielded a split of 4412 “easy” and 2700 “hard” examples. Adaptive methods should use less computation to solve the easy problems.

**Table 2: The length of generated reasoning chains across models and methods (mean number of tokens with 95% confidence intervals). Adaptive methods should maximize the difference in length between the two distributions. Our method (Metareasoning) reduces the overall length and increases the difference in the length distribution between the Hard and Easy splits (as seen in column Length Reduction), demonstrating an improvement in the model’s ability to adapt the reasoning length to the complexity of the task.**
| Model | Method | Hard Split | Easy Split | Length Reduction (%) $\uparrow$ |
| --- | --- | --- | --- | --- |
| Phi-2 | CoT Few-Shot | 145.0 ($\pm$ 4.0) | 86.0 ($\pm$ 2.0) | 40.7 |
| STaR | 135.0 ($\pm$ 4.0) | 91.0 ($\pm$ 1.0) | 32.6 |  |
| Metareasoning | 122.0 ($\pm$ 4.0) | 57.0 ($\pm$ 3.0) | 53.3 |  |
| Meta-Llama-3-8B | CoT Few-Shot | 132.0 ($\pm$ 3.0) | 57.0 ($\pm$ 1.0) | 56.8 |
| STaR | 126.0 ($\pm$ 3.0) | 64.0 ($\pm$ 2.0) | 49.2 |  |
| Metareasoning | 99.5 ($\pm$ 2.5) | 27.0 ($\pm$ 1.0) | 72.9 |  |
| Mistral-7B-v0.3 | CoT Few-Shot | 144.0 ($\pm$ 4.0) | 77.0 ($\pm$ 3.0) | 46.5 |
| STaR | 149.0 ($\pm$ 6.0) | 98.0 ($\pm$ 1.0) | 34.2 |  |
| Metareasoning | 125.0 ($\pm$ 3.0) | 32.0 ($\pm$ 1.0) | 74.4 |  |

We can empirically compare the results across methods for these two data splits. As shown in Table 2, all models and methods are able to differentiate between hard and easy problems, generating fewer tokens on easier problems.(^1^11We note that we specifically chose CoT few shot examples that demonstrated adaptive reasoning length; we expect the effects here to be dependent on the chosen examples.) Intriguingly, STaR *decreases* the difference in reasoning length between hard and easy problems, suggesting that it trains out this adaptive tendency, biasing the model towards using reasoning on all problems. In contrast, Rational Metareasoning *increases* the difference in reasoning between hard and easy problems, achieving a length reduction of up to 74.4% on Mistral 7B. This indicates that our approach trains models to reason adaptively, helping them recognize when detailed reasoning is necessary and when a shorter response is sufficient.

### 5.3 Generalization

We assess the out-of-distribution generalization of our method using the MMLU benchmark (Hendrycks et al., , see Section 4.1). As presented in Table 3, our approach achieves comparable performance while generating 28.8% to 54.8% fewer tokens compared to STaR.

**Table 3: Comparison of different methods based on accuracy and length metrics in an out-of-distribution setting on the MMLU benchmark. We report the mean with 95% confidence intervals. Metareasoning achieves comparable performance while generating significantly fewer output tokens compared to STaR.**
| Model | Method | Accuracy (%) $\uparrow$ | # Output Tokens $\downarrow$ |
| --- | --- | --- | --- |
| Phi-2 | STaR | 47.9 ($\pm$ 1.3) | 110.5 ($\pm$ 1.5) |
| Metareasoning | 48.1 ($\pm$ 1.3) | 78.7 ($\pm$ 1.5) |  |
| Meta-Llama-3-8B | STaR | 56.3 ($\pm$ 1.1) | 75.4 ($\pm$ 1.4) |
| Metareasoning | 54.5 ($\pm$ 1.4) | 48.8 ($\pm$ 2.1) |  |
| Mistral-7B-v0.3 | STaR | 54.5 ($\pm$ 1.3) | 112.0 ($\pm$ 1.6) |
| Metareasoning | 54.0 ($\pm$ 1.4) | 50.6 ($\pm$ 1.5) |  |

Figure: Figure 2: Cost and Performance for Phi-2 Across Selected MMLU Domains. Left panel: Input and output token distributions for different methods and domains. Right panel: While CoT improves performance in subjects like mathematics and chemistry, it appears less effective in history and macroeconomics. The latter domains show a greater reduction in output tokens (35% for history and 37% for macroeconomics), compared to the smaller reductions seen in chemistry (20%) and mathematics (12%), where intermediate reasoning chains provide more substantial benefits.
Refer to caption: extracted/5908727/plots/mmlu_subcategories.png

Among the 57 subcategories of MMLU, Fig. 2 highlights four—high school chemistry, mathematics, U.S. history, and macroeconomics—that effectively showcase the strengths of our model. We specifically chose two subjects (U.S. history and macroeconomics) where the use of chain-of-thought (CoT) reasoning appears to be counterproductive compared to Direct Few-Shot prompting, as well as two subjects (mathematics and chemistry) where the benefits of CoT are more pronounced. Consistent with our findings from the four in-domain datasets, we observe that the reduction in output length achieved by our method is inversely related to the benefits gained from intermediate reasoning chains. In the cases of U.S. history and macroeconomics, our method results in a substantial reduction in generated tokens. In contrast, in mathematics and chemistry, the reduction in token count is smaller.

## 6 Related Work

### 6.1 Reducing inference costs

The rising cost of deploying large language models (LLMs) has driven efforts to reduce inference costs. Techniques such as Speculative Decoding (Leviathan et al., ) and Medusa (Cai et al., ) improve efficiency through parallelization, while Mixture of Experts (Jacobs et al., ; Zhou et al., ) activates only a subset of LLM parameters during decoding. Though effective, these methods require significant architectural changes and don’t adapt computation based on task difficulty. Other approaches have developed neural architectures that enable adaptive computation (Graves, ; Banino et al., ; Dehghani et al., ; Mohtashami et al., ; Schuster et al., ) but involve new architectures or training methods. In contrast, our approach uses existing architectures and pretrained models, modifying only the fine-tuning process. More similar to our approach, model routing (Ong et al., ; Jiang et al., ) optimizes resource utilization based on query complexity by routing easier queries to smaller models and harder queries to larger ones. However, this necessitates multiple models and a router, while our approach trains a single model to adaptively adjust its own outputs to match task complexity.

### 6.2 Reasoning in LLMs

Techniques such as Chain of Thought (CoT) and related methodologies (Wei et al., ; Yao et al., ; Yasunaga et al., ; Madaan et al., ; Zheng et al., ) have proven effective at enhancing LLM performance across a wide range of tasks. CoT boosts LLMs’ performance on complex reasoning by guiding them through a series of intermediate reasoning steps, increasing inference costs to improve task performance. This method can be implemented through in-context learning (Wei et al., ), prompting (Kojima et al., ), or training (Li et al., ). The benefits of CoT can be attributed to both a greater computation depth (Goyal et al., ; Pfau et al., ) and the semantic values of the thought tokens, which function as intermediate variables in the computation of the answer (Prystawski et al., ). However, recent studies have raised concerns regarding the meaningfulness of such reasoning chains in reaching the target solution, and whether models effectively utilize them to solve tasks (Turpin et al., ; Paul et al., ; Sprague et al., ). We further demonstrate that standard prompting and training methods fail to teach the model to use CoT purposefully, resulting in inefficient inference. Reasoning can also be used to bootstrap language models. Self-improving techniques (Huang et al., ; Zelikman et al., ) consist of generating reasoning chain-augmented answers for unlabeled questions and fine-tuning the LLM using those self-generated solutions as target outputs. Similar techniques, such as ReST (Gulcehre et al., ), can also be used to better align LLMs with human preferences and needs. Our approach builds on these techniques to optimize the inference cost of reasoning chains in addition to their task performance. More recently, chat models that “think before answering” have been developed (OpenAI, ), using inference-time computation to enhance their outputs. Although these models outperform others, they expend more computational resources on all tasks, even when it may not be necessary. Our method could be incorporated into their training process to help the model determine when this additional computation is genuinely beneficial.

## 7 Conclusion

In this work, we introduced a novel method inspired by previous research in AI and cognitive science aimed at reducing inference costs in LLMs. We operationalize the concept of rational metareasoning – which formalizes humans’ adaptive use of cognitive resources – with a novel reward function based on the Value of Computation (VOC). This reward function trains the LLM to optimize the use of intermediate reasoning steps in task execution, enabling it to balance task performance with computational efficiency. Empirically, we find that this approach significantly reduces the number of generated tokens and input context length while maintaining comparable performance across diverse datasets.

However, we note several limitations to our approach. First, the focus of our work here is enhancing the efficiency of large language models in reasoning tasks, rather than the overall performance of such systems. Thus, we aimed to reduce unnecessary intermediate reasoning steps, rather than improve the quality of reasoning per se. It remains to be seen whether our approach can be extended to enhance task performance in addition to computational efficiency. Additionally, we tested our approach on well-established datasets in science, commonsense reasoning, and math, but it remains untested in more specialized domains. Broader testing could help assess its generalizability and effectiveness across diverse contexts.

Most excitingly, our work demonstrates how cognitively-inspired reward functions can endow LLMs with desirable inference-time properties, opening a broad avenue of future work. Given its flexibility, this method could be integrated into instruction tuning to potentially enhance performance, even in scenarios where verifying the correctness of answers is challenging. Since the utility measure within the reward function can be tailored to prioritize any desired, measurable property, this approach offers the potential to guide models toward achieving these enhanced qualities while still benefiting from the reduced computational costs.

## Appendix A Full results tables

**Table 4: Comparison of accuracy of different methods across datasets and models. We report the mean with 95% confidence scores.**
| Model | Method | ARC | Commonsenseqa | GSM8K | Proofwriter |
| --- | --- | --- | --- | --- | --- |
| Phi-2 | Direct Few Shot | 86.1 ($\pm$ 1.1) | 68.4 ($\pm$ 2.6) | 10.2 ($\pm$ 1.6) | 38.0 ($\pm$ 3.0) |
| Cot Few Shot | 82.8 ($\pm$ 1.2) | 66.0 ($\pm$ 2.6) | 41.2 ($\pm$ 2.8) | 36.3 ($\pm$ 3.1) |  |
| STaR | 82.9 ($\pm$ 1.1) | 63.4 ($\pm$ 2.9) | 51.6 ($\pm$ 2.8) | 54.8 ($\pm$ 2.8) |  |
| Metareasoning | 83.9 ($\pm$ 1.3) | 61.6 ($\pm$ 2.7) | 54.9 ($\pm$ 2.6) | 58.2 ($\pm$ 2.9) |  |
| Meta-Llama-3-8B | Direct Few Shot | 86.7 ($\pm$ 1.0) | 72.4 ($\pm$ 2.5) | 13.1 ($\pm$ 1.8) | 49.7 ($\pm$ 3.1) |
| Cot Few Shot | 87.5 ($\pm$ 1.2) | 70.6 ($\pm$ 2.5) | 44.7 ($\pm$ 2.6) | 52.1 ($\pm$ 3.0) |  |
| STaR | 83.6 ($\pm$ 1.2) | 67.4 ($\pm$ 2.7) | 39.7 ($\pm$ 2.5) | 64.6 ($\pm$ 2.7) |  |
| Metareasoning | 83.6 ($\pm$ 1.2) | 70.5 ($\pm$ 2.7) | 40.3 ($\pm$ 2.8) | 63.3 ($\pm$ 2.8) |  |
| Mistral-7B-v0.3 | Direct Few Shot | 84.9 ($\pm$ 1.3) | 69.6 ($\pm$ 2.6) | 10.2 ($\pm$ 1.6) | 45.8 ($\pm$ 2.9) |
| Cot Few Shot | 84.3 ($\pm$ 1.2) | 66.8 ($\pm$ 2.7) | 35.5 ($\pm$ 2.4) | 43.3 ($\pm$ 2.9) |  |
| STaR | 82.9 ($\pm$ 1.4) | 69.2 ($\pm$ 2.5) | 32.8 ($\pm$ 2.4) | 55.4 ($\pm$ 3.0) |  |
| Metareasoning | 81.3 ($\pm$ 1.2) | 69.0 ($\pm$ 2.3) | 36.9 ($\pm$ 2.8) | 54.9 ($\pm$ 3.2) |  |

**Table 5: Comparison of mean output length of different methods across datasets and models. We report the mean with 95% confidence scores.**
| Model | Method | ARC | Commonsenseqa | GSM8K | Proofwriter |
| --- | --- | --- | --- | --- | --- |
| Phi-2 | Direct Few Shot | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) |
| Cot Few Shot | 74.6 ($\pm$ 1.4) | 80.4 ($\pm$ 2.2) | 161.0 ($\pm$ 2.9) | 459.6 ($\pm$ 16.5) |  |
| STaR | 91.4 ($\pm$ 1.0) | 80.6 ($\pm$ 1.5) | 155.7 ($\pm$ 3.1) | 299.9 ($\pm$ 7.2) |  |
| Metareasoning | 55.2 ($\pm$ 1.2) | 53.0 ($\pm$ 1.7) | 143.8 ($\pm$ 2.6) | 240.4 ($\pm$ 6.8) |  |
| Meta-Llama-3-8B | Direct Few Shot | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) |
| Cot Few Shot | 59.5 ($\pm$ 0.9) | 69.9 ($\pm$ 2.0) | 157.1 ($\pm$ 2.8) | 307.1 ($\pm$ 11.4) |  |
| STaR | 61.3 ($\pm$ 1.2) | 55.7 ($\pm$ 1.9) | 155.3 ($\pm$ 2.8) | 206.4 ($\pm$ 6.2) |  |
| Metareasoning | 31.4 ($\pm$ 0.7) | 26.1 ($\pm$ 0.7) | 125.8 ($\pm$ 4.7) | 199.2 ($\pm$ 8.2) |  |
| Mistral-7B-v0.3 | Direct Few Shot | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) | 0.0 ($\pm$ 0.0) |
| Cot Few Shot | 71.0 ($\pm$ 1.4) | 66.2 ($\pm$ 2.3) | 174.8 ($\pm$ 3.6) | 418.8 ($\pm$ 16.9) |  |
| STaR | 93.1 ($\pm$ 1.3) | 85.0 ($\pm$ 1.9) | 175.7 ($\pm$ 3.8) | 275.9 ($\pm$ 5.7) |  |
| Metareasoning | 39.7 ($\pm$ 1.0) | 25.0 ($\pm$ 0.7) | 158.6 ($\pm$ 3.6) | 234.3 ($\pm$ 7.2) |  |

### A.1 Qualitative examples

Below we show one example of question, reasoning chain and answer per dataset, generated with each method: CoT Few Shots (Yellow), STaR (green) and Metareasoning (Red). The following examples were generated by Phi-2 (Javaheripi et al., ).

Figure: Figure 3: Qualitative examples of reasoning processes for ARC dataset
Refer to caption: extracted/5908727/Examples/arc_examples.png

Figure: Figure 4: Qualitative examples of reasoning processes for CommonsenseQA dataset
Refer to caption: extracted/5908727/Examples/cqa_examples.png

Figure: Figure 5: Qualitative examples of reasoning processes for GSM8K dataset
Refer to caption: extracted/5908727/Examples/gsm8k_examples.png

Figure: Figure 6: Qualitative examples of reasoning processes for Proofwriter dataset
Refer to caption: extracted/5908727/Examples/pw_examples.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. Re Section 4.3, how many expert iteration steps $n$ do you perform?
soundness: 3
strengths: - Computational efficiency of LLM deployments is a timely topic relevant to the ICLR community.
- The proposed method is sufficiently simple that something like it might well get used in practice.
- The paper is well-written and clearly structured.
summary: The paper proposes to fine-tune language models with an objective that maximizes task performance while minimizing the (logarithm of) the number of tokens used. This objective is optimized with expert iteration. Results suggest that this objective does indeed help models use the extra reasoning tokens only when they're actually needed, thus reducing compute cost on easier tasks.
weaknesses: ### a) Insufficient baselines and ablations -- I don't feel like I get the "shape" of the proposed method and the potential alternatives all that well.

A few notes here:

1. How necessary is rationalization (described in the paragraph on line 131)? I assume the authors only use it because it's also used in STaR, the main baseline? Relatedly (and importantly), STaR should be described in more detail in the paper, and justified as a relevant baseline -- I needed to open up the original STaR paper to remind myself of what it does.

2. Re design of the computational cost function: what happens when the cost is linear instead of logarithmic in the number of tokens? (Authors mention they tried linear cost but don't provide details.) How exactly is $\gamma$ chosen, and how sensitive is the method to this choice?

3. Did the authors try a prompting baseline such as asking the model to think however long it needs, but answer as soon as it's ready? I'd be interested in seeing this kind of baseline with a few prompt variations, as this can plausibly also help avoid spending many tokens on easy tasks & spend more tokens on harder tasks when needed -- especially in larger and more capable models. I think models might well already be tracking whether further reasoning is useful: such tracking could plausibly arise just from self-supervised pretraining on large corpora that include texts written by humans who're tracking this implicitly. (Efficiency of such a prompting baseline could be improved further with prompt distillation, but I don't think this would be important for this paper).

4. I'm also interested in variations of the proposed method with RLHF-style algorithms other than expert iteration (e.g. iterated DPO).

### b) I did not find Figures 1 and 2 particularly useful, especially the subplots on the left side. Also, is information from subplots on the right side duplicated in the tables?

1. One idea for improving the figures could be plotting task performance on the y axis and the computational cost (e.g. input + output tokens) on the x axis. It might then be clearer whether your method is on the pareto frontier, and generally help understand the performance-efficiency tradeoff.

2. Regarding Figure 2: I think here it'd be helpful to include 5-shot and/or (0-shot, CoT) benchmarks for MMLU, to get a sense of how STaR and your method compare to these.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: ### Questions
* Section 3.2: What is the likelihood of not generating a correct answer? Will this happen frequently? Any comment or justification on this?
* Are the input tokens of failed cases counted as part of the input and output tokens?
* What's the time cost of the proposed method itself, rather than the inference costs? The complexity is largely from the online reinforcement learning used in the proposed methods, i.e., EI?
*  The reliance on a single, VOC-inspired reward function might limit flexibility across diverse task types. Is there any justification on using alternative reward structures could reveal greater adaptability and robustness?

### Minors
* Line 97: Eq. 3.1 --> Eq. (2)?
soundness: 2
strengths: * The paper introduces a unique rational metareasoning approach that balances inference cost with performance, addressing a crucial need in the efficient deployment of LLMs.
* The integration of the Value of Computation (VOC)-based reward function is well-designed and thoughtfully applied, showing careful consideration of LLM efficiency.
* The approach is tested across a diverse set of benchmarks, covering science knowledge, commonsense reasoning, math problem-solving, and logical deduction, as well as an out-of-domain generalization task (MMLU).
* Empirical results demonstrate notable reductions in token generation (20-37%), indicating that the method can achieve similar performance with fewer computational resources.
* The approach is valuable for scenarios where computational resources are limited, potentially benefiting applications needing cost-effective, high-quality language model outputs.
summary: This paper introduces a rational metareasoning approach for LLMs, using a novel Value of Computation (VOC)-based reward function to reduce inference costs without compromising performance. Tested across diverse benchmarks, the method achieves significant token savings (20-37%) compared to standard prompting techniques. The results highlight this approach as an effective solution for cost-efficient reasoning in LLMs, though further evaluation on task variety would enhance the findings’ robustness.
weaknesses: * Limited Analysis of Time Complexity: While the paper focuses on token reduction, it lacks an analysis of the time complexity of the proposed method. A deeper investigation into time savings would provide a clearer picture of its practical efficiency.
* Narrow Range of LLMs and Tasks: The method was primarily tested on a limited selection of benchmarks and model architectures. Broader experimentation across different LLMs and a wider range of tasks would strengthen claims about the generalizability of the approach.
* Limited Robustness Testing: Although the results are promising, the paper lacks robustness checks to assess how performance holds under varied conditions, such as noisy inputs or more complex task requirements.
* Scalability Concerns: The feasibility of scaling this method to very large models or highly complex tasks is not fully addressed, leaving open questions about its applicability in more computationally intensive scenarios.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: N/A
soundness: 3
strengths: 1. The authors introduce an interesting problem in LLM reasoning, optimizing LLMs’ inference cost and performance at the same time. This is important issue in using LLMs especially considering the LLMs inference cost is becoming larger. 

2. The paper is well-written and easy to follow. Overall, I could follow the whole story that the authors want to present in this paper.
summary: This paper presents a method for fine-tuning large language models to enhance the cost-performance trade-off in reasoning procedure. The authors introduce a reward function that evaluates the computational value of various reasoning chains, enabling the ranking of these chains. By integrating this reward function with expert Iteration, the method trains LLMs utilizing intermediate reasoning steps selectively, employing them according to the rewards. This approach significantly reduces inference costs on specific datasets while preserving task performance.
weaknesses: 1. Lack experiments in more realistic datasets. LLMs are not limited to tasks in text space; they are frequently utilized as agents that interact with external tools to perform complex tasks in various environments. Incorporating experiments on more realistic datasets, such as GAIA [1] and ToolBench [2], would provide valuable insights into the model's performance in more complex reasoning scenarios.

2. Currently, the method and experiments focus exclusively on CoT reasoning, generating trajectories for model fine-tuning and serving as baselines. However, CoT may not always produce the optimal trajectory and should not be considered a strong baseline. It would be beneficial to explore alternative reasoning methods for fine-tuning the models. At the very least, a discussion or justification regarding the choice of reasoning methods should be included in the paper.

[1] Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023). Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983.
[2] Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., ... & Liu, Y. (2024). StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. arXiv preprint arXiv:2403.07714.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Could the authors elaborate on how VOC translates to the setting of reasoning with LLMs? What would the computations $c$, beliefs $b$ and actions $a$ correspond to with respect to the terms presented in section 3: input $x$, reasoning chain $z$ and answer $y$?
2. Why is the LLM policy fine-tuned only with respect to the reasoning’s $\hat{z}$ that maximise the reward within the sampled batch of rasoning chains? In expert iteration, we would typically retain all reasoning chains $z_i$ for which the reward is above a certain threshold. Could the authors provide either a theoretical motivation behind their choice or run additional empirical studies comparing their training method to alternatives (e.g. expert iteration with threshold rejection, PPO) to justify the choice of the training method?
3. Is the $T$ in lines 165-169 referring to the number of fine-tunning steps used in the $\mathrm{train}(\pi, \mathcal{D}_n^*)$ subroutine at each iteration or is it the number of examples sampled from $\mathcal{D}$? 
4. How many training iterations $N$ of the algorithm are needed for convergence and what is the relationship between the compute time for policy training vs. final performance?
soundness: 2
strengths: - The paper tackles an important and underexplored problem of lowering sampling costs at inference time without sacrificing model performance in an adaptive manner.
- The proposed method shows improved performance over a set of 3 baselines, including the STaR method which is a direct ablation of Rational Metareasoning.
- The method is evaluated on a good selection of benchmark datasets.
summary: This paper proposes a new fine-tuning approach for training LLMs to generate token-efficient and adaptive responses. The method encourages the LLM to produce longer reasoning chains only when necessary. Building on the STaR framework, it introduces a filtering step to fine-tune the model on correct answers with the highest reward, where the reward function penalises unnecessarily long responses. Experiments on a dataset constructed from a mixture of four benchmark datasets show that the method effectively reduces the token count of generated responses without sacrificing accuracy. The results also demonstrate that the method promotes adaptive behaviour, producing longer responses for more difficult questions and shorter ones for simpler questions.
weaknesses: - The link between VOC and the latter presented approach of modified expert iteration is unclear. Sections 2 and 3 are not well connected together.
- Incremental contribution. The method is an extension of existing work (STaR). More justification or analysis of this approach's novelty would strengthen the paper (see question 2).
- There a few places where writing could be improved (see the writing suggestions below).
- There are a few inconsistencies or unclear statements (see the clarity comments below).
- A potential limitation, not mentioned in the paper, is that while shorter responses may be preferred from the computational costs point of view, they may not necessarily be more human friendly. The paper would benefit from a small human study assessing the qualitative aspects of the generated reasoning chains with Rational Metareasoning.
- Experiments could include additional baselines and ablation studies (see below for details).


**Comments on clarity**

- Line 159: the statement: *“Initially, in the exploration phase, we approximate the optimal policy   by using rejection sampling on our student policy $\pi_\theta$”* is unclear.  First, the notion of an optimal policy $\hat{\pi}^*$ hasn’t been defined. Second, in rejection sampling, all reasoning chains with the reward above a certain threshold should be retained. Yet, the proposed algorithm only selects a single reasoning chain that maximises the reward. The motivation for this choice is unclear (see question 2).
- It seems that there are a few inconsistencies in the presentation of Algorithm 1. Assignment of $\pi_0$ is missing. In line 2, the entire dataset $\mathcal{D}$ is assigned to  $\mathcal{D}\_n$, but from the comment and the main text it follows that $\mathcal{D}\_n$ is subsampled from $\mathcal{D}$.  The reward function in the algorithm should should be made dependent on $\pi_{n-1}$, i.e.  $\mathcal{R}\_{\pi_{n-1}}$. Storing the rewards in $r_{i,k}$ (line 7 of Algorithm 1) seems redundant given the later $\arg\max$ again uses $\mathcal{R}$. The input to this argmax should be $\mathcal{R}\_{\pi_{n-1}}(x_i, z_{i,k}, y_i)$ rather than $\mathcal{R}(x, z_{i,k}, y)$, I believe. The quantifier $\forall i$ is also missing in this step.


**Experiments**

- It would be interesting to compare Metareasoning with Direct Few-Shot prompting, where the LLM is explicitly instructed to provide concise responses (e.g. “Keep your answer concise”.). It has been previously demonstrated that such a statement shortens the expected answer length, often without sacrificing the performance [1, 2].
- The batching technique with increasing $T$ is a design choice which should be tested in an additional ablation study.
- The proposed expert-iteration algorithm could be additionally compared to other fine-tunning algorithms, like PPO, to better motivate the particular choice of the training method.

**Minor writing suggestions**

- Line 032: “… many of these methods reduce costs at the expense of performance”—this statement would benefit by adding an example with an appropriate reference similar to the juxtaposed approach of chain-of-thought.
- Lines 071-075: the word “while” is repeated 4 times in 3 consecutive sentences, consider rephrasing.
- Line 097: There is no equation 3.1 in the paper

**References**

[1] https://arxiv.org/pdf/2401.05618 

[2] https://arxiv.org/pdf/2407.19825

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: I don’t have any questions, the paper was very clear.

What the authors would have to demonstrate to see an improved rating from me:
1. Compare directly to Quiet-STaR, and demonstrate whether their approach leads to orthogonal gains (E)
2. Add an ablation where the LLM is directly fine-tuned on the author’s training sets (F)
3. Demonstrate results on additional compositional reasoning benchmarks such as AQUA-RAT since ARC and CommonsenseQA aren’t informative (G)
4. Add an ablation where the LLM is used to summarize chains of reasoning and then fine-tuned on the resulting synthetic data (H)
soundness: 3
strengths: A) The paper is clear and easy to follow. Furthermore, the method is simple enough and the paper detailed enough so that it would, I believe, be easy to reproduce. 

B) The problem of reducing the number of output tokens of CoT-like approaches has practical relevance.

C) While the LLMs that were investigated are on the small end (2.7B-8B), the authors demonstrate that their approach leads to reduced number of output tokens on multiple LLMs and datasets.

D) I appreciate the inclusion of insightful qualitative results in the Appendix.
summary: This paper presents an approach for optimizing small LLMs towards generating shorter chains of reasoning while maintaining capabilities. To this end, the authors designed a reward function that balances the log-likelihood of generating a target answer with the log normalized cost of the number of generating chains of thought tokens. Experiments on a number of question-answering datasets demonstrate that the proposed approach can reduce the number of output tokens while maintaining the LLM’s performance.
weaknesses: ## Major
E) I appreciate the attempt to demonstrate the validity of the method for multiple chains of reasoning approaches (CoT and STaR). However, in my view, the paper is incremental, since the cost of generating chains of reasoning has already been addressed by follow up work like Zelikman et al. (2024). Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. arXiv. https://doi.org/10.48550/arXiv.2403.09629 The authors should directly compare to Quiet-STaR, and investigate whether their approach can provide orthogonal gains.

F) In my view, the following important ablation is missing to ensure that the models are indeed retaining their ability to reason (and decompose a problem into subproblem) as opposed to relying simply on directly producing an answer. Looking at Figure 3 and Figure 4 in the Appendix, it seems to me the model learned to simply rely on its internalized knowledge (one conclusion from this could be that ARC and CommonsenseQA simply aren’t great benchmarks to assess complex reasoning capabilities). As a baseline, directly fine-tune the LLM to produce the correct answer. I also need to point out that the reduction in reasoning lengths is much less impressive on the harder reasoning benchmarks GSM8K and Proofwriter (see Table 5). Thus, my current hypothesis is that the proposed approach works well for problems that are not hard reasoning problems, but is much less effective on harder reasoning problems where we actually need more CoT-like methods.

G) Related to the above, I believe the authors should test on additional hard compositional problems such as AQUA-RAT (https://github.com/google-deepmind/AQuA).  

H) Another ablation I am missing is investigating to what extent RL is really needed here. What happens if you generate chains of reasoning on the training data, ask the LLM to summarize these chains, and then directly fine-tune on this synthetically generated data?

## Minor
I) LLMs are on the smaller scale, so it remains to be seen whether gains in terms of reduced number of output tokens persist for larger models.

Meta-review: The paper proposes a fine-tuning approach to improve the token cost of LLMs on reasoning tasks. The approach is motivated by metareasoning, and trains a reward model to capture the value of computation when decoding tokens for reasoning tasks. Compared to Chain-of-Thought prompting and STaR as baselines, the proposed fine-tuning achieves significant reduction in tokens used without harming overall reasoning performance on some benchmarks.

All of the reviewers agreed that the problem is well-motivated and timely, and the solution approach is sound. However all of the reviewers concurred that the paper is borderline at present, and identified the following avenues for improving the experiments and the contributions:
1. Does VoC finetuning help improve token efficiency on other approaches to reasoning beyond CoT? (e.g. Quiet-STaR, other prompting approaches like Tree-of-Thought).
2. Ablations to identify the contributions from the different components of the proposed approach (e.g. is RL with the VoC reward model necessary, or could we accomplish similar results by distilling successful reasoning chains directly).
3. Experiments on more domains (e.g. AQuA) to ensure that the proposed approach is indeed adaptive to the hardness of tasks and domains.

Decision: Reject