reviews: 
Comments: This paper proposes a new fine-tuning approach for training LLMs to generate token-efficient and adaptive responses. The method encourages the LLM to produce longer reasoning chains only when necessary. Building on the STaR framework, it introduces a filtering step to fine-tune the model on correct answers with the highest reward, where the reward function penalises unnecessarily long responses. Experiments on a dataset constructed from a mixture of four benchmark datasets show that the method effectively reduces the token count of generated responses without sacrificing accuracy. The results also demonstrate that the method promotes adaptive behaviour, producing longer responses for more difficult questions and shorter ones for simpler questions.
Strengths: - The paper tackles an important and underexplored problem of lowering sampling costs at inference time without sacrificing model performance in an adaptive manner.
- The proposed method shows improved performance over a set of 3 baselines, including the STaR method which is a direct ablation of Rational Metareasoning.
- The method is evaluated on a good selection of benchmark datasets.
Weaknesses: - The link between VOC and the latter presented approach of modified expert iteration is unclear. Sections 2 and 3 are not well connected together.
- Incremental contribution. The method is an extension of existing work (STaR). More justification or analysis of this approach's novelty would strengthen the paper (see question 2).
- There a few places where writing could be improved (see the writing suggestions below).
- There are a few inconsistencies or unclear statements (see the clarity comments below).
- A potential limitation, not mentioned in the paper, is that while shorter responses may be preferred from the computational costs point of view, they may not necessarily be more human friendly. The paper would benefit from a small human study assessing the qualitative aspects of the generated reasoning chains with Rational Metareasoning.
- Experiments could include additional baselines and ablation studies (see below for details).


**Comments on clarity**

- Line 159: the statement: *“Initially, in the exploration phase, we approximate the optimal policy   by using rejection sampling on our student policy $\pi_\theta$”* is unclear.  First, the notion of an optimal policy $\hat{\pi}^*$ hasn’t been defined. Second, in rejection sampling, all reasoning chains with the reward above a certain threshold should be retained. Yet, the proposed algorithm only selects a single reasoning chain that maximises the reward. The motivation for this choice is unclear (see question 2).
- It seems that there are a few inconsistencies in the presentation of Algorithm 1. Assignment of $\pi_0$ is missing. In line 2, the entire dataset $\mathcal{D}$ is assigned to  $\mathcal{D}\_n$, but from the comment and the main text it follows that $\mathcal{D}\_n$ is subsampled from $\mathcal{D}$.  The reward function in the algorithm should should be made dependent on $\pi_{n-1}$, i.e.  $\mathcal{R}\_{\pi_{n-1}}$. Storing the rewards in $r_{i,k}$ (line 7 of Algorithm 1) seems redundant given the later $\arg\max$ again uses $\mathcal{R}$. The input to this argmax should be $\mathcal{R}\_{\pi_{n-1}}(x_i, z_{i,k}, y_i)$ rather than $\mathcal{R}(x, z_{i,k}, y)$, I believe. The quantifier $\forall i$ is also missing in this step.


**Experiments**

- It would be interesting to compare Metareasoning with Direct Few-Shot prompting, where the LLM is explicitly instructed to provide concise responses (e.g. “Keep your answer concise”.). It has been previously demonstrated that such a statement shortens the expected answer length, often without sacrificing the performance [1, 2].
- The batching technique with increasing $T$ is a design choice which should be tested in an additional ablation study.
- The proposed expert-iteration algorithm could be additionally compared to other fine-tunning algorithms, like PPO, to better motivate the particular choice of the training method.

**Minor writing suggestions**

- Line 032: “… many of these methods reduce costs at the expense of performance”—this statement would benefit by adding an example with an appropriate reference similar to the juxtaposed approach of chain-of-thought.
- Lines 071-075: the word “while” is repeated 4 times in 3 consecutive sentences, consider rephrasing.
- Line 097: There is no equation 3.1 in the paper

**References**

[1] https://arxiv.org/pdf/2401.05618 

[2] https://arxiv.org/pdf/2407.19825
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2