reviews: 
Comments: This paper presents an approach for optimizing small LLMs towards generating shorter chains of reasoning while maintaining capabilities. To this end, the authors designed a reward function that balances the log-likelihood of generating a target answer with the log normalized cost of the number of generating chains of thought tokens. Experiments on a number of question-answering datasets demonstrate that the proposed approach can reduce the number of output tokens while maintaining the LLM’s performance.
Strengths: A) The paper is clear and easy to follow. Furthermore, the method is simple enough and the paper detailed enough so that it would, I believe, be easy to reproduce. 

B) The problem of reducing the number of output tokens of CoT-like approaches has practical relevance.

C) While the LLMs that were investigated are on the small end (2.7B-8B), the authors demonstrate that their approach leads to reduced number of output tokens on multiple LLMs and datasets.

D) I appreciate the inclusion of insightful qualitative results in the Appendix.
Weaknesses: ## Major
E) I appreciate the attempt to demonstrate the validity of the method for multiple chains of reasoning approaches (CoT and STaR). However, in my view, the paper is incremental, since the cost of generating chains of reasoning has already been addressed by follow up work like Zelikman et al. (2024). Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. arXiv. https://doi.org/10.48550/arXiv.2403.09629 The authors should directly compare to Quiet-STaR, and investigate whether their approach can provide orthogonal gains.

F) In my view, the following important ablation is missing to ensure that the models are indeed retaining their ability to reason (and decompose a problem into subproblem) as opposed to relying simply on directly producing an answer. Looking at Figure 3 and Figure 4 in the Appendix, it seems to me the model learned to simply rely on its internalized knowledge (one conclusion from this could be that ARC and CommonsenseQA simply aren’t great benchmarks to assess complex reasoning capabilities). As a baseline, directly fine-tune the LLM to produce the correct answer. I also need to point out that the reduction in reasoning lengths is much less impressive on the harder reasoning benchmarks GSM8K and Proofwriter (see Table 5). Thus, my current hypothesis is that the proposed approach works well for problems that are not hard reasoning problems, but is much less effective on harder reasoning problems where we actually need more CoT-like methods.

G) Related to the above, I believe the authors should test on additional hard compositional problems such as AQUA-RAT (https://github.com/google-deepmind/AQuA).  

H) Another ablation I am missing is investigating to what extent RL is really needed here. What happens if you generate chains of reasoning on the training data, ask the LLM to summarize these chains, and then directly fine-tune on this synthetically generated data?

## Minor
I) LLMs are on the smaller scale, so it remains to be seen whether gains in terms of reduced number of output tokens persist for larger models.
Recommendation: 5
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3