Kati8/ICLR_2023_2025 huggingface dataset:
authors: C. Nicolò De Sabbata
authors: Theodore Sumers
authors: Thomas L. Griffiths
reviews: 
Comments: This paper presents a method for fine-tuning large language models to enhance the cost-performance trade-off in reasoning procedure. The authors introduce a reward function that evaluates the computational value of various reasoning chains, enabling the ranking of these chains. By integrating this reward function with expert Iteration, the method trains LLMs utilizing intermediate reasoning steps selectively, employing them according to the rewards. This approach significantly reduces inference costs on specific datasets while preserving task performance.
Strengths: 1. The authors introduce an interesting problem in LLM reasoning, optimizing LLMs’ inference cost and performance at the same time. This is important issue in using LLMs especially considering the LLMs inference cost is becoming larger. 

2. The paper is well-written and easy to follow. Overall, I could follow the whole story that the authors want to present in this paper.
Weaknesses: 1. Lack experiments in more realistic datasets. LLMs are not limited to tasks in text space; they are frequently utilized as agents that interact with external tools to perform complex tasks in various environments. Incorporating experiments on more realistic datasets, such as GAIA [1] and ToolBench [2], would provide valuable insights into the model's performance in more complex reasoning scenarios.

2. Currently, the method and experiments focus exclusively on CoT reasoning, generating trajectories for model fine-tuning and serving as baselines. However, CoT may not always produce the optimal trajectory and should not be considered a strong baseline. It would be beneficial to explore alternative reasoning methods for fine-tuning the models. At the very least, a discussion or justification regarding the choice of reasoning methods should be included in the paper.

[1] Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023). Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983.
[2] Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P., ... & Liu, Y. (2024). StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. arXiv preprint arXiv:2403.07714.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper introduces a rational metareasoning approach for LLMs, using a novel Value of Computation (VOC)-based reward function to reduce inference costs without compromising performance. Tested across diverse benchmarks, the method achieves significant token savings (20-37%) compared to standard prompting techniques. The results highlight this approach as an effective solution for cost-efficient reasoning in LLMs, though further evaluation on task variety would enhance the findings’ robustness.
Strengths: * The paper introduces a unique rational metareasoning approach that balances inference cost with performance, addressing a crucial need in the efficient deployment of LLMs.
* The integration of the Value of Computation (VOC)-based reward function is well-designed and thoughtfully applied, showing careful consideration of LLM efficiency.
* The approach is tested across a diverse set of benchmarks, covering science knowledge, commonsense reasoning, math problem-solving, and logical deduction, as well as an out-of-domain generalization task (MMLU).
* Empirical results demonstrate notable reductions in token generation (20-37%), indicating that the method can achieve similar performance with fewer computational resources.
* The approach is valuable for scenarios where computational resources are limited, potentially benefiting applications needing cost-effective, high-quality language model outputs.
Weaknesses: * Limited Analysis of Time Complexity: While the paper focuses on token reduction, it lacks an analysis of the time complexity of the proposed method. A deeper investigation into time savings would provide a clearer picture of its practical efficiency.
* Narrow Range of LLMs and Tasks: The method was primarily tested on a limited selection of benchmarks and model architectures. Broader experimentation across different LLMs and a wider range of tasks would strengthen claims about the generalizability of the approach.
* Limited Robustness Testing: Although the results are promising, the paper lacks robustness checks to assess how performance holds under varied conditions, such as noisy inputs or more complex task requirements.
* Scalability Concerns: The feasibility of scaling this method to very large models or highly complex tasks is not fully addressed, leaving open questions about its applicability in more computationally intensive scenarios.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The paper proposes to fine-tune language models with an objective that maximizes task performance while minimizing the (logarithm of) the number of tokens used. This objective is optimized with expert iteration. Results suggest that this objective does indeed help models use the extra reasoning tokens only when they're actually needed, thus reducing compute cost on easier tasks.
Strengths: - Computational efficiency of LLM deployments is a timely topic relevant to the ICLR community.
- The proposed method is sufficiently simple that something like it might well get used in practice.
- The paper is well-written and clearly structured.
Weaknesses: ### a) Insufficient baselines and ablations -- I don't feel like I get the "shape" of the proposed method and the potential alternatives all that well.

A few notes here:

1. How necessary is rationalization (described in the paragraph on line 131)? I assume the authors only use it because it's also used in STaR, the main baseline? Relatedly (and importantly), STaR should be described in more detail in the paper, and justified as a relevant baseline -- I needed to open up the original STaR paper to remind myself of what it does.

2. Re design of the computational cost function: what happens when the cost is linear instead of logarithmic in the number of tokens? (Authors mention they tried linear cost but don't provide details.) How exactly is $\gamma$ chosen, and how sensitive is the method to this choice?

3. Did the authors try a prompting baseline such as asking the model to think however long it needs, but answer as soon as it's ready? I'd be interested in seeing this kind of baseline with a few prompt variations, as this can plausibly also help avoid spending many tokens on easy tasks & spend more tokens on harder tasks when needed -- especially in larger and more capable models. I think models might well already be tracking whether further reasoning is useful: such tracking could plausibly arise just from self-supervised pretraining on large corpora that include texts written by humans who're tracking this implicitly. (Efficiency of such a prompting baseline could be improved further with prompt distillation, but I don't think this would be important for this paper).

4. I'm also interested in variations of the proposed method with RLHF-style algorithms other than expert iteration (e.g. iterated DPO).

### b) I did not find Figures 1 and 2 particularly useful, especially the subplots on the left side. Also, is information from subplots on the right side duplicated in the tables?

1. One idea for improving the figures could be plotting task performance on the y axis and the computational cost (e.g. input + output tokens) on the x axis. It might then be clearer whether your method is on the pareto frontier, and generally help understand the performance-efficiency tradeoff.

2. Regarding Figure 2: I think here it'd be helpful to include 5-shot and/or (0-shot, CoT) benchmarks for MMLU, to get a sense of how STaR and your method compare to these.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 1