Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper focuses on optimizing inference hyperparameters in Large Language Models (LLMs) to maximize the value of text generation while operating under a limited inference budget. It introduces EcoOptiGen, a framework designed for economical hyperparameter optimization and cost-based performance evaluation. It uses existing method BlendSearch which combines Bayesian Optimization (BO) and local search for cost efficiency. Experiments show that by tuning the inference parameters such as the number of responses, temperature, and max tokens on the utility and cost of text generation in LLMs, the GPT-3.5 can achieves better performance on several benchmarks compared to the default parameters. 
overall_review: Strengths:
- This paper is well written and easy to follow. The formulation is clear: optimizating the performance on the validation set within specific budget.
- The paper is well motivated and tackles an important problem - inference parameter optimiation for LLMs.

Weaknesses:
- Limited technical novelty. The challenges of the problem is not clear. 
- The design choices are not sufficiently demonstrated with experiments. More optimization baselines should be considered. The proposed approach is not demonstrated on other non GPT-3.5 models.
review_summary: This paper has limited technical novelty and the emipirical study is not comprehensive (although I believe this is a nice project that tackles important problems).
actions_required_to_increase_overall_recommendation: In general, I feel this paper conducts a good empirical study on how to leverage existing blackbox optimiation and small set of data for hyperparameter tuning. Although I believe this is an important problem for industry or LLM users, the current presentation is lack of technical novelty and cannot demonstrate the challenge in this task. Instead it leverages existing algorithms to build the system easily. Moreover, this paper does not fully justify the design choice - lack of comparisons with some other blackbox optimiation approaches. Finally, there are no insights or explanation provided for the optimized parameters.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: &nbsp;

The authors introduce EcoOptiGen, a black-box optimizer based on BlendSearch for inference hyperparameter optimization of LLMs. The scheme features a pruning strategy to filter invalid hyperparameter configurations that exceed a cost budget based on token count numbers for the LLM. The main experimental findings are:

&nbsp;

1. EcoOptiGen outperforms untuned LLM baselines on new tasks, indicating that there are performance gains to be achieved through optimization of LLM inference hyperparameters.

2. The authors perform a a series of experiments with definitive conclusions e.g. increasing the token budget improves the performance of EcoOptiGen.

&nbsp;
overall_review: &nbsp;

**__MAJOR POINTS__**

&nbsp;

1. In terms of suggesting a formal problem statement. It should be possible to frame the current problem as one of black-box optimization under unknown constraints [1], $\argmax  f(x)  s.t.  C(x) < Budget$ where f is an objective function measuring the utility of the LLM output and x is a vector representing the LLM inference hyperparameters (configuration). In the case of code generation for example this would be the supplied function signature and docstring. In this setting, the cost constraint is unknown but can be learned using a separate surrogate model to the surrogate for the utility. While learning the unknown constraint is simply an idea for future work, it would be good to consider how the LLM hyperparameter inference procedure fits into the unknown constraint framework.

2. For the baselines, could random search over the hyperparameter configurations be added as a baseline?

&nbsp;

**__MINOR POINTS__**

&nbsp;

1. In the introduction, it may be worth citing [2, 3] for some of the limitations of ChatGPT on math problem solving. With the current merge with Wolfram Alpha forthcoming it may be important to highlight that LLMs as a piece of standalone machinery still have great difficulties solving graduate level mathematics problems.

2. It appears as though much of the code in the repository comes from [4]. This is understandable as the method is used and acknowledged as a component of the paper. It would, however, be appropriate to mention explicitly in the README of the authors' codebase that portions of the code are derived from the codebase of Wang et al.

&nbsp;

**__REFERENCES__**

&nbsp;

[1] Gelbart et al. [Bayesian optimization with unknown constraints](http://auai.org/~w-auai/uai2014/proceedings/individuals/107.pdf). In 30th Conference on Uncertainty in Artificial Intelligence, UAI 2014 (pp. 250-259). AUAI Press.

[2] Frieder et al. 2023. [Mathematical capabilities of ChatGPT](https://arxiv.org/abs/2301.13867). arXiv preprint arXiv:2301.13867.

[3] Borji, A., 2023. [A categorical archive of ChatGPT failures](https://arxiv.org/abs/2302.03494). arXiv preprint arXiv:2302.03494.

[4] Wang et al. 2021. [Economic hyperparameter optimization with blended search strategy](https://openreview.net/forum?id=VbLH04pRA3). In International Conference on Learning Representations.

&nbsp;

review_summary: &nbsp;

The authors tackle a topical problem in this paper. If the issues on clarity and reproducibility can be addressed I will strongly consider raising my score.

&nbsp;
actions_required_to_increase_overall_recommendation: &nbsp;

1. Address reproducibility concerns stemming from the checklist answers (1 point raise).

2. Conditional on addressing the reproducibility concerns I will upgrade my score by a further point (2 points in total).

&nbsp;
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper presents a novel method for optimizing the inference hyperparameters of large language models (LLMs). The approach involves a cost-based pruning strategy paired with a black-box optimization technique, namely BlendSearch. 

The authors describe a dynamic pruning approach that eliminates regions of the search space where further text generation would exceed the allocated cost/utility. The method also iteratively doubles the number of samples while ensuring the cost does not exceed a pre-defined budget, reducing the linear task into a logarithmic scale. 

The proposed approach is evaluated using the HELM protocol and compared with similar strategies. The results demonstrate the superiority of the proposed approach, with further ablation studies supporting the claim.
overall_review: The paper offers several strengths:

- The authors explore the significance of optimizing inference hyperparameters for language models, an increasingly expanding field. They demonstrate that a straightforward tuning strategy can enhance performance over default hyperparameters.
- The proposed pruning approach is both easy to understand and efficient.
- The authors successfully employ progressive sampling in order to reduce the chance of incorrect pruning. 
- The suggested method surpasses other basic hyperparameter optimization methods, as well as the default hyperparameters recommended.

There are some concerns with the paper, including:

- As noted in the Clarity section, the rationale for using only 20 examples for tuning is not explained. Presenting the average performance of the tuning strategy across multiple sampling instances could help justify this approach further.

- The authors explore a limited search space and ignore some hyperparameters.

- The paper does not investigate the temporal aspect of using the proposed hyperparameter tuning strategy. It is not clear whether the minor improvement in performance on 75% of the datasets justifies the additional computational cost compared to using the default hyperparameters, which do not require any tuning.
review_summary: The paper addresses the significance of tuning inference hyperparameters for large language models and proposes a practical and effective tuning strategy. By utilizing progressive sampling of dataset examples and BlendSearch implementation, the authors demonstrate the quality of their approach over default hyperparameters on four datasets. This study fills a crucial gap in the rapidly growing field of generative AI.
actions_required_to_increase_overall_recommendation: In my overall review, I suggested that running the proposed approach with different examples sampled each time as the tuning dataset would increase the confidence in the results and investigate the scalability of the approach. Additionally, the authors should justify the cost of performing the tuning strategy against the minimal improvement achieved, such as 0.004 on XSum, 0.02 on APPS, and less than 0.1 on HumanEval.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper presents a study on hyperparameter optimization for text generation inference using Large Language Models (LLMs). It proposes a cost-effective hyperparameter optimization method named EcoOptiGen, which optimizes multiple hyperparameters jointly, including the number of responses, max tokens, temperature, probability mass, and prompts. The paper also introduces a cost-based pruning strategy to improve the optimization efficiency under budget constraints. The effectiveness of EcoOptiGen is evaluated on four datasets, and it is shown that EcoOptiGen can find higher quality hyperparameter settings than the default settings suggested by a recent LLM benchmark. The paper shows that the holistic hyperparameter optimization can prevent suboptimal results and provides evidence that a tuned alternative model can outperform the recommended model.
overall_review: 
Strengths:

The paper addresses a novel and important issue of exploring inference hyperparameter space for Large Language Models (LLMs).
Proposed EcoOptiGen method, is novel and provides a hyperparameter optimization framework for LLMs.
The paper demonstrates the effectiveness of the EcoOptiGen method on multiple datasets and shows that it can find higher quality hyperparameter settings than the default settings suggested by HELM.
The paper presents a cost-based pruning strategy that improves optimization efficiency under budget constraints.
The findings of the paper are important for researchers and practitioners who work with LLM-enabled applications.

Weaknesses:

The paper could have provided a more detailed explanation of the BlendSearch hyperparameter search method and its appropriateness for the current problem.
The paper could have presented more clearly analysis to make it easier for readers to understand the differences between the approaches, inference budget implications, and possible errors that occurred during optimization, which could have helped to understand the problem better.
The paper does not perform statistical significance tests for the results.




review_summary: Based on my analysis of the paper, I recommend that the paper be accepted. The paper addresses an important and timely issue of AutoML concerning Large Language Models (LLMs). The proposed EcoOptiGen method is novel and preliminary results demonstrates that it can find higher quality hyperparameter settings than the default settings suggested by recent LLM benchmarks.
actions_required_to_increase_overall_recommendation: Any action that addresses any of the weaknesses of the work
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: How to tune the hyperparameters or search for the architecture of Large Language Models is one of the very contemporary questions, considering the size of these models and the currently heuristically & manually driven search. This paper provides a first, very targeted step into a more feasible subproblem of the hyperparameter search surrounding LLMs. Particularly, they concern themselves with inference-related hyperparameters of GPT models and try to optimize the utility of the generated responses on a test set of queries (context) used for tuning their search space. They tend to this problem in a cost-constrained setting, where each response generated under a context and evaluated for a given hyperparameter setting has monetary implications. A software developer trying to optimize the usefulness of the generated text of his application very well would want to cut down on the hyperparameter tuning cost.
They clearly demonstrate that tuning hyperparameters jointly considerably can increase performance (as one would expect).

From an AutoML perspective, they use a local-search-augmented Bayesian Optimization setup called BlendSearch, which they adapt to their cost-constrained setup. They similarly do this to aggressive racing over instances, but adapted according to use case-specific properties of their search space and problem.
overall_review: Weakness: 
1. Noticeably, - and as the authors admit in the discussion section "Automatically searching for optimal numbers and choices of demonstration examples can potentially result in more effective ways of using the inference budget."
the distribution of "instances" i.e. user demonstrations needs to be well crafted for the end application which is a hindrance to their applicability.
review_summary: As outlined above I appreciate the divide-and-conquer strategy on LLM models and the very targeted & feasible approach the authors take here. This paper will likely spawn discussions in this area. I would hope for a little more than applying well known BO method with local search in a cost-constrained racing heuristic (which is why I suggest a weak accept), but this is still a valid first step in dealing with LLMs from an AutoML perspective.
actions_required_to_increase_overall_recommendation: NA
review_title: null

Decision: Accept(Main Track)