Kati8/ICLR_2023_2025 huggingface dataset:
authors: Lucas Spangher
authors: Tianle Li
authors: Rama Kumar Pasumarthi
authors: William F. Arnold
authors: Peter Grabowski
authors: Eugene Ie
authors: Daniel Gruhl
reviews: 
Comments: The paper introduces a new performance measurement for LLMs and tests it on some of the well-known LLMs. The measurement is two dimensional: one is for the response accuracy and the other is for the efficiency (QPS).
Strengths: This paper's topic is of interest since the community is suffering from the lack of a golden standard to measure an LLM's true performance.
Weaknesses: This paper appears to lack a solid foundation in terms of both theoretical motivation and empirical rigor. The idea seems to be introduced without substantial grounding, and the experimental methodology appears somewhat ad-hoc, merely applying the approach to several well-known LLMs without providing accompanying code, open-source datasets, or a thorough explanation of the measurement's validity. Moreover, the vague correlation presented in Figure 3, which is the main result, leaves the effectiveness of the measurement uncertain. Based on these limitations, I do not recommend accepting this paper.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper proposes a unified metric, "Model Performance and Goodness" (MPG), to benchmark LLMs based on their "Goodness" (accuracy) and their "Fastness" (queries per second, QPS). The authors aim to provide a cost-effective and computationally efficient metric that could serve users with limited resources, such as smaller companies or university labs. The authors evaluate 13 models and compare MPG's correlation to other benchmarks, such as LMSys and MMLU, demonstrating a fair level of correlation.
Strengths: Clear Motivation: The paper effectively highlights the gap in the benchmarking field, specifically the lack of an accessible, single aggregated metric that balances accuracy and computational efficiency. The intention of this benchmark is generally a good one.
Practical Application: The target audience is well-defined, and the authors justify the metric's utility for smaller organizations and rapid internal testing. If done right, this could be a helpful resource.
Weaknesses: There are actually many studies showing that MPG is not a good measure of gas consumption. See, for example, here:
https://www.science.org/doi/full/10.1126/science.1154983
https://journals.sagepub.com/doi/abs/10.1177/237946151500100109

The introduction already misses out on some related papers, such as:
tinybenchmarks: https://arxiv.org/abs/2402.14992 -> one of the first papers attempting to reduce benchmark size
metabench: https://arxiv.org/abs/2407.12844 -> essentially a very similar effort, measuring abilities via one number but over many more LLMs and --from what I can tell-- more principled?


For the score aggregation, the tree is just assumed (not estimated or derived). It's like imposing structure from the beginning, while this structure could have also been estimated. One can also see later that the factuality and problem solving benchmarks formed a cluster.
"We measured an average Queries-per-Second (QPS) by simply timing the response rate of every
prompt that was sent to the external servers for our specific benchmark questions." -> I think this might cuse all sorts of problems where time might not only reflect how long the query took. 

All of page 6 is a heatmap and a table, it really feels like there just isn't enough here for an ICLR paper. I think this would be better as a workshop paper. 

In the conclusion section, the project is --all of a sudden-- called IQ. I think this really seems to show that this has been a rushed submission. The first paragraph seems to be from an earlier version of the paper, whereas the second paragraph seems to be an exact repetition of what had been said before.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: The authors propose a method/metric that resource-constrained users can use to choose an LLM that best serves their purpose. They name it ``MPG'' to underline how one cannot summarize a performance with a single value.
Strengths: - The paper is well-written and easy to follow
- They get good correlations with LMSys (a difficult benchmark to re-produce due to online ranking LLM answers) for their benchmark evalution method, improving upon MMLU (however, see weaknesses as well).
Weaknesses: - If I'm not mistaken, I believe the starting point for the paper and the solution are not consistent. Especially the part where, who will need such a unified score to make a selection for which LLM to use, and why they cannot simply use other benchmarks. If a developer wants to compare different in-house models (that are not available online), can't they simply use offline benchmarks that are strictly related to the metrics they care about? Why would they need a unified score that may include unrelated metrics (such as social sensitivity)?
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 1
Clarity: 3
reviews: 
Comments: This work proposes a novel evaluation framework for assessing LLM performance using a dual-axis metric that captures both response quality and computational throughput (QPS). The authors validate their approach across several prominent LLM architectures. However, the paper suffers from insufficient theoretical justification and weak empirical validation. The proposed metric lacks proper motivation and grounding, while the experimental evaluation appears to rely on an informal methodology that simply applies the framework to existing models without releasing accompanying implementation code, reproducible datasets, or validation evidence. The primary contribution shown in Figure 3 exhibits weak correlations that cast doubt on the measurement's practical utility and discriminative power. These fundamental shortcomings preclude recommendation for publication.
Strengths: The research addresses a recognized gap in the field, as practitioners continue to struggle without an established benchmark for comprehensively evaluating LLM capabilities.
Weaknesses: The work demonstrates inadequate theoretical grounding and experimental rigor. The core concept is presented without sufficient motivation or prior foundation, and the empirical assessment uses a casual experimental design that merely applies the metric to several well-known models. Critical resources such as source code, benchmark datasets, and documentation justifying the measurement's validity are not provided. Additionally, the correlation analysis presented in Figure 3—representing the paper's central finding—is unconvincing and ambiguous, raising serious questions about whether the proposed metric actually measures what it claims to measure. Given these substantial limitations, acceptance is not warranted.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2