reviews: 
Comments: This paper introduces Project "MPG", an approach to measuring the capabilities and efficiency of Large Language Models. The main contribution of the work is to formalise two metrics: "Goodness" and "Fastness". Fastness is defined as the average response rate for an LLM when queried. Goodness, as far as I can tell, is a latent capability inferred from the distribution of performances across a series of benchmarks that purport to measure different capabilities. It looks like the authors use a hierarchical Bayesian model to infer this latent capability quotient. First, they define a hierarchy of nested capabilities, where the terminal nodes are candidate benchmarks (e.g., MMLU College Computer Science). In their case, they group benchmarks into three intermediary capabilities (Factual Knowledge, Social Sensitivity, Problem Solving), which are then grouped into a single Goodness capability. Second, They fit a Bernoulli to each item from each benchmark, representing the probability of success on that item. They then use that as a likelihood to fit a Beta distributed posterior for the intermediate capability. Then those posteriors are used to infer a Beta distributed posterior of Goodness, giving a single number that they take to denote the accuracy, or intelligence, of the LLM. This methodology allows us to aggregate performances from many different benchmarks with many different measurement schema, for which simply averaging over benchmark performance would lead to biased capability estimates.

The authors put their metrics to use, defining a Pareto frontier between goodness and fastness. This allows researchers using MPG to evaluate models by balancing performance and efficiency. The MPG methodology is generalisable beyond the benchmarks and capabilities that the authors employ here, meaning that there is scope for future extensions that study different capabilities or use more complete test beds.
Strengths: The paper has a number of strengths. First, the paper tackles an important lacuna in LLM research - accuracy is being prioritised over efficiency. The community must move towards measurement frameworks that balance performance against the efficiency of models. A preference for performance incentivises developers to constantly scale up their models and continue to use a vast number of resources. Benchmarks and schema like MPG that also incentivise efficiency can help to kickstart progress towards efficient architectures. Second, the authors cover a number of state-of-the-art LLMs, including closed- and open-source models, which is commendable. Third, they situate their work within the context of a number of other approaches and include comparisons between them and Project MPG.
Weaknesses: The paper contains many weaknesses. 

First, the presentation of the paper is poor. There are numerous spelling errors, syntactic disfluencies, and semantic ambiguities throughout. This makes it very difficult to follow the paper and discern the authors' methodology and the novel contributions of the paper. 

Second, I find the presentation of the 'Fastness' number as problematic. It is defined as the average time taken to respond to every prompt sent to external servers for benchmark questions. This seems incredibly noisy as a measure because it is contingent on the connection speeds of the authors, the traffic on the various servers being used, and rate limits on the proprietary APIs. I am not sure how reproducible these numbers would be and whether the QPS values taken from RunPod instances of open-source models are commensurable with the QPS values from the different APIs (e.g., Anthropic, OpenAI, Google). The authors are right to remark that the numbers would be different and give different rankings depending on the benchmarks and the questions, but there are so many other variables to consider here too. In any case, I would have liked to see the QPS values being normalised by the number of tokens in the query. I think that would go some way to improving comparability between benchmarks and between models. It is also not clear which kind of averaging is used to compute QPS, but I would recommend the geometric rather than the arithmetic mean since these are rates.

Third, the authors present their latent capability analysis as being in some sense novel. However, there is an extensive literature that proposes methods such as theirs, with added sophistication. In the cognitive sciences, the field of psychometrics has produced a number of latent capability estimation procedures, most notably, Item Response Theory. The proposal in this paper is also closely related to (Bayesian) Structural Equation Models which are used to infer latent abilities and cognitive capacities in social and developmental psychology and computational psychiatry. In recent years, these methodologies have been used and extended in the field of AI Evaluation, including for LLM capability evaluations. The authors should look at the papers listed at the end of this section. In particular, Burden et al. (2023), Kipnis et al. (2024), Polo et al. (2024) and Wang et al. (2023).

Fourth, the capability hierarchy in Figure 2 seems arbitrary. Finding names for the latent constructs is a common problem in latent factor estimation, but usually such hierarchies are accompanied by independent evidence for why certain test sets are deemed to probe certain capabilities and not others. Understandably, the MPG project and methodology extends beyond this exact capability hierarchy, but since it is in the paper, it needs some motivation.

Fifth, the methodology for latent capability estimation is difficult to follow. Is this a hierarchical Bayesian network? Is it a structural equation model with priors? What does it mean when the authors say 'we propose a form of a Monte-Carlo Markov chain (MCMC) to simulate latent questions from the aggregate beta distributions.' It is unclear what latent questions are.

Finally, the paper seems to end by ignoring the QPS measure and focus mostly on the latent capability measure and how well it recovers LMSys scores, etc. Much more time ought to be spent on the idea of a Pareto frontier between the two components of MPG and determining how we might create a balanced metric between these two facets. This is all left unsaid. 

References

Burden, J., Voudouris, K., Burnell, R., Rutar, D., Cheke, L., & Hernández-Orallo, J. (2023). Inferring Capabilities from Task Performance with Bayesian Triangulation. arXiv preprint arXiv:2309.11975.
Wang, X., Jiang, L., Hernandez-Orallo, J., Stillwell, D., Sun, L., Luo, F., & Xie, X. (2023). Evaluating general-purpose ai with psychometrics. arXiv preprint arXiv:2310.16379.
Hernández-Orallo, J. (2017). Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48, 397-447.
Hernández-Orallo, J. (2017). The measure of all minds: evaluating natural and artificial intelligence. Cambridge University Press.
Kipnis, A., Voudouris, K., Buschoff, L. M. S., & Schulz, E. (2024). metabench--A Sparse Benchmark to Measure General Ability in Large Language Models. arXiv preprint arXiv:2407.12844.
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024). tinyBenchmarks: evaluating LLMs with fewer examples. arXiv preprint arXiv:2402.14992.
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2023). AI Psychometrics: Using psychometric inventories to obtain psychological profiles of large language models. OSF preprint.
Ye, H., Xie, Y., Ren, Y., Fang, H., Zhang, X., & Song, G. (2024). Measuring Human and AI Values based on Generative Psychometrics with Large Language Models. arXiv preprint arXiv:2409.12106.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1