Lifelong Heterogeneous Benchmarking
Authors: Adhiraj Ghosh, Sebastian Dziadzio, Vishaal Udandarao, Ameya Prabhu, Samuel Albanie, Matthias Bethge


Reviewer paper summaries: This paper describes the findings of benchmarking language models (LMs) at the level of benchmark datasets' examples in different experimental settings under the standard prinicples of the social choice theory. The authors unify the leaderboard-based and arena-based benchmarking approaches and construct natural language processing and generation benchmark (LLM-Bench) and multimodal benchmark (LMM-Bench) using publicly available benchmarks/leaderboards. The results of empirical evaluation show that:

(i) fine-grained ranking of the systems in the domain of history on the multiple-choice qa (if I get the "MCQ" abbreviation correctly) differs from the global ranking (Section 4.1).

(ii) the ELO and Bradley-Terry rankings are more sensitive to removing performance criteria than the Plackett-Luce rankings (Section 4.2).

(iii) some benchmark examples are not representative w.r.t. the model rankings (Section 4.3).

(iv) systems on the OpenLLM leaderboard are similar to one another (Section 4.4).

(v) combining multiple benchmarks to aggregate the system rankings change the resulting rankings (Section 4.4).
Summary of strengths: - An attempt to combine different benchmarking approaches (standard leaderboard-based benchmarking and arena-based benchmarking).
- An attempt to reinterpret rankings of the systems on commonly used benchmarks and arenas.
Summary of weaknesses: - The paper needs improvement in terms of the presentation and methodology description. In particular, Section 2 introduces many abbreviations that are not used in the remaining part of the main text. This section can also benefit from adding a scheme, which describes the overall process for evaluating and ranking the systems in the user-oriented scenario. Description of the used aggregation procedure from the social choice theory is limited, especially for a non-expert reader. What about other critical properties in the social choice theory, e.g., the independence of irrelevant alternatives?
- There is a limited comparison of the proposed methodology with other related studies, which re-interpret benchmarking using the social choice theory, both at the level of instances and datasets: https://aclanthology.org/2023.eacl-main.48/, https://openreview.net/pdf?id=kvtVrzQPvgb, https://aclanthology.org/2022.acl-long.240/
- Some findings and conclusions represent a limited novelty. In particular, it could be expected that a system ranking in a fine-grained scenario could differ from the global ranking and that some arena-based benchmarks' examples can be less informative for a given set of systems (but could still be valuable for another set of systems, e.g., upcoming ones).
- The choice behind the ranking procedure needs more motivation. Why particular this method and not the others? At the same time, the rankings in the experiments are compared only against the global rankings but not to the other ranking procedures and models, which limtis the empirical comparison (how does the proposed solution differs from existing ones?)
- It is unclear how the user can utilize the proposed benchmarks and the ranking approach for their research and development purposes, and under which license.
Comments, suggestions and typos: The paper needs to be proofread. Missing references (e.g., L456), unneccesary parentheses (e.g., L270), broken text segments (L240). For other main suggestions please refer to the weaknesses above.
Overall assessment scores: 2