BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM
  Evaluation
Authors: name: Eunsu Kim, name: Haneul Yoo, name: Guijin Son, name: Hitesh Patel, name: Amit Agarwal, name: Alice Oh

As large language models (LLMs) continue to advance, the need for up-to-date
and well-organized benchmarks becomes increasingly critical. However, many
existing datasets are scattered, difficult to manage, and make it challenging
to perform evaluations tailored to specific needs or domains, despite the
growing importance of domain-specific models in areas such as math or code. In
this paper, we introduce BenchHub, a dynamic benchmark repository that empowers
researchers and developers to evaluate LLMs more effectively. BenchHub
aggregates and automatically classifies benchmark datasets from diverse
domains, integrating 303K questions across 38 benchmarks. It is designed to
support continuous updates and scalable data management, enabling flexible and
customizable evaluation tailored to various domains or use cases. Through
extensive experiments with various LLM families, we demonstrate that model
performance varies significantly across domain-specific subsets, emphasizing
the importance of domain-aware benchmarking. We believe BenchHub can encourage
better dataset reuse, more transparent model comparisons, and easier
identification of underrepresented areas in existing benchmarks, offering a
critical infrastructure for advancing LLM evaluation research.