LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation
  framework
Authors: name: Zecheng Tang, name: Haitian Wang, name: Quantong Qiu, name: Baibei Ji, name: Ruoxi Sun, name: Keyan Zhou, name: Juntao Li, name: Min Zhang

Long-context processing has become a fundamental capability for large
language models~(LLMs). To assess model's long-context performance, numerous
long-context evaluation benchmarks have been proposed. However, variations in
evaluation settings across these benchmarks lead to inconsistent results,
making it difficult to draw reliable comparisons. Besides, the high
computational cost of long-context evaluation poses a significant barrier for
the community to conduct comprehensive assessments of long-context models. In
this paper, we propose LOOM-Scope, a comprehensive and efficient framework for
long-context evaluation. LOOM-Scope standardizes evaluation settings across
diverse benchmarks, supports deployment of efficient long-context inference
acceleration methods, and introduces a holistic yet lightweight benchmark suite
to evaluate models comprehensively. Homepage: https://loomscope.github.io