HelloBench: Evaluating Long Text Generation Capabilities of Large
  Language Models
Authors: name: Haoran Que; name: Feiyu Duan; name: Liqun He; name: Yutao Mou; name: Wangchunshu Zhou; name: Jiaheng Liu; name: Wenge Rong; name: Zekun Moore Wang; name: Jian Yang; name: Ge Zhang; name: Junran Peng; name: Zhaoxiang Zhang; name: Songyang Zhang; name: Kai Chen
Published: 2024-09-24 15:38:11

Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable
capabilities in various tasks (e.g., long-context understanding), and many
benchmarks have been proposed. However, we observe that long text generation
capabilities are not well investigated. Therefore, we introduce the
Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
text generation tasks into five subtasks: open-ended QA, summarization, chat,
text completion, and heuristic text generation. Besides, we propose
Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
method that significantly reduces the time and effort required for human
evaluation while maintaining a high correlation with human evaluation. We have
conducted extensive experiments across around 30 mainstream LLMs and observed
that the current LLMs lack long text generation capabilities. Specifically,
first, regardless of whether the instructions include explicit or implicit
length constraints, we observe that most LLMs cannot generate text that is
longer than 4000 words. Second, we observe that while some LLMs can generate
longer text, many issues exist (e.g., severe repetition and quality
degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
methods, which show that HelloEval has the highest correlation with human
evaluation. We release our code in https://github.com/Quehry/HelloBench.