HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

Authors: Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Noah Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, Kai Chen
Primary area: datasets and benchmarks
Keywords: Large Language Models, Long Text Generation, Benchmark

Abstract:
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation.

Decision: reject
Reviewer scores: 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730349024655

thread_type:
official_review

<thread_object>
creator: reviewer_4n33

<content>
summary:
The paper introduces HelloBench, a benchmark designed to evaluate the long text generation capabilities of LLMs. It categorizes long text generation tasks into open-ended QA, summarization, chat, text completion, and heuristic text generation based on Bloom’s Taxonomy and proposes HelloEval, a human-design evaluation method. The authors conduct experiments on approximately 30 LLMs, revealing limitations in their long text generation capabilities.

strengths:
1. The proposed benchmark is well-categorized and well-grounded.

2. The metric HelloEval is quite innovative and wisely designed. It is also intuitive that using human judgments to induce metric parameters can lead to better human alignment.

3. The experiments are thorough and yield useful findings.

weaknesses:
1. I am not satisfied with the benchmark. While the benchmark is well-categorized, it is not large-scale and does not have good coverage. Its contribution is also very incremental. For example, regarding open-ended questions, the authors only collected 200 samples from a single source Quora. I believe there are existing human-collected benchmarks that are very large-scale. ELI5 is such a representative.

2. To the best of my knowledge, Quora does not allow the crawling of its data. The benchmark may cause policy violations.

3. The analysis of the results lacks depth. While the paper mentions limitations in the models' capabilities, it does not explore the underlying reasons for these limitations or suggest potential improvements.

4. The paper claims that current LLMs struggle with long text generation, but it does not adequately discuss the implications of these findings for the development of future models.

questions:
See weaknesses, plus, does Quora allow data crawling?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628673987

thread_type:
official_review

<thread_object>
creator: reviewer_jq6j

<content>
summary:
The paper proposes HelloBench, a benchmark for long text generation inspired by Bloom's taxonomy of cognitive abilities. Furthermore, it proposes HelloEval, a technique to quantitatively evaluate the performance of LLMs initially assessed as checklists. The authors propose training a linear regression model to adjust weights of different checks obtained by human judges to the overall evaluation score, and then use these trained weights when working with an LLM judge.

The authors evaluate a range of recent LLMs on HelloBench, proprietary and open-source, large and small. The evaluation elicited various insights like most LLMs tend to generate output at around 1000 tokens (even those with max_tokens of 16384 or more), text generation quality decreases with response length (especially going outside the usual length of 1000 tokens)

They also conducted a correlation analysis of the HelloEval comparing it to a series of traditional automated text generation metrics and found much stronger Spearman correlation and much lower p-value that all the other methods.

strengths:
* a new long text generation evaluation benchmark HelloBench is proposed which contains multiple tasks inspired by Bloom taxonomy of cognitive abilities
* an evaluation technique is presented for obtaining numerical scores from qualitative checklist-based assessments. It shows superior correlation to human judgements that a series of traditional metrics
* a comprehensive study of a wide range of LLMs is conducted on HelloBench eliciting insights of model's struggles to generate high-quality outputs at higher lengths.

weaknesses:
* the Bloom's taxonomy sounds inspiring but the mapping of different dimensions to text generation tasks looks superficial and containing overlaps. "...we have selected the most
 suitable task for each cognitive level" - this is not obvious to me, needs justification / proof

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680593466

thread_type:
official_review

<thread_object>
creator: reviewer_a2vb

<content>
summary:
This paper develops a new LLM evaluation benchmark, namely HelloBench, focusing on evaluating LLM’s long text generation capability, filling the missing piece in the current LLM evaluation landscape. HelloBench covers 5 tasks in 38 subcategories, totaling 647 examples, constructed by manual selection from the web and some existing benchmarks. It focuses on open-ended tasks and targets at generation over 1000 words. To evaluate LLMs, the authors further propose HelloEval, a checklist-based LLM-as-a-Judge method that shows positive correlation with human evaluation. Experiments on popular open-source and proprietary LLMs reveal their insufficiency in long text generation.

strengths:
* The benchmark, HelloBench, presents a timely effort on evaluating long text generation for LLM evaluation.
* The evaluation method, HelloEval, provides an automatic way to evaluate LLMs, saving time and effort.
* The experiments are conducted over many popular LLMs, making the findings more reliable and convincing.
* The findings offer insights on the insufficiency of existing LLMs on long text generation capability.

weaknesses:
While HelloEval shows the highest and significant correlation with human evaluation, its overall spearman correlation is just 0.32. It’s not high enough to assume that improvements on HelloEval indicate real gains on long text generation.

questions:
1. What’s the licence of HelloBench? This becomes more important for evaluation benchmarks.
2. Is the correlation analysis for HelloEval based on the annotations from the preparation stage? If so, there may be a risk of overfitting since HelloEval adopts checklist weights derived from this stage?
3. How many annotations did you use for correlation analysis? Is the number large enough to reach a significant conclusion? 0.32 is not a very high correlation value. Please include a plot for HelloEval score and human annotation score: I assume this plot looks more like a cloud rather than a line, so as to remind the others of the risk of HelloEval.
4. In lines 363-364, the authors compared the HelloEval scores between QA/text completion and summarization/chat. Is the scores of different tasks directly comparable?
5. Please provide a detailed section explaining what the HelloEval score means. For example, what does an increase of 1 in HelloEval indicate? What quality gains could be considered as significant? Can we compare scores across sub-tasks?
6. In line 1021, the authors state “By doing so, we guarantee that the data are not leaked for
the reason that they are all test samples and that their quality remains relatively high.” This is not true: test data may already leaked to LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730810153902

thread_type:
official_review

<thread_object>
creator: reviewer_bkwt

<content>
summary:
This paper presents HelloBench, a comprehensive benchmark designed to assess the long text generation capabilities of large language models (LLMs). Additionally, the authors introduce HelloEval, an automatic evaluation method that leverages LLMs-as-a-Judge to efficiently evaluate checklist results associated with each long text generation task. Through extensive experimentation across approximately 30 mainstream LLMs, the work reveals significant limitations in the long text generation capabilities of these models, including an inability to generate text exceeding 4000 words.

strengths:
HelloBench encompasses a diverse array of long text generation tasks, such as open-ended QA and summarization, thereby offering a holistic evaluation framework for assessing LLMs' long text generation capabilities. The proposed HelloEval methodology reduces the time and labor associated with human evaluation, while maintaining a strong correlation with human judgments. The authors conducted experiments across 30 mainstream LLMs, providing valuable insights into the current limitations of long text generation.

weaknesses:
1) Omission of Prior Work: The paper fails to adequately acknowledge and compare its methodology with ProxyQA [1], a pioneering framework specifically designed for evaluating long-form text generation capabilities of LLMs. Both methodologies assess generated content indirectly through evaluators to ensure adherence to specific standards. However, ProxyQA employs a query-specific checklist known as proxy questions, while HelloBench uses a more general checklist, which can not adaptively provide query- and semantic-aware checklists. The motivations and insights of both approaches appear to align closely.

2) To provide valuable context, it would be beneficial to include a comparison with ProxyQA in Table 1. Additionally, an analysis of the correlation between the results of ProxyQA and those of the proposed HelloBench would significantly strengthen this paper. If this analysis demonstrates that HelloBench aligns well with ProxyQA while offering more challenging and representative tasks, it would enhance the credibility of the work. 

3) It appears that all the key components of HelloBench or HelloEval, such as the six levels of Bloom's Taxonomy, the concept of LLM-as-a-Judge, the checklist-based evaluation method, and the dataset collection approach, have already been proposed in existing works. Furthermore, the ProxyQA work has already investigated the long-text generation benchmark and evaluation method, which overlaps with the focus of this study. The novelty and contribution of the work are quite limited. 

[1] Tan, H., Guo, Z., Shi, Z., Xu, L., Liu, Z., Feng, Y., ... & Song, L. ACL 2024. ProxyQA: An alternative framework for evaluating long-form text generation with large language models.

questions:
Lack of Robustness Analysis: Beyond examining the correlation between Hellobench and human evaluation, it would be beneficial to conduct win rate analysis or similar experiments (CI test). This would help determine if the proposed method consistently produces reliable and firm judgments, as even top-performing LLMs or human evaluators can generate inconsistent evaluation results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732467917394

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Thanks a lot for handling/reviewing our submitted manuscript. We would like to thank the reviewers for their thoughtful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our HelloBench can be improved a lot. The general responses are summarized as follows:

## The Revised Manuscript
Considering the constructive suggestions given by reviewers, we will provide a new version of our manuscript before the deadline of the author response period (November 27 at 11:59pm AoE).

## G.Q1: The coverage of HelloBench?
HelloBench has a high coverage rate and closely aligns with real user scenarios. To illustrate this point, we list the statistics we obtained from WildChat (see line 1049 - line 1118). The 15 most common long text generation task types in WildChat[1] are: report writing (HTG), guide generation (HTG), science problem solving (Chat), academic writing (HTG), content continuation (TC), question answering (OEQA), rewriting (TC), script writing (HTG), creative writing (HTG), idea generation (HTG), explanation (HTG), data analysis (Summ), character creation (HTG), curriculum development (Chat), and question generation (HTG). It can be observed that all 15 categories are covered by HelloBench's sub-tasks, achieving a **100%** coverage rate. In comparison, the previous benchmark such as ProxyQA only covered QA tasks, and LongBench-Write only covered creative writing tasks.
## G.Q2: The win-rate experiments of HelloEval.
Regarding the win-rate experiment mentioned in Reviewer bkWt.Question, we believe there are two ways to conduct it. The first method is to compare all models pairwise, but this approach is costly and may have the issues of length-bias, as analyzed in our paper (see line 2569 - line 2576). The second method is to compare all models against a single, relatively strong model (e.g., GPT-4o), which has a lower evaluation cost than the first method. Here, we used the second approach for our supplementary experiment. We had three annotators compare the win-rate of different models against GPT-4o, using checklists as the evaluation standard (see line 1728 - line 1772). Due to time constraints, we sampled 100 samples each and tested 5 LLMs. The results of the supplementary experiment are as follows:

| Model | winrate |
| --- | --- |
| Claude-3.5-Sonnet |  47.33 |
| Mistral-Lage-API | 55.67 |
| Gemma-2-27B | 47.00 |
| LLaMA-3.1-70B | 37.33 |
| Gemini-1.5-Pro | 45.33 |

We can observe that the results of the evaluated models show strong consistency in ranking with the main experiment. The main experiment (Mistral > Claude > Gemini > Gemma > LLaMA) and the supplementary experiment (Mistral > Claude > Gemma > Gemini > LLaMA), further prove the effectiveness of HelloEval. Thank you again for your suggestion. We have included this part in the appendix (see line 2261 - line 2279).
## Reference
[1] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732470358177

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summarization on the responses

comment:
Thanks for handling/reviewing our submitted manuscript: "HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models". We would like to thank the reviewers for their insightful and constructive comments and suggestions. By addressing each of the issues raised by the reviewers, we believe that the quality and clarity of our HelloBench can be improved a lot. The major responses are summarized as follows:

(1). We have carefully discussed the contributions of HelloBench and the difference between HelloBench and previous works (like ProxyQA) (See Reviewer bkWt. W1&W2&W3 and Reviewer 4N33. W1).

(2). We have additionally conducted the win-rate experiments and further demonstrated the effectiveness of HelloEval (See Reviewer bkWt.Q).

(3). We have discussed the Spearman Correlation of HelloEval and detailed the experimental settings of the effectiveness experiments of HelloEval (See Reviewer A2vB.W&Q2&Q3).

(4). We have provided more clarification on the license and data collection permission of HelloBench (See Reviewer A2vB.Q1, Reviewer 4N33. W2&Q).

(5). We have provided more clarification on the HelloEval scores and the sub-tasks of HelloBench is comparable (See Reviewer A2vB.Q4&Q5).

(6). We have discussed the reasons behind the choice of Bloom's Taxonomy (See Reviewer Jq6j.Weakness).

(7). We have provided more clarification on the potential improvements of long text generation capabilities and the research directions provided by our work (See Reviewer 4N33.W3&W4).

Again, we would like to sincerely thank you very much for these constructive comments and evaluation for our manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732693691773

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Paper Submitted and Looking forward to feedback

comment:
## Dear Reviewers
Considering the constructive suggestions given by reviewers, we have provided a new version of our manuscript and appended the old version of our manuscript in the supplementary material. We made the following changes:

1. Considering the suggestion given by Reviewer bkWt. W1&W2, we have appended the comparison between HelloBench and ProxyQA in Table 1 (see line 089 - line 096).
2. Considering the suggestion given by Reviewer A2vB. Q1, we have appended the LICENSE of HelloBench in Appendix O (see line 2612 - line 2634).
3. Considering the suggestion given by Reviewer A2vB. Q3, we have appended the plot of HelloEval Scores with Human Evaluation Scores in Figure 12  (see line 2221 - line 2245).
4. Considering the suggestions given by Reviewer A2vB. Q6 and Reviewer Jq6j.W, we have revised some of the sentences in our paper.
5. Considering the suggestion given by Reviewer 4N33.W3, we have detailed our future research directions in Appendix L (see line 2486 - line 2545).
6. Considering the suggestion given by Reviewer bkWt.Q, we have supplemented win-rate experiments and appended in the Appendix J.3 (see line 2261 - line 2279).

Thanks again for the thoughtful and constructive comments and suggestions, by addressing these issues raised by the reviewers, we believe that the quality and clarity of our HelloBench can be improved a lot. 

**Lastly, we would like to kindly ask you to take a look at our responses and revised paper and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address them. We appreciate your time and consideration!**
</content>
</thread_object>

</thread>


Metareview: