PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models (2024)

Abstract: Large Language Models (LLMs) have succeeded remarkably in understanding longform contents. However, exploring their capability for generating long-form contents, such as reports and articles, has been relatively unexplored and inadequately assessed by existing benchmarks. The prevalent evaluation methods, which predominantly rely on crowdsourcing, are recognized for their labor-intensive nature and lack of efficiency, whereas automated metrics, such as the ROUGE score, demonstrate discordance with human judgment criteria. In this paper, we propose PROXYQA, an innovative framework dedicated to assessing longtext generation. PROXYQA comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. LLMs are tasked to generate extensive content in response to these meta-questions, by engaging an evaluator and incorporating the generated texts as contextual background, PROXYQA assesses the generated content’s quality through the evaluator’s accuracy in addressing the proxy-questions. We examine multiple LLMs, emphasizing PROXYQA’s demanding nature as a high-quality assessment tool. Human evaluation demonstrates that the proxyquestion method is notably self-consistent and aligns closely with human evaluative standards. The dataset and leaderboard is available at https://proxy-qa.com.

heading: 1 Introduction
text: Recent Large Language Models (LLMs) have made significant advancements (Brown et al., 2020; Touvron et al., 2023a,b; OpenAI, 2022a, 2023b). GPU technology innovations and memory-efficient attention mechanisms (Dao et al., 2022; Dao, 2023) have further enabled LLMs to model context sequences
* Corresponding author. †These authors contributed equally

heading: Traditional Evaluation
text: spanning tens of thousands of tokens (Anthropic, 2023; OpenAI, 2023c), paving the way for sophisticated applications such as analyzing complex scientific essays and generating detailed reports. As long-context LLMs evolve, several benchmarks have emerged to evaluate their ability to handle extensive contexts (Shaham et al., 2023; Bai et al., 2023; An et al., 2023; Zhang et al., 2023). However, these assessments primarily focus on LLMs’ comprehension of lengthy passages, using automated metrics to measure performance. This leaves a significant gap in understanding LLMs’ proficiency in generating long-form texts, an essential aspect that requires further investigation.
6806
One primary roadblock to understanding LLMs’ capability to generate long-form texts is the lack of competent evaluation methods. Current methods, often involving a combination of automated metrics and crowdsourced annotations, leave much to be desired (Xu et al., 2023). For instance, automated metrics that use word-level string (Lin, 2004) or meaning representation matching (Yuan et al., 2021) rely on gold references, which unfortunately do not exist for many generation tasks, such as reports or essay writing. Furthermore, these automated metrics are inadequate for reliably assessing long-form content due to the considerable and unstructured space of potential outputs (Celikyilmaz et al., 2020; Krishna et al., 2021). Human evaluation has its own set of issues too. Crowdsourced workers may lack the necessary expertise for evaluating knowledge-rich content, and domain experts’ subjective preferences could result in inconsistent evaluations (Xu et al., 2023). While recent studies explored using LLMs for evaluation (Chiang and Lee, 2023; Liu et al., 2023), LLMs have been found to lack the most current information required for precise verification. Moreover, their assessments have been observed to be inconsistent (Shen et al., 2023). There is a clear need for more robust and precise evaluation methods.
To address this issue, we introduce PROXYQA, a benchmark comprising human-curated metaquestions covering a wide range of subjects, from computer science to history. These meta-questions require domain expertise and up-to-date knowledge, prompting LLMs to generate detailed and comprehensive responses. To assess knowledge coverage and informativeness, we pair each metaquestion with a series of proxy-questions and answers that capture its essential points. As illustrated in Figure 1, PROXYQA uses an evaluator to answer proxy-questions based on the long-form content produced by LLMs, rather than comparing the output to a reference. If the generated content is sufficiently detailed and accurate, it should equip the evaluator with enough information to thoroughly answer all associated proxy-questions.
PROXYQA offers several benefits. By employing proxy questions and an evaluator, it eliminates the need for direct comparison against a single gold reference, enabling a more accessible and subjective evaluation. Using this approach allows evaluators without specific domain knowledge to assess content. Additionally, unlike previous datasets compiled from online sources (Nguyen
et al., 2016; Fan et al., 2019a) that potentially leading to data contamination (Sainz et al., 2023), all the proxy-questions and answers are invisible to public, thereby preventing data leakage. We apply PROXYQA to extensively test different LLMs (Touvron et al., 2023a,b; Taori et al., 2023; Chiang and Lee, 2023; OpenAI, 2022a, 2023b), including the LLMs that enhanced iterative reasoning (Yao et al., 2023a) and retrieval augmentation (Bing, 2023; Gemini, 2023). A systematic human evaluation demonstrates that PROXYQA offers a highly consistent evaluation scope, surpassing inter-human agreement rates while maintaining strong correlations with the majority preferences of humans. In-depth analysis shows that ProxyQA effectively eliminates the bias that LLM-as-a-judge (Zheng et al., 2023) prefer content generated by GPT models, and the evaluators’ accuracy on the proxyquestions accurately reflects the quality of the generated contents.

heading: 2 Related Work

heading: 2.1 Long-Form Text Generation
text: Significant strides have been made in long-form text generation, particularly in story generation (Fan et al., 2019b; Xu et al., 2020), paragraph completion (Kang and Hovy, 2020), long-term conversation (Xu et al., 2022) and article generation (Hua and Wang, 2020; Hu et al., 2022). A closely related field is long-form question answering (Fan et al., 2019a; Dasigi et al., 2021; Stelmakh et al., 2022; Lee et al., 2023), which involves generating detailed responses to complex informationseeking questions. ELI5 (Fan et al., 2019a) was a pioneer dataset for generating explanatory paragraphs in response to open-ended questions, utilizing answers from Reddit. QASPER (Dasigi et al., 2021) and QASA (Lee et al., 2023) extend general factoid questions to the domain of scientific literature. Evaluating answers on these datasets relies on comparing the generated texts with the provided single reference. However, open-ended questions can be answered in myriad different ways. ASQA (Stelmakh et al., 2022) also introduces a set of disambiguated questions from AmbigQA (Min et al., 2023) for evaluating ambiguous questions. They assume that long-form answers to ambiguous questions should resolve ambiguity. In contrast, ProxyQA aims to gauge the informativeness and comprehensiveness of long-form answers, without being confined solely to ambiguous questions.

heading: 2.2 Text Generation Evaluation
text: Automated metrics such as surface form matching (Lin, 2004; Banerjee and Lavie, 2005) and semantic representation comparison (Zhang et al., 2020; Yuan et al., 2021), face challenges with longform content due to their inability to handle the diversity of the potential outputs (Celikyilmaz et al., 2020; Krishna et al., 2021). They often do not align with human judgment (Xu et al., 2023). Attempts to use LLMs for evaluation (Liu et al., 2023; Chiang and Lee, 2023; Liu et al., 2024) are hindered by their limited access to current information and inconsistency in performance (Shen et al., 2023). Evaluators also have difficulties, particularly if they lack expertise, which can impair their judgment on key dimensions like informativeness and factuality (Gillick and Liu, 2010; Iskender et al., 2020). Strategies to enhance human evaluations include A/B testing, as seen with HURDLES (Krishna et al., 2021) and WebGPT (Nakano et al., 2021), with the latter demonstrating that providing evidence helps annotators make more informed decisions. The list of proxy questions in PROXYQA can be viewed as evidence to assist the evaluator in making decisions. While some research focused on coherence (Goyal et al., 2022; Jiang et al., 2022; Deng et al., 2022) and factuality (Goyal and Durrett, 2020; Laban et al., 2022; Min et al., 2023) in related tasks like summarization, our work emphasizes informativeness and coverage.

heading: 3 A Long-form Generation Benchmark
text: An alternative framework for evaluating long-form text generation is created. 100 meta-questions are annotated to prompt LLMs to generate detailed and
informative responses, which cover subjects in artificial intelligence (AI) research, historical event investigations, sports and entertainment event analysis. The topic distribution is shown in Figure 2. Each meta-question accompanies various proxyquestions with annotated answers, which are invisible to the LLMs to be tested. To proxy the objective evaluation of the generated contents to subjective metrics, we adopt an evaluator that takes the generated contents as contextual background and answers the proxy-questions. We assume that only if the contextual background is informative and comprehensive, can the proxy-questions be well-addressed by the evaluator. Therefore, the quality of the generated contents is reflected in the evaluator’s accuracy on the proxy-questions.

heading: 3.1 Creating the Meta-questions
text: Meta-questions were manually raised by five experienced researchers, who were instructed to initiate meta-questions in areas with which they were most familiar or had a keen interest. Besides the mostconcerned topics such as AI research, sports and gaming, less popular domains such as infrastructure and agriculture are also included. We focus on questions that are aligned with real-life scenarios and should be well-addressable in reports or articles. For instance, a pertinent question within the Computer Science domain could be: “Could you elaborate on the development of Model Parallelism and Pipeline Parallelism, detailing key milestones and contributions?” This meta-question aligns with interests in parallel computing techniques. In contrast, questions of the sort “Did Aristotle use a laptop?” from StrategyQA (Geva et al., 2021) are omitted due to their lack of occurrence in realistic
settings.
Difficulty of Meta-question Meta-questions can be classified into two levels of difficulty: easy and hard. The annotation process was bifurcated; half of the experts annotated questions using only Wikipedia, denoting easy questions, while the rest utilized various resources to labeling difficult questions. This method was adhered to in subsequent annotation rounds, with experts verifying knowledge sources to accurately categorize question difficulty. Easy questions can be sufficiently answered using only information from Wikipedia, while hard questions demand the integration of Wikipedia content with insights derived from a wider range of open-domain knowledge sources to formulate a comprehensive response. Generally, most LLMs can effectively address easy questions given their extensive training on Wikipedia corpora. Conversely, hard questions pose a challenge to the models’ ability to acquire information beyond the commonly used pre-training data, necessitating access to specialized private corpora, web searches, or document retrieval for a comprehensive response. For more details, please refer to Appendix A.8.

heading: 3.2 Annotation of Proxy-Question
text: The evaluation of the generated reports is proxied to the evaluator’s accuracy on proxy-questions. Experts are tasked with identifying the pivotal content that a satisfactory answer to a meta-question must contain. Then they craft a series of proxyquestions that probe these identified key points. For instance, regarding the example of model parallelism and pipeline parallelism mentioned in Section 3.1, a thorough answer should incorporate in-depth information about Gpipe (Huang et al., 2019), Megatron (Shoeybi et al., 2019), and other pertinent subjects. Therefore, annotators develop proxy-questions that specifically focus on Gpipe, Megatron and other related topics. As our destination is not to stress-test the evaluator but to quantify the quality of the generated contents, we present straightforward and concise proxy-questions, deliberately avoiding multi-hop and complex reasoning queries. Each annotated response is provided in a boolean format, ensuring that evaluators can effortlessly answer these proxy-questions, given a sufficiently high-quality generated context. Conversely, if an evaluator struggles to address these simple proxy-questions based on the provided context, it indicates a significant deficiency in the generated
context, with crucial information being absent.

heading: 3.3 Iterative Annotation
text: As the foundation of the evaluation in PROXYQA, proxy-questions are curated through a multi-round annotation process, where experts iteratively exchange the meta-questions while supplementing and verifying proxy-questions annotated by the others. Each meta-question is thus repeatedly given to different experts to label different proxy-questions, until a consensus is reached that all experts agree that the points covered by the proxy-questions are sufficiently comprehensive. Such an alternate labeling process ensures a multi-perspective rubric, leading to an experts-consolidated benchmark. After the iterative annotation, each meta-question is coupled with 15.5 proxy-questions on average, we then measure the inter-annotator agreement of PROXYQA. We randomly extracted a subset of 50 proxyquestions and tasked the annotators to re-annotate them. Following the Kazemi et al. (2021), the Randolph’s free-marginal multi-rater κ (Randolph, 2010), an alternative to Fleiss’ κ are measured. PROXYQA achieves κ = 0.936 thanks for the introduction of the iterative annotation process.

heading: 3.4 Quality Assurance
text: The meta-question and its corresponding proxyquestions were annotated and quality-checked iteratively. During the annotation, we excluded three meta-questions that were offensive, politically sensitive, ethically concerning, or not safe for work (NSFW). Meta-questions that have been previously posted and well-addressed on relevant forums are replaced. This ensures that LLMs cannot generate answers by directly copying content from these platforms, therefore preserving the integrity of our dataset. Additionally, it is worth noting that realworld knowledge evolves over time. Therefore, to ensure the quality of PRXOYQA, our experts are required to review and update proxy-questions periodically, as detailed in the Appendix A.9.

heading: 3.5 Evaluators
text: To ensure the generated contents such as markdown tables and math formulas can be well encoded, GPT-4 and its variant, GPT-4-Turbo (GPT-4-1106Preview), are utilized as evaluators. Instead of applying retrieval augmented generation, evaluators are required to read the generated contents and answer the proxy-questions. We prompt the evaluator to formulate answers to the proxy-questions strictly
from the information presented within the contextual background. Evaluator’s accuracy can be high only if the generated contextual background is informative and comprehensive enough. The objective assessment of these reports is subsequently anchored to the precision of the GPT evaluator. Moreover, we take the mean value of evaluation results from GPT-4 and GPT-4-Turbo to reinforce the reliability and robustness of the assessment.

heading: 4 Experiments

heading: 4.1 Setup
text: Assessment and Models To measure LLMs’ performance on PROXYQA, we compute the accuracy of the proxy-questions across easy and hard splits. Open-sourced LLMs such as LLaMA and its instruction-finetuned variants are tested. We also evaluated closed-sourced LLMs (e.g. GPT) and web-augmented LLMs, including Bard (Gemini Pro) (Gemini, 2023) and New Bing (Bing, 2023). Details of all the tested models are in the Appendix A.3. Each LLM is prompted with “Write a well-structured and extensive report to answer the question: [META-QUESTION]” under the setting of zero-shot evaluation. The max decoding length of open-sourced and closed-sourced LLMs is set to their reported maximum length. All the other hyperparameters of decoding strategies are the same as their reported settings in the original paper or API documentation.
Setting of the Evaluator We feed GPT-4 and GPT-4-Turbo with the prompt shown in Appendix A.5. Both evaluators used the same decoding strategy with top_p=1, max_tokens=10, and frequency_penalty=0. Each evaluator only processes one sub-questions at a time. We calculate the evaluator’s average accuracy to represent the generated content’s overall quality.
Research Questions To carry out an in-depth investigation of long-form content generation and the effectiveness of our proposed PROXYQA, the following research questions are naturally raised: RQ1. How do open-sourced LLMs compare to proprietary models’ ability to generate extensive reports or articles? (Section 4.2) RQ2. How well do the modern LLMs grasp the knowledge with different difficulty levels? (Section 4.3) RQ3. How do LLMs perform in generating long content from different domains? (Section 4.4) RQ4. How does the iterative and alternate annotation of proxy-questions affect the performance of the LLMs? (Section 4.5) RQ5. Does PROXYQA give higher ratings for generated content with longer length? (Section 4.6)

heading: 4.2 Main Results
text: The ability of open-sourced LLMs to generate comprehensive and extensive content is far behind the proprietary models. As shown in Table 1, the base
versions of the LLaMAs series demonstrate a limited capacity to produce long-form content. However, notable enhancements are achieved through instruction-based supervised fine-tuning (SFT), as the Vicuna-13B and LLaMA2-13B-Chat transcend most other open-sourced models, evidencing their superior capability in delivering acceptable content. However, compared to the proprietary models, the open-source LLMs far lag behind GPT models. Even the GPT-3.5-Turbo outperforms the entire suite of open-source LLMs by a significant margin, and GPT-4-Turbo maintains a substantial lead. This underscores the considerable gap that open-source LLMs must bridge to match the performance of their proprietary counterparts.

heading: 4.3 Results on Different Levels of Difficulty
text: Well-pretrained LLMs surpass others on all fronts but struggle with hard questions, while welldesigned retrieval-augmented generation (RAG) significantly make up the shortcoming. Table 1 illustrates a pronounced decline in performance among most large language models (LLMs) on the more challenging subset of questions, which cannot be well-solved solely with information from Wikipedia. Notably, even the powerful GPT-4 exhibits a marked decrease in efficacy(6.55 ↓). In contrast, equipped with the GPT-4 with a search engine and well-designed searching strategy, New Bing Creative Mode performs more robustly than other LLMs, exhibiting a comparatively minor performance loss( 39.95 →38.75). However, RAG is not a one-size-fits-all solution, as the GPT-4 model equipped with the ReAct falls short of generating comprehensive content. This is attributed to the fact that ReAct repurposes the GPT into a role more akin to planner and executor, constraining its capacity as a parametric knowledge base.

heading: 4.4 Domain
text: Proprietary LLMs overwhelmingly outperform other competitors in all domains. The advantage is further extended by integrating the search engine. Figure 3 shows that the GPT-4-Turbo surpasses the open-sourced LLMs in all aspects. However, it is worth noting that in some domains, such as music and economics, the gap between open-source models and GPT-4-Turbo is very small, but opensourced LLMs are biased and inadequate to cover all domains. Training LLMs that excel at multiple domains remains sufficient exploration.

heading: Bard GPT-4-Turbo ReAct Llama2-Chat-13B New Bing Vicuna-13B

heading: 4.5 Impact of Alternate Annotation
text: Despite the impressive performance, proprietary LLMs are still unable to cater to the preferences of every individual. Figure 4 compares the LLMs’ performance on expert-consolidated and singleexpert-focused subsets. As the iterative verification and supplementation of proxy-questions proceed, the performance of all models decreases, suggesting that LLMs cannot cater to every individual’s preferences. Remarkably, New Bing outperforms all other baselines by a considerable margin, no matter the sub-split where only a single expert is involved or on the complete expert-consolidated set. However, despite the impressive performance, significant degradation could be observed. This suggests that the multi-perspective evaluation criteria in PROXYQA pose critical challenges to LLMs.

heading: 4.6 Generation Length

heading: Improving the readability and informativeness of
text: generated content within limited token budgets remains an area for systematic exploration. The average generation lengths are presented in Table 2. It is essential to emphasize that the degree of informativeness and comprehensiveness is not proportional to the length of the generated content. Specifically, LLaMA2-13B generates lengthy content, yet it exhibits the lowest quality in generating contextual background on PROXYQA. In contrast, GPT-4-Turbo produces concise content while conveying extensive and comprehensive information. Moreover, when GPT-4 is incorporated with the search engine, the New Bing Creative Mode yields highly informative and in-depth content, significantly surpassing all other baseline models with an acceptable increase in generation length.

heading: 5 Analysis

heading: 5.1 Win Rate
text: We study the pairwise win rate among various LLMs evaluated by PROXYQA and compare the results with human evaluation to validate the effectiveness of PROXYQA.
Setup Five well-educated postgraduate students are engaged; all have not participated in annotating the meta and proxy-questions of PROXYQA; they are required to score and rank the randomly sampled reports generated by different LLMs. The scoring guideline is shown in Appendix A.4. We sampled ten meta-questions from PROXYQA and employed four LLMs to generate comprehensive reports. As a further comparison, we also follow the settings in MT-Bench (Zheng et al., 2023) that adopt LLM-as-judges, which directly rate the generated report based on the scoring guideline. Similarly, we utilize GPT-Seperate (GPT-S), which
evaluates a single report at a time, and GPT-Batch (GPT-B), which evaluates and compares multiple reports simultaneously, to score and rank each report. Given that five human evaluators are involved in the comparison, we ensure fairness and robustness by requiring GPT-S, GPT-B, and our proposed PROXYQA to evaluate each report five times. We then calculate the average win rate based on the pairwise comparison.
Result GPT-as-judges over-confident on the contents generated by GPT models, while ProxyQA’s choice is highly correlated with humans. As shown in Figure 5, majority of the evaluation results highly correlated with the human’s choices. Specifically, evaluators generally recognize the quality of the reports generated by GPT-4-Turbo and New Bing, i.e., their win rates are much higher than those of Vicuna and LLama2-Chat. It is worth noting, however, that GPT-S and GPT-B exhibit overconfidence in the quality of the reports generated by GPT-4Turbo compared to New Bing. In contrast, both human evaluators and ProxyQA exhibit a preference for New Bing over GPT-4-Turbo. This outcome attests to the effectiveness of ProxyQA and demonstrates the correlation between ProxyQA and human evaluation.

heading: 5.2 Agreement Evaluation
text: To thoroughly examine the consistency of PROXYQA and the correlation against human evaluation
criteria, we investigate the agreement rate of human assessment and PROXYQA. Two categories of agreement rates are explored. Similar to MTBench (Zheng et al., 2023), to effectively gauge the consistency of our proposed PROXYQA, we assess the self-agreement rate, which calculates the inter-evaluator agreement rate. Furthermore, we establish that the evaluation method proposed in the PROXYQA exhibits a strong correlation with human judgment by determining the agreement rate between human evaluations, referred to as the human agreement rate. We employ GPT-Seperate and GPT-Batch, as discussed in section 5.1, for comparison purposes.
Setup Following the settings in section 5.1, 20 reports generated by GPT-4-Turbo and New Bing are evaluated and compared by experts that have not participated in the annotation of PROXYQA. We analyze the agreement between different evaluation methods and human evaluations.
Self-agreement Given a pair of meta-question and its corresponding generated report, each evaluation method is required to score and vote the preferred reports n times. Let V = {v1, . . . , vn} be the set of voting results, then the self-agreement rate is calculated as:
Rself ≜ 1
C2n
n
∑ i=1
n
∑ j=i+1 1{vi = vj} (1) Where 1{⋅} denote the indicator function. The self-agreement rate, denoted as Rself , quantifies the consistency of an evaluation method.
Human-agreement The calculation is divided into Majority-to-Group (M2G), Group-to-Majority (G2M) and Group-to-Group (G2G). The RM2G quantifies the proportion of the majority vote of a specific evaluation method in concordance with the overall votes of human evaluation. Conversely, RG2M calculates the proportion of overall votes of an evaluation method that concur with the majority vote of humans, indicating how well the evaluation criteria are aligned with the majority opinion of humans. RG2G provides a view of the overall agreement between an evaluation method and the human. Let Ve = {e1, . . . , en}, Vh = {h1, . . . , hn} represent the set of the voting results of an evaluation method and human, respectively. The majority vote of a set is represented as M(⋅). Then, the
agreement is calculated as:
RM2G ≜ 1 n
n ∑ i=1 1{hi =M(Ve)} (2)
RG2M ≜ 1 n
n ∑ i=1 1{ei =M(Vh)} (3)
RG2G ≜ 1
n2
n
∑ i=1
n ∑ j=1 1{ei = hj} (4)
Results Human preference varies from individual to individual, while ProxyQA shows firm consistency and is highly correlated to the majority of humans. Table 3 illustrates that PROXYQA offers a highly consistent metric for evaluating long content generation. Furthermore, PROXYQA strongly correlates with the majority opinion of human evaluators, thereby emphasizing its efficacy in validating long-form content. Notably, the subjective preferences of human experts vary, as the agreement rate reaches only 52.19%, leading to relatively inconsistent evaluations, which is consistent with the findings of Xu et al. (2023). In contrast, PROXYQA achieves an 88.00% agreement rate, indicating its potential as a highly consistent performance indicator. Moreover, when evaluating the consensus between different evaluation methods and the majority opinion of human experts, both GPT-S and GPT-B are contrary to human preference. GPT-asjudges is overconfident in the reports generated by GPT-4-Turbo, which becomes even more extreme when evaluating with GPT-B. However, the criteria of PROXYQA significantly align with the majority opinion of human experts, surpassing GPT-asjudges in all human-agreement rates with a substantial margin and attaining 66.19% and 63.33% M2G and G2M agreement rate respectively. These findings provide robust evidence that the proposed PROXYQA can effectively and reliably assess the capabilities of LLMs in generating long-form content.

heading: 5.3 Validation of the GPT Evaluator
text: To ensure the evaluator’s accuracy and confirm that answers to proxy-questions are derived strictly from the information within the generated content, we involved five human experts to conduct a human evaluation. If the GPT-evaluator fails to extract correct information from the generated content to answer a proxy-question, or answers a proxyquestion without using the knowledge presented in the content (answer using the internal knowledge), the experts will consider it a negative sample. We validated the accuracy of our proxy-evaluator. Given the reports generated by GPT-4-Turbo, New Bing, Vicuna-13B, and LLama2-Chat, 100 proxyquestions and the boolean answer generated by the GPT-4 evaluator are sampled. The GPT-4 evaluator reaches a 91% accuracy rate, demonstrating its capability as a reliable evaluator.

heading: 5.4 Validation of Prompts
text: Furthermore, we evaluated the model’s performance using different prompts such as “Please write a comprehensive long report on the following question”. We compared the results to the performance without any prompts. Our analysis indicated that, in most instances, the presence or absence of different prompts had minimal impact on performance. We attribute this to the fact that the majority of meta-questions in ProxyQA already necessitate the LLM to “explain/introduce/compare in detail” or “generate a detailed report about .”

heading: 6 Conclusion
text: In this work, we introduce PROXYQA, a framework designed to evaluate LLMs’ ability to generate long-form text. Unlike traditional methods that rely on a direct comparison with a reference text, by employing an evaluator to use the information provided in the LLM-generated text to answer proxy-questions, the framework assesses the LLMs’ ability without fixed references or crowdsource workers. By mitigating concerns over data contamination and ensuring the relevance and freshness of evaluation content, PROXYQA enhances our understanding of LLMs and drives innovation towards developing long-form generation methods with LLMs.

heading: 7 Limitation
text: In this study, PROXYQA mainly focuses on evaluating the informative and knowledge coverage of
the generated texts. However, multiple key dimensions such as factuality (Min et al., 2023), verifiability (Liu et al., 2023), and coherency (Deng et al., 2022) should be considered for long-form content generation. For instance, proxy questions cannot measure hallucination in long-form content, a critical issue for a long-form generation. On the other hand, each meta-question is annotated by five experts, but they cannot cover potentially all the proxy-questions. More advanced methods that consider these issues will be developed in future work. In addition, due to limited annotation resources, we prioritized adding as many proxyquestions as possible to maintain a high benchmark quality, rather than increasing the number of metaquestions. More meta-questions should be included in the future version.

heading: Ethical Considerations
text: To avoid potential ethical issues, we carefully checked all questions in multiple aspects, as discussed in Section 3.4. We try to guarantee that all samples do not involve any offensive, genderbiased, or political content, and any other ethical issues. The source code will be released with instructions to support correct use. The baseline model we tested are all open-sourced LLMs and public APIs, these previous works have already considered ethical issues when creating the models.

heading: Acknowledgements
text: We would like to express our profound gratitude to the anonymous reviewers for their invaluable and insightful feedback. This research has been partially funded by the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and the Collaborative Research Fund C1042-23GF. Additional support was provided by the National Natural Science Foundation of China under Grant 62371411, the InnoHK initiative, the Government of the HKSAR, and the Laboratory for AI-Powered Financial Technologies.

heading: A Appendix

heading: A.1 Advancements in Long-Context LLMs
text: As LLMs gain global traction, their ability to adeptly process extensive sequences—such as protracted conversation histories or intricate scientific documents—becomes increasingly significant. Recent years have witnessed the emergence of LLMs capable of managing long context windows, a feat made possible by the advent of more powerful GPUs with expanded memory capacities, along with innovations in memory-efficient attention mechanisms (Dao et al., 2022; Dao, 2023). Historically, the context window size for models like GPT-2 encompassed 1024 tokens (Radford et al., 2019), which was then extended to 2048 in GPT-3 (Brown et al., 2020). Modern iterations, such as GPT-4-turbo, boast an impressive 128K token capacity (OpenAI, 2023c), while Claude 2.1 extends this even further to 200K tokens (Anthropic, 2023). Nevertheless, scaling the context window during the pretraining phase remains a daunting task as the computational demands surge quadratically with the length of the attention span, and a majority of texts within standard corpora, like Common Crawl, tend to be comparatively brief.
A novel approach gaining momentum among researchers is the augmentation of the LLMs’ context window through the process of continued training or fine-tuning. For instance, Tworkowski et al. (2023) successfully refined the 3B and 7B OpenLLaMA checkpoints, employing contrastive training techniques to adeptly handle contexts stretching up to 8K tokens. Similarly, Mohtashami and Jaggi (2023) achieved an expansion of the context length from 4K to 32K for LLaMA 7B by incorporating “landmark tokens” that effectively encapsulate blocks of the existing context. These tokens allow for focused fine-tuning of attention mechanisms, which in turn facilitates the selection of pertinent contextual blocks.
Furthermore, Chen et al. (2023) introduced a method known as positional interpolation, to be
used with LLMs that incorporate Rotary Position Embeddings (RoPE) as their choice of positional encoding (Su et al., 2021). This technique yielded promising outcomes when applied to LLaMA models ranging from 7B to 65B in size, requiring minimal fine-tuning efforts—a mere 1000 optimization steps. A different paradigm, ALiBi (Press et al., 2022), circumvents the necessity of fine-tuning altogether for expanding the context window. By eschewing positional embeddings and instead applying a linear bias to the attention scores—which is proportionate to the distance between tokens—it elegantly adjusts to handle longer contexts.
Lastly, the strategy proposed by Ratner et al. (2022) partitions extensive contexts into several sub-windows, employing the same positional embeddings across them. This innovative reuse of embeddings enables the models to cope with extended contexts without the need for additional finetuning. This collective body of work represents the ongoing evolution of strategies to enhance the capabilities of LLMs in accommodating long context sequences, a critical requirement for their effective deployment in complex, real-world applications.

heading: A.2 Evaluation for Long-Context LLMs
text: The advent of long-context LLMs has ushered in an era where evaluating performance over extensive text sequences is crucial. Benchmarks like ZeroSCROLLS (Shaham et al., 2023) have emerged to challenge these models’ understanding of expansive texts in a zero-shot setting. ZeroSCROLLS extends the foundation laid by the SCROLLS benchmark (Shaham et al., 2022)—originally designed to handle longer texts through fine-tuning—by introducing four new tasks: query-based summarization, multi-hop question answering, sentiment aggregation, and ordering book chapter summaries. It distinguishes itself by focusing on zero-shot performance, using simple natural language prompts and eschewing training data, relying on non-public, high-quality references.
Another contribution to this domain is LongBench (Bai et al., 2023), a suite of 21 datasets across 6 categories of tasks such as single- and multi-document question answering, summarization, few-shot learning, specific synthetic tasks, and code completion. What sets LongBench apart is its uniform format for all datasets, promoting a unified and automated evaluation process with metrics like F1 and ROUGE. Bamboo (Dong et al., 2023) also provides a valuable framework for analyzing
comprehension over lengthy texts, offering a selection of 10 datasets from 5 diverse activities that range from question answering to hallucination detection, text sorting, language modeling, and code completion. Bamboo specifically tackles potential data contamination by exclusively using sources released no earlier than 2023, maintaining the relevance and contemporaneity of its material.
L-Eval (An et al., 2023) introduces a bifurcated approach to evaluate LLMs, featuring both closedended and open-ended tasks. Closed-ended tasks focus on the model’s reasoning and comprehension skills in a protracted context. In contrast, its open-ended tasks provide a variety of summarization challenges that require models to synthesize information from lengthier documents. InfiniteBench (Zhang et al., 2023) is tailored to assess LLMs that process, understand, and infer information from contexts that span over 100,000 tokens. It should be noted that these datasets prioritize the assessment of long-context understanding, and as a result, a significant portion of tokens are used as inputs for the LLMs rather than outputs.

heading: A.3 Baseline Models
text: All the baselines are prompted with “Write a wellstructured and extensive report to answer the question: [META QUESTION]”. We then employ the generated results as the contextual background and force the GPT-4 and GPT-4-turbo to answer the proxy-questions accordingly. Multiple competitive baselines are tested under PROXYQA.
Base LLaMA is a set of open-sourced LLMs pretrained on diverse sources spanning multiple domains (Touvron et al., 2023a,b). The pretraining corpora include the June 2022 Wikipedia dumps, which should enable the LLaMA family to effectively address most ‘easy’ meta-questions in PROXYQA. Our experiment evaluates LLaMA-7B, LLaMA2-7B, and LLaMA2-13B.
Instruction-Finetuned LLaMA includes Vicuna (Chiang et al., 2023), Alpaca (Taori et al., 2023), LLaMA2-Chat (Touvron et al., 2023b). Vicuna is a chat assistant trained by fine-tuning LLaMA on around 70k user-shared conversations collected from ShareGPT (Eccleston, 2022). Similarly, Alpaca is trained with 52k self-instructed demonstrations adapted text-davinci-003 (OpenAI, 2022b). As an extension of base LLaMA2, LLaMA2-Chat is optimized specifically for dialogue usage of over 1 million instructions.
OpenAI APIs includes GPT-3.5-turbo(OpenAI, 2023a), GPT-4 and GPT-4-turbo (OpenAI, 2023b). The default decoding configuration is utilized to generate responses, while the maximum decoding length is set as its maximum limitation. Both GPT3.5-turbo and GPT-4-turbo are of version 1106, while GPT-4 corresponds to GPT-4-0613. The training data for GPT-4-turbo is up-to-date as of April 2023, while the remaining models are trained with data up to September 2021.
Web-Augmented LLMs utilize external search APIs are evaluated. Specifically, GPT-4 and GPT-4turbo are integrated with the Google Search API under the configuration of ReAct (Yao et al., 2023b). These models are tasked with processing metaquestions, reasoning through search traces, and extracting relevant content from search results from the internet across multiple turns. The implementation is adopted from LangChain (Chase, 2022). In addition to ReAct, the performance of New Bing (creative mode) (Bing, 2023) and Bard with Gemini Pro (Gemini, 2023) are also assessed.

heading: A.4 Scoring Guideline for Human Evaluation and LLM-as-Judges
text: Scoring Guideline
Please rate the knowledge coverage of the reports provided, using a scale of 0-5. Assess how well the report covers the necessary information related to the question. Knowledge Coverage Scale: 0 - Nonsense: The report offers no useful information and is completely irrelevant to the question. 1 - Poor: The report provides very little useful information and barely addresses the question. 2 - Fair: The report offers some useful information but lacks depth and detail, leaving the question partially unanswered. 3 - Average: The report presents a decent amount of information, addressing the question adequately but not exceptionally. 4 - Good: The report provides comprehensive information, covering the question well with appropriate depth and detail. 5 - Excellent: The report thoroughly covers all aspects of the question, offering a high level of detail and leaving no gaps in knowledge.

heading: A.5 Prompts for tested LLMs and Evaluator
text: All the tested baselines are prompted with “Write a well-structured and extensive report to answer the question: [META QUESTION]”, while the evaluators are prompted with:
Prompt for Evaluator
Read the provided document and determine whether the statement below is "True" or "False". Use only the information in the text to make your decision. Do not rely on prior knowledge or information outside of the given text. If the text does not provide enough information to make a decision, respond with "Not mentioned". Format your answer as "True", "False", or "Not mentioned". Document: [generated_report] Statement: [proxy_question]

heading: A.6 Annotation Guideline for Formulating Meta-Questions and Proxy-questions
text: The meta-questions should be based on the topic the experts are most familiar with or keen on. Meta-question: The meta-question should be: a) Answerable through thorough research. b) Aligned with real-life scenarios. c) Avoid of offensive or ethical concerns. d) Without an absolute or unique answer. e) Addressable in long-form reports or articles. f) Ensure the questions are open-ended, promoting in-depth research and discussion. Proxy-questions: For each meta-question, determine the essential proxy-questions that cover the critical contents of the topic. proxy-questions should be: a) Directly related to the meta-question. b) Comprehensive enough to cover different angles of the meta-question. Golden Answer: For each proxy-question, label the golden answer, which refers to the most accurate and relevant information. Steps for Annotation: a) Choose a topic you are familiar with or interested in. b) Formulate a metaquestion following the criteria mentioned above c) Determine the essential proxyquestions that cover the critical contents of the topic, as per the guidelines above. d) For each proxy-question, label the golden answer. e) Review the meta-question, proxyquestions, and golden answer to ensure they adhere to the guidelines and criteria provided. f) Collaborate with your peers to review and refine the meta-questions and proxy-questions. Verify and supplement the proxy-questions in an alternate way.

heading: A.7 Case Study and Error Analysis
text: Table 4- 9 show the case study of the reports generated by LLaMA2-13B-Chat, New Bing and GPT4-turbo. Although LLaMA2-13B-Chat generates lengthy contents, very little important information are contained. Therefore GPT-4 evaluator is unable to fetch useful information from the generated contents, result in low accuracy on the proxy-questions. In contrast, both New Bing and GPT-4-turbo generate more concise results conveyed with essential
contents. Specifically, critical hallucination can be observed in the output of the LLaMA2-13B-Chat, while none of the contents can be used to answer the proxy-questions. However, New Bing covers as much points as it can, providing a comprehensive overview of the given topic, while results of GPT-4-turbo is well-structured and offers detailed and helpful introduction to some of the critical contents.

heading: A.8 Distribution of the ProxyQA
text: 51% of meta-questions are categorized as easy, signifying that they can be addressed utilizing information solely from Wikipedia. In contrast, 49% of meta-questions are deemed difficult. Moreover, 53% of these questions are time-sensitive, as their answers may dynamically alter over time. A representative example of a time-sensitive question is: "Could you kindly offer a comprehensive overview of the historical development of pre-training in NLP, encompassing notable contributions?"

heading: A.9 Maintaince of the Leaderboard
text: Concerning meta-questions of varying difficulty levels, the average accuracy of evaluators will be reported for both easy and hard subsets of PROXYQA. In addition, for time-sensitive questions, proxyquestions will be incrementally updated, and the leaderboard will be dynamically adjusted to evaluate the LLMs’ possession of current knowledge. The leaderboard is updated on a weekly basis, while the proxy-question of time-sensitive metaquestion is incrementally updated on a bi-monthly basis. Each periodic update of proxy-questions will also renew the leaderboard.

heading: Introduction:

heading: A.12.2 Representative Works and Frameworks

heading: A.12.1 Definition and Overview

heading: A.12 Model Parallelism

heading: A.11.2 Representative Works and Frameworks

heading: A.11.1 Definition and Overview

heading: A.11 Data Parallelism

heading: A.14 Conclusion

heading: A.13.2 Representative Works and Frameworks

heading: A.13.1 Definition and Overview

heading: A.13 Pipeline Parallelism

heading: Future Work
text: More advanced methods that consider these issues will be developed in future work. In addition, due to limited annotation resources, we prioritized adding as many proxyquestions as possible to maintain a high benchmark quality, rather than increasing the number of metaquestions. More meta-questions should be included in the future version