Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization
Authors: Weixuan Wang
Minghao Wu
Barry Haddow
Alexandra Birch
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Summarization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Long document summarization remains a significant challenge for current large language models (LLMs), as existing approaches commonly struggle with information loss, factual inconsistencies, and coherence issues when processing excessively long documents. We propose SummQ, a novel adversarial multi-agent framework that addresses these limitations through collaborative intelligence between specialized agents operating in two complementary domains: summarization and quizzing. Our approach employs summary generators and reviewers that work collaboratively to create and evaluate comprehensive summaries, while quiz generators and reviewers create comprehension questions that serve as continuous quality checks for the summarization process. This adversarial dynamic, enhanced by an examinee agent that validates whether the generated summary contains the information needed to answer the quiz questions, enables iterative refinement through multifaceted feedback mechanisms. We evaluate SummQ on three widely used long document summarization benchmarks. Experimental results demonstrate that our framework significantly outperforms existing state-of-the-art methods across ROUGE and BERTScore metrics, as well as in LLM-as-a-Judge and human evaluations. Our comprehensive analyses reveal the effectiveness of the multi-agent collaboration dynamics, the influence of different agent configurations, and the impact of the quizzing mechanism. This work establishes a new approach for long document summarization that uses adversarial agentic collaboration to improve summarization quality.

LEARNING TO SUMMARIZE BY LEARNING TO QUIZ: ADVERSARIAL AGENTIC COLLABORATION FOR LONG DOCUMENT SUMMARIZATION
Weixuan Wang
Minghao Wu
Barry Haddow
Alexandra Birch
School of Informatics, University of Edinburgh
Tongy Lab, Alibaba Group
3 Avenn
[weixuan wang, bhaddow, a. bbirchheedddac.. uc minghao. wu@alibaba-inc. com
ABSTRACT
Long document summarization remains a significant challenge for current large language models (LLMs), as existing approaches commonly struggle with infor- mation loss, factual inconsistencies, and coherence issues when processing ex- cessively long documents. We propose SUMMQ, novel adversarial aggntt framework that addresses these limitations through collaborative intelligence be- tween specialized agents operating two complementary domains: summariza- tion and quizzing. Our approach employs summary generators and reviewers that work collaboratively to create and evaluate comprehensive summaries, while quiz generators and reviewers create comprehension questions that serve as contin- uous quality checks for the summarization process. This adversarial dynamic, enhanced by examinee agent that validates whether the generated summary contains the information needed to answer the quiz questions, enables iterative refinement through multifaceted feedback mechanisms. We evaluate SUMMQ on three widely used long document summarization benchmarks. Experimental re- sults demonstrate that our framework significantly outperforms existing state-of- the-art methods across ROUGE and BERTScore metrics, well as in LLM-as-a- Judge and human evaluations. Our comprehensive analyses reveal the effective- ness of the multi-agent collaboration dynamics, the influence of different agent configurations, and the impact the quizzing mechanism. This work establishes new approach for long document summarization that uses adversarial agentic collaboration to improve summarization quality. In addition, we release the code to foster research along this line
INTRODUCTION
Summarization has become increasingly critical in modern natural language processing, as organi- zations and individuals face ever-growing volume of textual information that requires efficient processing and comprehension (Gambhir & Gupta. 2017 Zhao et al.) 2020 Zhang et al.] 2021). Prior works in summarization have primarily focused short medium-
documents, where models can effectively capture the essential content and generate coherent summaries (See et al.) 2017 Fabbri al. 2019). Recently, there has been increasing interest in long document doeu- ment summarization, driven by the need to process extensive texts such as research articles, legal documents, and books (Huang al. 202 Kryseinski al. 2022 Saxena Keller 2024).
Recent large language models (LLMs) have shown promising results for summarization tasks (Pu al. 2023 Laban al. 2023 Keswani et al., 2024). However, existing methods struggle with long documents, often leading to significant information loss, factual inconsistencies, and difficulty
Equal contribution.
https://gi thub. www.w//m//eeeeeeeeee.omoommommee wei xuan wangn// Summq

---

Summary Generators
Generate
Quiz Generators
Feeeback
Generate 6u2
Examinee
Stmenatee
Gamenate surmory
Ganerate feedbook
Geneeate
feadback
Quiz
Summary Reviewers
Quiz Reviewers
conterted
debate
debate
agreed
ogneed
summary
surmory feedbock
quiz feedbook
Examirattor feedback
Figure I: The overall framework of SUMMQ. It consists of two tasks, summarization and quizzing, and two types of agents: generator and reviewer, resulting in four groups of agents: Summary Generators, Quiz Generators, Summary Reviewers, and Quiz Reviewers. Additionally, we include an Examinee agent to check if quiz questions can be answered by the summary.
maintaining coherence across lengthy texts (Khh et al., 2023). Current approaches often fail to capture the nuanced relationships between distant parts of a document, resulting in summaries that may miss crucial information or introduce hallucinations (Chrysostomou et al. 2024, Tang et al.] 2024a Xia et al. 2024). Recently, multi-agent systems has demonstrated potential for improving complex reasoning tasks through collaborative interactions (Guo et al. 2024), yet their application to long document summarization remains underexplored (Fang al., 2024 Kim & Kim 2025).
To address these challenges, we propose SUMMQ, an adversarial multi-agent framework that leverages collaborative intelligence to generate high-quality summaries for long documents. Our approach divides specialized agents into two complementary task domains: summarization and quizzing. Within the summarization domain, we deploy summary generators that collaboratively create comprehensive summaries through independent drafting, aggregation, and collective voting, alongside summary reviewers that rigorously evaluate content quality through independent review and structured debate mechanisms. Simultaneously, the quizzing domain employs quiz generators to create comprehension quizzes that test the completeness and accuracy of generated summaries, while quiz reviewers ensure the quality, coverage, and appropriateness of these assessments. This dual-task framework creates a natural adversarial dynamic where the quiz generation process serves continuous quality check for summarization. The summary aims to provide comprehensive coverage of the document, enabling the quiz questions to be answered correctly, while the quiz challenges the information coverage, factuality, and coherence of the summary. Furthermore, an examinee agent is introduced to provide additional feedback, ensuring that the quiz questions can be accurately answered using only the generated summary. Through iterative refinement guided by multifaceted feedback, SUMMQ ensures that the final summaries are not only comprehensive and coherent but also factually accurate and verifiable.
To validate the effectiveness of SUMMQ, we conduct extensive experiments on three long document summarization tasks including MENSA (Saxena Keller 2024), BookSum (Kryscinski et al., 2022). and GovReport (Huang et al., 2021). Our results demonstrate that SUMMQ significantly outper- forms existing state-oftthe-an methods in terms of ROUGE (Lin) 2004) and BERTScore (Zhang et al. 2020), as well as LLM-as-a-Judge and human evaluations. Furthermore, more in-depth anal- yses highlight the effectiveness of our multi-agent framework in enhancing summary quality, the coverage of the generated quizzes, and the impact of various agent configurations.
Our contributions are summarized as follows:
We introduce SUMMQ, novel adversarial multi-agent framework that integrates summa- rization and quizzing tasks to enhance the long document summarization (see SectSoniion 3). three long document summarization bench-
We conduct comprehensive experiments marks, demonstrating that SUMMQ achieves ffat-of--the-n performance across multiple evaluation metrics and human assessments (see Section 4).
2

---

We provide in-depth analyses of the multi-agent collaboration, the dynamics of the quizzing mechanism, and the impact of various agent configurations (see Section 5).
RELATED WORK
Multi-Agent Systems Recent advances in LLMs have enabled the development of multi-agent
systems that harness the strengths of multiple agents to tackle complex tasks (Wang et al. [024]
Guo et al. 2024, Xi ct al.) 2025). These systems typically involve agent collaboration to boost per- formance, seen in multi- ageett debating (Du et al.] [2024] Xiong al. 2023; Chen et al. 2023a
2024b) and discussion (Chen et al. [202| Saha et al. | 2024) for reasoning over short texts (Du al.
Tang et al.
2024 Tang et al.) 2024b), paper review (Xu al. (023). dataset synthesis (Wang al.
2025b), machine translation (Wu et al. (024), and code generation (Huang et al., 2023 Wang et al.
2025ap.
Collaboration among agents introduces diverse perspectives and complementary skills,
leading to higher-quality and more robust outputs.
Long document Summarization Long document summarization has seen various methods, in- cluding architectural optimizations (e.g., sparse attention (Liu et al. 2021 Ivgi et al. 2023 Bertsch
al.
2023), Long-context finetuning (Beltagy al. 2020 Guo al. 2021 memory augmentation
(Cui & Hu
2021 Saxena et al.) (025), window extension PPress et al. 2021 Chen et al. 2023b Su
al. 2024
Yen et al. 2024)) and chunking strategies (e.g., sliding window (Zaheer et al. 2020
Pang et al. 2023). LLMs with improved long-context abilities have shifted the field toward leverag-
ing their strong language skills for summarization (Goyal al. 2022, Ratner et al. 2023 Keswani
et allaal12000 (202)), but still face challenges with context limits and maintaining coherence (Pu al. 2023
Liu et al. (002). Multi-agent systems have been explored to address these issues, enabling collab-
orative, more accurate summaries (Zhao et al., 2024 Fang al. 2024 Jeong al.
2025 though
many still rely on self-verification, leading to biases and missed errors.
METHODOLOGY
In this section, we present SUMMQ for long document summarization, as illustrated in Figure I We first introduce the overall workflow in /Section 3.1 and then describe the collaboration between the generator and reviewer agents in Section 3.2 and Section 3.3] respectively.
Algorithm 1: Overall SUMMQ Workflow
Input: Document D; Summary Generators G.; Quiz Generators Gy; Summary Reviewers Rai Quiz Reviewers Ryi Examinee E; Number of iterations Taer Output: Accepted summary S*, accepted quiz Q*
{(0) - 0: (0) 0
/1 Initialize summary s(0) quis Q10)
for iteration t = =1 to Ther do
S(t) + GENERATE D. ((t-1)
17 Summary Generatera produce candidate aumarias
Q(9) GENERATE Gg. D. (1-(1=1))=
Qui Generatora produce cand quiazes
F(!) + REVIEW (R,, S(1), ((1,(())) D):
11 Summary Reviewera produce reedback aummary
F(!) REVIEW RR _ D);
I/ Usiz Reviewers produce feedback on quiz
F(!) TAKEQUIZ(E,
Examinas E
quiz basad en aummary
F(1) < F(C)) U ((() summary
// Merga faedback relevant aummary
F(1) < (((1) U (()) quiz
// Marga feedback relevant to quiz
10
F! and p(t)
0 then
return (S(O) (()))
11
11 IE no lssues, accept and return the summary and quiz
12 return ((TT),(TT)
3

---

3.1 OVERALL WORKFLOW
The overall workflow of SUMMQ, depicted in Figure 1 involves two primary tasks: summarization and quiz generation, supported by two types of agents: generators 9 and reviewers R. These tasks and agents combine to form four distinct components: Summary Generators Ge, Quiz Generators 99- Summary Reviewers R., and Quiz Reviewers Rg. Additionally. an Examinee agent is incorporated to validate that the quiz questions can be accurately answered using the summary.
The interaction between summarization and quiz generation creates natural adversarial framework that continuously improves summarization quality. In this framework, the summary aims to provide comprehensive coverage that enables correct answers quiz questions, while the quiz challenges the information coverage, factuality, and coherence of the summary. This dual-task approach ensures that both components evolve together, resulting summaries that are not only informative but also verifiable through targeted questioning.
The iterative workflow of SUMMQ, as detailed in Algorithm I operates through a systematic pro- cess of generation, reviewing, and refinement. Beginning with an input document D, the system initializes empty summary and quiz states and enters an iterative loop for up to Titer iterations. In each iteration the process unfolds in four key stages. First, the Summary Generators Gs produce candidate summary (K() based on the document and any previous summary, while Quiz Generators 9g simultaneously generate a candidate quiz ((t). Second, the reviewing phase begins with Sum- mary Reviewers R2 evaluating the generated summary against both the quiz and original document to produce feedback e and Quiz Reviewers Ry assessing the quiz quality to generate feedback F!") Third, an Examinee agent & attempts to answer the quiz questions using only the generated summary, providing additional feedback F(!)) that is then merged with the respective summary and quiz feedback streams. Finally, the system performs an acceptance check: if both feedback sets are empty, the current summary and quiz are accepted and returned. This iterative refinement continues until either high-quality outputs are achieved or the maximum iteration limit is reached. Note that GENERATE() and REVIEW() are detailed in Section 3.2 and [Section 3.3 respectively.
This comprehensive workflow design ensures allows for continuous improvement based on concrete feedback, while the dual-task approach of simultancous summary and quiz generation creates natural consistency check that enhances the overall reliability and coherence of the final outputs.
Algorithm 2: GENERATE(): Generator Collaboration
Input: Document D; Previous summary/quiz z'; Generator agents G = {9:} Aggregator agent A Ranker agent ARanker
Output: Final summary/quiz =²
Phase 1: Independent Draft Generation;
for each generator agent g: E G do
L + DRAFT((FF D,2));
// Senerate independent draft aummary/quiz // set of all draft sumariies/quizze
Z = =(21. 22.,,,,,,,,,
Phase 2: Draft Aggregation:
Znge + AGGREGATE( (AAgg. Z);
If Aggregate drafts an unified summary/quiz
Phase 3: Best Draft Selection;
Thest <- BESTSELECT/ (ETEEEEET(AAaeeee,, 2):
// Select best individual draft
Phase 4: Collective Voting;
10 C. C+ (Zagg: Phest)
Candidate summartes/quiznee for voting
11 for each agent 9j E G do
12 vote, & PREFER C, D)
// Agent 8, votes for preserzed candidate Select the candidate with most votes
2" arg ma Kj vote, 2}|
return 2*
Quiz Generation The Quiz Generators 9, are responsible for producing a diverse range of ques- tion types, including multiple-choice, true-false, and short-answer questionn. Through collaboration
4

---

among multiple Quiz Generators, the system generates 10 questions for each type well as the corresponding answers, resulting in a total of 30 question-answer pairs per quiz.
3.2 GENERATOR COLLABORATION
The generator collaboration SUMMQ built around a multi- process that combines strengths of multiple generator agents. As shown in Alggorithmm 2] this process unfolds in four phases:
Phase 1: Independent Draft Generation The process begins with each generator agent E G working independently to create its own draft summary/quiz 2, from the input document D and any previous summary quiz z'. This parallel approach naturally leads to diverse initial drafts, since different agents may emphasize various aspects of the document. We collect all these draft summaries quizzes into a set Z {21, 22, 2n where denotes the total number of generator agents involved the process.
Phase Draft Aggregation In the second phase, an aggregator agent AAgs combines the individ- ual draft summaries or quizzes into a unified summary quiz. Zage- This agent selectively combines the strengths of each draft and incorporates complementary information that individual agents may have overlooked. By drawing upon the collective knowledge across all drafts, AAge creates more comprehensive summary or quiz that harnesses the diverse perspectives and insights.
Phase Best Draft Selection Concurrently with the aggregation process, a ranker agent ARanker evaluates each of the individual draft in 2 to identify the highest-quality draft Shea- This parallel selection process serves as an important safeguard, ensuring that when a particular agent produces an exceptionally strong summary or quiz, it remains visible and is not overshadowed by A
Phase 4: Collective Voting In the final phase, we bring together the collective wisdom of all gen-
erator agents to make the ultimate decision between two candidates: the aggregated summary/quiz Zugy and the best individual draft bbea- Each generator agent 9j carefully evaluates both candidates from the set = {zugg- Zbeu and casts their vote for the one they believe best captures the essence of the original document. The final summary/quiz is the candidate that receives the most votes.
Algorithm 3: REVIEW(): Reviewer Collaboration
Input: Document D; Summary/Quiz z; Reviewer agents R (r:) Number of debate
rounds Tebbate
Output: Decision dec {ACCEPT, REJECT): Issue list
Phase 1: Independent Reviewing:
for each reviewer R
L AG + ANNOTATE(T,, D);
Review annotate the summary/quiz
Phase 2: Issue Categorization
M + {a e A; and a 2}:
// Agreed issues // Contested issues Initislize valid issues with evideace from and Debate validicy
+ {a a E A; and ii: A:)l < 2):
Phase 3: Contested Issue Validation via Debate
KC + 0
for each contested issue e do
10 for debate round = 1 to Taebute do
11
for each reviewer T; ARGUE(r,,c, 2)
13
vote. MAJORITYVOTF((RCC vote, = VALID then
// Vote whether fa valid
14
L K. (c):
Keep velid contested issue
Phase 4: Final Decision
17 t- MUK; // Combine major issues and kept contested issues into the Sinal issue list return I;
// Return 1at
5

---

3.3 REVIEWER COLLABORATION
The reviewer collaboration in SUMMQ takes systematic approach to quality assessment, where multiple reviewer agents work together to thoroughly evaluate generated summaries/quizzes and catch potential errors. As shown in Algorithm 3, this reviewing process also includes four phases:
Phase 1: Independent Reviewing The reviewing process begins with each reviewer agent r; conducting an independent and comprehensive review the generated summary/quiz 2 against the original document D. During this phase, each reviewer meticulously examines the summary/quiz to identify various types of quality issues, including factual errors, omissions of important information, redundant content, and other potential problems. The reviewers produce individual annotation sets A that capture their unique perspectives and assessment criteria.
Phase 2: Issue Categorization Once all reviewers have completed their independent reviews, the system categorizes the identified issues based the level of agreement among reviewers. Issues that are flagged by at least two reviewers are classified as agreed issues M, indicating a strong consensus that these problems genuinely exist. Conversely, issues identified by fewer than two reviewers are categorized as contested issues C. suggesting disagreements that require further discussion.
For contested issues in C, where initial reviewer agreement is lacking, SUMMQ employs structured debate mechanism determine their validity. Each contested issue e undergoes Taebate rounds of debate, where all reviewer agents E R engage evidence-based argumentation using the original document and summary or quiz z as supporting materials. During each debate round, reviewers present their reasoning for against the validity issue e. After the debate rounds conclude, all reviewers participate in a majority vote to determine whether the contested issue should be considered valid.
Phase 3: Contested Issue Validation via Debate
Phase 4: Final Decision In the final phase, the reviewer collaboration reaches its final decision by consolidating all validated issues. The system combines the agreed issues M from Phase with the valid contested issues IC from Phase 3 to form the comprehensive final issue list I -M U K. This issue list is returned to guide subsequent iterations of the generation process, ensuring that identified problems are systematically addressed in future revisions.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Evaluation We evaluate SUMMQ on the long document summarization task using the MENSA
Saxena & Keller 2024), BookSum (Kryscinski et al., (022), and GovReport (Huang al. 2021)
benchmarks. Following standard protocols, wC assess the generated summaries using ROUGE (R-
1), ROUGE-2 (R- 2). ROUGE-L (R-L) (Lin 2004), and BERTScore- (BSF,) (Zhang al 2020
We also employ LLM LLM-as-a-Jull Judeee evaluations with GPT-4. and GPT-5 (detailed in Appendix A), and conduct human evaluations. A case study is presented in Appendix F
Baselines We compare SUMMQ against strong baselines from three categories: (1) Supervised
Fine- Tine-Tunigg: TEXTRANK (Mihalcea & Tarau 2004 Jeong et al.) 2025), LONGT5 (Guo et al.
2021 U.FORMER (Bertsch et al.] (023), SLED (Ivgi et al. 2023), and CACHED (Saxena al.
2025
(2) Prompting: Proprictary LLMs including GPT-40, GPT-4. GPT-5, 03, and open-
source models such DEEPSEEK-RI and QWEN3-32B. (3) Multi-Agent Systems: HM-SR
[Jeon et al. 2025) and C.MULTILLM (Fang et al. 2024).
Ours In our experiments, we consider two configurations of our approach: SUMMQ where each component employs single agent for efficient and straightforward deployment, and SUMMQcoMBo. where each component leverages multiple agents in an ensemble manner to facili- tate collaborative generation and review. By default, we use GPT-40 as the agent backbone for both
BeETTScore model: bert berttbaae-ae -base-uncased.

---

Table I: Overall results given by different methods on MENS?, BookSum, and GovReport. The best results are highlighted in bold.
MENSA
BookSum
GovReport
R-1
R-2
R-L
BSF
R-1
R-2
BSF
R-1
R-2
BSp
Supervised Fine- Tinennuning
TEXTRANK
34.37
4.60
12.84
48.10
LONGTS
20.77
2.26
10.03
45.01
U.FORMER
36.70
7.30
15.50
51.50
56.60
26.30
27.60
68.20
SLED
38.90
7.50
15.80
52.40
57.50
26.30
27.40
66.90
CACHED
42.80
10,50
18.80
54.40
57.00
26.30
28.19
67.30
Prompting
GPT-40
25.78
7.24
13.59
59.67
23.02
1.81
12.23
58.52
31.42
11.87
17.61
63.43
GPT-4.1
30.31
8.36
15.39
55.01
23.05
5.54
11.47
58.12
40.84
12.96
19.13
62.95
GPT-5
37.38
9.14
17.11
60.44
23.98
5.69
12.38
58.55
41.52
12.52
19.23
62.55
03
32.84
8.54
17.09
59.27
22.00
5.24
11.51
58.44
38.28
9.93
17.47
61.19
DEEFŠEEK-RY R1
27.63
7.66
14.82
56.82
18.86
4.69
9.71
55.85
35.42
9.58
16.66
61.07
QWEN3-32B
23.49
5.58
12.77
55.76
20.19
4.68
10.68
56.51
35.52
10.80
17.07
61.08
Multi-Agent Systems
HM-SR
34.26
9.74
13.46
60.22
C.MULTILLM
47.90
19.70
SUMMQ saLo
39.30
9.70
17.12
61.84
33.33
8.35
15.47
60.41
48.71
17.26
21.21
65.21
SUMMQcOMBa
41.58
10.96
18.24
62.76
44.62
11.14
20.38
61.49
52.79
18.47
21.76
65.46
iter.
77.0%
13.0% 10.0%
iter. 1
66.
33.0%
16
iter.
78.0%
11.0%11.0%
2
0%
33. 03.0%
iter.
80.0%
11.0% 0%
Iter. 3
74. 74.0%
26.05
SummQ
Baseline wirs
(a) SUMMQUOMER
GPT-40
(b) SUMMQ COMBO 03
Figure 2: The comparison between SUMM and baselines judged by GPT-5 on MENSA during iteration, where there are three GPT-40 agents in each component of SUMMQcOMBO
SUMMQSOLO and SUMMQunMBOO deploying three agents in each component for SUMMQ un- less otherwise specified. Moreover, the number of iterations Titer is set to three for all confi configurations and the number of debate rounds Techaro is set one. Prompts used in SUMMQ are in Appendix G
4.2 AUTOMATIC EVALUATION RESULTS
SUMMQcoMmp achieves strong performance with notable improvements on challenging datasets. Table Trrports the automatic evaluation results all methods on MENSA, BookSum, and GovReport benchmarks. Our SUMMQcoMBO configuration demonstrates strong performance across all datasets, achieving the best results on MENSA and BookSum across all metrics. On GovReport, while some supervised fine-tuning baselines (U.FORMER, SLED, CACHED) achieve competitive or superior performance on specific metrics due to their large-scale task-specific train- ing, SUMMQcoMMO still outperforms all prompting baselines and shows substantial improvements over the SUMMQsora variant. Notably, SUMMQcoMBO yields the most significant improvements on the challenging BookSuu dataset, where it substantially outperforms all baselines across all metrics. Furthermore, SUMMQ configuration also performs strongly, consistently surpassing prompting baselines. These results confirm the advantages of our proposed multi-agent summarization frame- work, particularly the SUMM OCoMEO configuration, in handling diverse long documents.
Figure 2 presents LLM-as LLM-as-a-Judge -uddge results, comparing SUMMQcoMBO strong baselines the MENSA benchmark over multiple iterations. LLM judges (GPT-5) compare summary pairs and select the superior one, with each subfigure showing the winning rate of SUMM against different L.LM agents
Judeee evaluation highlights the effectiveness of

---

Table 2: Results on MENSA obtained by SUMMQ, where component contains multiple agents while other components contain only single agent.
86.0%
12.0%
Summo GPT4o
SummQuase Vs.
65.0%
15.0%
20.09
R-1
R-2
R-L.
BSp
BSR
BSF,
Summ Summ@Qanie Qembel Vs. GPT4e
60.0%
20.0%
SUMM SSoLo
39.30
9.70 17.12 62.19 61.55 61.84
Sum
SUMMQcoser
41.58
11.08 18.24 63.28 62.28
62.76
SummQ wins
Baseline wins
Tie
Only one component with agents
Quiz Rev.
40.81
10.63 17.78 62.17 61.61 61.87
Figure 3: Human evaluation results com- paring GPT-4o, 03, SUMM and SUMMQcOMBOR-
Summary Rev.
41.20 10.80 17.93 62.29 61.72 61.99
Quiz Gen.
40.40
10.80 17.95 62.54
61.72 62.11
Summary Gen.
40.72
10.96
18.07 62.70 62.39 62.53
Table 3: Results with different number of iter- ations The on MENSA with the SUMMQcOMBO- All agents are GPT-40.
Table 4: Results with different number of agents each component on MENSA with the SUMMQCOMED. All agents are GPT-40.
#iter.
R-1
R-2
R-L
BSp
BSR
BS
#agents
R-1
R-2
R-L
BSp
BSR
BS
38.14
10.44
17.85
62.77 61.50 62.60
39.30
9.70
17.12 62.19 61.55 61.84
2
40.55
10.74
17.80 63.06 61.85 62.43
40.49
10.20
18.02 62.23 62.11 62.16
3
41.58
11.08
18.24 63.28 62.28 62.76
41.58
11.08
18.24 63.28 62.28 62.76
4
41.62
11.19
18.11 63.26
62.25 62.73
41.81
10.94
18.30
62.82 62.46 62.34
41.53
11.01
18.21
62.90
62.23 62.55
42.52
11.49
18.56
63.53
62.99 62.96
(GPT-40 or 03). judged by GPT-5. Across all settings, SUMMQcoMBO consistently outperforms baselines, achieving higher winning rates and demonstrating the effectiveness and generalizability of SUMMQcOMEO- The specific prompts and additional results judged by GPT-4. are Appendix A.
4.3 HUMAN EVALUATION RESULTS
Setup We conduct human evaluation using 20 randomly selected NLP papers published after June 2024, with five Ph.D. students as judges. Each judge compares two summaries considering Informativeness, Coherence, and Factuality. The performance is measured by the winning rate. To address length bias, we include both SUMMQcoMBO and rephrased version SUMMQcOMBOR with shortened summaries. Details on the evaluation protocol and selected papers Anpppedix E
presents the results of our human evaluation, comparing SUMM and SUmMQcoMBOR against strong basclines, including GPT-40 and 03. The results indicate that SUMMQcoMBB is preferred over GPT-40 and 03 with winning rates of 88% and 82%, respectively. Even after mitigating the potential length bias through rephrasing. SUMMQcOMBOR still outperforms GPT-40 and 03 with winning rates of 65% and 60%, respectively. These findings underscore the effectiveness of our collaborative multi-agent framework in generating high-quality summaries that are favored by human judges, even when accounting for differences in summary length.
Results Figure 3
5 ANALYSIS
Multi-agent collaboration consistently excels for each component in SUMMQ. We analyze each component of SUMMQ by replacing the single-agent setup with a multi-agent ensemble while keeping the other components as single-agent. As shown in Table 2 all components benefit from multi-agent collaboration, especially the Summary Generators and Summary Reviewers. These re- sults highlight that collaboration and diverse perspectives significantly improve summary quality.
More iterations of refinement does not always lead to better summaries. We analyze the effect of varying the number of iterations The in SUMMQ AAgoriitmm 1) summary quality for MENSA Table 3). Performance generally improves from to 3 iterations, with BERTScore-Fi peaking at 62.76. 76, but further iterations yield diminishing negative returns. This suggests that too few
8

---

Table 5: Results given by SUMMQcosIO with different LLMs as agent backbone on MENSA.
Proprictary Models
Open-source Models
R-1
R-2
R-L
BSp
BSa
BSA
R-1
R-2
R-L
BSp
BS
BSp,
GPT-4O-MINI
DEEPSEEK-
baseline
26.61
6.56
13.77
57.66
55.87
58.15
baseline
27.63
7.66
14.82
54.98
58.86
56.82 60.30
SUMMQ
35.18
7.97
15.67
58.02
58.64
58.31
SUMMQ 30.71
7.77
15.04
61.99
58.77
GPT-4.1
QWEN3-32B
baseline 30.31
8.36
15.39
56.00
54.12
55.01
baseline
23.49
5.58
12.77
55.86
55.99
55.76
SUMMQ 49.17
12.25
18.98
59.58
62.42
60.95
SUMMQ 26.50
5.80
13.14
57.13
59.09
58.02
03
baseline
32.84
8.54
17.09
57.38
59.81
59.27
baseline 26.66
6.35
13.77
58.83
55.57
57.07
SUMMQ
46.69
10.37
19.09
61.90
61.30
60.82
SUMMQ 31.07
6.99
14.70
58.80
57.15
57.83
iterations fail fully leverage collaborative reasoning, while too many can introduce noise or over- refinement, indicating an optimal balance is needed.
More agents lead to better performance, but with diminishing returns and increased cost. As shown in Table 4| increasing the number of agents in each component of SUMMQcOMBO generally improves performance, but gains become smaller as more agents are added. For example, ROUGE-L rises from 12.12 (one agent) to 18.07 18.02 (two agents), but further increases yield only minor improve- ments. This highlights trade-off between summarization quality and managing computational cost, as adding agents increases costs without proportional benefits.
SUMMQ consistently achieves superior performance with diverse LLM agent backbones.
Ta-
ble 5 shows that SUMMQcoMBO outperforms all baselines across various proprietary and open-source
LLM backbones. Summarization quality strongly depends on the agent backbone: advanced models
like GPT-4.1 and 03 perform better than their weaker counterparts. This highlights the importance of choosing robust LLMs to maximize multi-agent summarization performance. We also explore the impact of combining different LLMs within SUMMQ in Appendix
Quizzing mechanism effectively improves the summarization quality. To evaluate the contribution of the quizzing mechanism in
Table 6: Results with and without quiz on MENSA.
SUMMQ, we ablate the quizzing mechanism in SUMMQ by removing generators and re- viewers for quizzing and the examinee. As shown in Table 6] this leads to consistent per- formance drops across all metrics MENSA. For SUMMQ R-1 drops by 5.00 and BS₁; by 6.40; for SUMMQcoMMo R-1 and BSF, de- crease by 2.09 and 2. 17, respectively. These results show that the quizzing mechanism is crucial for enhancing summarization quality by comprehensively challenging the generated summaries.
R-2
R-L
BS BS BS
SUMMQ
39.30 9.70 17.12 62. 61.55 61.84
w/o quiz
34.30
8.99
15.85 59.71 51.80 55.44
SUMMQcoMBO
41.58 11.08 18.24 63.28 62.28 62.76
w/o quiz
39.49 9.33 17.13 61.10 60.39 60.59
Quiz coverage gets more balanced across document seg- ments as the iteration proceeds. We divide each document into five equal segments and use GPT-4. to map quiz ques- tions to segments. As shown in Figure 4 quiz questions ini- tially focus on the beginning and end, which aligns well with the findings Liu et al. (2024), but coverage becomes more balanced by iteration 3, with increased attention to middle segments. This shift indicates more holistic document un- derstanding. which crucial for generating summaries that accurately reflect the entire content.
Percentage
0.1
0.0
5eg.1
S0g.2
500
seg
seg.5
her.1
Figure 4: Quiz question distribution MENSA with the SUMMQ

---

CONCLUSION
In this work, we introduce SUMMQ, a novel adversarial multi-agent framework that addresses criti- cal challenges long document summarization through collaborative intelligence between special- ized summarization and quizzing agents. Our approach creates natural adversarial dynamic where quiz generation serves as continuous quality check, ensuring comprehensive coverage, factual ac- curacy, and verifiability of summaries through iterative refinement. Extensive experiments three benchmarks demonstrate that SUMMQ achieves superior performance across multiple evaluation metrics including ROUGE, BERTScore, LLMM-as-a-judg,, and human assessments. Our compre- hensive analyses reveal the effectiveness of multi-agent collaboration, the impact of the quizzing mechanism summary quality, and the influence of various agent configurations.
ETHICS STATEMENT
This work introduces SUMMQ. an adversarial multi-agent framework for long document summa- rization using large language models (LLMs). All experiments were conducted using publicly avail- able datasets and LLMs, strictly adhering to their respective licenses and usage policies. No human subjects were involved in the development or evaluation of the system, except for the human evalu- ation, which performed by consenting Ph.D. students with relevant expertise. We acknowledge that LLMs and datasets may contain inherent biases, which could affect the generated summaries and quiz questions. We encourage responsible use of our framework, with attention to fairness, transparency, and accountability in downstream applications.
REPRODUCIBILITY STATEMENT
We are committed to reproducibility in this work. Detailed descriptions of the SUMMQ frame- work, including algorithms, agent configurations, and collaboration mechanisms, are provided in Section 3] Experimental setups, including model backbones, datasets, evaluation metrics, and base- line comparisons, are thoroughly described in (nectiinn 4 All datasets used are standard benchmarks. and references included for accessibility. Prompts and implementation details provided in Appendix. To further support reproducibility, we will release our code and experiment scripts upon publication, enabling other researchers replicate and extend results.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
In preparing this work, we utilize large language models (LLMs) as general-purpose tools assist with writing polish and grammar correction. The LLMs are not involved in research ideation, exper- imental design, substantive content generation. Their role is limited improving the clarity and readability of the text, ensuring grammatical accuracy, and refining the presentation of our findings. All scientific contributions, analyses, and conclusions are solely the work of the authors.
ACKNOWLEDGEMENT
This project has received funding from UK Research and Innovation (UKRI) under the UK govern- ment's Horizon Europe funding guarantee [grant numbers 10039436).
The computations described in this research were performed using the Baskerville Tier 2 HPC ser- vice http://wwwwbbbkkeiille..e.. the World Class Labs scheme (EP/TO02221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Research Computing the University of Birming- ham.
Baskerville was funded by the EPSRC and UKRI through
REFERENCES
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long document transformer. CoRR, abs/2004.05150, 2020. URLhtttp
05150
10

---

Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unliniformer: Long-range range transformers with unlimited length input. Advances in Neural Information Processing Systems, (6.35522-335443. 2023.
Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. Multi-agent consensus seeking via large language models. CoRR, abs/2310.20151, 2023a. doi: 10.4550/ARXI.2300200511 URL
https://doi wwg/oo.............o...o........ 48550/
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), ACL. 2024, Bangkok, Thailand, August / 11-16, 2024, pp. 7066-7085, Association for Computational Linguistics, 2024, doi: 10. 10.18653/V1/2024. ACL-LONG.381. URL httthhtts: http://doo.ooggg/// 186.88663333 22244 acl 1ong.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306. 2023b. doi: 10. 48550/ARXIV.2366.1 15595. URL
George Chrysostomou, Zhixue Zhao, Miles Williams, and Nikolaos Aletras. Investigating halluci- nations in pruned large language models for abstractive summarization. Transactions of the As- sociation for Computational Linguistics, 12:1163- 2024. doi: 10.1162/ URL
Peng Cui and Hu. Sliding selector network with dynamic memory for extractive summarization of long documents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 5881-5891, 2021.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024. URL /
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluis Märquez (eds.), Proceedings of the 57th Anmual Meeting of the Associ- ation for Computational Linguistics, pp. 1074- 1084, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 18653 1/199/11//191100 URL ftttp::/aaclantee ogy.
org/P19-11027
Jiangnan Fang, Cheng- Tse Liu. Jieun Kim, Yash Bhedaru, Ethan Liu, Nikhil Singh, Nedim Lipka, Puneet Mathur, Nesreen K Ahmed, Franck Dernoncourt, et al. Multi-llm text summarization. arXiv preprint arXhv:2412.15487 2024.
Mahak Gambhir and Vishal Gupta. Recent automatic text summarization techniques: survey. Artif. Intell. Rev,, 47(1) 2017. doi: 10. 10.1007/510462-016-99777 10. 1007/00/ 00007/s104550 016 9475-1 9
URL httpss://ot org/
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209. /2356, 2022.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint
arXiv:21 2021.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference Artificial Intel- ligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 8048-8057. ijcai.org, 2024, URL https: www. wwt.:aaar.............//.....ooo httg://ooceeiigg////O0o0

---

Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi- agent-based code generation with iterative testing and optimisation. CoRR, abs1231..11000 2023. doi: 10.45500ARRXV.2312.11100 URL htLhttts::/ /ooii w2t.10..45500//r.1111....
13010
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/022/ htwp://///......lllee.a.aa naacl-main. 112. URL https: //acl ac antho org www.0....aaall... naacc ma in..in.... 112/
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text
models. Transactions of the Association for Computational Lingwistics, 11:284-299, 2023.
Yeonseok Jeong, Minsoo Kim, Seung-wor won Hwang, and Byung-Hak Kim. Agent-as-judge for factual summarization of long narratives. arXiv preprint arXiv:2501.0999) 2025.
Gunjan Keswani, Wani Bisen, Hirkani Padwad, Yash Wankhedkar, Sudhanshu Pandey, and Ayushi Soni. Abstractive long text summarization using large language models. International Journal of Intelligent Systems and Applications Engineering, 12(12s):
(003)))6:-1110 110166066666 168, 2024.
Hyuntak Kim and Byung-Hak Kim. NexusSum: Hierarchical LLM agents for long-form narrative summarization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 00120-10157, Vienna, Austria. July 2025. Association for Computational Linguistics. ISBN 879---89176-251-11 doi: 10. N0.18863/v1//000 aet22225.cclloooo acI-long.500. wwwphhhoooooo.rg////////
URL
ac 1onollon...0000
Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey on long document sum- marization: Datasets, models, and metrics. ACM Comput. Surv, 55:8):154:11154:33 2023. doi: 10.1145/3545176.
Wojeiech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Yoav Gold- berg, Zornitsa Kozareva, and Yue Zhang (eds.), Finulings of the Association for Compulational Linguistics: EMNLP 2022. pp. 6536-6558, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 18.18//3/v 1 1/2022. h/20022.inningssonn... findings-cmmlp.488.. URL https:://cclatthi cenethoooggg
2022. Fiindingg-eee 1 p. 488/
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9662 9676, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023. emnlp-main.600 6000. URL https:ttp.....
aataoaaoohoyy org 202.3 emn 1 p ma i n. 600/
Chin-Yew, Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: http:///aaaaattoocl..cooo
1013/]
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1/6//tacl a 00638. URL https://ach h
antho wgw.oog/..22....cc./gg httpas//cccaatttooooooooo///////
12

---

Ye Liu, Jian-Gus Guo Zhang, Yao Wan, Congying Xia, Lifang He, and Philip S. Yu. HETFORMER:
heterogeneous transformer with sparse attention for long-text extractive summarization. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 146-154. Association for Computational Linguisties, 2021, doi: 10. 10.18653/V
URL https:
7/doi. //doi.org/10. 10. 10w.11.11111111111
emnlp main.13
Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pp. 404-411, 20044
Bo Pang, Erik Nijkamp, Wojeiech Kryscinski, Silvio Savarese, Yingbo Zhou, and Caiming Xiong- Long document summarization with top-down and bottom-up inference. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 1237-1254, Association for Computational Lin- guistics, 2023. doi:
EACL.944 URL UtL.https:///o..crg/
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 2021.
Xiao Pu, Mingqi Gao, and Xiaojun Wan. arXiv:2309. 095558988. 2023.
Summarization is (almost) dead. arXiv preprint
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Anna Rogers, Jordan Boyd-Graben, and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 6383-6402. Asso- ciation for Computational Linguistics, 2023. doi: 1.1865NN1203AAL-LLNN...3 URL httpss://doi.... http:////.....ar......./////////
352
Swamnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch- solve-merge improves large language model evaluation and generation. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume I: Long Papers), NAACL, 2024, Mexico City, Mexico, June 16-21, 2024, pp. 8352- 8370. Association for Computational 462. (tth::t::// //oii www./o.......... .02.0.188655 18651886333313 h0t.///////222.2.. naacl -ong-oon......... 462
Linginaaiiiniiiiiii Linguistics, 2024. doi:
Rohit Saxena and Frank Keller. Select and summarize: Scene saliency for movie script summa- rization. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3439-3455, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2024. findings-naaclln.
URL
/tttps://acllnnthlloog...o 2024, nnd finiiiggnnniiinnnnnnnn---n- naacl 218/
Rohit Saxena, Hao Tang, and Frank Keller. End-to-end long document summarization using gradient caching. arXiv preprint arXiv:2501.01805, 2025.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generatgr networks. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18653/v1/P17-1100 URL wwt.h:ttts/.......t....t huu_apppp///elaattoooooooooo////
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024. doi: 2023 127063
URL htthhttpss... :/tp:i///oooo//// ott.:o..gg11..010 neucom.
13

---

Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu'an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKe- own. TofuEval: Evaluating hallucinations of LLMs on topic-focused dialogue summarization. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 4455-4480, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v http://..aac.--.//eaaaa https: halplaccaannhooggggo org/ 2024 www.nnaallllaoo 251/
URL
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. -99-621- Association for Computational Linguistics. 2024b. doi: 10. 16553///////2 FINDINGS-ACL.33 URL httpt::::///ooo oro.orgg////o /10. . 8 06336533vv2 / /2024 f nd ngg acl
33
Bing Wang, Changyu Ren, Jian Yang. Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan. Qian- Wen Zhang. Di Yin, Xing Sun, and Zhoujun Li. MAC-SQL: A multi-agent collaborative framework for text-to-sql. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al- Khalifa, Barbara Di Eugenio, and Steven Schoekaert (eds.), Proceedings of the 31st Interna- tional Conference Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 540-557, Association for Computational Linguistics, 2025a. URL https: //aclantholoog.orq/// http:///22....ccoooooee...oo main.. 36/
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):183455, 2024.
Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. Odysseybench: Evaluating Ilm agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09122 2025b.
Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. CoRR, abs/2405. 1 1804, 2024. doi: 00-48550/ARXX..2405.11 1804. URL htttps://ddte / /ttp:////ooo.o/..../ 18550/arXis 2405.11804
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121 101, 2025.
Yu Xia, Xu Liu, Tong Yu, Sungchul Kim, Ryan Rossi, Anup Rao, Tung Mai, and Shuai Li. Hal- lucination diversity-aware active learning for text summarization. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp.
Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.1106...0 1/2254311/224.maaa-l haw.2244.aal--ooo.44. URL htaps:://acaatthoooggoooo org/ 2024. nwac long. .799
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics; EMNLP 2023, Singapore, December 6-10, 2023, pp. 7572-7590. Association for Computa- tional Linguistics, 2023. doi:
/AL.///22...NIIINNSEEEEEEEP.... URL https:
7/doi. orooi.org/100 10. 1.e.186555 18653/v1 /20/l ii ngs- 508
Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. To- wards reasoning in large language models via multi-agent peer review collaboration. CoRR, abs/2311.08152. 2023. doi: 10.4850/ARXI.2311.000 1.08152. URL https: :topps///d.. http///..........////
48550/ 2311 08152
14

---

Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). ACL 2024, Bangkok, Thailand, August 1 11-1,, 2024, pp. 2588-2611 Association for Computational Linguistics, 2024. doi: 10.
142. Utthtttss: //cp::://doi.. wttg///..rg//...
18653/v1/2020 acl long.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In 8th International Conference on Learning Representa- lions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL.
httts::/oppeneev ew net/ noet/forumm.ddsSeeFfCCCCC
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. Summ n: A multi-stage summariza- tion framework for long input dialogues and documents. arXiv preprint arXiv:21 10. 10150, 2021.
Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. Longagent: Scaling language models to 128k context through ageet collaboration. CoRR, abs/2402.11550, 2024. doi: 1048850/ARXV.2400...1 11500. URL sttpps:/ddos org/10.
44850/arxiv.24000 211.HV....02.11555..
Yao Zhao, Mohammad Saleh, and Peter J Liu. Seal: Segment-wise extractive-abssrratiie long-form text summarization. arXiv preprint arXiv:2006.10215 2020.
15

---

iter.
68.0%
22.0%
10.0%
iter. 1
73.0%
24.0%
3,0%
iter.
73.0%
17.0%
10.0%
iter. 2
74.0%
26.0%
Iter.
79.0%
11.0%10.00
iter. 3
77.0%
23.0%
SummQ wins
Baseline wins
Tie
SummQ wins
Baseline wins
Tie
(a) SUMMQcoMBO GPT-40
(b) SUMMQcoamo Vs. 03
The comparison between SUMMQCOMBO and baselines on MENSA judged by GPT-4.1 during iteration, where there three GPT-40 agents in each component of SUMMQcoMBO-
Figure 5:
Table 7: Results with different combinations of LLMs in each component on MENSA with the SUMMQcoMOo- / indicates the LLM is used.
GPT-4o GPT-4.1 03
GPT-5
R-1
R-2
R-L
BSp
BSR
BS
41.58
11.08
18.24
63.28
62.28
62.76
48.82
13.79
21.46
63.32
64.66
63.97
/
48.06
13.17
21.44
62.63
64.25
63.42
49.42
13.26
22.90
63.71
65.17
64.42
A LLM-AS-A-JUDGE EVALUATION
A.1 LLM-AS-A-JUDGE SETUP
We employ an LLM-as-a-Judge evaluation framework to assess the relative quality of generated summaries. For each document, we compare pairs of summaries using the evaluation prompt shown below. To mitigate positional bias, we reverse the order of summaries for each pair, ensuring that each comparison is evaluated twice with alternating positions. The judge is instructed to determine which summary better meets the evaluation criteria whether they are of equal quality.
LLMeas-a-Judge I: Evaluation Prompt
Sys PROMPT:
You are expert evaluator tasked with objectively assessing the quality of text summariza- tions.
Your response must strictly follow this format:
Reasoning: [Brief, precise explanation based on the criteria above.]
Better one Equal: [Summary Summary Equal]
USER PROMPT:
Evaluate the following document and two summaries provided below. Determine which summary better meets the evaluation criteria provided, or whether they are equal. Document: "(text]"
Summary 1: "(summary 13"
Summary 2: "( "(summary 2)"
A.2 ADDITIONAL. RESULTS USING GPT-4.1 AS THE JUDGE
We provide additional LLM-as-a-judge results judged by GPT-4.1 in Figure complementing the GPT-5 results in Figure 2 These results show that SUMMQ consistently outperforms all baselines across iterations, further validating the effectiveness of our method.
16

---

B COMBINATION OF DIFFERENT LLMS
Diverse agent combinations leverage complementarr strengths. We have demonstrated the ef- fectiveness of SUMMQcomBO with multiple GPT-40 agents, but we also explore the impact of com- bining different LLMs within our framework. Table presents results for various strategies that mix agent types, including GPT-40, GPT-4.1, 03, and GPT-5. The results show that ensembles com- posed of diverse LLMs generally outperform those using a single model type. These combinations leverage complementary strengths, such as distinct reasoning capabilities, knowledge domains, or summarization styles. For instance, the pairing of 03 and GPT-5 achieves the highest scores, likely due to 03's 's advanced reasoning and GPT-5's robust routing and selection abilities. This diversity enables the system to generate more comprehensive and higher-quality summaries, as agents can mitigate each other's weaknesses and collectively address broader range of content and quality challenges.
C ACCURACY OF QUESTIONS EVOLVE
Quiz answer accuracy improves throughout iteration. As the iteration proceeds, the accuracy answering quiz questions based on the generated summaries improves steadily. Figure 6 illustrates that all the question types ex- hibit this upward trend: multiple-choice (MC) and true-false (TF) questions achieve higher accuracy more rapidly, while short-answer (SA) questions, which demand deeper compre- hension, show a more gradual improvement. This pattern un- derscores that agentic collaboration and iterative refinement in SUMMQ enhance summary quality, as reflected in the im- proved performance increasingly complex quiz questions.
1.0
0.8
0.6
0.4
0.2
0.0
MC
TF
SA
Iter1
Rer.2
te:3
D COST ANALYSIS
Figure 6: Accuracy of multiple- choice (MC), True-False (TF), and Short-Answer (SA) questions evolve during iteration on MENSA with the SUMM
The cost analysis in Table 8 reveals significant differences in computational resource requirements across different ap- proaches. While simple prompting baseline maintains the lowest cost at $0.15 18 per instance with minimal input and out- put token usage, SUMM Q demonstrates a moderate increase in resource consumption, requiring $1.95 per instance with 0,43M input tokens and 6.97K output tokens. This reflects the additional computational overhead of our iterative refinement process compared to baseline approaches. The SUMMQ COMBO variant shows the highest resource requirements at $14.45 per instance, consuming 3.30M input tokens and generating 24.96K output tokens, which is attributable to the collaborative multi-agent framework involving multiple summary and quiz generators, reviewers, and the iterative debate process. Despite the higher computational cost, the substantial improvements in summary quality and quiz generation accuracy demonstrated throughout our evaluation justify this invest- ment, particularly for applications where high-quality outputs are prioritized over computational efficiency.
Table 8: Average token usage and cost (in USD) per example of different methods on MENSA with GPT-40.
Input Tokens
Output Tokens
Cost (USD)
Prompting
0.04M
0.23K
0.18
SUMMQ
0.43M
6.97K
1.95
SUMMQcoNRO
3.30M
24.96K
14.45
17

---

arXiv ID
Title
2410.07095
MLE-bench: Evaluating machine learning agents machine learning engineering
Al Safety should prioritize the Future Work
1410.15522
M RewardBench: Evaluating Reward Models in Multilingual Settings
17557
The Fin Datasets: Decanting the Web for the Finest Text Data at Scale
2411 2411.01493
Sample-Efficient Alignment for LLMs
2407.1900
Benchmarking Language Agents across Multiple Applications for Office Automation Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM''s Reason- ing Capability
2411.19943
2412.03679
Evaluating Language Models as Synthetic Data Generators
2412.03555
PaliGemma 2: A Family of Versatile VLMs for Transfer
2412.09871
Byte Latent Transformer: Patches Scale Better Than Tokens
2412.06555
ProcessBench: Identifying Process Errors in Mathematical Reasoning
2412. 14161
TheAgenCoomppny: Benchmarking LLM Agents on Consequential Real World Tasks
2406.06144 44
Language Models Resist Alignment: Evidence From Data Compression
2410. 12883
Scaling Laws for Multilingual Language Models
2410.04840
Strong Model Collapse
2406,15480
On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion
2410.08964
Language Imbalance Driven Rewarding for Multilingual Self improving
2411.19799
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowl- edge Extracting Transferring Abilities Building Multi-lingual Ability-enhanced Large Language Models Scaling L.M Pre-training with Vocabulary Curriculum
2410.07825
2502.17910
Table List of papers used for human evaluation, including arXiv ID and title.
E HUMAN EVALUATION
We conduct a human evaluation to assess the quality of summaries generated by different methods. It is impractical for human judges read the entire long documents from unfamiliar domain, so we randomly select 20 NLP papers published after June 2024, and employ five Ph.D. students who published at least one NLP paper as judges. Each judge presented with the source document and two summaries generated by different methods, and they are asked to choose the better summary considering the following aspects, including Informativeness, Coherence, and Factuality. Based on the feedback from the judges that the summaries generated by SUMMQcoMmO (2084 words on average) are significantly longer than those from other methods (1450 words on average by 03 and 982 words average by GPT-40), we also include rephrased version of SUMMQcoMBO. denoted as SUMMQcOMBOR where we prompt GPT-40 to shorten the summary generated by SUMMQcoMBO- The rephrased summaries have an average length of 1573 words, which is more comparable to the baselines. The selected papers are presented in Table 9
F CASE STUDY
We present a case study in Table 10 comparing the summaries generated by GPT-40, 03, and SUMMQcoMBo for the movie "Good Will Hunting". The text highlighted in red indicates details that are covered by SUMMQcOSBO but missed by either GPT-40 or 03. Conversely, the text highlighted in blue represents details that included in either GPT-40 or 03 but omitted by SUMM Qcommo- This comparison illustrates how SUMM captures a broader range of important plot points and character developments, while also highlighting some specific elements that were overlooked. Overall, the case study demonstrates the strengths of our approach in generating comprehensive and detailed summaries.
18

---

Model
Summary
GPT-40
The text describes various scenes and interactions from a screenplay set in South Boston and at M.I.T. It opens with a St. Patrick's Day parade and moves into bar where four young men-Chuckie Will, Billy, and Morgan-share crude story. Will Hunting, 20- year-old with remarkable intelligence, is revcaled to working as janitor at M.I.T. and anonymously solving complex math problems on chalkboards. Professor Lambeau discovers Will's talents and arranges for him to avoid jail by meeting with therapists, including Sean Maguire, tough psychologist from South Boston. Will's 's relationship with Skylar, Harvard student, is explored, revealing his fear of intimacy and abandon- ment. Scan helps Will confront his past and emotional barriers. Will ultimately decides to pursue job at Tri-tech, but leaves a note for Sean saying he had to go See about a girl.indicating he chose to follow Skylar to California. The story ends with Will driving away, symbolizing his decision to embrace change and new opportunities.
03
Will Hunting. a 20- year-olod orphan from South Boston, spends his days drinking with blue-collar friends and nights working an MIT janitor. When he anonymously solves an impossihle math proof, Professor Gerald Lambeau tracks him down, then saves him from jail (after an assault arrest) in exchange for weekly math work and therapy. Will sabotages several therapists until he meets Lambcau's 's former roommate, Scan Maguire, community-con coollege psychologist who grew up in the same neighborhood and is still mourning his late wife. Their confrontations force Will to face the childhood abuse, class anger and fear abandonment that make him hide behind brilliance and bravado. While Lambeau lines up lucrative jobs, Will half- heartedly dates Harvard pre-med Skylar but pushes her away when she asks him to move to California. Chuckie, his best friend, chal- lenges Will not to waste his "lottery ticket." At last, a breakthrough with Sean convinces Will he deserves love and future of his own choosing. Rejecting the safe corporate of- fer, Will leaves Boston-after gifting his therapist a note that reads, "Sorry, I had to go see about a girl"-and drives west to reunite with Skylar, finally ready to risk both his talent and his heart.
SUMMQcOMBO
The screemplay "Good Will Hunting" follows Will Hunting, a young man from South Boston with a troubled past and extraordinary mathematical abilities. Will spends his time with his close-knit group of friends, including Chuckie, Morgan, and Billy, in their working-class Irish neighborhood. Despite his genius, Will works as a janitor at MIT, where he anonymously solves complex mathematical problems posted by Professor Lam- bcau, a Field's Medal winner. The story begins with scenes from the South Boston St. Patrick's Day Parade and humorous yet dark tale from Chuckic about his cousin Mikey Sullivan's 's misadventure with an injured cat. Will's life takes a significant turn after he gets into fight with Bobby Champa, who hurts a woman with bottle, and faces jail time. Professor Lambeau intervenes, offering Will a deal to avoid prison if he agrees to study mathematics and attend therapy sessions. Will reluctantly starts therapy with several psychologists but only begins open up with Sean Maguire, therapist with similar background who has faced personal hardships, including the loss of his wife to cancer and his military service. Through their sessions, Sean helps Will confront his past traumas and fcars of abandonment. Will's relationship with Skylar, a Harvard student, becomes strained as he struggles with his emotions and the idea of leaving his familiar life. Despite being offered job at the National Security Ageney, Will refuses, citing ethical concerns. He initially considers a position at Tri-Tech Laboratorics but ultimately decides to follow his heart and go after Skylar, who has moved to California. The story concludes with Will leaving his old life behind, symbolized by his friends discovering his departure and Sean finding a note from Will explaining his decision to seek future with Skylar. Chuckie gifts Will car for his 21st birthday, which Will uses drive away, leaving his friends and old life behind.
Table 10: Case study on the summaries given by GPT-40, 03, and SUMMQcOMRO about the movie "Good Will Hunting". The text in red highlights the details covered by SUMMQcOMBO but missed by either GPT-40 or 03. The text in blue highlights the details covered by either GPT-40 and 03 but missed by SUMMQcOMBB
19

---

G PROMPTS
G.1 PROMPTS OF SUMMARY GENERATORS
In this section, we present the detailed prompts used for summary generation in SUMM covering draft summary generation, refinement, aggregation, best summary selection, and voting.
Summary Generators 1: Draft Summary Generation Prompt
Sys PROMPT:
You are a helpful assistant tasked with summarizing long text. Summarize the following text concisely and accurately, ensuring that all key points are covered. The summary should be clear and coherent, avoiding unnecessary details or repetition. Use precise language and maintain the original meaning of the text.
USER PROMPT:
Original Text: "(Document""
Summary:
Summary Generators 2: Refine Summary Generation Prompt
sys PROMPT:
You are helpful assistant tasked with refining summaries. Given the original text, the initial summary. feedback from the evaluator, and feedback from quiz testing, refine the summary to better capture the key points in the original text and address any shortcomings.
USER PROMPT:
Original Text: " "(Document)"
Previous Summary: "[Summary)]"
Reviewers Feedback: "[Summary reviewers feedback)"
Quiz Testing Feedback:
The summary could not answer the following questions correctly: "[Examinee feedback))
Refined Summary:
Summary Generators 3: Summary Aggregation Prompt
Sys PROMPT:
You are an expert synthesiser. You will be given several candidate summaries of the same original text. Merge them into ONE superior summary that retains every important detail but avoids redundancy.
USER PROMPT:
Original Text: " "(Document)" Candidate Summaries: "(Candidates)"
Merged Summary:
20

---

Summary Generators 4: Best Summary Selection Prompt
Sys PROMPT:
You are expert summarisation judge. Rank the candidate summaries from best to worst according to coverage, factual accuracy and conciseness. Return the best summary.
USER PROMPT:
Original Text: " "Document)" Candidate Summaries: "[Candidates)"
Best Summary:
Summary Generators 5: Voting Prompt
Sys PROMPT:
You are an expert and strict summarization judge. Given two summaries, determine which is better according coverage, factual accuracy and conciseness. ONLY Return 1 or 2, where means the first summary is better and means the second summary is better. If both are equally good, return 2. Reply with nothing else.
uSER PROMPT:
Original Text: " "Doeument)" Candidate Summaries: "(Candidates)"
Best One (1 or 2):
G.2 PROMPTS OF QUIZ GENERATORS
In this section, we present the detailed prompts for quiz generation in SUMMQcoMBO- covering draft quiz generation, refinement, aggregation, best quiz selection, and voting.
21

---

Quiz Generators 1: Draft Quiz Generation Prompt
sy's PROMPT:
Multiple-choice questions:
Format:
1. Question?
A) Option
B) Option 2
C) Option
D) Option 4
True/False questions:
Format:
Statement. (True/False)
Short-answer question:
Format:
Question?
You are a helpful assistant tasked with generating questions from long text. Generate quiz questions clearly covering key points. Include: "(num of me)" Multiple-choice questions, "(num of tf)" True/False questions, and "(num of sa)" Short-answer question.. The Ques- tion Format is above.
USER PROMPT:
Original Text: " "(Document)"
Quiz:
Quiz Generators 2: Refine Quiz Generation Prompt
Sys PROMPT:
You are helpful assistant tasked with refining generated questions. Given the text, the initial generated questions, feedback from the evaluator, and feedback from quiz testing, refine the questions to ensure they cover important information clearly and avoid trivial or overly detailed content. Return "(num of mc)" Multiple-choice questions, "(num of tf)" True/False questions, and "(num of sa)" Short-answer question.
USER PROMPT:
Original Text: "(Document)"
Previous Quiz: "[Quiz}"
Reviewers Feedback: "(Quiz reviewers feedback)"
Quiz Testing Feedback:
The following questions could not be answered correctly based the key information: "(Examinee feedback)"
Refined Quiz:
22

---

Quiz Generators 3: Quiz Aggregation Prompt
Sy's PROMPT:
You are expert synthesiser. You will be given several candidate generated questions of the same text. Merge them into superior questions that retains every important detail but avoids redundancy with "[num of me]" Multiple-choice questions, "(num of tf) True/False questions, and "(num of sa}" Short-answe question.
USER PROMPT:
Original Text: "{ "Document)" Candidate Quiz Sets: "(Candidates)"
Merged Quiz:
Quiz Generators Best Quiz Selection Prompt
sys PROMPT:
You are an expert question generation judge. Rank the candidate questions sets from best worst according to coverage, difficulty and clarity. Return the best question set.
USER PROMPT:
Original Text: " "(Document)"
Candidate Quiz Sets: "(Candidates)"
Best Quiz:
Quiz Generators 5: Voting Prompt
Sys PROMPT:
You are an expert and strict question generation judge. Given two question sets, determine which one is better according to coverage, difficulty and clarity. ONLY Return 2, where I means the first question set is better and 2 means the second question set is better. If both are equally good, return 2. Reply with nothing else.
uSER PROMPT:
Original Text: " (Document)" Candidate Quiz Sets: "(Candidates)"
Best One (1 or 2):
G.3 PROMPTS OF SUMMARY REVIEWERS
In this section, we provide the detailed prompts for summary review in SUMMQ including summary review annotation, merging agreed issues, and dehating contested issues.
23

---

Summary Reviewers I: Annotate Summary Prompt
SyS PROMPT:
You are strict generated summary reviewer.
1. Coverage at least 90% of key facts needed to answer every quiz question appear.
2. Faithful no hallucinations contradictions.
3. Brevity I 25 % tokens of source OR I 500 words.
4. Clarity precise, coherent language.
If **aal four** points are satisfied output exactly 'PASS' and reply with nothing else. Otherwise list concrete problems.
For every problem output ONE line in the form:
-CATEGORY: short description
where CATEGORY is in (COVERAGE, FAITHFUL, BREVITY, CLARITY).
If there no problem with this category, do output this category.
USER PROMPT:
Original Text: " "(Document)"
Quiz Questions: "[questions)"
Summary to Review: "(summary)"
Feedback:
Summary Reviewers 2: Agreed Issues Merged Prompt
sys PROMPT:
You expert synthesiser. You will be given several feedback for the generated sum- mary. Merge them into ONE superior feedback that retains every important detail but avoids redundancy.
uSER PROMPT:
Original Text: " "(Document)"
Summary: "( "(Summary)"
Candidate Feedback: "(Candidates)"
Merged Feedback:
Summary Reviewers 3: Contested Issues Debate Prompt
SyS PROMPT:
You are participating in a one-turn debate about the following alleged issue in generated summary. Reply with ONE line starting with either KEEP (keep the issue) or DISCARD (discard the issue) followed by brief justification.
USER PROMPT:
Original Text: " "(Document)"
Quiz Questions: "(questions)"
Summary: " "(Summary))
Issues to Debate: "[Issues)"
Feedback:
24

---

G.4 PROMPTS OF QUIZ REVIEWERS
In this section, we provide the detailed prompts for quiz review in SUMM including quiz review annotation, merging agreed issues, and debating contested issues.
Quiz Reviewers I: Annotate Quiz Prompt
Sys PROMPT:
You are a strict question reviewer.
QUESTION-review rubric:
A. Coverage Distribution
1. Every *major* section / scene argument of the chapter is addressed.
2. No cluster: questions are spread across the beginning, middle, end.
Cognitive Depth
40 % Remember / Understand
20 % Evaluate Create
C. Format Balance
Required counts of MC, True/False, Short-answer respected.
Short-answer asks for 1-2 sentences, names, dates, or concepts.
MC: exactly options, correct; distractors plausible and non-overlapping.
True/False: clear factual statements, no double-neegttive.
D. Difficulty Gradient
30 easy, 50 % medium, 20 % hard.
Easy answer is stated explicitly.
Medium:
answer needs light inference synthesis.
Hard answer needs multi-sentence reasoning.
E. Clarity & Quality
Questions are grammatically correct, unambiguous, no trivia.
2. Each question targets *one* idea only.
3. No repeated facts across different questions.
If **al points are satisfied output exactly 'PASS' and reply with nothing else.
Otherwise list concrete problems.
For every problem output ONE line in the form:
-CATEGORY: short description
where CATEGORY is in (Coverage Distribution, Cognitive Depth, Format Balance, Diffi
culty Gradient, Clarity & Quality }.
If there is no problem with this category, do not output it.
USER PROMPT:
Original Text: " "(Document)"
Key Information: "[summary]]
Quiz Review: "(Quiz)"
Feedback:
25

---

Quiz Reviewers 2: Agreed Issues Merged Prompt
Sy's PROMPT:
You are expert synthesiser. You will be given several feedback for the generated ques- tions. Merge them into ONE superior feedback that retains every important detail but avoids redundancy.
USER PROMPT:
Original Text: " "(Document)" Quiz: "[Quiz]"" Candidate Feedback: "(Candidates))
Merged Feedback:
Quiz Reviewers 3: Contested Issues Debate Prompt
Sys PROMPT:
You are participating in a one-turn debate about the following alleged issue in the generated questions. Reply with ONE line starting with either KEEP (keep the issue) or DISCARD (discard the issue) followed by brief justification.
USER PROMPT:
Original Text: " "(Document)"
Key Information: "(Summary)"
Quiz: "[Quiz)""
Issues to Debate: "[Issues)"
Feedback:
G.5 PROMPTS OF EXAMINEE
In this section, we present the detailed prompt for the examinee module in SUMMQcoMmo. which is responsible for answering the generated quizzes based on the provided summaries.
Examinee 1: Take Quiz Prompt
sys PROMPT:
For every question below select the answer ** the required formar**;
Multiple-choice return only the correct letter (A/B/C/D).
True/False - return only the word True or False.
Short-answer return short phrase or sentence taken verbatim from the text (no commentary).
USER PROMPT:
Text: "(Summary)"
Questions: "(Quiz Questions)"
Return one answer per line in the same order.
26


Reviews:
[{"review_id": "rJziqTJ02l", "reviewer": "Reviewer_o7me", "summary": "This paper proposes SUMMQ, an adversarial multi-agent framework for long-document summarization. It introduces two complementary domains, summarization and quizzing, each involving generator and reviewer agents. A quiz-examinee loop acts as a continuous quality check to improve coverage, coherence, and factuality. The framework iteratively refines both summaries and quizzes. Experiments on MENSA, BookSum, and GovReport show strong results over several baselines, including GPT-4/5 and prior multi-agent methods.\n\nThis paper is well executed and produces strong results, but its methodological novelty is modest given the growing body of multi-agent summarization and self-evaluation work. The framework mainly integrates existing ideas in a new context. The contribution feels incremental rather than conceptually new.", "questions": "See my above weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe idea of using quizzes as a factuality and coverage signal is creative and intuitive.\n2.\tExperiments are extensive and include human and LLM-as-a-judge evaluations.\n3.\tThe paper is clearly written and supported by systematic ablations that examine agent configuration, iteration effects, and backbone models.\n4.\tThe framework shows consistent gains across datasets and metrics.", "weaknesses": "1.\tLimited novelty relative to prior multi-agent work. The generator–reviewer–debater structure is already common in recent multi-agent reasoning systems, and the quizzing mechanism mainly extends existing self-evaluation ideas rather than introducing a fundamentally new algorithmic principle.\n2.\tThe framework appears expensive to run, requiring multiple high-capacity agents and several iterations, which may limit practical use.\n3.\tThe paper lacks clear explanation of what is actually learned during the process and whether any parameter update or fine-tuning occurs.\n4.\tFactuality improvements are claimed but not evaluated with specialized metrics beyond ROUGE and BERTScore.\n5.\tThe contribution of the examinee agent is not well isolated in the ablations.\n\nMinors:\n1.\tPage 2: Remove the duplicate word in “long document document summarization.”\n2.\tPage 2: Change “systems has” to “systems have.”\n3.\tPage 4: Add a space in “setZ” → “set Z.”\n4.\tPage 4: Use plural “drafts” in “each of the individual drafts.”\n5.\tPage 7: Add “To” at the start of “To address length bias.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HgJgzcXbit", "reviewer": "Reviewer_dAYE", "summary": "This paper introduces SUMMQ, an adversarial multi agent setup for long document summarization. The idea is to make a summarizer and a quizzer work together so that the quiz keeps checking whether the summary actually contains the information from the document. On MENSA, BookSum, and GovReport, SUMMQ, especially the multi agent version, outperforms strong LLM prompting baselines and prior multi agent summarizers", "questions": "- What is the average number of model calls and tokens per document for SUMMQCOMBO on MENSA and BookSum, including quiz agents and reviewers.\n\n- You generate 30 questions per quiz, 10 per type. Did you try fewer questions. How fast does performance drop if you ask only 5 or 10 questions.\n\n- How sensitive is the system to reviewer aggressiveness. If reviewers flag too many contested issues, does the system fail to converge.\n\n- Can you share inter annotator agreement and how you controlled for length, since SUMMQ often writes longer summaries.\n\n- You show that SUMMQ works with different backbones. Did you try a cheaper generator with a stronger reviewer.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper targets a real issue in long summarization, missing content and hallucination when the document is very long.\n\n- On MENSA and BookSum, SUMMQ COMBO gets the top ROUGE and BERTScore numbers, better than GPT 4O and GPT 5 prompting and also better than earlier multi agent methods.\n\n- The evaluations are performed over the automatic and human evaluations, which is good for summarization, since ROUGE often disagrees with human preference. The human study still prefers SUMMQ even after they shorten the summaries.", "weaknesses": "- Algorithm 1 stops when both summary and quiz feedback are empty. In real LLM interactions, reviewers almost always find something, so you rarely get empty feedback. The paper limits to 3 iterations, so in practice it is just a fixed number of rounds, not a true convergent loop.\n\n- The paper combines two ideas that are already around: multi agent summarization with draft/aggregate/vote, and LLM as a judge or QA style verification.\n\n- The paper shows that adding more agents keeps improving results, but with diminishing returns. This is not the scaling pattern you want, since every new agent is an extra LLM call. There is no attempt to share context, cache intermediate decisions or use cheaper models for some agents. So the system is not cost aware.\n\n- The whole framework relies on carefully designed prompts for generator, reviewer, quiz writer, quiz reviewer and examinee. The main paper does not analyze which parts of the prompts are critical. This makes the method brittle in practice, because small prompt changes could break the debate or the voting.\n\n- The paper do show that quiz coverage becomes more balanced over iterations, but they do not analyze question difficulty, answerability or hallucinated questions, which are all common problems with LLM generated quizzes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2MVDzVfBf", "reviewer": "Reviewer_E8j5", "summary": "This paper introduces SUMMQ, an adversarial multi-agent framework for long document summarization. The core idea is interesting: use quiz generation as a way to verify and improve summary quality. The system has four types of agent groups - Summary Generators, Quiz Generators, Summary Reviewers, and Quiz Reviewers - plus an Examinee agent. These agents collaborate iteratively, with the quiz questions serving as a check on whether the summary captures all important information. The authors test their approach on three benchmarks (MENSA, BookSum, GovReport) and show improvements over existing methods using both automatic metrics and human evaluation.", "questions": "- Q1) Could you provide concrete examples showing a document excerpt, the generated quiz questions, and the corresponding summary? This would help readers understand how the quizzing mechanism works in practice and assess the quality of generated questions.\n- Q2) Have you had human experts evaluate the quality of generated quizzes - for example, assessing whether questions are well-constructed, have appropriate difficulty, and test key concepts rather than trivial details?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The quiz-based verification mechanism is a reasonable idea for checking summary completeness. The results on BookSum show substantial improvements, which is noteworthy.\n- The experimental coverage is decent - three benchmarks with multiple evaluation approaches including human judges. Adding the rephrased version (SUMMQCOMBOR) in human evaluation was sensible to address length bias.\n- The ablations in Section 5 help understand component contributions. The quiz coverage evolution analysis showing shifts from beginning/end to middle sections is mildly interesting.\n- Including all prompts in the appendix aids reproducibility.", "weaknesses": "- The computational cost is a real concern for practical deployment: SUMMQCOMBO costs $14.45 per document—about 80× simple prompting ($0.18). The paper would benefit from end-to-end inference-time reporting in Table 8 to contextualize these costs.\n- Performance varies dramatically across datasets. On BookSum, R-1 improves from 23.98 (GPT-5) to 44.62 - nearly doubling performance. However, on GovReport, SUMMQcombo (52.79) significantly underperforms supervised baselines like U.FORMER (56.60) and SLED (57.50) as shown in Table 1. The paper doesn't explain what document characteristics lead to these differences, making it unclear when this expensive approach is warranted.\n- The quizzing mechanism lacks quality validation. While Table 6 shows quizzing improves performance and Quiz Reviewers apply detailed criteria (Appendix G.4), we never see actual quiz examples or human assessment of whether questions are well-constructed and test meaningful content versus trivial details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YsCmorMIqM", "reviewer": "Reviewer_Bj2b", "summary": "The paper introduces SUMMQ, an adversarial multi-agent framework for long document summarization via multi-task summarization and quizzIng. Under the SUMMQ approach, summary generators and reviewers work collaboratively to summarize text and refine the summary respectively, and generators and reviewers create comprehension questions via which summary is evaluated.\n\nThe authors evaluate the approach on 3 long summarization benchmarks (MENSA, BookSum, GovReport) and compare it to 3 types of baselines: Supervised Fine-Tuning, Prompting, and alternative Multi-Agent approaches. The systems are evaluated against automatic metrics (Rouge, BERTScore, LLM-as-a-Judge) and human evaluation. Results demonstrate that SUMMQ strongly outperforms other techniques, including previous multi-agent methods, in most evaluation scenarios.", "questions": "A multi-agent summ eval framework relevant for the related work section: https://arxiv.org/abs/2502.08514v1", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* the paper introduces a novel multi-agent framework for long document summarization in a multi-task setup of summarization itself and quizzing\n* compared to a series of baselines including Supervised Fine-Tuning, plain Prompting and other Multi-Agent approaches, SUMMQ outperforms them in both automatic and human evaluation on three long document summarization benchmarks", "weaknesses": "* given that the framework is targeted at long document summarization, there is little to no discussion (apart from cost analysis in the Appendix) of how the proposed framework results in increased context usage, what are the limits, and how the technique can scale with context in mind (even longer documents, multi-document summarization, increasing the number of iterations / agents - what would be the effect on the context?)\n* there seems to be little to no justification of what exactly in the technique targets long document summarization. And if nothing in particular, why not include other summarization datasets into comparison?\n* automatic metrics like Rouge and BERTScore are considered outdated even for short documents. Their applicability specifically to long documents is highly questionable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)