Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Authors: Yusen Zhang
Ruoxi Sun
Yanfei Chen
Tomas Pfister
Rui Zhang
Sercan O Arik
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Long Context Tasks
Multi-agent Collaboration
LLM Agents
Primary area: natural_language_processing
TL;DR: We propose Chain-of-Agents leveraging LLMs collaboration to solve long context tasks and outperform RAG and long context LLMs.

Abstract: Addressing the challenge of effectively processing long contexts has become a critical issue for Large Language Models (LLMs). Two common strategies have emerged: 1) reducing the input length, such as retrieving relevant chunks by Retrieval-Augmented Generation (RAG), and 2) expanding the context window limit of LLMs. However, both strategies have drawbacks: input reduction has no guarantee of covering the part with needed information, while window extension struggles with focusing on the pertinent information for solving the task. To mitigate these limitations, we propose Chain-of-Agents (CoA), a novel framework that harnesses multi-agent collaboration through natural language to enable information aggregation and context reasoning across various LLMs over long-context tasks. CoA consists of multiple worker agents who sequentially communicate to handle different segmented portions of the text, followed by a manager agent who synthesizes these contributions into a coherent final output. CoA processes the entire input by interleaving reading and reasoning, and it mitigates long context focus issues by assigning each agent a short context. We perform a comprehensive evaluation of CoA on a wide range of long-context tasks in question answering, summarization, and code completion, demonstrating significant improvements by up to 10% over strong baselines of RAG, Full-Context, and multi-agent LLMs.

Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
Yusen Z Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang + Sercan Ö. Arik ff
Penn State University, Google Cloud AI Research
fyfz5488, rmz5227) edu, {ruoxis, yanfeichen, tpfister, soarik)@google.com
Abstract
Addressing the challenge of effectively processing long contexts has become a critical issue for Large Language Models (LLMs). Two common strategies have emerged: reducing the input length, such as retrieving relevant chunks by Setrieval-Augmenteé Generation (RAG), and 2) expanding the context window limit of LLMs. However, both strategies have drawbacks: input reduction has no guarantee of covering the part with needed information, while window extension struggles with focusing on the pertinent information for solving the task. To miti- gate these limitations, we propose Chain-of-Agent (CoA), a novel framework that harnesses multi-agent collaboration through natural language to enable informa- tion aggregation and context reasoning across various LLMs over long-context tasks. CoA consists of multiple worker agents who sequentially communicate handle different segmented portions of the text, followed by manager agent who synthesizes these contributions into a coherent final output. CoA processes the entire input by interleaving reading and reasoning, and it mitigates long context focus issues by assigning each agent a short context. We perform a comprehensive evaluation of CoA summarization, and code completion, demonstrating significant improvements by up 10% over strong baselines of RAG, Full-Context, and multi-agent LLMs.
wide range of long-context tasks in question answering,
Introduction
Despite their impressive performance across a wide range of scenarios, LLMs struggle with tasks that involve long contexts [8, 63, 57]. Numerous application scenarios demand extremely long contexts, such as question answering [85, 22, 691, document and dialogue summarization [25, 93, 91, 90, 12], and code completion [20, 43], where the inputs contain entire books [32, 33] and long articles [16].
To tackle the challenge with long context tasks, two major directions have been explored as shown in Table 1: input reduction and window extension. Input reduction reduces the length of the input context before feeding to downstream LLMs. Truncation approaches [1, 67] directly truncate the input. Retrieval Augmented Generation (RAG) [81] extends this direction by retrieving the most relevant chunks through embedding similarity. However, because of low retrieval accuracy, LLMs could receive incomplete context for solving the task, hurting performance. Window extension extends the context window of L.MMs via finetuning to consume the whole input [13, 44, 48). For example, Claude-3 [5] directly allows reading 200k tokens for each input. However, when the window becomes longer, LLMs struggle to focus on the needed information to solve the task, suffering from ineffective context utilization such as the "lost in the middle" issue [37, 3, 42].
Work done while the author was student researcher at Google Cloud Al Research.
1 Last authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Chunk
Hobert Aldrch...Ver Cruz, Kiss Me Deady...
Who starred in her final film role in the 1964 film directed by the man also did Vera Cruz and Kiss Me Deadly?
Source Docs
Chunk
Mary Astor..
Cg.
cameeo role as...
Robert Aldrich, an iconoclastic American film director, His notable films include "Vera Cruz" (1954), "Kiss Me Deadly' (1955)
p CU
Query
Chunk 3
No related information
Fill
Fil
Robert Aldrich, director "Vera Cruz" and "Kiss Deadly," also directed "Hush..Hussh Hush, Sweet Charlotte" (1964), ... Mary Astor appeared in cameo role as little old lady waiting to die.
Cluory: (Cuurry), Current Source: (Chunk You need to reed current source lext and summiry of préviouus source text any), and generate a include them bott...
CU2
Mary Astor, ... appeared cameo role little old lady waiting to die. Robert the director of "Vera Cruz" and "Kisa Me Deadly," also directed "Hush... Hush, Sweet Charlotte" (1964).
3
Stage Worker Agent: Segment Iw Comprehension Chain Communicatonn
CUs
been summarized. you need answeer based the summary.
source long
M
Gold Arrower y May Astor
The answer is: Mary Astor
1.00
Stage Manager Agent: Information IM Integration and Response Generation
Chain-of-Agents Communication
Input preparation of Agents
Figure I: Overview of Chain-of-Agents, a training-free, task agnostic, and highly-interpretable framework that harnesses multi-agent collaboration for long-context tasks. Blue boxes indicate communication unit CU; between worker agents W; and W++1- It consists of multiple worker agents who sequentially communicate to handle different segmented portions of the text, followed by a manager agent who synthesizes these contributions into coherent final output.
Table 1: Comparison between Chain-of Afentss and prior methods for long congecontxt tasks. Rec./Foc.:
being able to mitigate inaccurate receptive field/long context focusing issues. Read: the number of tokens model input, where is the total input length, l is the context window limit of LLMs. Inter.: the interpretability of the approach. Note that RAG is 'medium interpretable" because of the re-ranked chunks.
Category
Example Work
Rec.
Foc.
No Train
Read
Agent
Applicability
Inter:
Truncation (50)
x
d
&
Single Single
Generic Query-based
Low Medium
Input Reduction
RAG [81]
x
x
+ k
Window Extension
Position Interpolation [13]
/
n
Single Single Generic
Generie
Low Low
Long Context [S]
x
Multi-agent LLMs
Chain-of-Agents (Ours)
/
/
n
Multiple
Generic
High
Motivated by the aforementioned challenges, we propose a novel framework, Chain-sf-Agents (CoA), inspired by the way humans interleave reading and processing of long contexts under limited working memory constraints of brains [15]. The key idea of CoA is harness multi-agent communication to enable information aggregation and context reasoning capabilities across different LLMs. As shown in Figure 1, CoA contains two stages. In stage 1, a series of worker agents in charge of different chunks of long context collaborate and aggregate evidence for answering the given query. To this end, the workers read and process sequentially, each receiving the message from previous worker and transferring the useful updated information to the next. In stage 2, the manager agent receives the complete evidence from last worker agent and generates the final response.
As shown in Table 1, CoA is a training free, task agnostic, and highly interpretable framework and mitigating the long context
processing entire "receptive field" by interleaved focusing issue by assigning each agent a short context. Different from input reduction where LLMs need start processing with low receptive field over reduced inputs ("read-then-prooe then-process"), workers in CoA start to process each chunk before reading all input ("interleaved read- -radd-poesss") tackling the problems that input reduction struggles with, such as, generic summarization or counting of passages [6]. Different from context extension, CoA leverages the capability of communication rather than trying to feed many tokens into an LLM. This is more natural solution for complex tasks because we assume that each LLM has its limit and there are always complex context tasks surpassing its limit. Compared with Full-Context, CoA is also cost effective by reducing time complexity from n² to nk, where n is input tokens and A is the context limit of LLMs.

---

We conduct intensive experiments on nine datasets, including question answering, summarization, and code completion tasks with six LLMs, with PaLM 2 [4]. Gemini [67], and Claude [5] models. We compare CoA with two strong baselines chosen from input reduction and window extension approaches, respectively: (i) RAG, which uses sfate-of-the-art retriever to obtain the most relevant information to feed into the LLM and (ii) Full-Context (Vanilla), which feeds all input into the LLM until reaching the window limit. Our results show that on all nine datasets, CoA obtains significant improvement over all baselines by up to 10%. Noting that there is not enough research on multi-agent for long context tasks, we carefully create two multi-agent baselines, including a hierarchical structure and result merging approach to further demonstrate that CoA is superior among other possible multi-agent frameworks.
2
Related work
Multi-agent LLMs. Multi-agent LLMs has become a popular topic [21]. A large proportion of
works focus on social simulation [52, 53] and frameworks [35]. Based on the success of them, some works explore the game settings [39, 74, 82, 83, 47]. world wars [23], economy markets [36, 76],
recommendation systems [88], and pandemics [19]. Others advance problem solving, focusing on reasoning of short text via multi-agent debating [18, 80, 10, 66] and discussing [11, 58] for
different tasks in reasoning [18, 66], mechanics problems [49], paper review [84], knowledge graph construction [86], and code intelligence [70, 24]. Different from the above works, we improve problem-solving on long context tasks. To the best of our knowledge, the closest work LongAgent [92] utilizes tree structure to do multi-hop QA over long input context and the sibling worker agents do not communicate, while CoA utilizes a chain structure allowing the communication units to flow between workers (detailed comparison in Appendix G and F.3).
Long Context Modeling for LLMs. Input Reduction: RAG is broadly leveraged to solve long context basee tasks [81, 2]. Combined with strong retriever [79, 41, 73), LLMs are expected to handle long context questions in open domains. Previous studies have augmented L.LMs during pretraining [26, 72], finetuning [34], inference [87], or directly integrating [28, 61]. Moreover, some token-level retrieval approaches are proposed [38]. Longllmlingua [27] removes tokens from long prompt to compress long context prompt to a desired budget. Window Extension: The context windows of LLMs are getting longer and longer thanks to the development of GPUs. For instance, the context window increases from 1024 (GPT-2 [56]), 2048 (GPT-3 [7]). to 128k (GPT-4 [50]). Moreover, the newest version of 3laude-3 [5] supports 200k context windows. To save the cost of L.M training, some continual learning or finetuning approaches are proposed to extend the context window of pretrained L.MSs 55, 44, 48). For instance, position interpolation [13] modifies rotary position encoding [64] and extends the context length of LLaMA [68] to 32k. Different from the above works, CoA does not reduce the input length or extend the window length of LLMs, but rather leverages multi-agent collaboration and communication obtain the full receptive field.
Recently, a few studies have employed text chunking algorithms to divide and process lengthy text inputs. Notably, RecurrentGPT [95] adopts chain strueture generate long outputs in chunk- by-chunk manner, maintaining continuity in the storyline. WalkMaze [9] utilizes a centralized tree structure without communication between sibling agents to do single-hop QA over a long input context. Different from these works, CoA focuses on long input tasks with a chain structure with inter-sibling communication (Append G).
Complex Task Reasoning. Previous works on complex reasoning have focused on decomposing the complex question into sub-questions to solve them step-by-step. [54] decompose the questions with an unsupervised model and answer them separately with another model. Decomposed Prompting [30] leverages some predefined modules to classify each decomposed sub-question, then further decom- pose if needed. Additionally, decomposing is used for human-computer interaction [77]. and prompter training [71]. Recently, many work has been proposed for LLMs, such as Chain-of -fouuht [75] most mossttto-mott/t prompting [94] and Pearl [65]. However, the length of the prompt does not exceed the context limit of single agent. By contrast, our Chain of Agents framework is proposed to effectively reason across multiple agents to support the unlimited length of source text.
Least-to-
3 Method
3

---

Figure/ 1 shows the overview of our Chain-of- ofrt--ot---esss (CoA) framework, containing two stages. In stage I, long context is split into chunks where each chunk can be processed by a worker agent. Then, the worker agents communicate sequentially to produce evidence over the entire context. In stage 2, a manager agent consumes the knowledge from the chain of workers to generate the final answer.
To formulate the task, we denote long-context sample as (x. y/ q). where I is the input of n tokens, is the output of m tokens, q is an op- tional query. Given an LLM with tokens (usu- ally k < n.) as the context window limit, the target to generate y with the limited input con- text window. Therefore, we divide each source text x into chunks {c1, c2....)), ) so that each chunk can be completely fed into the LLM agent backbone model.
Algorithm 1 Chain of Agents (CoA).
Input: Source input z, query g. agent window size ls large language model [+.
Output: Answer to the query. Split æ into chunks {ci, "i,,,,," "2,""". "cc)
where is shorter than &
Initialize CUu + empty string.
for 1, 2,2, do
CU; + LLLM,, (Ww, ci,
end for return LLM CUG,
3.1 Stage 1: Worker Agent: Segment Comprehension and Chain-Communication
In Stage 1, CoA contains a sequence of l number of worker agents. Each worker W; inputs the concatenation of a chunk c; from source text æ, a query q. instruction for specific task for worker agent Iw, and the message passed from the previous agent, denoted as "communication unit" CU;-1 The communication is unidirectional, passing from worker agent i -- 1 to i. The worker agents process them and output the message CU, for next worker, expressed as:
(I) CUs produced by worker agents vary across different tasks. For question answering, CU contains the evidence for the manager to answer the question. For summarization, CU contains the summary of the previous texts. For code completion, CU contains the code summary with function/class names and explanation. Effeetivenness on diverse tasks demonstrates the flexibility of CoA (Appendix C).
CU LLMM; (Ww, CU,-I,
C.,
The multi-step worker communication in CoA expands the model context to the full receptive field, meaning that the last worker can read the full input no matter how long the input is. Therefore, CoA is extensible inputs with different lengths by adjusting the number of worker agents.
The left side of Figure 1 underscores the necessity of collaborative communication among workers to effectively address complex, long-context reasoning tasks. We observe that 1) Although the question unanswerable given 61, W1 generates related evidence that is useful for answering the question; 2) with the partial answer from the previous worker, W2 further reasons with the current source to complete the full reasoning chain across agents and generate the interpretative reasoning chain; 3) W3 finds no related information in the chunk 3, it directly rewrites CU2 by putting the correct answer as the first token of CU3 without adding any unrelated information (Appendix F.4). This shows that if workers are independent (such as tree structure communication), it is impossible to answer hop two while the answer of hop one is held by another worker (Appendix G).
3.2 Stage 2: Manager Agent: Information Integration and Response Generation
In Stage 2, after multiple steps of information extraction and comprehension by worker agents, the manager agent produces the final solution. While worker agents extract relevant information in long-context source, the manager agent synthesizes relevant information accumulated by the end of "worker-agent-chain to generate the final answer. Specifically, given the instruction for manager IM and query g, the manager agent consumes accumulated knowledge from last worker CU; and generates the final answer Response:
Response = M (T,, CU., g)
(2)
The benefit of using separate LLM as the manager agent is to decompose the duty of analyzing chunks in the long-context source (" ("worker agents") ") and producing the final answer ("manager agent"), "), so that every agent can fulfill its duty to the most?.
*Other design choices: Our experiments show that using the last worker Wi to directly generate the final result leads to performance drop. Besides, feeding the manager with all CU, or some CU that is related to the answer (decided by Wi) also hurts the performance because of confusion led by conflicting CU..

---

Table 3: Dataset Statisties, Avg. Input/Agents is the average words/agents (8k) for source imput.
Question Answering
Summarization
Code
HatporQA
MuSiQue
NarrativeQA
Qasper
QUALITY
QMSum
GovReport
BonkSum
RepoBench
Avg. Input
10603
12975
71787 12.45
4236
4936
12524
9239
108478
7105
Avg. Agents
2.35
2.88
1.12
1.31
2.57
2.03
18.63
1.69
Query-based
3.3 Time Complexity Analysis
We compare the time cost of full-context input and Chain-of-A Agetss theoretically in decoder-only setting. We assume the response generated by LLMs contains r tokens on average, the input has n tokens, the context limit of LLM is k, and the length of each chunk RAG k. The time complexity is shown in Table (Ap- pendix A). As can be seen, the encoding time of CoA is less than full-Context because k n in long context tasks, while they have the same decoding time. This demonstrates the efficiency of CoA compared with the Full-Context baseline.
Table 2: Time complexity.
Encode
Decode
Full-Context CoA
O(n2)
O(nr)
O(nk)
RAG
O(nk') + O(P))
(m//k)) + (kr)
Experiment
4.1 Experiment Setup
Datasets.
We conduct experiments on nine long context datasets across three task types (Table 3):
Question Answering. We consider five QA datasets from the LongBench [6] and SCROLL [60]. HotpotQA [85] is a Wikipedia-based multi-hop QA dataset. It requires reasoning across multiple passages to find the answer. MuSiQue [69] a multi-hop QA dataset. is much more difficult than HotpotQA as it contains more hops in one sample, unanswerable questions, and harder distracting content. NarrativeQA [31] is QA dataset over entire books or movie transcripts. The answers can be abstract or extractive, yes/no, and unanswerable. Qasper [17] is question answering dataset over NLP papers. It also contains extractive, abstractive, yes/no, and unanswerable questions. QuALITY [51] is dataset based on stories and articles with multiple-choice questions for each sample. The model needs to select the correct answer among choices.
Summarization. We pick two summarization datasets from SCROLLS. QMSum [93] is a query- based summarization dataset, formed by meeting transcripts from multiple domains such as academic and industrial products. GovReport [25] is generic summarization dataset containing long reports published by the U.S. Government Accountability Office. We also use one dataset for long context memorization tasks. BookSum [33] is collection of datasets for long- long-form narrative summarization, including novels, plays, and stories. We use the book-level partition of the BookSum dataset for experiments.
Code Completion. We pick RepoBench-P [43] which collected from GitHub repositories, and the model needs to generate the next line of code given the long code base.
Metrics. We report the geometric mean of ROUGE [40] for Summarization tasks, code similarity score [6] for Code Completion task, exact match for QuALITY dataset [60], and Fl score for the rest of the Question Answering datasets [6].
LLMs. We use six LLMs total as the backbone of CoA across all experiments (Appendix B). PaLM 2 [4] is series of models with a dense left-to-right, decoder-only language model pretrained on high-quality corpus of 780 billion tokens. We use text-bison @001 and text-unicorn @001 for the experiments with an 8k maximum context window. Gemini 1.0 [67] is family of LLMs proposed by Google. We use gemini-ultra for experiments. The input limit is 32k tokens for Gemini. Claude 3 [5] is family of large language models developed by Anthropic. The family includes models in ascending order of capability: claude-3-haiku, claude-3-sonnet,
three state-of-the- and claude-3-opus. These models are capable of consuming 200k tokens in the context window, providing strong baseline for long context tasks. Although our framework is flexible to use diverse types of LLMs workers and manager,
the same model for each W; and M if not specified.
5

---

Table 4: Overall results CoA. CoA significantly outperforms Vanilla and RAG using various backbone LLMs on all datasets.
Question Answering
Summarization
Code
LLMs
Baselines
HotpotQA
MuSiQue
NarrativoQA
Qeaper
QuALITY
QMSum
GovReport
RepoBeach-P
Vanilla (8k)
45,57
26.87
11.96
26.56
61.86
15.45
20.60
56.30
lext-bison
RAG (8k)
51.91
33.83
14.20
27.20
55.25
15.59
20.83
33.63
CoA (8k)
53.62
37.09
25.26
37.17
65.42
16.77
26.11
58.25
Vanilla (8k)
51.09
29.67
14.45
27.81
83 40
16.61
23,50
53.87
lext-unicom
RAG (8k)
58.01
40.38
19.12
24.44
83.00
16.83
21.43
50,49
CoA (8k)
62.04
42.49
20.37
38.01
83.80
17.67
26.48
60.39
Vanilla (8k)
40.62
23.61
7.71
20.59
57.40
12.10
26.18
49.09
Vanilla (32k)
45.09
27.93
7.21
21.71
58.60
10.24
26.96
73.04
gemini-ultra
RAG (8k)
51.13
31.56
14.51
18.70
62.40
12.70
25.87
72,94
CoA (8k)
54.26
35.09
25.26
35.10
80.60
12.84
26.98
73.05
Baselines. Our principle of choosing baselines is to find the strongest and most typical approaches from input reduction and window extension. The first baseline Vanilla. It directly consumes tokens until the context window of LLM is fully utilized, implying a 200k window LLM if using Claude 3. The other one is Retrieval Retrieval-Augmennt Augmented Generation (RAG). We use the state-of-the-an retriever [79]. Following [81], we first segment the source text into chunks of 300 words and re-rank them using retriever. Top-n chunks are then fed into the downstream LLM until the context window is fully utilized. GovReport dataset does not contain a query initially, we create a pseudo query "What is the summary of the whole government report?" as the query to rerank.
To evaluate the performance of CoA compared with possible multi-agent approaches, we carefully construct two approaches. Similar to CoA, also assign each chunk c; to W; using similar instructions to generate CU., In these approaches, worker agents are parallel and independent while CoA sequential. Multi-Agent Voting (Merge) Each agent directly generate an answer a according to c, A majority voting is applied to all answers ag to decide the final answer. Multi-Agent Hierarchical Structure (Hierarchical). Inspired by [9], we propose hierarchical framework, where the communication forms a tree structure between workers W. and manager M. For each worker, it first judges whether o; contains useful information. If true, generates a communication unit CU;- Then, all CU; are sent to the manager M to come up with final answer. Besides, we append an integer number L at the end every approach to clearly remind the window size limit of LLM. For instance, "CoA (8K)" refers to the base LLM used in CoA with window size 8K.
4.2 Overall Results of CoA
Question Answering. Table 4 shows the results of Question Answering tasks on all three models. CoA (8k) outperforms Vanilla (8k) by a large margin on all 8 datasets, including 13.30% on NarrativeQA, 12.82% on MuSiQue, and 22.00% on Quality, for text-bison, text-unicorn, and gemini- ultra, respectively. Also, CoA (8k) outperforms RAG (8k) model for all 8 datasets using all three LLMs, demonstrating that CoA achieves higher performance than RAG. In other words, using multi- agent L. outperforms RAG models. It is also worth noting that for gemini-ultra, Vanilla (32k) improves the Vanilla (8k) baseline, yet it is still lower than CoA (8k). CoA also obtains compatible performance with with state-of-the.. -f- models (Appendix E), while keeping robustness of window size (Appendix F.2).
Summarization and Code Completion. Table 4 shows the results of Summarization and Code Completion tasks. Similarly, CoA (8k) also outperforms all Vanilla (8k) and (32k) baselines on all three datasets, demonstrating the strong capability of CoA on various tasks. It is worth noting that for GovReport, RAG fails to improve the baseline with pseudo query. By contrast, CoA improves the performance significantly, showing that CoA can also be applied in non-query tasks (Appendix F.
Long Context LLMs.. As Claude models support 200k of tokens, we call these models long context models (LCM). Table 5 shows the performance of the LCM on two datasets. As can be seen, CoA (8k) outperforms Vanilla (200k) significantly, showing that with only an Sk context window, CoA achieves much higher performance than LCM with a 200k context window. Also, CoA improves the performance with the samples that can be fed into 200k context window (no truncation). Moreover, the improvements over the Vanilla (200k) and RAG (8k) become higher when the model

---

Table 5: Comparison with long context LLMs on N and BookSum. CoA significantly outperforms Claude 3 with 200k context limits. No Trun//rrun indicates the souree text in the sample less/more than 200k tokens which does not need/needs truncation for vanilla (200k) baseline. Average is the mean value across all samples.
claude-3 h-hki
claude-3-sonne
claude-3-ope -ooud---opuus
No Trun.
Trun.
Average
No Trun.
Trun.
Average
No Trun.
Trun.
Average
NarrativeQA
Vanilla (200k)
8.00
2,50
7.17
5.58
2.44
5.15
7.23
2.35
6.56
RAG (8k)
5.94
4.22
5.71
9.09
5.17
8.50
6.13
4.29
5.86
CoA (8k)
18.31
21.34
18.80
16.63
16.47
16.51
24.38
21.26
23.96
BookSum
Vanilla (200k)
11.98
11.70
12.04
12.17
11.90
12.10
14.11
13.67
14.00
CoA (8k)
13.28
13.73
13.70
14.92
15.05
14.96
17.74
16.68
17.47
Table Comparison between CoA and other multi-agent frameworks. CoA with sequential agents outperforms other designs with multiple parallel agents including Merge and Hierarchical.
HotporQA
MuSiQue
NarrativeQA
Qasper
QuALITY
QMSum
GovReport
RepoBench-P
Vanilla (8k)
45.57
26.87
11.96
26.56
61.86
15.45
20.60
$6.30
Merge (8k)
42.96
26.66
11.27
26.78
59.30
9.42
25.38
33.66
Hierarchical (8k)
50.62
29.40
17.04
31.39
64,20
15.19
16.54
27.96
CoA (8k)
53.62
37.09
25.26
37.17
65.42
16.77
26.11
58.25
size increases from Haiku to Opus 116.6/11.3//111111 for NarrativeQA, 1.66/2.86/3.4 for BookSum). This demonstrates that CoA benefits from stronger models achieve higher improvements.
Other Multi-Agent Frameworks. As shown in Table 6, Hierarchical (8k) outperforms Vanilla (8k) on five out eight datasets, demonstrating the hierarchical approach can also improve the vanilla baselines significantly. Merge (8k) is lower than Vanilla (8k) except for GovReport, showing that merging is especially effective for long summarization tasks such as GovReport. As can be seen, CoA outperforms Hierarchical and Merge all eight datasets. The reason behind the results is because Hicrarchical and Merge do not allow workers to communicate with each other due to their parallel designs. Thus, each worker only maintain the information in its chunk which blocks the understanding of the whole text, hurting the performance greatly.
5
Analyses
5.1 CoA Improvement is More Obvious When RAG Fails to Retrieve Gold Answer
To demonstrate this, we first classify the samples in NarrativeQA dataset into different bins according to the position (index) of the chunk in RAG processed input that contains the gold answer. Then, compute the average score the CoA and RAG results of different bins. Figure shows the results. As shown in the figure, RAG performs better when the index is smaller (the gold answer is nearer to the top), showing that downstream LLMs rely significantly on the quality of RAG re-ranking. Besides, the performance of RAG is positively correlated CoA's when successfully retrieves the gold answer. However, when RAG fails, CoA can greatly improve the performance (much higher than the tendency line).
0.16
0.175
8.200
ROUGE
Avrongge ROUGE
0.34
0.2
8.500
0338
0.88
6.775
30h-10
26-308
5430
(Unit: Thousand)
Imput
(a) Claude Haiku
(b) Claude Sonnet
(c) Claude Opus
Figure 2: Performance of Claude on BookSum. Improvement is more obvious for longer inputs.
7

---

0.3
50.00
0. (000g)
Full CoA
CoA (F1 Scores)
648 (m))
45.00
0.2
(129)
F1 Scores
Fall (1SN)
40.00
0.1
35.00
0.0
30.00
0.08
0.10
0.12
0.14
0.16
5 10
15 20 25
RAG (F1 Scores)
Position of Document with the Answer
Figure 3: Comparison on NarrativeQA. X-axis/Y- axis indicate RAG/CoA performance while each point represents a bin. The number indicates the
Figure 4: Performance of CoA and Full on Nat- ural Questions. CoA mitigates the lost-in-the- middle issue. X-axis is the index of document
chunk index of gold answer (ratio of number of samples in bracket), and the size of the point indi- cates the improvement of CoA over RAG.
with gold answer where small number indicates
gold answer is closer to start.
Q: Gary L. Bennett was part of the space missions that have primary destination of what celestial body? Gold: Sun
Gary L. Bennett ( bom January 17 1940 is an American scientist and engineer ... the Voyager. Galileo. and Ulysses space missions, He joined the NASA headquarters in June 1988 as Manager of Advanced Space Power Systems Earth in the case of sample-retum missions.
Gary L. Bennett, scientist and engineer, has contributed various space missions, including Voyager, Gallleo, Ulysses. has worked on advanced space power and ... the scientific integrity of celestial bodies
Worker 1:
Worker 3: Gary L. Bennett, in various space missions, particularly Voyager, Galileo, and Ulysses. Ulysses, launched in 1990, embarked on unique trajectory to Medal and the COSPAR explore both the southern Distinguished Service Award.
Worker 2: Gary L.
Bennett.... Bennett has received numerous accolades for his work, including the NASA Exceptional Service
and northern polar regions of the Sun
Planetary protection ... celestial bodies
Celestiel Space
Bodies
Missians
Mars
Space Missions
Ulyses
Sun
Sun
Worker Directly Forwards
Wrong
Usaful Reasoning
Final Pradict
Bennett
Space Power
Answer
Gary L. Bennett
Awards
NASA
COSPAR
Other Reasoning
RAG:
Only Information around question
CoA: Complex Reasoning across Agents for Multi-hop Reasoning
Figure 5: A case study of RAG (left) and CoA (right) on HotpotQA. The sequential agent communi- cation enables CoA perform complex multi-hop reasoning long contexts.
5.2 CoA Improvement is More Obvious When Long Context Models Meet Longer Inputs
We compare the performance of CoA and Vanilla with Claude 3 on BookSum. As shown in Figure CoA can outperform the vanilla baseline by a large margin on various source lengths. It is worth noting that, when the length of the sample increases, the performance even increases for CoA, and the improvement over Vanilla (200k) baseline becomes more significant. The improvement of CoA reaches around 100% when the length is larger than 400k. Thus, we can conclude that 1) CoA can still enhance the LLM performance even though the model has a very long context window limit; and 2) CoA delivers more performance gains when the input is longer.
5.3
CoA Mitigates -Loss-in-tth--Middlee Phenomenon
To assess the "lost-in-the-middlis [42] effect on Vanilla and CoA models, we replicated the original study by randomly selecting 500 samples from their dataset to create QA dataset. The results displayed in Figure 4. The Vanilla model exhibits a significant performance range of 6.13 (7). In contrast, CoA shows resilience against this issue, with narrower performance gap of 4.89 (+11) demonstrating that CoA effectively mitigates this problem by providing each agent a shorter context focus
teat-in-th--mddlle issue, with a
5.4 Multi-agent Collaboration in CoA Enables Complex Reasoning over Long Context
Figure displays a sample prediction from HotpotQA. To find the correct answer, RAG retrieves text chunks with high semantic similarity with the query. However, conducting multi-hop reasoning is challenging as the critical first-hop answer often lacks semantic relevance to the query. contrast, CoA operates differently: the first agent explores related topics without knowing the query's 's answer, aiding subsequent inference. The second agent, also unaware of the answer, broadens the topic scope by incorporating new information. The third agent finally discovers the answer, synthesizing
8

---

Table Ablation on CoA. Manager plays an important role in CoA, and left-to-right yields the best performance among other reading orders including Right-to-Left and Permutation.
HotpotQA
MuSiQue
NarrativeQA
Qasper
QuALITY
QMSum
RepoBench-E
CoA
53.62
37.09
25.26
37.17
65.42
16.77
58.25
w/o Manager 48.58
26.79
20,80
29.66
58.80
16.50
56.16
Right-to-Lefi
51.83
29.77
21.57
36.60
62.80
15.91
55.10
Permutation 56.05
34.55
23.60
37.42
64.60
16.50
58.43
Table 8: Comparison of three multi-pati augmentation through judge voting. Multi-pat path CoA
furthers enhance the performance.
HotpotQA
MuSiQue
NarrativeQA
Qasper
QuALITY
QMSum
RepoBench-P
Bi-direction: left-to-right and right-to-left paths (2-way)
w/ judge
54.11
36.97
24.47
39.23
65.00
16.41
49.69
w/ vote
51,46
34.16
21.59
37.42
64.60
9.51
38.70
oracle 62.12
48.02
32.84
46.37
71.80
18.83
59.67
Self- Self-Consistency: five left-to-right reasoning paths (5-way)
w/ judge 57.17
38.82
21.58
36.24
62.80
17.06
46.97
w/ vote 57.49
40.78
25.56
39.15
68.60
8.35
35.56
oracle 67.07
55.74
39.89
52.74
80.40
20.81
63.52
Permutation: five random order reasoning paths (5-way)
w/ judge 59.17
42.37
25.47
37.65
63.40
17.81
52.45
w/ vote 58.29
39.17
26.58
38.09
67.60
8.31
35.44
oracle 75.73
60.16
39.58
52.22
79.80
20.88
67.80
information from earlier agents and new data to complete the reasoning chain. This collaborative approach highlights CoA's ability to facilitate complex reasoning across long context tasks.
5.5
Ablation Study: Effectiveness of Manager and Alternative Design Choices
To demonstrate the effect of the manager, conduct ablation study that uses the last worker to generate results directly. As shown in Table 7, "w/o Manager" hurts the performance significantly, dropping more than 10% MuSiQue. This demonstrates the important role of the manager. Next, to empirically verify that left-to-right yields the best performance, we evaluate other orders of reading, including Right-to-Left by reading from the last chunk to the first one and Permutation which reads in random order. As shown in Table 7, on most of the datasets, left-to-right yields the highest score, demonstrating the advantages of natural reading order.
5.6 Multi-path Chain-of-Agents Further Enhances Performance
We manually investigated the results these three orders (left-to-right, right-to-left, permutation), and we found that other orders sometimes can produce better answers than left-to-right. Inspired by this observation,
explore two approaches to select the best result among multiple paths. w/ vote applies majority voting over the final results while w/ judge uses an LLM to judge the most reliable CU, of diverse paths and generate the final answer. Oracle picks the best path by evaluating score of each path, yielding the upper bound performance. Table 8 compares three multi-path augmentation approaches. Surprisingly, results show that 1) all ensemble approaches (Bi-direction, Self-consistency, and Permutation) can further enhance the performance of CoA and 5-way Permutation yields the best improvement, 2) majority voting (w/ vote) of final answer is better than using an LLM as judge (w/ judge) in Self-consistenece but worse in Bi-direction, 3) using LLM judge (w/ judge) works well on long result generation tasks (QMSum, RepoBench-PP) and 4) there is large space to improve because oracle (choose as answer the one with highest performance) is much higher than either w/ judge or w/ vote. We leave the direction of multi-path reasoning future study.
5.7 Practical Time Complexity
We run an analysis to show the practical time consumption of the proposed CoA on HotpotQA dataset. We choose llama3-8b as the backbone model for preventing additional untrackable latency due the

---

Table Practical time analysis on HotpotQA dataset. Avg. # of Input/Output shows the total input and output tokens for each model.
Running Time (s)
Avg. # of Input
Avg. # of Output
Avg. # Agent Output
Vanilla (8k) RAG (8k)
1.33
5912.85
2.40
2.40
2.41
16479.91
2.75
2.75 1 1..00
CoA (8k)
3.10
10840.95
38.38
Table 10: Information loss of text-bison model on different datasets.
HotpotQA
MuSiQue
Qasper
NarrativeQA
QMSum
Coλ performance 54.87
40.38
37.03
24.04
17.01
Information Loss
1.46
1.65
3.92
3.88
0.91
network of API queries. As can be seen in the Table 9, Vanilla consumes the least amount of tokens and generates the least of tokens as well. However, truncates the input thus maintaining a low performance. Although RAG generates fewer tokens than CoA (by adding retrieval and downstream input together), it needs to read the whole input by Retrieval models which also consumes additional time. Overall, RAG is faster than CoA by around 30% in this example. Parallel decoding. For decoder-only LLMs, CoA agents can run in parallel. Before CUs are produced, agents can start to read their assigned paragraphs and wait for the CUs to come. We conduct an approximation by asking the model to generate one token for each sample. The output time is shown to be short and negligible, mimicking the encoding time of each sample. We have found that the running time of CoA can be reduced by 57.21 % on average, leading to a 1.32-second running time for each sample, closing to the running time of the Vanilla baseline.
5.8 Information Loss
To probe the information loss during information propagation in CU, we further propose metric to analyze. For each sample, we compute the highest score between the communication unit and the gold answer, and if this score higher than the score of the final prediction, refer to this information loss, formulated as Infomation_ = max (0, max score(CU, Y) score(Y, Y)) where score(*) is the performance metric for each task and dataset, Y is the gold answer and Y is the prediction. Table 10 shows the information loss of text-bison. If choosing the communication unit with the highest performance, around 1%-4% performance gain can be obtained, meaning that 1%-4% of the information is lost during the chain communication. While every method might have information loss, CoA design ensures final results yielding negligible information loss.
compute the difference and
6 Conclusion
In this paper, we propose Chain-of-Agent a multi-agent LLM collaboration framework for solving long context tasks. It is a training free, task/length agnostic, interpretable, and cost-effective frame- work. Experiments show that Chain-of-Agents outperforms RAG and Long Context LLMs by large margin despite of its simple design. Analysis shows that by integrating information aggregation and context reasoning, CoA mitigates lost-in-the-middle and performs better on longer samples.
Limitations. While CoA features with simple and effective design, future directions can address the following limitations to further improve prowess and efficiency. First, communication effectiveness can be further improved via finetuning or in-context learning because current LLMs are aligned with human norms which is not optimal for communication between LLMss. Second, CoA does not explore other forms of communication approaches, such as debating complex discussions. Third, the cost and latency running CoA can further reduced, such as replacing some L.LMs with more effective models via model routing [62].
10

---

Acknowledgement
We thank colleagues in Cloud AI Research team for providing helpful feedback for this paper. We also thank the anonymous reviewers for their helpful comments.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt. Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv23303.08774, 2023.
[2] Qingyao Ai, Ting Bai, Zhao Cao, Yi Chang, Jiawei Chen, Zhumin Chen, Zhiyong Cheng. Shoubin Dong, Zhicheng Dou, Fuli Feng, et al. Information retrieval meets large language models: strategic report from chinese community. Al Open, 4:80-90, 2023.
[3] Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian lan-Guang Lou. Make your Ilm fully utilize the context. arXiv preprint arXiv:2404. 1681 2024.
[4] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10400, 2023.
[5] AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
[6] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. (at::2238..450000 14508, 2023.
[7] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05022, 2023.
[10] Huaben Chen, Wenkang Ji, Lufeng Xu, and Shiyu Zhao. Multi-agent consensus seeking via large language models. arXiv preprint arXiv:23 2023.
[11] Justin Chih- Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13000 2023.
[12] Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. SummScreen: A dataset for abstractive screenplay summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8602-8615, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[13] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window large language models positional interpolation. arXiv preprint arXiv:2306. 15595, 2023.
11

---

[14] Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable multi- collaboration with large language models: Centralized or decentralized systems? In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4311-4311 IEEE, 2024.
[15] Nelson Cowan. Working memory underpins cognitive development, learning, and education. Educational psychology review, 26: 16:197-223, 2014.
[16] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, Online, June 2021. Association for Computational Linguisties.
[17] Pradcep Dasigi. Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner, A dataset of informattion-seeinng questions and answers anchored in research papers. arXiv preprint arXiv:2/05.03011 2021.
(18) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning language models through multiagent debate. arXiv preprint arXiv:2305. /4325, 2023.
[19] Navid Ghaffarzadegan, Aritra Majumdar, Ross Williams, and Niyousha Hosseinichimeh. Gen- erative agent- based modeling: Unveiling social system dynamics through coupling mechanistic models with generative artificial intelligence. arXiv preprint arXiv:2309. 1446,, 2023.
[20] Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long-range pre-trained language model for code completion. In International Conference on Machine Learning, pages 12098-12107 PMLR, 2023.
[21] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.0 68800 2024.
[22] Xanh Ho, Anh- Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics. pages 6609-6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
[23] Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311. 2023.
[24] Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13011 2023.
[25] Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur: Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online, June 2021. Association for Computational Linguistics.
[26] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023.
[27] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuging Yang, and Lili Qiu. Longlimlingua: Accelerating and enhancing ml long context scenarios via prompt compression. arXiv preprint arXiv:2310.06833 2023.
12

---

[28] Zhengbao Jiang, Luyu Gao, Zhiruo Wang, Jun Araki, Haibo Ding, Jamie Callan, and Graham Neubig. Retrieval attention: End-to-end learning of retrieval and reading within a single transformer. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2336-2349, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[29]
Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend Il context window without tuning, 2024.
[30] Tushar Khot, H. Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. ArXiv, abs/2210.02406,, 2022.
[31] Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transac- tions of the Association for Computational Linguistics, 6:317-328, 2018.
[32] Tomák Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gá- bor Melis, and Edward Grefenstette. The NarrativeQA Reading Comprehension Challenge. Trunsactions of the Association for Computational Linguistics, 6:317-328, 05 2018.
[33] Wojciech Kryšcinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08200 2021.
[34] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[35] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society, Advances in Neural Information Processing Systems, 19t:1999-5200,, 2023.
[36] Nian Li, Chen Gao, Yong Li, and Qingmin Liao. Large language model-empowered agents for simulating macroeconomic activities. arXiv preprint arXiv:2310.1044 2023.
[37] Tianle Li, Ge Zhang, Quy Due Do. Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning. arXiv preprint Kii:22404.02066 2024.
[38] Yucheng Li. Unlocking context constraints of Ilms: Enhancing context efficiency of llms with self-information- based content filtering. arXiv preprint arXiv:2304. 1022 2023.
[39] Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon. NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
[40] Chin Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summa- rization Branches Out, pages 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
[41] Sheng- Chiih Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen- tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 6385-6400, Singapore, December 2023. Association for Computational Linguistics.
[42] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
[43] Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.0309, 2023.
13

---

[44] Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu. Hao Zhang, Jonathan May. Luke Zettlemoyer. Omer Levy, and Chunting Zhou. Megalodon: Efficient Ilm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801, 2024.
[45] Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 286-299. IEFE, 2024.
[46] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random- infinite context length for transformers. arXiv preprint arXiv:2305.16300 2023.
[47] Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. arXiv preprini arXiv:2310.08901 2023.
[48] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024.
[49] Bo Ni and Markus J Buehler. Mechagents: Large language model multi-agent collaborations solve mechanics problems, generate new data, and integrate knowledge. Extreme Mechanics Letters, page 102131, 2024.
[50] OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774 2023.
[51] Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States, July 2022. Association for Computational Linguisties.
[52] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Anmual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[53] Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-18, 2022.
[54] Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho. and Douwe Kiela. Unsupervised question decomposition for question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864 8880, Online, November 2020. Association for Computational Linguistics.
[55] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
[56] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[57] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXi5:2403.005530 2024.
[58] Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-mergge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123,
2023. 2023.
14

---

[59] Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977-7989, Singapore, December 2023. Association for Computational Linguisties.
[60] Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[61] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: preprint arXiv:2301 12652, 2023.
black-box language models. arXiv
[62] Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thomp- son, and Mikhail Yurochkin. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309. 15789, 2023.
[63] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonsi al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXivv2206.04660 2022.
[64] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[65] Simeng Sun, Y. Liu, Shuo Wang, Chenguang Zhu, and Mohit lyyer. Pearl: Prompting large language models to plan and execute actions over long documents. ArXiv, abs/2305. 2023.
[66] Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning, arXiv preprint wrXiv:2311.10532 2023.
[67] Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /8805 2023.
[68] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13711, 2023.
[69] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-5555 -54,55442200 2022.
[70] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian Wen Zhang, Zhao Yan, and Zhoujun Li. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arXiv:2312.11242. 2023.
[71] Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models for chain of thought. In Yoav Goldberg, Zornissa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 27 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[72] Boxin Wang, Wei Ping. Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong. Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, and Bryan Catanzaro. Shall we pretrain autoregressive language models with retrieval? comprehensive study. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 7763-7786, Singapore, December 2023. Association for Computational Linguistics.
15

---

[73] Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training arXiv preprint arXiv:2212.03533: 2022.
[74] Shenzhi Wang. Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's 's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.00 10.01320, 2023.
[75] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 1103,, 2022.
[76] Martin Weiss, Nasim Rahaman, Manuel Wuthrich, Yoshua Bengio, Li Erran Li, Bernhard Schölkopf, and Christopher Pal. Rethinking the buyer's inspection paradox in information markets with language agents, 2023.
[77] Tongshuang Wu, Michael Terry, and Carrie Jun Cai. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. Proceedings of the 2022 CHI Conference Human Factors in Computing Systems, CHI '22, New York, NY, USA, 2022. Association for Computing Machinery.
[78] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources advance general chinese embedding. arXiv preprint arXiv:2309.0759) 2023.
[79] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[80] Kai Xiong, Xiao Ding, Yixin Cao. Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[81] Peng Xu, Wei Ping. Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subrama- nian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023.
[82] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games; An empirical study on werewolf. arXiv preprint arXiv: 04658, 2023.
[83] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.189400 2023.
[84] Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. Towards reasoning in large language models via multi-agent peer review collaboration. arXiv preprint arXiv:2311.000
2023.
[85] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[86] Hongbin Ye, Honghao Gui, Aijia Zhang, Tong Liu, Wei Hua, and Weiqiang Jia. Beyond isolation: Multi-agent synergy for improving knowledge graph construction. arXiv preprint arXiv:2312.03022. 2023.
[87] Dani Yogatama, Cyprien de Masson d' Autume, and Lingpeng Kong. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362-373, 2021.
[88] An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. arXiv preprint arXiv:2310.10108 2023.
16

---

[89] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum. Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint 1rXiv:23377.22485. 2023.
[90] Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed H Awadallah, Dragomir Radev, and Rui Zhang. Summ": A multi-stage summarization framework for long input dialogues and documents. arXiv preprint
2021.
[91] Yusen Zhang, Ansong Ni, Tao Yu, Rui Zhang, Chenguang Zhu, Budhaditya Deb, Asli Celikyil- maz, Ahmed Hassan Awadallah, and Dragomir Radev. An exploratory study on long dialogue summarization: What works and what's next. arXiv preprini arXiv:2 1002200000000 100.04609, 2021.
[92] Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. Longagent: Scaling language models to 128k context through multi-agent collaboration. arXiv preprint arXiv:2402.14 11550, 2024.
[93] Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-base multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Makkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-59211 Online, June 2021. Association for Computational Linguistics.
[94] Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Huai hsin Chi. Least-to-most prompting enables complex reasoning in large language models. ArXiv, abs/2205. 10625, 2022.
[95] Wangchunshu Zhou, Yuchen Eleanor Jiang. Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint 2rX.v:2305.13:3 2023.
17

---

A Proof of Time Complexity
Assuming the source text containing n tokens, window limit of LLM is & tokens, and the responses contain tokens in average. For decoder-only LLM, grasp the operations for attention calculation as the time cost unit. Then, for Full-Context LLM, total operation for encoding input source text Truil is:
Enne (1 + 2 ................................ +n) (n 1)m = On²)
(3)
2
Similarly, decoding starts when the model already generate all input. Thus, the first decoded token attends to positions. Total operation for decoding response is (r < n):
Dec (n + 10+1+++220 + +n+r)= (n + 1 r)r
O(nr + r²) ((r))
(4)
2
For Chain-of-Agente we first split the source into [n/k] chunks. Thus, total encoding time for all input is:
[n/A] (I: + 1)h x n/kk] O(k² x n/kk) O(k))
Tene (1 21+2+
(5)
2
Decoding starts when the model already generates tokens. Thus, the first decoded token attends to A positions. The total operation for decoding response is (r < k):
(k: + 1+ k + r)r [n/k] 0 O(rr)
Tee (a01++++-.--+4))>>)))
(6)
Similarly, for RAG models, encoding time is retrieval time O x n/kk) = O(nk') plus the encoding time of downstream LLM O(k2). Decoding cost is the time of retrieval embedding generation On//k), plus the generation the downstream LLM (kr)).
B Implementation Details
For all experiments, we use Vertex model garden 4 API to use all six models. Maximum generation token set to 2048 for gemini-ultra and set to 1024 for rest of the models. We set temperature to 0 for all experiments except for Self-consistency setting. Table 11 shows the prompt for all models. for task specific requirement
datasets, we follow the original LongBench [6] and SCROLLS [60]. For RAG model, we use the model provided by Huggingface" and run on A 100 GPUs to rerank the chunks. Algorithm shows the algorithm to chunk the input to cater the window size.
Algorithm 2 Chain of Agents (CoA) Input Chunking Algorithm.
Input:
Source input æ, query q, agent window size k instruction I...
Output: A list of chunks C = {ci, ..). Split the source into sentences $1, "2, .n
Initialize e +- empty string, +- cmpty ordered set, Initialize length budgets f- & count_token count_token((a)) for in 81, if count token(c) + count token(s) B then Append to C, initialize +- empty string end if
Sn. do
t + s end for if count token(c) 7 then
// // indicates concatenating two strings with a blank.
Append C'
end return C
httpe::////ooodooogg..o wow.gle..o//oooeegggraeaaaeeeeaa ltttpss/huugdingfacecco
18

---

Table 11: Prompt of all models for query-base based tasks.
[Task specific requirement)
[Source Input with truncation if needed)
Vanilla
Question: (Question q}
Answer:
(Task specific requirement)
[Retrieved Chunks of Source Input x:
RAG
Question: (Question q)
Answer:
Worker Wi:
(Input Chunk c.
Here is the summary the previous source text: (Previous Communication Unit (CU+-1))
Question: (Query 4]
You need to read current source text and summary of previous source text (if any) and generate summary include them both. Later, this summary will be used for other agents to answer the Query, if any. please write the summary that can include the evidence for answering the Query: Manager M:
CoA
[Task specific requirement]
The following given passages. However, the source text is too long and has been summarized, You need answer hased the summary: (Previous Communication Unit CU1)
Question: (question]
Answer:
Table 12: Prompt all models for non-query-based tasks.
[ask specific requirement)
Vanilla
(Souree Input x with truncation if needed)
Answer:
Task specific requirement)
RAG
|eetrieved Chunks of Source Input x)
Answer:
Worker Wi:
[Input Chunk c
Here the summary the previous source text: (Previous Communication Unit (CU,-1)
You need to read the current source text and summary of previous source text (if any) and generate a summary to include them both. Later, this summary will be used for other agents to generate a summary for the whole text. Thus, your generated summary should be relatively long.
CoA
Manager
(Task specific requirement)
The following are given passages. However, the source text is too long and has heen summarized, You need to answer hased on the summary: [Previous Communication Unit CUi)
Answer:
C Example Outputs
We pick one sample from HotpotQA in Table 13, QMSum in Table 14, and RepoBench-P in Table 15 show the prediction Chain-of-Ageetss, in the order of question q. Communication Unit CU of agents, and final prediction of manager. We also observe that 1) the first worker usually explores the most number paths, including 3 or more topics because the agent is not sure about the result and willing provide more information for the next ones. 2) The final worker usually narrows down to one answer with a much shorter CU, and 3) The speed of narrowing down will be faster for simpler samples, producing small reasoning graph.
19

---

Table 13: Example output on HotpotQA dataset.
Question
Gary L. Bennett was a part of the space missions that have primary destination of what celestial body?
Worker
Gary L. Bennett, scientist and engineer. has contributed to various space missions, including Voyager, Galileo, and Ulysses. He has worked on advanced space power and propulsion systems and has been involved in planetary protection measures. Bennett's s expertise has heen instrumental ensuring the scientific integrity of celestial bodies and preventing harmful contamination. He has received numerous awards and accolades for his contributions to space exploration and is recognized as a leading expert in the field of planetary protection.
Worker
Gary L. Bennett, renowned scientist and engineer, has made significant contributions to space missions, including Voyager, Galileo, and Ulysses. His expertise in advanced space power and propulsion systems, as well as planetary protection measures, has been crucial in ensuring the scientific integrity of celestial bodies and preventing harmful contamination. Bennett has received numerous accolades for his work, including the NASA Exceptional Service Medal and the COSPAR Distinguished Service Award.
Worker
Gary L. Bennett, a distinguished scientist and engineer, played a pivotal role in various space missions, particularly Voyager, Galilco, and Ulysses. His expertise in advanced space power and propulsion systems, coupled with his focus planctary protection measures, has been instrumental in safeguarding the scientific integrity of celestial bodies. Ulysses, launched in 1990, embarked explore both the southern northern polar regions of the Sun. During its extended mission, Ulysses provided invaluable data on the Sun's 's magnetic field, solar wind, and the presence of dust in the Solar System.
unique trajectory to
Manager
Sun
RAG
Mars
Gold
Sun
Table 14: Example output QMSum dataset.
Question
Summarize the discussion about industrial components.
Worker
The industrial designer proposed that the remote control should he made of titanium or rubber, and the buttons should be simple. The marketing expert said that the remote control should be fancy, innovative, and easy to use. They also discussed the trend of making products look like fruit and vegetables.
Worker
The industrial designer proposed that the remote control should be made of titanium rubber, and the buttons should be simple. The marketing expert said that remote control should be fancy, innovative, and casy to use. They also discussed the trend of making products look like fruit and vegetables. The industrial designer said that they would explore the two options of titanium and rubher, and see if rubber is expensive. The user interface designer said that they would pretend that the modelling clay is titanium and paint it afterwards. The project manager said that they would stick with two batteries and not reinvent the wheel.
Manager
The industrial designer proposed that the remote control should be made of titanium or rubber, and the buttons should be simple. The marketing expert said that the remote control should be fancy, innovative, and easy to use. They also discussed the trend of making products look like fruit and vegetables. The industrial designer said that they would explore the two options of titanium and rubber, and see if rubber is expensive. The user interface designer said that they would pretend that the modelling clay is titanium and paint it afterwards. The project manager said that they would stick with two batteries and not reinvent the wheel. (ROUGE Score: 21.38)
RAG
The industrial designer and marketing manager discussed the industrial components control. They agreed that the remote control should have a simple design, with a few buttons and speech recognition technology. They also discussed the possibility of adding I_C display. but decided that would be too expensive. The designer suggested that the remote control could be made of titanium rubber, and the marketing manager agreed that spongy design would be more appealing to consumers. (ROUGE Score: 16.54)
remote
Gold
The industrial designer provided several options respectively for energy, material and interface. Among these options, the industrial designer preferred traditional battery, titanium and simple push buttons so that they would have enough money for speech recognition. After that, the user interface designer proposed an LCD display only for output, which might be taken into account. The group also talked about how to enable the users to find their controllers.
20

---

Table 15: Example output RepoBench-P dataset.
Question
What is the next line of code: if(type == type =
Worker
The method CansoDDaaColllcctttMaaggggg SataaaamggeettttSSSSooonnttaCCCo database cache for the given sensor type. If the type is 0, all sensor types flushed. The method first checks if the type valid, and if it calls the flushDBCache method for the appropriate sensor collector. CaansorDaaaColllcoooaaaagee SaaagggrffuusSSenoDDaaaaCCCCCC database cache for the given sensor type. If the type is 0, all sensor types are flushed. The method first checks if the type is valid. and if it is, it calls the flushDBaache method for the appropriate sensor collector.
type, String devicelD) flushes the
type, String devicelDD flushes the
Worker
The method database cache for the given sensor type. If the type is 0, sensor types are flushed. The method first checks if the type valid, and if is, calls the flushDBCache method for the appropriate sensor collector.
type, String devicelD) flushes the
Manager
RAG
Gold
Table 16: Compare between CoA and previous state-of tfateoof-the-art models on nine datasets. the model needs further training.
indicates
HotpotQA
MuSique
Qusper
Narrative
Quality
QMSum
GovReepor
BookSum
RepoBench-PP
Previous Best
$4.4 [29]
40.4 [29]
53.90 [60]
26 [29]
89,2 [59]
22.44 [60]
26.3 [60]
18.5 [78]
56.47 [29]
Ours Best
62.04
42.49
25.26
17.67
26.98
73.05
D Broader Impacts
Chain-of-Agent is generic framework for long context tasks. users can apply this to diverse tasks not restricting to the mentioned ones. It will greatly increase the efficiency of individuals or companies to solve complex long context tasks. Besides, the interpretablity of such approach can reduce the misuse of the LLMS because users can check the correctness of results and decrease the possibility of making faults. However, similar to all prompt based approaches, this framework requires careful prompt design for unseen large language models, users may not get optimal solution on certain newly proposed LLMs. Besides, may increase the number of the calls for API, causing higher network traffic and higher latency for user pools.
E
Comparison with State-of-the-Art Models
Table 16 shows the comparisons of CoA with the previous state-of-the-are performance, including the ones requiring training (indicated with *). As can be seen, CoA achieves better or comparable results on all datasets, improving HotpotQA and RepoBench-p by large margin. The performance on some datasets is lower than that of state-of-the-art models because the datasets are trained on some domain-specific models, such as Qasper.
More Experiments and Analysis
F.1
Short LLMss on BookSum Dataset
We have run the BookSum dataset with Text-bison and Text-unicorn models. As shown in the Table 17, similar the results with the LLMs that support longer contexts, CoA outperforms baselines by significant margin.
F.2 Robustness against Context Window Size
We set the default context window of CoA to 8k due to the limitation of text-bison and unicorn models. To test the influence of CoA against context window change, set window sizes to 4k, 8k,
21

---

Table 17: Performance of text-bison and text-unicorn on BookSum dataset.
text-bison
text-unicorn
Vanilla (8k)
9.13
8.15
RAG (8k)
9.38%
8.01
CoA (8k)
14.51
14.41
21
20
Performance
19
18
17
16k
32k
Agent Context Window
Figure 6: Performance of CoA on Claude 3 Haiku on the NarrativeQA dataset with various context window sizes of an agent. Results show the robustness of CoA towards different choices of context lengths.
16k, 32k, 64k, and 128K of Claude Haiku model and evaluate on Narrative dataset and see the performance change. As shown in Figure 6, the performance of the model increases from 4k to 16k and stabilizes to around 20 with the context window further increases. This result shows that CoA will benefit from increasing length and keep stable when the length touches a bound.
We further analyze different agent lengths and reported the scores of text-bison-32k Table 18. Our results show that it can improve the baseline with various window sizes. Thus, we choose 8k for short context models, such as text-bison.
F.3 Evaluation on NIAH Test
We conduct evaluations the NIAH test. We follow the LongAgent [92] paper to NIAH PLUS test for evaluating the long context understanding capability of LLMs. Different from the original NeedleInAHaystac, test, NIAH PLUS more challenging because LLMs need answer the questions rather than simply retrieve the information (needle). The results are shown in Figure 7. As can be seen, the CoA greatly increases the accuracy Text-bison from 26.0% to 97.8%, showing that CoA significantly increases the capability of LLMs to understand long contexts.
E4 Preventing Collapse of Reasoning Chain
Experiments show that when certain irrelevant content is present in the communication unit CU,- 1. it often prevents the worker unit W; from generating meaningful response. This meaningless response then propagates to the next unit, causing W to produce similarly meaningless content and eventually leading to a collapse of the entire reasoning chain. We call this phenomenon catastrophic collapse. In such cases, the model's 's overall performance may even fall below that of a single agent Wi. A common instance of this phenomenon is rejection response. If CU, includes rejection cues such as "I don't 't know the answer to the query" or "not mentioned". W, is likely to repeat
Table 18: Comparison between vanilla and CoA with various context lengths using text-bison-324
8k
16k
32k
text- text-bison-393 bison-32k
45.69
53.55
59.14
48.54
Ours (same base)
54.95
60.34
63.11
50.25
22

---

Jo
(a) Vanilla (8k)
(b) CoA (8k)
Figure 7: NIAH PLUS test results of Vanilla model and CoA model. Greener/Redder cells indicate higher/lower performance. CoA with 8k improves the performance of the NIAH test greatly compared with the Vanilla baseline.
Table 19: Comparison of related work and their topologies.
Name or Source
Task or Challenge
Agent Structure
Communication Schema
[14]
Multi-robot planning
Decentralized
Circle
RoCo [45]
Multi-robot collaboration
Decentralized
Planning Pipeline
CoELA [89]
Multi-Agents cooperation
Decentralized
Memory Module
MAD [18]
Improving Factuality
Decentralized
Debate
Reconclie [11]
Reasoning
Decentralized
Round Table
WalkMaze [9]
Long Input
Centralized
Tree Structure
LongAgent [92]
Long Input
Centralized
Tree Structure
RecurentGPT [95]
Long Output
Decentralized
Memory Gating
CoA (Ours)
Long Input
Decentralized
Chain of Agents
similar content without adding any meaningful information, often leading to final output of "not mentioned".
To mitigate this issue, three simple methods can be used: (I) repeatedly rumning the sample until the final result is not a rejection; (2) postprocessing the communication unit to remove irrelevant words; and (3) using prompt engineering instruct worker agents to ignore irrelevant information in CU, We speculate that this issue may be influenced by reinforcement learning from human feedback (RLHF) chat fine-tuniag fine-tuning, causing verbose responses that introduce excessive noise. Further fine-tuning may be needed to address this problem, which we leave to future work.
G Detailed Comparison with Related Works
Table 19 illustrates the comparison of related work on multiagent LLM systems. Broadly, prior work on multi-agent LLM systems for long context tasks has a centralized design, where the sibling worker agents do not communicate during inference the communication unit only propagates between manager and worker, and existing decentralized designs do not extend effectively to long context tasks. To the best our knowledge, CoA is the first to use a decentralized structure on long input tasks and is more effective than baselines such as RAG.
The challenge of long dependency on input remains under-explored (as illustrated in Table 20). Our target is to novelly mitigate challenging context dependency issues in long-conter tasks particularly those requiring complex reasoning. To this end, we explored various structures (WalkMaze, Merge, Debate, Group Discussion, etc.) and found that a variant of decentralized well-existing chain communication and this simple approach can work more effectively than others. Our contribution is to mitigate a challenging issue (long input dependency) with a simple intuitive approach (decentralized chain communication). We want to emphasize that centralized communication and our work solve the long context problems in different ways, and they are not in conflict with each other. They can be merged into much stronger framework (e.g., adding our chain reference stage to mitigate context dependency issues before the stages in LongAgent [92]).
23

---

Table 20: Example for comparison of Centralized and Decentralized models.
Input Source
Who is the grandson of A? Source: [1],12][[11[4] (chunks) Evidence in each chunk: A's husband is DJ, [2: A's son BJ, [3: No evidence), [4: B's son is C]
Question:
Centralized [92]
Round I: Manager: Who is the son of A? Worker with [2]: It is B. Others say unknown Round 2: Manager: Who is the son of B? Worker with [4]: It is C. Others say unknown Final answer: It C.
Decentralized (ours)
Who the grandson A? Worker with [I]: A's husband is (topic exploration), Worker with [2]: A's son is B (answer first hop), Worker with [3]: A's son is B (forward previous evidence), Worker with [4]: A's son is B, B's son Thus, A's grandson C. (complete reasoning) Final answer: is C.
Manager:
Table 20 showcases the comparison between CoA and centralized designs in mitigating agent dependency issues for long inputs. Specifically, previous work for long input tasks, despite showing significant improvements, usually uses disentangling algorithms such question decomposition, to address long-distance dependency in long inputs, and then they use centralized structure to answer disentangled questions one by one. Different from them, CoA does not disentangle the question but leaves the entire question to the agents. Although chain communication is well-introduced in the literature, to the best of our knowledge, CoA the first work to apply decentralized communication to long-term dependency challenges. In addition, CoA is not in conflict with existing work as it can be considered as a plugin for other multi-agent LLM systems to improve them.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim that CoA improves LLM performance long-context tasks accurately reflects our contribution and is supported by experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention that the performance of CoA drops when the answer is in the end of long input context. Also, we show that performance of CoA can be further enhanced via multi-path ensemble and the space for improving is significant.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We display all needed information in the appendix, including hyper-parametery, templates, example output, and details. These ensure reproducing the main results is easy.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
25

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide open access to the data and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
wwriiissssoooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // cttipp.cc/publicc/eee wtwpggddddddeedbbmmimiiiiiia /CodeSubmiieeeeeeeeeee issi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
26

---

Answer: [Yes]
Justification:
We list the dataset details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We fix the temperature of the large language models. Thus, the running results are fixed. For RAG model, we also fix the random seed so that the results can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We conduct time complexity anlaysis to show the computer resources needed. Also, we state each of the API call clearly so that the resource information can be easily accessed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
27

---

9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpps///neuripi htcp/ullic/Bthhccuuuiiieeeeeeeee
Answer: [Yes]
Justification:
We conform with the NeurlPS Code of Ethics including preserving anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We add section in appendix to discus all potential social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
28

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets in our experiments are public and properly credited with license properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "4Wodf8oOqC", "reviewer": "Reviewer_vGWD", "summary": "This paper is about addressing the issue of lengthy inputs when using language models. Predominant approach is RAG, but is hampered by retrieval performance. Window extension extends the architecture of the model to handle lengthy inputs, but doesn’t guarantee that the model is able to extract the relevant information from lengthy inputs. This work proposes a simple approach where text is split into multiple chunks and the chunks are sequentially processed, left-to-right, where information from all chunks observed so far is consolidated into a summary. The response to a query (if there is one) is calculated based on the aggregate summary of the text. Despite the simplicity of the approach, the method performs well across various models and tasks.", "questions": "* How sensitive is the model to choice of the size of chunks?\n* Was the RAG baseline properly optimized? \n* Table 2 should also compare against RAG models. For some practical applications, RAG could be more beneficial due to inference speed.", "limitations": "Limitations were discussed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Idea is quite simple and seems top be effective\n* Positive results on many tasks and benchmarks\n* It is interesting that even long context models capable of processing lengthy inputs benefit from the proposed method, which reinforces the hypothesis that sifting the relevant information from lengthy inputs directly can be difficult. Consistent improvements are observed on Claude models.\n* Appreciate the ablations including effect of direction (left-to-right, right-to-levet) and lost in the middle experiment. \n* Seems to work especially well when the inputs are long.", "weaknesses": "* Technical depth is limited\n* Framing/presentation is somewhat misleading\n  - From Figure 1, is summarization all that’s being done? Is CoA is just a fancy term to describe this sequential summarization process?\n  - ‘Chain of Agents’ makes it sound like there are different agents doing different tasks, which is misleading\n* Fairly local view of performance results. Only comparisons against RAG/Vanilla baselines are presented. How about comparisons to best published numbers on these datasets? \n* Clarity issues\n  - Figure 1 does not explain the architecture well\n  - 127: W1 generates related evidence useful for answering question - Without knowing CU1, the reader cannot verify this. Basically unable to follow the discussion in 126-134 without knowing what CU1,2,3 are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback that has helped us to improve our submission! \n\nW1: **Technique depth**. \n\nThe depth of our work lies in that we propose Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks. It is a training free, task/length agnostic, interpretable, and cost-effective framework. CoA (1) can generalize to many tasks, (2) achieve considerable improvements, (3) while the framework is simple, including diverse communication patterns. Our experiments show that Chain-of-Agents outperforms commonly-used RAG and Long Context LLMs by a large margin despite its simple design.\n\nW2 (1) **Is summarization all that’s being done?** \n\n We would like to clarify that CoA’s applicability is not restricted to the summarization task. While the communicated results summarize the contents in the source, CoA agents play various roles beyond summarization, because they perform content selection, information aggregation, context utilization, and reasoning abilities to solve complex problems from diverse tasks. such as extracting and refining information through agent communication, and performing analysis and reasoning with them. Tables 11-13 exemplify CoA agents' capabilities to generate evidence for answering the query, a summary of chunks, and code comments and usages. \n\nW2 (2) **Naming as Chain of Agents**: \n\nWe name it an \\it{agent} for two reasons. First, chain-of-agents contain two different roles, including manager agent and worker agents. Worker agents analyze, aggregate, and digest segment information, whereas manager agents synthesize these contributions into a coherent final output worker. Second, we employ agents to emphasize “communication” between agents. We will clarify this.\n\nW3: **Comparison with SOTA**: \n\nWe choose RAG and Full-context as baselines because they represent the commonly-used and solid approaches of input reduction and window extension. Thanks for your great advice. We have further added comparisons of CoA with the previous SOTA below, even including the ones requiring training (indicated with *). As can be seen, CoA achieves better or comparable results on all datasets, improving HotpotQA and RepoBench-p by a large margin. The performance on some datasets is much lower than SOTA because the datasets are trained on some domain-specific models, such as the Qasper dataset\n|               | HotpotQA | MuSique  | Qasper | NarrativeQA | Quality  | QMsum      | GovReport | BookSum | RepoBench-P |\n|---------------|----------|----------|----------------|-------------|----------|------------|-----------|---------|-------------|\n| Previous Best | 54.4 [1] | 40.4 [1] | 53.9* [2]      | 26 [1]      | 89.2 [3] | 22.44* [2] | 26.3 [3]  | 18.5*[4]   | 56.47 [1]   |\n| Ours Best     |    62.04 |    42.49 |          38.01 |       25.26 |     83.8 |      17.67 |     26.98 |   17.47 |       73.05 |\n\nW4 **Clarification of Figure 1**: \n\nWe have revised Figure 1 in pdf. The new figure clarifies the working flow and communication units. It is worth noting that the blue boxes on the left are CUs themselves. \n\nQ1 **Context window sizes**: \n\nWe show that the CoA becomes stable when the context window is larger than 8k, We also find it can improve the baseline with various window sizes. As shown in Figure 6, the performance becomes stable when it is larger than 8k. We have further analyzed different agent lengths and reported the scores of text-bison on the NarrativeQA dataset in the following table. Our results show that it can improve the baseline with various window sizes. Thus, we choose 8k for short context models, such as text-bison.  \n\n| Model\\context len | 4k    | 8k    | 16k   | 32k   |\n|-------------------|-------|-------|-------|-------|\n| text-bison-32k    | 45.69 | 53.55 | 59.14 | 48.54 |\n| Ours (same base)  | 54.95 | 60.34 | 63.11 | 50.25 |\n\n\nQ2 **Is RAG well-optimized?**: \n\nThe specific RAG [5] implementation we use is indeed well-optimized. It is the SOTA model from the RAG MTEB leaderboard on Hugging Face, and we follow the recent approaches that combine RAG and LLM for the best prompting. Moreover, the RAG model is fine-tuned on the HotpotQA dataset, making it even better adapted for the tasks as the HotpotQA dataset is evaluated in this paper. Overall, CoA outperforming such an optimized RAG implementation highlights the effectiveness.\n\nQ3 **Time complexity analysis**:\n\n We agree that RAG can be more beneficial due to inference speed. However, it hurts the performance because semantic similarity cannot ensure retrieval of the needed information. It is difficult for RAG models to answer the question that needs multiple reasoning hops (such as multihop-QA) or entire input tokens (such as counting tokens in a passage). We compare the time complexity of RAG with CoA and will add the table in the final version. $k’$ is the chunk size of the RAG model, $k$ is the context window of downstream LLMs, $r$ is the average response length and $n$ is the input source length. We have also compared the time cost and show that RAG is around 30% faster than CoA. Moreover, CoA can be further improved by parallel reading. Please refer to GR 3 for details of inference speed analysis.\n\n|     | Encode          | Decode       |\n|-----|-----------------|--------------|\n| RAG | O(nk') + O(k^2) | O(n) + O(kr) |\n| CoA | O(nk)           | O(nr)        |\n\n\n[1] Jin et al. Llm maybe longlm: Self-extend llm context window without tuning. ICML  2024.\n\n[2] Shaham et al. Scrolls: Standardized comparison over long language sequences. EMNLP 2022.\n\n[3] Shaham et al. Zeroscrolls: A zero-shot benchmark for long text understanding. EMNLP Findings 2023 \n\n[4] Pu D, Demberg V. RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization. NAACL 2024.\n\n[5] Xiao S, Liu Z, Zhang P, et al. C-pack: Packaged resources to advance general chinese embedding. SIGIR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zBGqAVhCOs", "reviewer": "Reviewer_FiRU", "summary": "The paper proposed Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks, where multiple worker agents sequentially comprehend and communicate to handle different segmented portions of the text, and a manager agent, at last, synthesizes these contributions into a coherent final output. The paper conducted experiments on 9 long-context tasks in qa, summarization, and code completion, showing a significant performance gain over vanilla long-context LMs and RAG baselines. The paper also provides abundant analysis.", "questions": "Please see Weaknesses", "limitations": "As the paper also noted, the communication effectiveness and inference efficiency could be further improved.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is intuitive.\n\n- The experiments are comprehensive, covering 6 different LLMs as backbones and evaluated across 9 benchmarks.\n\n- The results are good.\n\n- The paper provided useful analysis for more insights.\n\n- The paper is well-written.", "weaknesses": "- There is no support in the paper for the claim in L35 \"inspired by human-like processing of long-context task\", weakening the motivation.\n\n- Results on BookSum are missing in the main table, and no RAG baselines are provided on this benchmark.\n\n- Though the overall results are promising, I'm curious whether there is some information loss during the sequential \"read and consume\", and how this may affect the performance and the design choice.\n\n- For the case study in Figure 5, for worker 1, there seems to be more than one line of clues to answer the query (a.k.a other space missions such as \"Voyager\"), which may result in a complicated reasoning graph, how is this phenomenon dealt with the proposed method?\n\n- As also noted in the paper, there is no interactive communication between the agents, but only uni-directional and one-time. Whether the approach could be considered \"communication\" is debatable.\n\n- Although the time complexity, in theory, is true for Table 2, how's the actual inference speed on GPU considering the overhead of multiple times of extra prompting and decoding?\n\n- It might be helpful to also discuss the related works on communication between language agents, e.g.\n  - Camel: Communicative agents for\" mind\" exploration of large language model society, NeurIPS 23\n  - Building Cooperative Embodied Agents Modularly with Large Language Models, ICLR 24", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed feedback and for acknowledging the comprehensiveness of our experimental studies. We address the questions and concerns raised below, one by one. \n\nW1: **Human motivation**: To clarify this point, the motivation is that humans would not try to read the whole textbook and then start to do reasoning. Instead, it is better for them to learn one section and do exercises over it then move to the other one with the memory of the previous section because the human has limited working memory [1], similar to LLMs. This inspires proposing an approach based on interleaved reading, as one of the fundamental principles of CoA, rather than putting all information in one window and processing the information (read-then-process).\n\nW2: **BookSum results**: Thanks for the great suggestion. Following the suggestion, we have run the BookSum dataset with Text-bison and Text-unicorn models. As shown in the table below, similar to the results with the LLMs that support longer contexts, CoA outperforms baselines by a significant margin. We will add these results and discussions to the paper.\n\n|      | text-bison | text-unicorn |\n|------------- |-----------|-------------|\n| Vanilla (8k) |      9.13 |        8.15 |\n| RAG (8k)     |      9.38 |        8.01 |\n| CoA (8k)     |     **14.51** |       **14.41** |\n\n\n W3: **information loss**: We have added an analysis to further probe the information loss. For each sample, we compute the highest score between the communication unit and the gold answer, and if this score is higher than the score of the final prediction, we compute the difference and refer this information loss, as formulated below:\n\n$$ Loss = max(0,  \\max_{i=0}^{l} score(CU_i, Y) - score(\\hat{Y}, Y)) $$\n\nThe estimated information loss results of various datasets of CoA withext-bison are as follows:\n\n|                  | HotpotQA | MuSiQue | Qasper | NarrativeQA | QMSum  |\n|------------------|----------|---------|--------|-------------|--------|\n| performance      |   54.87 |  40.38 | 37.03 |      24.04 | 17.01 |\n| Information Loss |    1.46 |   1.65 |  3.92 |       3.88 |  0.91 |\n\nThis table shows that if choosing the communication unit with the highest performance, around 1%-4% performance gain can be obtained, meaning that 1%-4% of the information is lost during the chain communication. While every method might have information loss, COA design ensures final results yielding negligible information loss.\n\nHowever, although sometimes the information is lost, \"it is still acceptable within the overall framework as we tested using all CUs as the input to the manager and observed a 5% performance drop (Page 4, footnote). We leave mechanisms to further mitigate information loss to future work of multi-agent communication.\n\nW4: **Handling complex reasoning**: We do not give LLMs a predefined plan. When the CoA starts to process, the agents automatically explore the nodes and leave the processed information for the next agent. However, we indeed observed some patterns: 1) the first worker usually explores the most number of paths, including 3 or more topics because the agent is not sure about the result and willing to provide more information for the next ones. 2) The final worker usually narrows down to one answer with a much shorter CU, and 3) The speed of narrowing down will be faster for simpler samples, producing a small reasoning graph.\n\nW5: **Agent communication one directional or more**: Thanks for pointing this out. We agree that we have agent-to-agent communication in one direction and the communication is critical for performance. We will clarify the definition in the next version, such as renaming it as a Unidirectional Communication Unit. Regarding the design choices, we used unidirectional rather than bi-directional (or more complex communication) because experiments show that the bi-directional (or more complex communication) communication of two agents sometimes brings more noise and hallucinations thus hurting the performance.\n\nW6 **Actual inference time**: We further analyze the actual time cost of running samples. Please refer to GR 3 for more details.\n\n\nW7 **Related work discussion**: Thanks. We will add the following papers and compare them with them in the next version.\n\n[1] Cowan N. Working memory underpins cognitive development, learning, and education[J]. Educational psychology review, 2014, 26: 197-223.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04J85OS2iS", "reviewer": "Reviewer_FfHC", "summary": "The paper \"Chain of Agents: Large Language Models Collaborating on Long-Context Tasks\" introduces a novel framework called Chain-of-Agents (CoA) to address the challenge of effectively processing long contexts in large language models (LLMs). The CoA framework leverages multi-agent collaboration, where multiple worker agents sequentially handle different segments of the text and a manager agent synthesizes these contributions into a coherent final output. This method aims to mitigate the limitations of input reduction and context window extension strategies by enabling effective information aggregation and context reasoning across various LLMs. The framework is evaluated on a range of long-context tasks, including question answering, summarization, and code completion, demonstrating significant improvements over existing methods.", "questions": "- Could you test the CoA framework on smaller long-context models, such as qwen2-7b and llama3-8b, which both support an 8k context? I am curious to see how these smart, smaller models perform with CoA.\n\n- Regarding token cost, the input tokens are slightly higher than those in baseline methods. What about the output tokens? How are they related to the number of workers ($l$) and how each worker agent summarizes each chunk?\n\n- In Section 5.2, how do you handle the Full-200k method, which is the Vanilla (200k), when processing contexts over 200k length?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Task Agnostic and Training Free**: CoA is a versatile framework that does not require task-specific training or fine-tuning, making it applicable to various long-context tasks without additional training overhead.\n- **Significant Performance Improvement**: The framework shows significant improvements, up to 10%, over strong baselines like Retrieval-Augmented Generation (RAG) and full-context models in multiple long-context tasks, including question answering, summarization, and code completion.\n- **Mitigation of Long-Context Issues**: CoA effectively addresses common issues associated with long contexts, such as the \"lost in the middle\" phenomenon, by assigning each agent a shorter context, thus maintaining focus on relevant information.", "weaknesses": "- **Complexity in Implementation**: Implementing a multi-agent system could be more complex and resource-intensive compared to single-agent systems.\n- **Communication Overhead**: The sequential communication between agents might introduce latency and inefficiencies.\n- **Evaluation Scope**: While the paper evaluates on multiple datasets, more diverse real-world applications could further validate the robustness of CoA, like NIAH test.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. We appreciate your time and efforts spent on this paper. With your insightful suggestions, our paper can improve significantly.\n\n**Complexity in Implementation**\n\nIndeed, one of the design principles behind CoA is to propose a simple yet effective multi-agent system with multiple agents collaborating towards solving a complex task, unlike many other works on multi-agent system design. As can be inferred from the pseudocode in Algorithm 1 in the paper, the proposed approach is very straightforward to implement - with only O(100) lines of code for the end-to-end system. Also, due to its training-free nature, it is not necessary to prepare a training code for it, and the end-to-end system can be effectively operationalized in a highly controllable way. We will also release our code upon acceptance to further help the community implementing the approach. Besides, we compare the time complexity in GR 3, further demonstrating the practicality for deployment, for even scenarios with tight latency budgets.\n\n**Communication Overhead**\n\nWe compare the time cost of full-context input and Chain-of Agents theoretically in a decoder-only setting. We assume the response generated by LLMs contains r tokens on average, the input has n tokens, and the context limit of LLM is k. The time complexity is shown in Table 2 (Appendix A).  As can be seen, the encoding time of CoA is less than Full Context because $k <<  n$ in long context tasks, while they have the same decoding time. This demonstrates the efficiency of CoA compared with the Full-Context baseline.\n\n We have also conducted an experiment of communication cost and latency tests. We run an analysis to show the practical time consumption of the proposed CoA on the HotpotQA dataset. We choose to use LLaMA-3-8b as the backbone model for preventing additional latency due to the network of API queries. As can be seen in the table, Vanilla consumes the least amount of tokens and generates the least of tokens as well. However, it truncates the input thus maintaining a low performance. Although RAG generates fewer tokens than CoA (by adding RAG and downstream input together), it needs to read the whole input by Retrieval models which is also time-consuming. Overall, RAG is faster than CoA by only ~30% in this example.\n\n|              | Running Time (s) | Avg. # of Input  | Avg. # of Output | Avg. # Agent Output |\n|--------------|------------------|--------------|--------------|--------------|\n| Vanilla (8k) |             1.33 |     5,912.85 |         2.40 |         2.40 |\n| RAG (8k)     |             2.41 |    16,479.91 |         2.75 |         2.75 |\n| CoA (8k)     |             3.10 |    10,840.95 |        38.38 |        11.30 |\n\n**Parallel decoding analysis**: For decoder-only LLMs, the agents can run in parallel. Before CUs are produced, agents can start to read their assigned paragraphs and wait for the CUs to come. Unfortunately, current APIs or models do not support such dynamic reading. Thus, we conduct an approximation by asking the model to generate one token for each sample. The output time is shown to be short and negligible, mimicking the encoding time of each sample. We have found that the running time of CoA can be reduced by 57.21% on average, leading to a 1.32-second running time for each sample, closing to the running time of the Vanilla baseline! We will integrate discussions on this speedup approach in the final version of the paper. \n\n\n\n\n\n**Evaluation Scope**:\n\nWe have further conducted evaluations on the NIAH test. We follow the LongAgent [1] paper to run a NIAH PLUS test for evaluating the long context understanding capability of LLMs. Different from the original NeedleInAHaystack test, NIAH PLUS is more challenging because LLMs need to answer the questions rather than simply retrieve the information (needle). The results are shown as follows:\n\n| NIAH Test       | Accuracy |\n|-----------------|----------|\n| GPT-4 (128k)    |    62.00 |\n| LongAgent       |    81.53 |\n| text-bison (8k) |    26.00 |\n| CoA (8k)        |    **97.80** |\nAs can be seen, the CoA greatly increases the accuracy of Text-bison from 26.0% to 97.8%, showing that CoA significantly increases the capability of LLMs to understand long contexts. We also append the figures of NIAH test results in the attached pdf. \n\n[1] Zhao J, Zu C, Xu H, et al. LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration[J]. arXiv preprint arXiv:2402.11550, 2024.\n\n\nQ1 **Performance on Smaller long context models**: We have added experiments with Llama-3-8b on the NarrativeQA dataset, and the results are listed in the table below. As can be seen, the CoA framework on such LLMs can also boost the Vanilla performance a lot by 10.27% and surpass the RAG score by 5.9%! Noting that there is more space to improve since the prompt of LLama-3 is not even adjusted for this experiment. \n\n| Model        | F1    |\n|--------------|-------|\n| Vanilla (8k) | 8.78 |\n| RAG (8k)     | 13.15 |\n| CoA (8k)     | **19.05** |\n\nQ2 **Output Token Cost**: Regarding output tokens, we compute the average token generated by the model in General Response 3. As shown in the table, CoA outputs more tokens than vanilla and RAG baselines because it needs to produce CUs and final results. We found that the relation between generated tokens and workers holds an almost linear correlation, showing that basically, each worker generates the same amount of tokens. Including one more agent will increase around 11.3 total generated tokens.\n\nQ3 **Full-200k Chunking**: It is the same as Vanilla (8k). Please refer to the Algorithm in pdf. We first split the source into $n$ sentences, then we add $i$-th sentence to the input one by one so that the total length is less than the context window limit while $i+1$ sentences are longer than the total length.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "th5Hh8xsfc", "reviewer": "Reviewer_KMmx", "summary": "This paper proposes a new method, \"Chain-of-Agents\" (CoA), to augment the long-context handling capabilities of large language models (LLMs). CoA is a framework designed to enhance the processing of long contexts by sequentially using multiple agents to handle different chunks of input text. In CoA, worker agents manage different segments sequentially and communicate their findings. These findings are then synthesized by a manager agent into a coherent final output, effectively aggregating information and reasoning across the entire context. The author conducted experiments on a wide range of long-context tasks to verify the advantages of CoA.", "questions": "1. How is the chunk split? I did not find a clear description of how the chunk is processed. Could there be a risk of text being abruptly truncated in the middle?\n\n2. As mentioned in the abstract, \"input reduction has no guarantee of covering the part with needed information.\" I am wondering if sequential chunk processing will also encounter these challenges. Could the communication unit potentially drop necessary information if some information is only useful in the context of the following content?\n\n3. In Appendix 1, regarding the calculation of time complexity in lines 592-595, if the model's processing length limit is k and no additional mechanisms are introduced, is the computational complexity of attention calculation O(k^2) or O(n^2)? Please educate me if I miss something important.\n\n4. \"a LLM\" in line 104 -> \"an LLM\"", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Chain-of-Agents method is an interesting approach, and the authors have experimentally validated its effectiveness.\n2. The writing and presentation of the article are great, making it easy to read and follow overall.", "weaknesses": "1. My primary concern is the novelty of this submission. Breaking long texts into multiple chunks and processing them sequentially is a well-established practice in both long-content processing and generation. For example, in RecurrentGPT [1], the authors introduced a long-short memory mechanism to store intermediate states during processing, improving the quality of long content generated by LLMs. This is similar to the communication unit (CU) mechanism mentioned in this paper, but the authors do not sufficiently discuss this similarity. Additionally, previous work such as LongAgent [2] proposed segmenting the input long text into several chunks and assigning them to corresponding members. The authors lack sufficient discussion and experimental comparison on utilizing memory mechanisms or agent mechanisms for processing long information.\n\n2. The baselines used in the author's experiments are relatively weak. While the related works section discusses some studies on long text processing, such as references [9] and [13], the implementation process only compares RAG, Vanilla, and COA. The author primarily compares their own designed merge and hierarchical methods in Agent and mechanisms for processing long texts. So, it is challenging to determine the advantages of the proposed methods over existing strong baselines in the field based on the current experiments.\n\n3. Given the existing work on agents collaborating to process multiple chunks, the authors could enhance the novelty of their submission by delving deeper based on the discussion in Section 5.6. Specifically, they could explore the most effective methods of collaboration in multi-chunk processing and agent cooperation. This could add significant technical depth to the paper.\n\n\n[1] Zhou, Wangchunshu, et al. \"Recurrentgpt: Interactive generation of (arbitrarily) long text.\" arXiv preprint arXiv:2305.13304 (2023).\n\n[2] Zhao, Jun, et al. \"LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration.\" arXiv preprint arXiv:2402.11550 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable suggestions. We provide answers to all your weakness points and questions below. We hope these resolve your concerns.\n\nW1: **Novelty and other baselines**:\n\nWhile chunking the input into multiple segments seems intuitive, the novelty of CoA lies in the chain communication of multiple agents using natural language on inputs, which is not explored by previous studies.\n\nLongAgent [1] does not allow communication between workers, making it hard for each worker to understand what is happening at other agents, hurting the performance of tasks that need long dependency. Besides, the authors of LongAgent state that the decomposition of the question also is a limitation for LongAgents. The failure of decomposition multi-hop or complex questions would directly lead to wrong answers. Additionally, some of the structures of LongAgents are transferable to CoA, such as conflict resolution. In contrast, CoA concentrates on improving the long dependency reasoning (such as multi-hop question answering) and does not depend on the decomposition capability of the manager. \n\nRegarding RecurrentGPT [2], although it uses a chain structure to do long story generation, the task and motivation are different. They focus on memorizing the context to have a better plan to generate a longer story. In contrast, CoA focuses on the long-dependency and complex reasoning of the source text rather than output. \n\nWe also compare the performance of CoA with RecurrentGPT and LongAgent, where COA outperforms LongAgent by 16% on NIAH tasks and outperforms RecurrentGPT by 20% on Hotpot QA . More results can be found in GR 2 and Figure 2 in PDF. The results are listed in the Table:\n\n| LongAgent       | Accuracy |\n|-----------------|----------|\n| GPT-4 (128k)    |    62.00 |\n| LongAgent       |    81.53 |\n| Text-bison (8k) |    26.00 |\n| CoA (8k)        |    **97.80** |\n\n| HotpotQA     |       |\n|--------------|-------|\n| RecurrentGPT | 32.54 |\n| CoA (8k)     | **53.62** |\nWe will include these discussions and comparisons in our paper.\n\nW2: **Stronger baselines**:\n\n We believe the baselines used are already strong given the literature in this direction. We are open to benchmarking on any specific suggestions. We have considered other baselines that we modified as they were weaker. For example, WalkingMaze [5] also split the input into chunks; we found it did not work for our task (less than 40% on HotpotQA). Thus, we modify its structure and build a new baseline named Hierarchical which obtains a much stronger performance (50.62% on HotpotQA). \n\nAlso, as stated in W1, LongAgents and RecurrentGPT performances are much lower than CoA and thus weaker than the multiagent baseline we have used.\n \nWe have further added comparisons of CoA with the previous SOTA below, even including the ones requiring training (indicated with *). As can be seen, CoA achieves better or comparable results on all datasets, improving HotpotQA and RepoBench-p by a large margin. The performance on some datasets is lower than SOTA because the datasets are trained on some domain-specific models, such as Qasper \n|               | HotpotQA | MuSique  | Qasper | NarrativeQA | Quality  | QMsum      | GovReport | BookSum | RepoBench-P |\n|---------------|----------|----------|----------------|-------------|----------|------------|-----------|---------|-------------|\n| Previous Best | 54.4 [1] | 40.4 [1] | 53.9* [2]      | 26 [1]      | 89.2 [3] | 22.44* [2] | 26.3 [3]  | 18.5*[4]   | 56.47 [1]   |\n| Ours Best     |    62.04 |    42.49 |          38.01 |       25.26 |     83.8 |      17.67 |     26.98 |   17.47 |       73.05 |\n\nWe will include WalkingMaze, LongAgents,  RecurrentGPT, and other SOTA models in our final paper. Thank you for the great suggestions!\n\nW3: **More agent collaboration**: \n\nWe aim to emphasize a simple yet effective framework with strong performance to create a higher impact. It is indeed helpful to include more analysis and another complex framework, we prefer to use this paper to demonstrate the potential of chain communication. As mentioned by Reviewer vGWD and FiRU, being simple and intuitive but effective are the first strengths of the proposed framework. \n\nAs described, we have experimented with fairly complex agent collaboration, including bi-directional, permutation, hieratical (multi-layers of information processing), WalkMaze, RecurrentGPT, etc, and the current CoA framework performs the best. In the future, we want to explore more directions, such as high efficiency in communication, complex communication schema, etc. \n\nQ1 **Chunking**: Please refer to the Algorithm in pdf. While there is a small risk of chunking text in the middle, CoA is robust to chunking because the information of previous segments is sent to the next agent for processing, while other baselines such as LongAgents cannot communicate with siblings, losing the context if chunking in the middle.\n\nQ2 **Information loss**: This happens in CoA occasionally and is neglectable.  We have added an analysis to further probe the information loss, as described in W3 of Reviewer FiRU. It shows that COA  ensures final results yielding negligible information loss (only 1%-4%).\n\nQ3 **Time complexity**: It should be O(k^2) because auto-regressive LLMs such as GPT and LLaMA only pay attention to the left side of the input. Thus, each token generated will attend to $k$ tokens [6]. We will clarify this in our paper.\n\n\n[1] Llm maybe longlm: Self-extend llm context window without tuning. ICML  2024.\n\n[2] Scrolls: Standardized comparison over long language sequences. EMNLP 2022.\n\n[3] Zeroscrolls: A zero-shot benchmark for long text understanding. EMNLP Findings 2023 \n\n[4] RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization. NAACL 2024.\n\n[5] Walking down the memory maze: Beyond context limit through interactive reading. arXiv, 2023.\n\n[6] Improving language understanding by generative pre-training. arXiv, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the valuable feedback and comments from reviewers that have helped to improve our paper! We also thank the reviewers for appreciating the intuitive and interesting design of the CoA (Reviewer vGWD, FiRU, FfHC, KMmx), the effectiveness of CoA (Reviewer vGWD, FiRU, FfHC, KMmx), comprehensiveness of the evaluation (Reviewer vGWD, FiRU, FfHC), interesting analysis (Reviewer vGWD, FiRU, FfHC) as well as great presentation and writing (Reviewer FiRU, KMmx). 

We address several key questions in the following paragraphs:

GR 1 **Novelty and Technical Depth**

While chunking the input into multiple segments seems intuitive, the novelty of CoA lies in the chain communication of multiple agents using natural language on inputs, which is not explored by previous studies, demonstrating that such a simple yet effective framework with strong performance can create a higher impact.

LongAgent does not allow communication between agents, making it hard for each agent to understand what is happening at others, hurting the performance of tasks that need long dependency. Besides, the authors of LongAgent state that the decomposition of the input question also is a limitation for LongAgents. The failure of decomposition multi-hop or complex questions would directly lead to wrong answers. Additionally, some of the structures of LongAgents are transferable to CoA, such as conflict resolution. In contrast, CoA concentrates on improving the long dependency reasoning (such as multi-hop question answering) and does not depend on the decomposition capability of the manager. 

Regarding RecurrentGPT, although it uses a chain structure to do long story generation, the task and motivation are different. They focus on memorizing the context to have a better plan to generate a longer story.  In contrast, CoA focuses on the long-dependency and complex reasoning of the source text rather than generation. 

The depth of our work lies in that we propose Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks. It is a training-free, task/length agnostic, interpretable, and cost-effective framework. While the framework is conceptually straightforward, CoA can (1) generalize to many tasks, (2) achieve considerable improvements, and (3) demonstrate diverse communication patterns. Our experiments show that CoA outperforms commonly-used RAG and Long Context LLMs by a large margin despite its simple design. Analysis shows that by integrating information aggregation and context reasoning, CoA mitigates lost-in-the-middle phenomenon effectively, and performs better on longer samples.


GR 2 **Evaluation Scope**

We used nine representative, diverse datasets for three broad categories of long context tasks, and we believe the baselines used are already strong given the literature in this direction. As described in individual responses, we will include comparisons with more baselines including WalkingMaze, LongAgents, RecurrentGPT, and other SOTA models in our final paper. These additional results further corroborate consistent and significant performance improvements of CoA.

We follow the LongAgent paper to run a NIAH PLUS test for evaluating the long context understanding capability of LLMs. Different from the original NeedleInAHaystack test, NIAH PLUS is more challenging because LLMs need to answer the questions rather than simply retrieve the information (needle). The results are shown as follows:

| NIAH PLUS Test  | Accuracy |
|-----------------|----------|
| GPT-4 (128k)    |    62.00 |
| LongAgent       |    81.53 |
| text-bison (8k) |    26.00 |
| CoA (8k)        |    **97.80** |
As can be seen, CoA greatly increases the accuracy of Text-bison from 26.0% to 97.8%, showing that CoA significantly increases the capability of LLMs to understand long contexts.  We also append the figures of NIAH test results in the attached pdf.



GR 3 **Practical Time Complexity**

We run an analysis to show the practical time consumption of the proposed CoA on HotpotQA dataset. We choose to use LLaMA-3-8b as the backbone model for preventing additional latency due to the network of API queries. As can be seen in the table, Vanilla consumes the least amount of tokens and generates the least of tokens as well. However, it truncates the input thus maintaining a low performance. Although RAG generates fewer tokens than CoA (by adding RAG and downstream input together), it needs to read the whole input by Retrieval models which is also time-consuming. Overall, RAG is faster than CoA by only ~30% in this example.

|              | Running Time (s) | Avg. # of Input  | Avg. # of Output | Avg. # Agent Output |
|--------------|------------------|--------------|--------------|--------------|
| Vanilla (8k) |             1.33 |     5,912.85 |         2.40 |         2.40 |
| RAG (8k)     |             2.41 |    16,479.91 |         2.75 |         2.75 |
| CoA (8k)     |             3.10 |    10,840.95 |        38.38 |        11.30 |

**Parallel decoding analysis**: For decoder-only LLMs, CoA agents can run in parallel. Before CUs are produced, agents can start to read their assigned paragraphs and wait for the CUs to come. Unfortunately, current APIs or models do not support such dynamic reading. Thus, we conduct an approximation by asking the model to generate one token for each sample. The output time is shown to be short and negligible, mimicking the encoding time of each sample. We have found that the running time of CoA can be reduced by **57.21%** on average, leading to a 1.32-second running time for each sample, closing to the running time of the Vanilla baseline! We will integrate discussions on this speedup approach in the final version of the paper.

Meta-review: In this work, the authors propose Chain-of-Agents (CoA), a training-free framework that leverages multi-agent collaboration through natural language to solve long context tasks. Specifically, CoA first split long text into multiple chunks, then each text chunk is processed in a sequential manner (from left to right). At any step, with the awareness of the query, information from all chunks observed so far is consolidated into a summary, and being passed along to the agent processing the next chunk. Finally, a manager agent performs a final information integration and subsequently generates the response. 

This work has many merits. Reviewers agree that the idea is intuitive, simple yet effective; the method is task agnostic and training free, makes it accessible; the authors conducted a comprehensive set of experiments, on 9 benchmarks with 6 LLM backbones, CoA show strong experimental results on all tasks, this further highlighted the accessibility of the work; reviewer find the authors' analysis useful and insightful; they find the paper well-written, with clear presentation; they believe the work provides interesting insights, the method can be empirically impactful. The all-positive scores suggest the reviewers' overall sentiment to this work.

Before the author-reviewer discussion period, reviewers' concerns were mainly about:
- It was unclear how this work connects/differs from prior works such as LongAgent and RecurrentGPT where they also have similar system design that breaks long text into multiple chunks. 
- Reviewers felt unsatisfied about the baselines because the work only compared with RAG and Vanilla LLMs, stronger baselines were missing.
- The details of the agent communication was unclear.
- The inference speed/latency/efficiency was unclear.
- There lacks of evidence how CoA could generalize to smaller/weaker LLMs (with long context support).

The author did a great job addressing the concerns and answering the questions during the discussion period, they successfully convinced reveiwer FfHC to raise their score from 4 to 5; and reviewer KMmx to raise their score from 3 to 5. They both acknowledge that the authors had addressed their concerns effectively/adequately.

During the discussion period, the authors made their work stronger, thanks to the reveiwers' suggestions:
- It is much clearer now why CoA is unique among prior/concurrent work that has similar design. The table provided in their response to Reviewer KMmx is very helpful, and should be included in their cameraready.
- The additional experiments using LLaMA-3-8b as backbone made it more convincing that CoA can be applied to open-sourced models.
- The authors additionally compared their work on all the 9 datasets with previous best systems, they show that CoA could achieve better or comparable results on all datasets (comparable even with prior systems that require additional in-domain training). 
- The authors additonally tested CoA against a set of baselines on an extreme test, the Needle in a Haystack (NIAH) test set, as suggested by Reviewer FfHC. CoA achieves near perfect performance and outperforms other systems by a large margin. 

Overall, I believe if the authors could well integrate the additional clarification and experiments/analyses into their cameraready, this work could be well received by the NeurIPS community. Although this work presents a training-free method (which I personally less in favor of), but I like the fact the method is straightforward and effective, plus the authors did a good job justifying how this simple method could be applied to a potentially wide scenario, I do see good empirical impact this work could offer.
Decision: Accept (poster)