Chain of Agents: Large Language Models Collaborating on Long-Context Tasks

Authors: yusen  zhang
sun ♢ ruoxi
yanfei  chen
tomas  pfister
rui  zhang
sercan ö arık
robert aldrich vera
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:06:19

Abstract: Addressing the challenge of effectively processing long contexts has become a critical issue for Large Language Models (LLMs). Two common strategies have emerged: 1) reducing the input length, such as retrieving relevant chunks by Retrieval-Augmented Generation (RAG), and 2) expanding the context window limit of LLMs. However, both strategies have drawbacks: input reduction has no guarantee of covering the part with needed information, while window extension struggles with focusing on the pertinent information for solving the task. To mitigate these limitations, we propose Chain-of-Agents (CoA), a novel framework that harnesses multi-agent collaboration through natural language to enable information aggregation and context reasoning across various LLMs over long-context tasks. CoA consists of multiple worker agents who sequentially communicate to handle different segmented portions of the text, followed by a manager agent who synthesizes these contributions into a coherent final output. CoA processes the entire input by interleaving reading and reasoning, and it mitigates long context focus issues by assigning each agent a short context. We perform a comprehensive evaluation of CoA on a wide range of long-context tasks in question answering, summarization, and code completion, demonstrating significant improvements by up to 10% over strong baselines of RAG, Full-Context, and multi-agent LLMs.

Main paper: 
 introduction: 
 Despite their impressive performance across a wide range of scenarios, LLMs struggle with tasks that involve long contexts [8, 63, 57] . Numerous application scenarios demand extremely long contexts, such as question answering [85, 22, 69] , document and dialogue summarization [25, 93, 91, 90, 12] , and code completion [20, 43] , where the inputs contain entire books [32, 33] and long articles [16] .To tackle the challenge with long context tasks, two major directions have been explored as shown in Table 1 : input reduction and window extension. Input reduction reduces the length of the input context before feeding to downstream LLMs. Truncation approaches [1, 67] directly truncate the input. Retrieval Augmented Generation (RAG) [81] extends this direction by retrieving the most relevant chunks through embedding similarity. However, because of low retrieval accuracy, LLMs could receive an incomplete context for solving the task, hurting performance. Window extension extends the context window of LLMs via finetuning to consume the whole input [13, 44, 48] . For example, Claude-3 [5] directly allows reading 200k tokens for each input. However, when the window becomes longer, LLMs struggle to focus on the needed information to solve the task, suffering from ineffective context utilization such as the "lost in the middle" issue [37, 3, 42] .Figure 1 : Overview of Chain-of-Agents, a training-free, task agnostic, and highly-interpretable framework that harnesses multi-agent collaboration for long-context tasks. Blue boxes indicate communication unit CU i between worker agents W i and W i+1 . It consists of multiple worker agents who sequentially communicate to handle different segmented portions of the text, followed by a manager agent who synthesizes these contributions into a coherent final output.Table 1 : Comparison between Chain-of-Agents and prior methods for long-context tasks. Rec./Foc.: being able to mitigate inaccurate receptive field/long context focusing issues. Read: the number of tokens as model input, where n is the total input length, k is the context window limit of LLMs. Inter.: the interpretability of the approach. Note that RAG is 'medium interpretable' because of the re-ranked chunks. Motivated by the aforementioned challenges, we propose a novel framework, Chain-of-Agents (CoA), inspired by the way humans interleave reading and processing of long contexts under limited working memory constraints of brains [15] . The key idea of CoA is to harness multi-agent communication to enable information aggregation and context reasoning capabilities across different LLMs. As shown in Figure 1 , CoA contains two stages. In stage 1, a series of worker agents in charge of different chunks of long context collaborate and aggregate evidence for answering the given query. To this end, the workers read and process sequentially, each receiving the message from previous worker and transferring the useful updated information to the next. In stage 2, the manager agent receives the complete evidence from last worker agent and generates the final response.As shown in Table 1 , CoA is a training free, task agnostic, and highly interpretable framework processing entire "receptive field" by interleaved reading-processing and mitigating the long context focusing issue by assigning each agent a short context. Different from input reduction where LLMs need to start processing with low receptive field over reduced inputs ("read-then-process"), workers in CoA start to process each chunk before reading all input ("interleaved read-process"), tackling the problems that input reduction struggles with, such as, generic summarization or counting of passages [6] . Different from context extension, CoA leverages the capability of communication rather than trying to feed many tokens into an LLM. This is a more natural solution for complex tasks because we assume that each LLM has its limit and there are always complex context tasks surpassing its limit. Compared with Full-Context, CoA is also cost effective by reducing time complexity from n 2 to nk, where n is input tokens and k is the context limit of LLMs.We conduct intensive experiments on nine datasets, including question answering, summarization, and code completion tasks with six LLMs, with PaLM 2 [4] , Gemini [67] , and Claude 3 [5] models. We compare CoA with two strong baselines chosen from input reduction and window extension approaches, respectively: (i) RAG, which uses a state-of-the-art retriever to obtain the most relevant information to feed into the LLM and (ii) Full-Context (Vanilla), which feeds all input into the LLM until reaching the window limit. Our results show that on all nine datasets, CoA obtains significant improvement over all baselines by up to 10%. Noting that there is not enough research on multi-agent for long context tasks, we carefully create two multi-agent baselines, including a hierarchical structure and result merging approach to further demonstrate that CoA is superior among other possible multi-agent frameworks. 
related work: 
 Multi-agent LLMs. Multi-agent LLMs has become a popular topic [21] . A large proportion of works focus on social simulation [52, 53] and frameworks [35] . Based on the success of them, some works explore the game settings [39, 74, 82, 83, 47] , world wars [23] , economy markets [36, 76] , recommendation systems [88] , and pandemics [19] . Others advance problem solving, focusing on reasoning of short text via multi-agent debating [18, 80, 10, 66] and discussing [11, 58] for different tasks in reasoning [18, 66] , mechanics problems [49] , paper review [84] , knowledge graph construction [86] , and code intelligence [70, 24] . Different from the above works, we improve problem-solving on long context tasks. To the best of our knowledge, the closest work LongAgent [92] utilizes a tree structure to do multi-hop QA over a long input context and the sibling worker agents do not communicate, while CoA utilizes a chain structure allowing the communication units to flow between workers (detailed comparison in Appendix G and F.3).Long Context Modeling for LLMs. Input Reduction: RAG is broadly leveraged to solve long context query-based tasks [81, 2] . Combined with a strong retriever [79, 41, 73] , LLMs are expected to handle long context questions in open domains. Previous studies have augmented LLMs during pretraining [26, 72] , finetuning [34] , inference [87] , or directly integrating [28, 61] . Moreover, some token-level retrieval approaches are proposed [38] . Longllmlingua [27] removes tokens from long prompt to compress long context prompt to a desired budget. Window Extension: The context windows of LLMs are getting longer and longer thanks to the development of GPUs. For instance, the context window increases from 1024 (GPT-2 [56] ), 2048 (GPT-3 [7] ), to 128k (GPT-4 [50] ). Moreover, the newest version of Claude-3 [5] supports 200k context windows. To save the cost of LLM training, some continual learning or finetuning approaches are proposed to extend the context window of pretrained LLMs [46, 55, 44, 48] . For instance, position interpolation [13] modifies rotary position encoding [64] and extends the context length of LLaMA [68] to 32k. Different from the above works, CoA does not reduce the input length or extend the window length of LLMs, but rather leverages multi-agent collaboration and communication to obtain the full receptive field.Recently, a few studies have employed text chunking algorithms to divide and process lengthy text inputs. Notably, RecurrentGPT [95] adopts a chain structure to generate long outputs in a chunkby-chunk manner, maintaining continuity in the storyline. WalkMaze [9] utilizes a centralized tree structure without communication between sibling agents to do single-hop QA over a long input context. Different from these works, CoA focuses on long input tasks with a chain structure with inter-sibling communication (Append G).Complex Task Reasoning. Previous works on complex reasoning have focused on decomposing the complex question into sub-questions to solve them step-by-step. [54] decompose the questions with an unsupervised model and answer them separately with another model. Decomposed Prompting [30] leverages some predefined modules to classify each decomposed sub-question, then further decompose if needed. Additionally, decomposing is used for human-computer interaction [77] , and prompter training [71] . Recently, many work has been proposed for LLMs, such as Chain-of-thought [75] Least-to-most prompting [94] and Pearl [65] . However, the length of the prompt does not exceed the context limit of a single agent. By contrast, our Chain of Agents framework is proposed to effectively reason across multiple agents to support the unlimited length of source text.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We mention that the performance of CoA drops when the answer is in the end of long input context. Also, we show that performance of CoA can be further enhanced via multi-path ensemble and the space for improving is significant.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Figure 1 shows the overview of our Chain-of-Agents (CoA) framework, containing two stages. In stage 1, long context is split into chunks where each chunk can be processed by a worker agent. Then, the worker agents communicate sequentially to produce evidence over the entire context. In stage 2, a manager agent consumes the knowledge from the chain of workers to generate the final answer.Algorithm 1 Chain of Agents (CoA).Input: Source input x, query q, agent window size k, large language model LLM( * ). Output: Answer to the query.Splitx into l chunks {c1, c2, • • • , c l } where ci is shorter than k Initialize CU0 ← empty string. for i in 1, 2, • • • , l do CUi ← LLMW i (IW , CUi-1, ci, q) end for return LLMM (IM , CU l , q)To formulate the task, we denote a long-context sample as (x, y, q), where x is the input of n tokens, y is the output of m tokens, q is an optional query. Given an LLM with k tokens (usually k ≪ n) as the context window limit, the target is to generate y with the limited input context window. Therefore, we divide each source text x into chunks x = {c 1 , c 2 ...c l }, so that each chunk can be completely fed into the LLM agent backbone model.In Stage 1, CoA contains a sequence of l number of worker agents. Each worker W i inputs the concatenation of a chunk c i from source text x, a query q, instruction for a specific task for worker agent I W , and the message passed from the previous agent, denoted as "communication unit" CU i-1 . The communication is unidirectional, passing from worker agent i -1 to i. The worker agents process them and output the message CU i for next worker, expressed as:EQUATIONCUs produced by worker agents vary across different tasks. For question answering, CU contains the evidence for the manager to answer the question. For summarization, CU contains the summary of the previous texts. For code completion, CU contains the code summary with function/class names and explanation. Effectiveness on diverse tasks demonstrates the flexibility of CoA (Appendix C).The multi-step worker communication in CoA expands the model context to the full receptive field, meaning that the last worker can read the full input no matter how long the input is. Therefore, CoA is extensible to inputs with different lengths by adjusting the number of worker agents.The left side of Figure 1 underscores the necessity of collaborative communication among workers to effectively address complex, long-context reasoning tasks. We observe that 1) Although the question is unanswerable given c 1 , W 1 generates related evidence that is useful for answering the question; 2) with the partial answer from the previous worker, W 2 further reasons with the current source to complete the full reasoning chain across agents and generate the interpretative reasoning chain; 3) W 3 finds no related information in the chunk 3, it directly rewrites CU 2 by putting the correct answer as the first token of CU 3 without adding any unrelated information (Appendix F.4). This shows that if workers are independent (such as tree structure communication), it is impossible to answer hop two while the answer of hop one is held by another worker (Appendix G).In Stage 2, after multiple steps of information extraction and comprehension by worker agents, the manager agent produces the final solution. While worker agents extract relevant information in a long-context source, the manager agent synthesizes relevant information accumulated by the end of "worker-agent-chain" to generate the final answer. Specifically, given the instruction for manager I M and query q, the manager agent consumes accumulated knowledge from last worker CU l and generates the final answer Response:Response = LLM M (I M , CU l , q)(2) The benefit of using a separate LLM as the manager agent is to decompose the duty of analyzing chunks in the long-context source ("worker agents") and producing the final answer ("manager agent"), so that every agent can fulfill its duty to the most3 . Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment setup: 
 Datasets. We conduct experiments on nine long context datasets across three task types (Table 3 ):• Question Answering. We consider five QA datasets from the LongBench [6] and SCROLL [60] .HotpotQA [85] QuALITY [51] is a dataset based on stories and articles with multiple-choice questions for each sample. The model needs to select the correct answer among choices. • Summarization. We pick two summarization datasets from SCROLLS. QMSum [93] is a querybased summarization dataset, formed by meeting transcripts from multiple domains such as academic and industrial products. GovReport [25] is a generic summarization dataset containing long reports published by the U.S. Government Accountability Office. We also use one dataset for long context memorization tasks. BookSum [33] is a collection of datasets for long-form narrative summarization, including novels, plays, and stories. We use the book-level partition of the BookSum dataset for experiments. • Code Completion. We pick RepoBench-P [43] which is collected from GitHub repositories, and the model needs to generate the next line of code given the long code base.Metrics. We report the geometric mean of ROUGE [40] for Summarization tasks, code similarity score [6] for Code Completion task, exact match for QuALITY dataset [60] , and F1 score for the rest of the Question Answering datasets [6] .LLMs. We use six LLMs in total as the backbone of CoA across all experiments (Appendix B).PaLM 2 [4] is a series of models with a dense left-to-right, decoder-only language model pretrained on a high-quality corpus of 780 billion tokens. We use text-bison@001 and text-unicorn@001 for the experiments with an 8k maximum context window. Gemini 1.0 [67] is a family of LLMs proposed by Google. We use gemini-ultra for experiments. The input limit is 32k tokens for Gemini.Claude 3 [5] is a family of large language models developed by Anthropic. The family includes three state-of-the-art models in ascending order of capability: claude-3-haiku, claude-3-sonnet, and claude-3-opus. These models are capable of consuming 200k tokens in the context window, providing a strong baseline for long context tasks. Although our framework is flexible to use diverse types of LLMs as workers and manager, we use the same model for each W i and M if not specified. The other one is Retrieval-Augmented Generation (RAG). We use the state-of-the-art retriever [79] .Following [81] , we first segment the source text into chunks of 300 words and re-rank them using a retriever. Top-n chunks are then fed into the downstream LLM until the context window is fully utilized. GovReport dataset does not contain a query initially, we create a pseudo query "What is the summary of the whole government report?" as the query to rerank.To evaluate the performance of CoA compared with possible multi-agent approaches, we carefully construct two approaches. Similar to CoA, we also assign each chunk c i to W i using similar instructions to generate CU i . In these approaches, worker agents are parallel and independent while CoA is sequential. Multi-Agent Voting (Merge) Each agent directly generate an answer a i according to c i . A majority voting is applied to all answers a i to decide the final answer. Multi-Agent Hierarchical Structure (Hierarchical). Inspired by [9] , we propose a hierarchical framework, where the communication forms a tree structure between workers W i and manager M . For each worker, it first judges whether c i contains useful information. If true, it generates a communication unit CU i . Then, all CU i are sent to the manager M to come up with a final answer. Besides, we append an integer number L at the end of every approach to clearly remind the window size limit of LLM. For instance, "CoA (8K)" refers to the base LLM used in CoA with window size 8K.Question Answering. Table 4 shows the results of Question Answering tasks on all three models. CoA (8k) outperforms Vanilla (8k) by a large margin on all 8 datasets, including 13.30% on NarrativeQA, 12.82% on MuSiQue, and 22.00% on Quality, for text-bison, text-unicorn, and geminiultra, respectively. Also, CoA (8k) outperforms RAG (8k) model for all 8 datasets using all three LLMs, demonstrating that CoA achieves higher performance than RAG. In other words, using multiagent LLMs outperforms RAG models. It is also worth noting that for gemini-ultra, Vanilla (32k) improves the Vanilla (8k) baseline, yet it is still lower than CoA (8k). CoA also obtains compatible performance with with state-of-the-art models (Appendix E), while keeping robustness of window size (Appendix F.2).Summarization and Code Completion. Table 4 shows the results of Summarization and Code Completion tasks. Similarly, CoA (8k) also outperforms all Vanilla (8k) and (32k) baselines on all three datasets, demonstrating the strong capability of CoA on various tasks. It is worth noting that for GovReport, RAG fails to improve the baseline with pseudo query. By contrast, CoA improves the performance significantly, showing that CoA can also be applied in non-query tasks (Appendix F.1).Long Context LLMs. As Claude 3 models support 200k of tokens, we call these models long context models (LCM). Table 5 shows the performance of the LCM on two datasets. As can be seen, CoA (8k) outperforms Vanilla (200k) significantly, showing that with only an 8k context window, CoA achieves a much higher performance than LCM with a 200k context window. Also, CoA improves the performance with the samples that can be fed into a 200k context window (no truncation). Moreover, the improvements over the Vanilla (200k) and RAG (8k) become higher when the model This demonstrates that CoA benefits from stronger models to achieve higher improvements.Other Multi-Agent Frameworks. As shown in Table 6 , Hierarchical (8k) outperforms Vanilla (8k) on five out of eight datasets, demonstrating the hierarchical approach can also improve the vanilla baselines significantly. Merge (8k) is lower than Vanilla (8k) except for GovReport, showing that merging is especially effective for long summarization tasks such as GovReport. As can be seen, CoA outperforms Hierarchical and Merge on all eight datasets. The reason behind the results is because Hierarchical and Merge do not allow workers to communicate with each other due to their parallel designs. Thus, each worker can only maintain the information in its own chunk c i which blocks the understanding of the whole text, hurting the performance greatly.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We display all needed information in the appendix, including hyper-parameters, templates, example output, and details. These ensure reproducing the main results is easy. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
coa improvement is more obvious when rag fails to retrieve gold answer: 
 To demonstrate this, we first classify the samples in NarrativeQA dataset into different bins according to the position (index) of the chunk in RAG processed input that contains the gold answer. Then, we compute the average score of the CoA and RAG results of different bins. Figure 3 shows the results.As shown in the figure, RAG performs better when the index is smaller (the gold answer is nearer to the top), showing that downstream LLMs rely significantly on the quality of RAG re-ranking.Besides, the performance of RAG is positively correlated to CoA's when it successfully retrieves the gold answer. However, when RAG fails, CoA can greatly improve the performance (much higher than the tendency line). We compare the performance of CoA and Vanilla with Claude 3 on BookSum. As shown in Figure 2 , CoA can outperform the vanilla baseline by a large margin on various source lengths. It is worth noting that, when the length of the sample increases, the performance even increases for CoA, and the improvement over Vanilla (200k) baseline becomes more significant. The improvement of CoA reaches around 100% when the length is larger than 400k. Thus, we can conclude that 1) CoA can still enhance the LLM performance even though the model has a very long context window limit; and 2) CoA delivers more performance gains when the input is longer.To assess the "lost-in-the-middle" [42] effect on Vanilla and CoA models, we replicated the original study by randomly selecting 500 samples from their dataset to create a QA dataset. The results are displayed in Figure 4 . The Vanilla model exhibits a significant "lost-in-the-middle" issue, with a performance range of 6.13 (±2.17). In contrast, CoA shows resilience against this issue, with a narrower performance gap of 4.89 (±1.91), demonstrating that CoA effectively mitigates this problem by providing each agent a shorter context to focus on.Figure 5 displays a sample prediction from HotpotQA. To find the correct answer, RAG retrieves text chunks with high semantic similarity with the query. However, conducting multi-hop reasoning is challenging as the critical first-hop answer often lacks semantic relevance to the query. In contrast, CoA operates differently: the first agent explores related topics without knowing the query's answer, aiding subsequent inference. The second agent, also unaware of the answer, broadens the topic scope by incorporating new information. The third agent finally discovers the answer, synthesizing information from earlier agents and new data to complete the reasoning chain. This collaborative approach highlights CoA's ability to facilitate complex reasoning across long context tasks.To demonstrate the effect of the manager, we conduct an ablation study that uses the last worker to generate results directly. As shown in Table 7 , "w/o Manager" hurts the performance significantly, dropping more than 10% on MuSiQue. This demonstrates the important role of the manager. Next, to empirically verify that left-to-right yields the best performance, we evaluate other orders of reading, including Right-to-Left by reading from the last chunk to the first one and Permutation which reads in random order. As shown in Table 7 , on most of the datasets, left-to-right yields the highest score, demonstrating the advantages of natural reading order.We manually investigated the results over these three orders (left-to-right, right-to-left, permutation), and we found that other orders sometimes can produce better answers than left-to-right. Inspired by this observation, we explore two approaches to select the best result among multiple paths. w/ vote applies majority voting over the final results while w/ judge uses an LLM to judge the most reliable CU l of diverse paths and generate the final answer. Oracle picks the best path by evaluating score of each path, yielding the upper bound performance. Table 8 compares three multi-path augmentation approaches. Surprisingly, results show that 1) all ensemble approaches (Bi-direction, Self-consistency, and Permutation) can further enhance the performance of CoA and 5-way Permutation yields the best improvement, 2) majority voting (w/ vote) of final answer is better than using an LLM as judge (w/ judge) in Self-consistency, but worse in Bi-direction, 3) using LLM judge (w/ judge) works well on long result generation tasks (QMSum, RepoBench-P), and 4) there is large space to improve because oracle (choose as answer the one with highest performance) is much higher than either w/ judge or w/ vote. We leave the direction of multi-path reasoning to future study.We run an analysis to show the practical time consumption of the proposed CoA on HotpotQA dataset. We choose llama3-8b as the backbone model for preventing additional untrackable latency due to the 9 , Vanilla consumes the least amount of tokens and generates the least of tokens as well. However, it truncates the input thus maintaining a low performance. Although RAG generates fewer tokens than CoA (by adding retrieval and downstream input together), it needs to read the whole input by Retrieval models which also consumes additional time. Overall, RAG is faster than CoA by around 30% in this example. Parallel decoding. For decoder-only LLMs, CoA agents can run in parallel. Before CUs are produced, agents can start to read their assigned paragraphs and wait for the CUs to come. We conduct an approximation by asking the model to generate one token for each sample. The output time is shown to be short and negligible, mimicking the encoding time of each sample. We have found that the running time of CoA can be reduced by 57.21% on average, leading to a 1.32-second running time for each sample, closing to the running time of the Vanilla baseline.To probe the information loss during information propagation in CU i , we further propose a metric to analyze. For each sample, we compute the highest score between the communication unit and the gold answer, and if this score is higher than the score of the final prediction, we compute the difference and refer to this information loss, formulated as Infomation_Loss = max(0, max l i=0 score(CU i , Y )score( Ŷ , Y )) where score(*) is the performance metric for each task and dataset, Y is the gold answer and Ŷ is the prediction. Table 10 shows the information loss of text-bison. If choosing the communication unit with the highest performance, around 1%-4% performance gain can be obtained, meaning that 1%-4% of the information is lost during the chain communication. While every method might have information loss, CoA design ensures final results yielding negligible information loss. 
conclusion: 
 In this paper, we propose Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks. It is a training free, task/length agnostic, interpretable, and cost-effective framework. Experiments show that Chain-of-Agents outperforms RAG and Long Context LLMs by a large margin despite of its simple design. Analysis shows that by integrating information aggregation and context reasoning, CoA mitigates lost-in-the-middle and performs better on longer samples.Limitations. While CoA features with a simple and effective design, future directions can address the following limitations to further improve its prowess and efficiency. First, communication effectiveness can be further improved via finetuning or in-context learning because current LLMs are aligned with human norms which is not optimal for communication between LLMs. Second, CoA does not explore other forms of communication approaches, such as debating or complex discussions. Third, the cost and latency of running CoA can be further reduced, such as replacing some LLMs with more effective models via model routing [62] . Here is the summary of the previous source text: {Previous Communication Unit (CUi-1)} You need to read the current source text and summary of previous source text (if any) and generate a summary to include them both. Later, this summary will be used for other agents to generate a summary for the whole text. Thus, your generated summary should be relatively long. Manager M : {Task specific requirement} The following are given passages. However, the source text is too long and has been summarized. You need to answer based on the summary: {Previous Communication Unit CU l } Answer:Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We list the dataset details in the appendix. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We fix the temperature of the large language models. Thus, the running results are fixed. For RAG model, we also fix the random seed so that the results can be reproduced. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We conduct time complexity anlaysis to show the computer resources needed. Also, we state each of the API call clearly so that the resource information can be easily accessed.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We pick one sample from HotpotQA in Table 13 , QMSum in Table 14 , and RepoBench-P in Table 15 to show the prediction of Chain-of-Agents, in the order of question q, Communication Unit CU of agents, and final prediction of manager. We also observe that 1) the first worker usually explores the most number of paths, including 3 or more topics because the agent is not sure about the result and willing to provide more information for the next ones. 2) The final worker usually narrows down to one answer with a much shorter CU, and 3) The speed of narrowing down will be faster for simpler samples, producing a small reasoning graph. Question Summarize the discussion about industrial components.Worker 1 The industrial designer proposed that the remote control should be made of titanium or rubber, and the buttons should be simple. The marketing expert said that the remote control should be fancy, innovative, and easy to use. They also discussed the trend of making products look like fruit and vegetables.Worker 2 The industrial designer proposed that the remote control should be made of titanium or rubber, and the buttons should be simple. The marketing expert said that the remote control should be fancy, innovative, and easy to use. They also discussed the trend of making products look like fruit and vegetables. The industrial designer said that they would explore the two options of titanium and rubber, and see if rubber is expensive. The user interface designer said that they would pretend that the modelling clay is titanium and paint it afterwards. The project manager said that they would stick with two batteries and not reinvent the wheel.Manager The industrial designer proposed that the remote control should be made of titanium or rubber, and the buttons should be simple. The marketing expert said that the remote control should be fancy, innovative, and easy to use. They also discussed the trend of making products look like fruit and vegetables. The industrial designer said that they would explore the two options of titanium and rubber, and see if rubber is expensive. The user interface designer said that they would pretend that the modelling clay is titanium and paint it afterwards. The project manager said that they would stick with two batteries and not reinvent the wheel. (ROUGE Score: 21.38)The industrial designer and the marketing manager discussed the industrial components of the remote control. They agreed that the remote control should have a simple design, with a few buttons and a speech recognition technology. They also discussed the possibility of adding a L_C_D_ display, but decided that it would be too expensive. The designer suggested that the remote control could be made of titanium or rubber, and the marketing manager agreed that a spongy design would be more appealing to consumers. (ROUGE Score: 16.54)The industrial designer provided several options respectively for energy, material and interface. Among these options, the industrial designer preferred traditional battery, titanium and simple push buttons so that they would have enough money for speech recognition. After that, the user interface designer proposed an LCD display only for output, which might be taken into account. The group also talked about how to enable the users to find their controllers. Chain-of-Agents is a generic framework for long context tasks. users can apply this to diverse tasks not restricting to the mentioned ones. It will greatly increase the efficiency of individuals or companies to solve complex long context tasks. Besides, the interpretablity of such approach can reduce the misuse of the LLMs because users can check the correctness of results and decrease the possibility of making faults. However, similar to all prompt based approaches, this framework requires careful prompt design for unseen large language models, users may not get optimal solution on certain newly proposed LLMs. Besides, it may increase the number of the calls for API, causing higher network traffic and higher latency for user pools.Table 16 shows the comparisons of CoA with the previous state-of-the-art performance, including the ones requiring training (indicated with *). As can be seen, CoA achieves better or comparable results on all datasets, improving HotpotQA and RepoBench-p by a large margin. The performance on some datasets is lower than that of state-of-the-art models because the datasets are trained on some domain-specific models, such as Qasper.We have run the BookSum dataset with Text-bison and Text-unicorn models. As shown in the Table 17 , similar to the results with the LLMs that support longer contexts, CoA outperforms baselines by a significant margin.We set the default context window of CoA to 8k due to the limitation of text-bison and unicorn models. To test the influence of CoA against context window change, we set window sizes to 4k, 8k, We further analyze different agent lengths and the scores of text-bison-32k in Table 18 . Our results show that it can improve the baseline with various window sizes. Thus, we choose 8k for short context models, such as text-bison.We conduct evaluations on the NIAH test. We follow the LongAgent [92] paper to run a NIAH PLUS test for evaluating the long context understanding capability of LLMs. Different from the original NeedleInAHaystack test, NIAH PLUS is more challenging because LLMs need to answer the questions rather than simply retrieve the information (needle). The results are shown in Figure 7 .As can be seen, the CoA greatly increases the accuracy of Text-bison from 26.0% to 97.8%, showing that CoA significantly increases the capability of LLMs to understand long contexts.Experiments show that when certain irrelevant content is present in the communication unit CU i-1 , it often prevents the worker unit W i from generating a meaningful response. This meaningless response then propagates to the next unit, causing W i+1 to produce similarly meaningless content and eventually leading to a collapse of the entire reasoning chain. We call this phenomenon catastrophic collapse. In such cases, the model's overall performance may even fall below that of a single agent W 1 . A common instance of this phenomenon is a rejection response. If CU i-1 includes rejection cues such as "I don't know the answer to the query" or "not mentioned", W i is likely to repeat NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Other design choices: Our experiments show that using the last worker W l to directly generate the final result leads to a performance drop. Besides, feeding the manager with all CUi or some CU that is related to the answer (decided by Wi) also hurts the performance because of confusion led by conflicting CUi. 


Decision: In this work, the authors propose Chain-of-Agents (CoA), a training-free framework that leverages multi-agent collaboration through natural language to solve long context tasks. Specifically, CoA first split long text into multiple chunks, then each text chunk is processed in a sequential manner (from left to right). At any step, with the awareness of the query, information from all chunks observed so far is consolidated into a summary, and being passed along to the agent processing the next chunk. Finally, a manager agent performs a final information integration and subsequently generates the response. 

This work has many merits. Reviewers agree that the idea is intuitive, simple yet effective; the method is task agnostic and training free, makes it accessible; the authors conducted a comprehensive set of experiments, on 9 benchmarks with 6 LLM backbones, CoA show strong experimental results on all tasks, this further highlighted the accessibility of the work; reviewer find the authors' analysis useful and insightful; they find the paper well-written, with clear presentation; they believe the work provides interesting insights, the method can be empirically impactful. The all-positive scores suggest the reviewers' overall sentiment to this work.

Before the author-reviewer discussion period, reviewers' concerns were mainly about:
- It was unclear how this work connects/differs from prior works such as LongAgent and RecurrentGPT where they also have similar system design that breaks long text into multiple chunks. 
- Reviewers felt unsatisfied about the baselines because the work only compared with RAG and Vanilla LLMs, stronger baselines were missing.
- The details of the agent communication was unclear.
- The inference speed/latency/efficiency was unclear.
- There lacks of evidence how CoA could generalize to smaller/weaker LLMs (with long context support).

The author did a great job addressing the concerns and answering the questions during the discussion period, they successfully convinced reveiwer FfHC to raise their score from 4 to 5; and reviewer KMmx to raise their score from 3 to 5. They both acknowledge that the authors had addressed their concerns effectively/adequately.

During the discussion period, the authors made their work stronger, thanks to the reveiwers' suggestions:
- It is much clearer now why CoA is unique among prior/concurrent work that has similar design. The table provided in their response to Reviewer KMmx is very helpful, and should be included in their cameraready.
- The additional experiments using LLaMA-3-8b as backbone made it more convincing that CoA can be applied to open-sourced models.
- The authors additionally compared their work on all the 9 datasets with previous best systems, they show that CoA could achieve better or comparable results on all datasets (comparable even with prior systems that require additional in-domain training). 
- The authors additonally tested CoA against a set of baselines on an extreme test, the Needle in a Haystack (NIAH) test set, as suggested by Reviewer FfHC. CoA achieves near perfect performance and outperforms other systems by a large margin. 

Overall, I believe if the authors could well integrate the additional clarification and experiments/analyses into their cameraready, this work could be well received by the NeurIPS community. Although this work presents a training-free method (which I personally less in favor of), but I like the fact the method is straightforward and effective, plus the authors did a good job justifying how this simple method could be applied to a potentially wide scenario, I do see good empirical impact this work could offer.

Reviews: review: paper_summary: paper_summary: This paper is about addressing the issue of lengthy inputs when using language models. Predominant approach is RAG, but is hampered by retrieval performance. Window extension extends the architecture of the model to handle lengthy inputs, but doesn’t guarantee that the model is able to extract the relevant information from lengthy inputs. This work proposes a simple approach where text is split into multiple chunks and the chunks are sequentially processed, left-to-right, where information from all chunks observed so far is consolidated into a summary. The response to a query (if there is one) is calculated based on the aggregate summary of the text. Despite the simplicity of the approach, the method performs well across various models and tasks.
strength_weakness: strength_weakness: * Technical depth is limited
* Framing/presentation is somewhat misleading
  - From Figure 1, is summarization all that’s being done? Is CoA is just a fancy term to describe this sequential summarization process?
  - ‘Chain of Agents’ makes it sound like there are different agents doing different tasks, which is misleading
* Fairly local view of performance results. Only comparisons against RAG/Vanilla baselines are presented. How about comparisons to best published numbers on these datasets? 
* Clarity issues
  - Figure 1 does not explain the architecture well
  - 127: W1 generates related evidence useful for answering question - Without knowing CU1, the reader cannot verify this. Basically unable to follow the discussion in 126-134 without knowing what CU1,2,3 are.
questions: questions: * How sensitive is the model to choice of the size of chunks?
* Was the RAG baseline properly optimized? 
* Table 2 should also compare against RAG models. For some practical applications, RAG could be more beneficial due to inference speed.
limitations: limitations: Limitations were discussed
paper_summary: paper_summary: The paper proposed Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks, where multiple worker agents sequentially comprehend and communicate to handle different segmented portions of the text, and a manager agent, at last, synthesizes these contributions into a coherent final output. The paper conducted experiments on 9 long-context tasks in qa, summarization, and code completion, showing a significant performance gain over vanilla long-context LMs and RAG baselines. The paper also provides abundant analysis.
strength_weakness: strength_weakness: - There is no support in the paper for the claim in L35 "inspired by human-like processing of long-context task", weakening the motivation.

- Results on BookSum are missing in the main table, and no RAG baselines are provided on this benchmark.

- Though the overall results are promising, I'm curious whether there is some information loss during the sequential "read and consume", and how this may affect the performance and the design choice.

- For the case study in Figure 5, for worker 1, there seems to be more than one line of clues to answer the query (a.k.a other space missions such as "Voyager"), which may result in a complicated reasoning graph, how is this phenomenon dealt with the proposed method?

- As also noted in the paper, there is no interactive communication between the agents, but only uni-directional and one-time. Whether the approach could be considered "communication" is debatable.

- Although the time complexity, in theory, is true for Table 2, how's the actual inference speed on GPU considering the overhead of multiple times of extra prompting and decoding?

- It might be helpful to also discuss the related works on communication between language agents, e.g.
  - Camel: Communicative agents for" mind" exploration of large language model society, NeurIPS 23
  - Building Cooperative Embodied Agents Modularly with Large Language Models, ICLR 24
questions: questions: Please see Weaknesses
limitations: limitations: As the paper also noted, the communication effectiveness and inference efficiency could be further improved.
paper_summary: paper_summary: The paper "Chain of Agents: Large Language Models Collaborating on Long-Context Tasks" introduces a novel framework called Chain-of-Agents (CoA) to address the challenge of effectively processing long contexts in large language models (LLMs). The CoA framework leverages multi-agent collaboration, where multiple worker agents sequentially handle different segments of the text and a manager agent synthesizes these contributions into a coherent final output. This method aims to mitigate the limitations of input reduction and context window extension strategies by enabling effective information aggregation and context reasoning across various LLMs. The framework is evaluated on a range of long-context tasks, including question answering, summarization, and code completion, demonstrating significant improvements over existing methods.
strength_weakness: strength_weakness: - **Complexity in Implementation**: Implementing a multi-agent system could be more complex and resource-intensive compared to single-agent systems.
- **Communication Overhead**: The sequential communication between agents might introduce latency and inefficiencies.
- **Evaluation Scope**: While the paper evaluates on multiple datasets, more diverse real-world applications could further validate the robustness of CoA, like NIAH test.
questions: questions: - Could you test the CoA framework on smaller long-context models, such as qwen2-7b and llama3-8b, which both support an 8k context? I am curious to see how these smart, smaller models perform with CoA.

- Regarding token cost, the input tokens are slightly higher than those in baseline methods. What about the output tokens? How are they related to the number of workers ($l$) and how each worker agent summarizes each chunk?

- In Section 5.2, how do you handle the Full-200k method, which is the Vanilla (200k), when processing contexts over 200k length?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a new method, "Chain-of-Agents" (CoA), to augment the long-context handling capabilities of large language models (LLMs). CoA is a framework designed to enhance the processing of long contexts by sequentially using multiple agents to handle different chunks of input text. In CoA, worker agents manage different segments sequentially and communicate their findings. These findings are then synthesized by a manager agent into a coherent final output, effectively aggregating information and reasoning across the entire context. The author conducted experiments on a wide range of long-context tasks to verify the advantages of CoA.
strength_weakness: strength_weakness: 1. My primary concern is the novelty of this submission. Breaking long texts into multiple chunks and processing them sequentially is a well-established practice in both long-content processing and generation. For example, in RecurrentGPT [1], the authors introduced a long-short memory mechanism to store intermediate states during processing, improving the quality of long content generated by LLMs. This is similar to the communication unit (CU) mechanism mentioned in this paper, but the authors do not sufficiently discuss this similarity. Additionally, previous work such as LongAgent [2] proposed segmenting the input long text into several chunks and assigning them to corresponding members. The authors lack sufficient discussion and experimental comparison on utilizing memory mechanisms or agent mechanisms for processing long information.

2. The baselines used in the author's experiments are relatively weak. While the related works section discusses some studies on long text processing, such as references [9] and [13], the implementation process only compares RAG, Vanilla, and COA. The author primarily compares their own designed merge and hierarchical methods in Agent and mechanisms for processing long texts. So, it is challenging to determine the advantages of the proposed methods over existing strong baselines in the field based on the current experiments.

3. Given the existing work on agents collaborating to process multiple chunks, the authors could enhance the novelty of their submission by delving deeper based on the discussion in Section 5.6. Specifically, they could explore the most effective methods of collaboration in multi-chunk processing and agent cooperation. This could add significant technical depth to the paper.


[1] Zhou, Wangchunshu, et al. "Recurrentgpt: Interactive generation of (arbitrarily) long text." arXiv preprint arXiv:2305.13304 (2023).

[2] Zhao, Jun, et al. "LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration." arXiv preprint arXiv:2402.11550 (2024).
questions: questions: 1. How is the chunk split? I did not find a clear description of how the chunk is processed. Could there be a risk of text being abruptly truncated in the middle?

2. As mentioned in the abstract, "input reduction has no guarantee of covering the part with needed information." I am wondering if sequential chunk processing will also encounter these challenges. Could the communication unit potentially drop necessary information if some information is only useful in the context of the following content?

3. In Appendix 1, regarding the calculation of time complexity in lines 592-595, if the model's processing length limit is k and no additional mechanisms are introduced, is the computational complexity of attention calculation O(k^2) or O(n^2)? Please educate me if I miss something important.

4. "a LLM" in line 104 -> "an LLM"
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Reponses from Authors (Part 2)
Reponses from Authors (Part 1)
comment: We are glad to know our responses addressed your concerns! We would also like to thank you for your thoughtful questions and insightful discussion, and we will include all our discussion results in the final version.
We are really glad that our responses address your concerns! Thanks so much for your thoughtful questions and insightful discussion. We will carefully include our discussion in the final version. We deeply thank the effort you made during the whole process!
Thank you for the clear response, which effectively addresses my concerns regarding how the CoA distinguishes itself from existing multi-agent research. Including these discussions in the paper will enhance its quality and emphasize your contributions. I  would like to raise my score to 5.
I would like to thank the authors for the response.   
My concerns are adequately addressed. Thus I have improved my score to 5.
Dear Reviewer,

Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.
Thank you for your time and consideration.

Sincerely, 

All Authors
Dear Reviewer,

Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.
Thank you for your time and consideration.

Sincerely, 

All Authors
Dear Reviewer,

Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have.
Thank you for your time and consideration.

Sincerely, 

All Authors
>  the central agents in centralized communication essentially function as the communication units described in your paper.

Although they all accomplish the task of communication, we think these two types of communication, centralized communication (e.g., LongAgent) and decentralized communication (our CoA), differ in their approach to solving long context dependency issues. When a centralized approach faces a multi-hop question, it leverages question decomposition to disentangle the long dependency issue, while decentralized communication leverages the interleaved reading-processing. Below is an example:

```
Question: Who is the grandson of A?
Source: [1],[2],[3],[4] (chunks)
Evidence in each chunk: [1: A’s husband is D], [2:  A’s son is B], [3: No evidence], [4: B’s son is C]
```

```
Centralized communication (e.g., LongAgent):

Round 1: 
Manager: Who is the son of A?
Worker with [2]: It is B!  Others say unknown
Round 2: 
Manager: Who is the son of B?
Worker with [4]: It is C. Others say unknown
Final answer: It is C.
```

In this approach, a worker with [i] and a worker with [i+1] do not communicate. Thus, it is difficult to deal with the dependency when the answer to the question is split into the end of agent i and the start of the agent i+1.

```
Decentralized communication (Our CoA):
Manager: Who is the grandson of A?
Workers: [1]: A’s husband is D (topic exploration), [2]: A’s son is B (answer first hop), [3]: A’s son is B (forward previous evidence), [4]: A’s son is B, B’s son is C. Thus, A’s grandson is C. (complete reasoning)
Final answer: It is C
```

This exemplifies how adjacent workers communicating in CoA would differ from the centralized one as CoA agents receive previous evidence in addition to the question itself. 

We want to emphasize that LongAgent and our work are solving the long context problem in different ways, and they are not in conflict with each other. They can be merged into a much stronger framework (e.g., adding our chain reference stage to mitigate context dependency issues before the stages in LongAgent).

> Could you clarify how CoA addresses this issue? From my understanding, CoA does not involve the decomposition of the input question.

The key aspect of CoA decentralized communication is that workers can communicate so that the question does not need to be decomposed. Our experiments highlight the effectiveness of this design, especially when question decomposition is difficult. For instance, when the passage does not mention “A’s son” directly but mentions “A’s husband is D”, and mentions “D’s son is B”, the first question in the centralized approach should be “Who is A’s husband” rather than “Who is A’s son”, which is difficult to propose. 

> However, this further diminishes the novelty of the proposed method.

We acknowledge that chain communication is not first used in our work. But we are the first ones that use chain communication for long dependency in the **input of long context** tasks and we show its effectiveness on generic tasks. This approach is independent of RecurrentGPT, as RecurrentGPT solves **long output**. For the output, the issue they mitigate is memory and planning for story generation tasks. Combining CoA and RecurrentGPT to solve Long-Input-Long-Output generation could be promising as well and we leave that to future work.

> many components of the proposal are common practices in multi-agent system design and long-content generation/processing.

We truly appreciate your thoughtful summary. We understand your concerns and would like to re-emphasize the value of our contributions: our work presents a novel and straightforward solution to the long-standing challenge of long context reasoning, and our results demonstrate significant effectiveness. We are hopeful that our findings will inspire further innovation in the research community, encouraging others to integrate our simple yet powerful components into their future work in multi-agent systems.

> it should be comprehensively tested as a stronger baseline (e.g. LongAgent) across all benchmarks you tested.

Thanks for the suggestion. Indeed, LongAgent is an important work in this direction. We try to compare all datasets in our paper. However, they did not open source their code, making it difficult to reproduce and compare with them fully in this short-time rebuttal window. Since we found that CoA is 15% higher than that of LongAgent (97% vs. 82%) on NIAH PLUS (Figure 2 in pdf), we believe that given the strong improvement, the performance of CoA will be promising. We will include detailed comparisons across all datasets in the final version of our paper.

We hope our response has addressed your concerns and clarified the contributions of our work. We welcome any further questions you may have and would be happy to provide additional clarification if needed. We deeply appreciate the time and effort you've taken to provide feedback on our work.
Thanks so much for carefully reading our responses, and providing such insightful questions and thoughts! Below, we first re-state our novelty aspects more clearly and then comment on the raised concerns one by one, hoping to make our claims clearer to you and address your concerns!

CoA has a unique positioning in the multi-agent LLM literature. The table below illustrates the comparison of related work and their topologies  (after carefully reading the survey [1], [2], and other literature). Broadly, prior work on multi-agent LLM systems for long input tasks has a centralized design, and existing decentralized designs do not extend effectively to long context tasks. To the best of our knowledge, CoA is the first to use a centralized structure on long input tasks and is more effective than baselines such as WalkMaze and RAG.
| Work                            | Task                      | Type          | Communication Schema |
|---------------------------------|---------------------------|---------------|-------------------|
| [Chen et al., 2023d]           | Multi-robot planning      | Decentralized | Circle            |
| RoCo [Mandi et al., 2023]       | Multi-robot collaboration | Decentralized | Planning Pipeline |
| CoELA [Zhang et al., 2023c]     | Multi-Agents cooperation  | Decentralized | Memory Module     |
| MAD [Du et al., 2023]           | Improving Factuality      | Decentralized | Debate            |
| Reconclie [Chen et al., 2023]   | Reasoning                 | Decentralized | Round Table       |
| WalkMaze [Chen et al., 2023]    | Long Input                | Centralized   | Tree Structure    |
| LongAgent [Zhao et al., 2024]   | Long Input                | Centralized   | Tree Structure    |
| RecurentGPT [Zhou et al., 2024] | Long Output               | Decentralized |  Memory Gating    |
| CoA (Ours)                      | Long Input                | Decentralized | Chain of Agents   |

Besides, our contribution is not to claim a new topology for communication. The challenge of long dependency on long input tasks is well-known and remains under-explored (as illustrated with the sample in Part 2). Our target is to *novelly* mitigate **challenging context dependency issues** in long-context tasks particularly those requiring complex reasoning. Thus, it is better if we can find a **simpler yet effective approach**. To this end, we explored various structures (WalkMaze, Merge, Debate, Group Discussion, etc.) and found that a variant of **decentralized well-existing chain communication and this simple approach can work more effectively** than others. Our contribution is not to claim a new topology for communication but to mitigate a challenging issue (long dependency) with a simple intuitive approach (decentralized chain communication). In addition, CoA is not in conflict with existing work as it can be considered as a plugin for other multi-agent LLM systems (e.g. serve as an additional stage in LongAgent) to improve them.

Specifically, previous work for long input tasks, despite showing significant improvements, usually uses disentangling algorithms such as question decomposition, to address long dependency, and then they use centralized structure to answer the disentangled questions one by one. Different from them, CoA does not disentangle the question but leaves the entire question to the agents. Although chain communication is well-introduced in the literature, to the best of our knowledge, CoA is the first work to apply decentralized communication to long-term dependency challenges - please let us know if there is any paper we are missing on this. We show the high effectiveness of CoA in mitigating agent dependency issues for long inputs and we believe it would constitute a significant value add in the multi-agent system literature. We will better clarify these novelty points in the Introduction section of our paper. 

Below are the detailed answers to specific points:

> various communication mechanisms exist in multi-agent systems

Indeed, chain communication has been introduced as a very simple topology of communication. Our contribution is not to claim a new topology for communication but to mitigate the challenge of long dependency. We bring a novel perspective to the challenge with the simple chain communication mechanisms. Also, to the best of our knowledge, CoA is the first to use a decentralized structure on long input tasks (see table above). We note that no other multi-agent method in the literature has such results of outperforming RAG for long inputs.
I appreciate the author's clear and point-by-point rebuttal, as well as the supplementary experiments provided. However, after thoroughly re-examining the paper, the rebuttal, the feedback from the other reviewer, and the related works in this area, I still have **significant concerns** regarding the novelty and the experimental validation of this work.

First, as I mentioned in my initial review, the novelty of this submission remains my primary concern. I agree with reviewer *vGWD*'s comment that "*the technical depth of this work is limited.*"

Although the authors compare LongAgent and RecurrentGPT in their general response, asserting that "LongAgent **does not allow communication between agents**, making it hard for each agent to understand what is happening with others," various communication mechanisms exist in multi-agent systems (see Section 3.3 in [1] and the vertical and horizontal multi-agent architecture definition in [2]). While CoA uses decentralized communication and LongAgent employs centralized communication, based on my understanding, the central agents in centralized communication essentially function as the communication units described in your paper. Additionally, you mention that LongAgent identifies the decomposition of the input question as a limitation. Could you clarify how CoA addresses this issue? From my understanding, CoA does not involve the decomposition of the input question.

Furthermore, as the authors stated, "Regarding RecurrentGPT, although it uses a chain structure to perform long story generation, the task and motivation are different." However, this further diminishes the novelty of the proposed method.

In my view, many components of the proposal are common practices in multi-agent system design and long-content generation/processing. The work appears to be a fine-tuned combination of existing methods rather than a deeply innovative approach. In my opinion, the limited novelty does not meet the standards of NeurIPS.

Second, regarding the stronger baseline, the authors only tested LongAgent in the NIAH PLUS. Given that LongAgent is closely related to your work and was published three months before your submission, it should be comprehensively tested as a stronger baseline across all benchmarks you tested. Without this, it is difficult to claim that your communication unit or designed agent system is more efficient.

Overall, I appreciate the effort the authors have put into the rebuttal. Unfortunately, it does not address my primary concern regarding the work's novelty. I remain open to further discussion with the authors in the coming days.

References:

[1] Guo, Taicheng, et al. "Large language model based multi-agents: A survey of progress and challenges." arXiv preprint arXiv:2402.01680 (2024).

[2] Masterman, Tula, et al. "The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey." arXiv preprint arXiv:2404.11584 (2024).