Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Authors: Zhangyue Yin
Qiushi Sun
Cheng Chang
Qipeng Guo
Junqi Dai
Xuanjing Huang
Xipeng Qiu
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models; Model Communication; Chain-of-Thought

Abstract: Large Language Models (LLMs) have recently made significant strides in complex reasoning tasks through the Chain-of-Thought technique. Despite this progress, their reasoning is often constrained by their intrinsic understanding, lacking external insights. To address this, we propose Exchange-of-Thought (EoT), a novel framework that enables cross-model communication during problem-solving. Drawing inspiration from network topology, EoT integrates four unique communication paradigms: Memory, Report, Relay, and Debate. This paper delves into the communication dynamics and volume associated with each paradigm. To counterbalance the risks of incorrect reasoning chains, we implement a robust confidence evaluation mechanism within these communications. Our experiments across diverse complex reasoning tasks demonstrate that EoT significantly surpasses established baselines, underscoring the value of external insights in enhancing LLM performance. Furthermore, we show that EoT achieves these superior results in a cost-effective manner, marking a promising advancement for efficient and collaborative AI problem-solving.

Exchange-of-Thoughe
Enhancing Large Language Model Capabilities
through Cross-Model Communication
Zhangyue Yin
Qiushi Sun
Cheng Chang
Qipeng Guo
Junqi Dai
Xuanjing Huang
Xipeng Qiu
School of Computer Science, Fudan University
National University of Singapore Shanghai Al Laboratory
(yinzy21, changc21, jqdai22 fudan. edu. cn piushisun@u. nus. edu {qpguo16, xjhuang, xpqiu]@fudan. edu. cn
Abstract
Q
Q
Large Language Models (LLMs) have recently made sigmificant strides in complex reasoning technique.
tasks through the Chain-of-Thooe Despite this progress, their reasoning is often constrained by their intrinsic understanding, lacking external insights. To address this, we propose Exchange-of- ofhTuhh (EoT), a novel framework that enables cross-model communi- cation during problem-solving. Drawing inspi- ration from network topology, EoT integrates four unique communication paradigms: Mem- ory, Report, Relay, and Debate. This paper delves into the communication dynamics and volume associated with each paradigm. To counterbalance the risks incorrect reason- ing chains, we implement robust confidence evaluation mechanism within these communi- cations. Our experiments across diverse com- plex reasoning tasks demonstrate that EoT sig- nificantly surpasses established baselines, un- derscoring the value of external insights in en- hancing LLM performance. Furthermore, we show that EoT achieves these superior results in cost-effective manner, marking promis- ing advancement for efficient and collaborative problem-solving.
R
R₁
R₂
R3
R
Review
Rg
R₂
Rs
A
A
A2
A
(a) CoT
(b) Sclf-Correction
(c) BoT
Figure I: Comparison CoT, celf-Correction, and EoT. Both CoT and Self-Correction rely the model's innate abilities to generate and refine output, lacking external insights. FoT enhances the model's reasoning ability by incorporating the thoughts of other models external insights.
reasoning tasks, and this limitation cannot be over- come solely by increasing the size of models (Rae et al., 2022; bench authors, 2023).
To overcome this shortcoming, Wei et al. (2022b) proposed chain-of-thought (CoT) prompt- ing, which guides the model to generate a series of intermediate reasoning steps before reaching the final answer. At the same time, series of self- correction methods (Madaan et al., 2023; Welleck et al., 2023; Shinn et al., 2023) have been proposed. which aim iteratively improve the quality of an- swers by using the models' feedback on previous outputs (Bai et al., 2022b).
"Two heads are better than one."
English Proverb
1 Introduction
Large Language Models (LLMs) such as GPT- 4 (OpenAI, 2023) are revolutionizing the field of Natural Language Processing (NLP) by utilizing vast training сorpoга and huge computational re- sources (Bai et al., 2022a; Ouyang et al., 2022: Chowdhery et al., 2022: Zhang et al., 2022; Tou- vron et al., 2023a, inter alia). Although LLMS achieve exemplary performance across wide range of NLP tasks (Wei et al., 2022a; Chung et al., 2022), they consistently struggle perform well in Corresponniig author.
However, CoT and self-correction solely base on the model's own understanding and perspective of the question during the reasoning process. Recent studies (Huang et al., 2023; Valmeekam et al., 2023; Stechly et al., 2023) indicate that LLMs struggle revise their responses without external feedback. This be attributed the model's complete de- pendence on internal representations to generate responses, which makes it difficult to overcome inherent limitations in capability (Yin et al., 2023).

---

100
Contain conoet
Noi custain corret
thermore, we delve into the intricacies of each com- munication paradigm, analyzing the dynamics of information flow and the volume of communica- tion. With the awareness that both correct and incorrect reasoning chains propagate within com- munications, we introduce confidence evaluation mechanisms that employ the analysis of answer variations to assess models' confidence levels. It is designed to mitigate the influence of erroneous reasoning. thereby ensuring the integrity and relia- bility of the problem-solving process.
0-
GSMAK
AQUA
SVAMP
Figure 2: Pilot experiments on three reasoning datasets. The number of erroneous samples containing the correct answer is significantly higher than those not containing the correct answer.
Experiments across various complex reasoning tasks demonstrate that EoT significantly outper- forms established strong baselines, underscoring the critical role of external insights in augment- ing the capabilities of LLMs. We summarize our contributions as follows:
Despite the undeniable importance of external insights (Yao et al., 2023), acquiring high-quality external insights remains a challenge. Wang et al. (2023c)'s research suggests that the single reason- ing chain generated by CoT limits the model's rea- soning performance. By increasing the temperature sample diverse reasoning chains and selecting answers through majority voting, the model's rea- soning performance can be further improved. How- ever, when confronted with difficult questions, the model often yields a higher number of incorrect responses. In Figure 2, our analysis of correct and incorrect answers within erroneous samples from three reasoning datasets reveals that in most cases the model can deduce the correct answer.
We introduce Exchange-of-Thonee
(EoT), a pioneering framework for cross-model com- munication that incorporates external insights from other LLMs during problem-solving.
We present and examine four communication paradigms coupled with a confidence evalua- tion mechanism that assesses model certainty through the variability of answers, mitigating the impact of incorrect reasoning.
Experimental results on various complex rea- soning tasks underscore the efficacy and cost- effectiveness of EoT, highlighting the signif- icance of incorporating external insights and communication in problem-solving.
In human society, the truth, even when held by a minority, can gain widespread acceptance and recognition through clear and persuasive commu- nication (Le Bon, 1897). The correct reasoning of others can serve as high-quality external insights, which enrich and enhance our collective under- standing. Thus, we propose Exchange-off Thouhttt (EoT)), novel framework that fosters cross-model communication during the problem-solving pro- cess. This initiative enables models to incorporate the reasoning of others as external insights.
2
Related Work
2.1 Chain-of-Though prompting in LLMs
Wei et al. (2022b) highlight that LLMs can man- ifest enhanced reasoning capabilities when being prompted by demonstrations with intermediate rea- soning steps. This technique can effectively im- prove the performance of LLMs on complex rea- soning tasks (Wei et al., 2022a; Kaplan et al., 2020). A series of strategies for enhancing CoT has been proposed to further improve the performance of LLMs. One such method is Program aided lan- guage models (Gao et al., 2022; Chen et al., 2022), which aims to decouple reasoning and computa- tion through program synthesis. Moreover, com- plex tasks can also be transformed into delegable sub-tasks through modular approaches (Khot et al., 2023). Choosing appropriate demonstrations can also enhance the performance of CoT (Li et al.,
Figure 1 contrasts BoT with CoT and self- correction methods, highlighting the unique ap- proach of EoT in integrating external perspec- tives. Inspired by the principles of network topol- ogy (Bisht and Singh, 20155 and agent communi- cation (Parsons and McBurney, 2003), propose four communication paradigms: Memory, Report, Relay, and Debate. These paradigms are designed facilitate the exchange of ideas and reason- ing chains among models, enriching the problem- solving process with a diversity of insights. Fur-

---

2023a; Li and Qiu, 2023a), Notable among these, AutoCoT (Zhang et al., 2023b) uses an automated way to construct and sample diverse demonstra- tions. Active-Prompt (Diao et al., 2023) selects the most helpful samples for labeling based on the model's uncertainty in the outputs. Recently, Li and Qiu (2023b) employ a strategy of storing high-confidence thoughts as external Memory and retrieves these insights aid the reasoning process.
correct answer by using previously generated an- swers as hints. With the aid of external knowledge, Wang et al. (2023a) introduce chain-of-knowllddee prompting that employs evidence triples to curb the generation of unfactual and unfaithful answers. Taking model interactions into account, mulli-agent debates (Du et al., 2023; Liang et al., 2023) have been introduced to enhance the factual accuracy of generated content and reduce fallacies and hal- lucinations. EoT differs from these efforts as we prioritize reasoning chains over simply providing answers. Our communication method involves mul- tiple LLMs, which not only considers the historical information of a single LLM but also takes into account the interactive information among them.
2.2 Ensemble of Reasoning Paths
LLMs have the ability explore multiple rea- soning paths using techniques such as tempera- ture adjustment and prompt sampling (Chu et al., 2023). Wang et al. (2023c) suggest that for com- plex questions, there may be several correct paths to approach problem, leading to the proposal of Self- Self-Consistency. This method replaces the greedy decoding strategy with the sampling of multiple rea- soning paths and selecting the most consistent an- swer, resulting in significant performance improve- ments. Beyond that, Fu et al. (2023b) discover that prompts with higher reasoning complexity could achieve better performance in multi-step reason- ing tasks, lcading to the proposal of complexity- based prompting. While other methods, such as re-ranking (Thoppilan et al., 2022; Cobbe et al., 2021), have also been applied to select suitable rea- soning paths, they often rely on heuristic or trained smaller models. Recently, Li et al. (2023b) sam- ple different demonstrations and use step-by-step verification to filter out incorrect answers. How- ever, obtaining step-level labels can be challenging. and using smaller models for judgment struggles to handle complex reasoning processes. In contrast, method fully utilizes the communication and decision-making capabilities of LLMs reach the final answer.
3 Preliminary
Firstly, we define the current methods that use LLMs to solve problems. We denote LLM with parameter size of 0 as pa, and the sequence length as t, which includes tokens [$1, : s:]. The LLM predicts the next token based on the prior tokens in the sequence. The probability of the 8, token is pe (s, 82, ,2,-..,8(11).. Therefore, the probability of the whole sentence is pe(s) = I (-)1--)
Standard prompting. Standard prompting in- volves deriving answer from question using Po(alg). To bolster the comprchension of LLM, an instruction prompt I can be included before the question, resulting in po(a) 9)). In-Context Learning, proposed by Brown al. (2020), aims to improve LLMs performance by adding demon- strations D {d1, ddd1,d..- dz..... ...,,,,)) d to the input, which can be expressed as pe(a) p D,
CoT prompting. As identified by Wei et al. (2022b), the incorporation of intermediate reason- ing steps can improve the proficiency of LLMs in tackling complex reasoning challenges. To fa- cilitate this, a rationale r; is added to examples d, = {q, ri, a to guide the LLMs in explicitly generating reasoning steps. Kojima et al. (2022) propose that including "Let's think step by step" in the instruction I can achieve the same effect, eliminating the need for hand-crafted examples.
2.3 Reasoning Path Refinement
Although CoT (Wei al., 2022b) effectively en- hances the performance of LLMs in complex rea- soning tasks, they remain susceptible to errors dur- ing the reasoning process, leading to incorrect an- swers (Bai et al., 2022b; Lyu et al., 2023). To mitigate this issue, starting from the model's 's own thoughts, Shinn et al. (2023) and Madaan et al. (2023) employ the LLMs' own outputs and past mistakes to refine the reasoning process, Yao et al. (2023) explore the synergies between reasoning traces and action plans. For numerical problems, Zheng et al. (2023) gradually guides models to the
Self- CaffConsistency Self-Consistency proposed by Wang et al. (2023c), is a method that consoli- dates answers obtained from multiple independent reasoning chains. It returns the answer with the highest frequency of occurrence, represented as

---

A
A
A
A
B
C
B
C
B
C
B
C
Bus
Star
Ring
Tree
B
B
C
B
B
Memory
Report
Relay
Debate
B
Fully Visible
Central Visible
Neighbor Visible
Peers Visible
Figure 3: Correspondence between communication paradigms and network topologies. The top row depicts four network topologies. The second row correlates these with the corresponding communication paradigms. The bottom row offers analysis of the communication volume associated with each mode. The horizontal axis represents the information that models can receive, while the vertical axis indicates the information that models can send.
a argmax,, (a,), where fa;; represents the frequency of occurrence of answer a₃. This ap- proach enables the model to explore a wider array reasoning paths, thereby enhancing its inferen- tial capabilities. However, it remains constrained by the intrinsic limitations of LLMs' capabilities.
sons and McBurney, 2003), we propose four com- munication paradigms determine the counter- parts for model communication. As illustrated in Figure 3, we propose Memory, Report, Re- lay, and Debate communication paradigms each corresponding to the Bus, Star, Ring, and Tree network topologies. respectively. Assume in j- th round of communication, given set of LLMs {M} {m1, m2..... m,), the model m, gener- ates the corresponding rationale Ti
4 Methodology
To incorporate external insights, we introduce Exchange-of-Thoughi (EoT), novel framework that facilitates communication and the exchange of ideas between models during the problem-solving process. This innovative approach leverages the communicative abilities of LLMs to promote the sharing of reasoning chains r and answers a among participating models, fostering collaborative envi- ronment for thought and analysis. The implemen- tation of BoT encounters three key challenges:
(t) and the an-
swer a; based the (rx (j-1) (3-1)), where K is the from which model i can receive reason- ing chains. In the first round, method proposed by Wei et al. (2022b) to generate (1) () (o( (
.aK
use the CoT
Memory Under the Memory paradigm, all mod- els record their rationale and answer a in a log- book, which is fully visible from all models. This means that in the j-th round, any model, such as model m. can access the reasoning chains and an- swers from all models (i-1) am ( 1),, m E (M). As depicted in Figure 3. this paradigm facilitates the fastest flow of information and also incurs the highest communication cost among all paradigms.
1. How to identify the appropriate counterparts for model communication?
2. What the conditions for ceasing communi- cation between models?
3. How minimize the influence of incorrect reasoning during the communication process?
Report Under the Report paradigm, we designate model m,1 as the central node, which can obtain the rationale and answer from all other models (-1) ( m {M)){m)) Both mp and
4.1 Communication Paradigm
Inspired by network topology (Bisht and Singh, 2015) and intelligent agent communication (Par-

---

B Turn I: First, we noed to convert 20 dozen cups cups... cusii..Divdddnn both sides by 240 x $70 Therefore, the total cost is $70.
Turn 1: 20 dozen cups aquicalent to 20 12- cups...
$70.
Turn 2: One dozen is equal cups is x 240 cups... solution accurate. So, the total cost is 5145
dozen
Low Confidence
Both solutions... ... Dividing both sides by 240 e $70. So, of buying each 570.
provided
High Confidence
convert number of cupe: x 12 240 cups,.. Therefore, sotal buying each cup dollarrs.
Turnd:
cups to
Turn3:
Dased on the solutions provided by A and
So cup
An illustrative comparison between a confident model and an unconfident model. Model A's answers fluctuate over three dialogue rounds, indicating uncertainty about the answer, while Model B consistently believes in single answer.
Figure 4:
mc only receive information from m and do not interact with each other. Consequently, mA plays a pivotal role in the communication process. This paradigm also allows for rapid information flow, but it demands higher capacity for processing and analysis from the central node.
In the Memory paradigm, every node receives information from all other nodes in the previous round, resulting in communication volume of n² Any piece of information requires only one transmission to reach the corresponding node.
Under the Report paradigm, the central node processes picces of information in cach round, while the remaining n - 1 nodes process each. Therefore, the communication volume is 3n - 2. and any picce of information requires at most two transmissions via the central node reach the cor- responding node.
Relay Under the Relay paradigm, order the models by number and connect them in circle. Each node capable of receiving information from the preceding node and transmitting its own infor- mation to the subsequent node, For example, in the j-th round, A passes (1-1) a (1-1)) mo and receives (r (j-1) ag (1-1)) from the previous round m. This distributed mode of communication can alleviate the demands on information process- ing capacity of a single node, yet it may result in slower information flow.
In the Relay paradigm, each node receives infor- mation from itself and the previous node from the last round, leading to a communication volume of 2n. Node i receives information from node i - -1 in one transmission, while receiving information from node i+ requires n- transmissions, giving average propagation speed of n-1 2
Debate We have adapted the tree topology de- vise the Debate paradigm. This paradigm permits Ieaf nodes to exchange information with each other, while parent nodes are solely responsible for ag- gregating information, meaning that information flow is directed upward from child to parent. As illustrated in Figure 3, mB and mc, as child nodes, are able communicate, whereas m. parent node, can only receive information from its chil- dren. This communication paradigm balances the model's information processing capacity with the speed of information flow.
For the Debate paradigm, we assume the nodes form a binary tree with a height of [logg + 1)}. Each pair child nodes has a communication volume of 2, and the parent node has a volume of 3, so a subtree composed of two children and one parent has a communication volume of 7. The number of such subtrees in full binary tree is 2 1. Assuming n is odd, the calculation for the communication volume V proceeds as follows:
V 7 - 2 m)
(1)
2
which simplifies to (n 1). The aggregation of in- formation between nodes, without considering other communications, requires most 2(h 1) transmissions.
4.2
Communication Volume
The last row of figure 3 displays the information that can be transmitted and received through dif- ferent communication paradigms. The communi- cation volume is measured by the number of mes- sages received, assuming there are models partic- ipating in the communication, with each node trans- mitting its information from the previous round to the next.
4.3 Termination Condition
Utilizing the model's current round output and the answers from previous rounds, we have devised two criteria for terminating communication: con- sistent output and majority consensus.

---

Consistent Output Termination. Inspired by Zheng et al. (2023), we implement a consistent output termination in EoT. The termination con- dition is triggered when the output model m, in the j-th round is the same as the output in the
plex reasoning tasks: (1) Mathematical Reason- ing: This involved six datasets, which includes GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), SingleEQ Kkoncel-Kedzione et al., 2015), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), and SVAMP (Patel et al., 2021). (2) Commonsense Reasoning: We utilized the CommonsenseeQACCSOA., Talmor et al., 2019)
j j- 1-th round, a; = 2,; (j-1) In this case, m, will stop receiving or sending information and exit the current communication.
(5)
and StrategyQA (Geva et al., 2021). (3) Symbolic Reasoning: We employed two datasets from Big- Bench (bench authors, 2023; Suzgun et al., 2023), namely Penguins in a Table (Penguins) and Date Understanding. In Appendix B, we provide de- tailed description and statistics of the datasets.
Majority Consensus Termination. Du et al. (2023) observed that LLMs can converge on con- sensus after several rounds of debate, suggesting that LLMs fine-tuned with reinforcement learn ing from human feedback (RLHF) (Ouyang al.. 2022) are more likely to reach an agreement. In- spired by this finding, we propose the termination condition majority rule, where LLMs cease com- munication with each other once majority of them reach an agreement. This approach serves as global termination condition, distinguishing it from the consistent output termination, which acts cessation criterion an individual model basis.
We compare FoT with a series of strong baselines, which include (1) Chain-of- Thought prompting (CoT; Wei et al., 2022b), (2) Complexity based prompting (ComplexCoT; Fu et al., 2023b), (3) Self-Consistency (SC; Wang et al., 2023c), (4) Progressive Hint Prompting (PHP: Zheng al., 2023). Specifically, CoT and ComplexCoT are prompting methods, while SC and PHP are reasoning chain ensemble methods. For simplicity in notation, we use "CoT-SC(10)" to denote the approach that employs the CoT prompt method to sample 10 reasoning chains and then utilize the SC method to select the answer.
Baselines.
4.4 Confidence Evaluation
An intriguing aspect of human behavior is that in- dividuals tend to be less susceptible to incorrect answers when they are confident in their own re- sponses. Conversely, they are more likely to be in- fluenced by others' opinions when uncertain about their answers. Moreover, Wang al. (2023c) found that the probability of a correct answer decreases as the generated responses become more contra- dictory. Therefore, there's a high probability that the model's answer is incorrect when it frequently changes during the communication. We propose calculating the model's confidence based on the variation in responses. This aids the recipient of the information in ascertaining its reliability, thereby safeguarding the problem-solving process from the disruption of erroneous information.
Implementation Details. We accessed the GPT models through the OpenAl API. In the main experiments, employed 101T-33.-Turbb-005 (GPT-3.5) and GPT-4-0314 (GPT-4) to evaluate the effectiveness of EoT in comparison other strong baselines. We set the temperature at dur- ing the generation. The prompts for CoT and Com- plexCoT are sourced from Wei et al. (2022b) and Fu et al. (2023b). By default, we employ three 5TT-3.5-Turbo-B01 0001 to engage in the EoT com- munication. We apply the majority consensus ter- mination and confidence evaluation, selecting the majority answer as the final outcome. Taking into account the impact of temperature, we report the average performance and standard deviation across five runs. Additionally, in section 5.3, to further validate the performance of different LLMs on BoT, we incorporated the Claude-2 2 model. The further implementation details are listed in Appendix C.
In a communication with & rounds, model m, (1)
(k) generates a set of answers {a; ....,a a }. Let f(a;) argmax,, #(alaa a; d()) denote the most frequently occurring answer from model m,- Consequently, we obtain the model's confidence level C, [a,)) in the current round.
k
5 Experiments
5.2 Performance of EoT
5.1 Experimental Setups
Mathematical Reasoning. According to the re- sults presented in Table 1, the four communication paradigms of EoT have shown significant improve-
Tasks and Datasets. In our experiments, we eval- uated the performance of EoT across three com-

---

Method
GSM8K
MultiArith
SingleEQ
AddSub
AQuA
SVAMP
Avg.
Single Reasoning Chain
CoT
79. 19.150
97.2710.65
92.80=0.27
86.2330.52
55,122100
79.5210.81
81.67
ComplexCoT
19.3210.65
95.40
21.3420.33
84.4610.86
56.4610.59
77.7010.54
80.78
CoT (GPT-4)
94.90
97.80
93.10
89.30
77.50
90.50
90.51
Ensemble Methods
CoT-SC(3)
82.82+0.32
98.20+0.43
93.31=0.12
87.19+0.47
12.13+1330
81.98+0.49
84.27
CoI-SC((5)
80.57
98.60+0.08
93.70=0.25
87.49+0.38
64.02+0.95
83,76+0.81
85,50
CoT-SC(10)
87.57
98.9740.12
94.06=0.36
87.5940.58
66.38+1.72
84,96+0.33
86.59
CompleXCoT-SC(33
20.17
97.43+0.31
92.95=0.53
86.13
6.547
81.44+0.79
83.77
ComplexCoT-SC(55)
87.26+0.33
98.13+0.22
94.02=0.29
86.48+0.61
62.05+2.40
83.86+0.92
85,30
ComplexCoT:SC(1U)
89.23
98.23+0.37
94.21+0.16
86.58+0.58
64.96+1991
85.58+0.87
86,46
РHP
85.10
98.00
92.90
85.30
60.60
83.10
84.16
Exchange-of-TNe Thouughh
BoT-Memory
88.9820.89
98.8020.16
94.0920.48
87.6520.49
69.37
84.2810.48
87.20
EoT-Report
88.61
99.03
94.06=0.47
87.95+0.44
70.31
84.78+0.75
87.46
EoT-Relay
88.4220.72
98.97+0.16
94.1320.49
87.59+0.58
70.87
85.04+0.31
87.50
EoT-Debate
88.52+0.76
98.90+0.17
94.25+0.19
87.70+0.34
69.69+124
85.10+0.24
87.36
Table 1: Comparison of BoT performance with series of strong baselines mathematical reasoning tasks. Best results are highlighted in bold, while the best results among different BoT modes are underlined. The experimental results for CoT (GPT-4) and PHP were sourced from Zheng al. (2023).
ment over both CoT and ComplexCoT in mathe- matical reasoning tasks. Compared to the currently strongest baseline method, PHP, the Memory, Re- port, Relay, and Debate paradigms have respec- tively increased the average performance by 3.04%, 3.30% 3,34,, and 3.20%. EoT comprehensively outperforms CoT-SC(S), achieving performance comparable to, or even surpassing, that of CoT- SC(10). When compared to the current best LLM GPT-4, three G with EoT surpassed single GPT-4 with CoT the MultiArith and SingleEQ datasets. This indicates that through inter-model communication and collaboration, three less ca- pable models can compensate for their individual weaknesses and outperform more powerful model, showcasing the potential of EoT to enhance model capabilities and address inherent shortcomings by incorporating external insights.
paradigms outperform the CoT-SC(10) method, which samples 10 reasoning chains, demonstrat- ing the superior performance of EoT.
Symbolic Reasoning Figures 5c and 5d com- pare the performance of EoT with CoT and CoT- SC methods on symbolic reasoning tasks. On the Penguins dataset, the Memory, Report, Relay, and Debate paradigms EoT achieve improvements of 2.01%, 1.92%, 2.33%, and 2.05% respectively, compared to the CoT-SC(3) method which samples 3 reasoning chains. On the Date Understanding dataset, the performance gains of EoT are even more pronounced, with all four methods showing an average improvement of 2.1 over CoT-SC(10).
5.3 Discussions
Commonsense Reasoning. The comparison of EOT with CoT and CoT-SC methods on common- sense reasoning tasks is illustrated in Figures 5a and 5b. EoT significantly outperforms the COT. Specifically, on the StrategyQA dataset, Memory, Report, Relay, and Debate respectively achieve im- provements of 8.06%, 8.24%. 8.42%, and 8.67% compared to COT. Similar significant gains are observed on the CSQA dataset. Furthermore, across both commonsense reasoning tasks, four
Communication Mode. We propose four com- munication paradigms and analyze the communica- tion volumes in section 4. and section 4.2. In the results illustrated in Table 1, we observe that differ- ent communication paradigms have their respective strengths. For instance, Report performs best on MultiArith and AddSub, while Debate achieves op- timal performance on SingleFQ and SVAMP. This suggests that various communication paradigms are best suited for different scenarios.

---

85
84
84
79.0
83
Performances Ptormanceess
70
78.5
81-
78.0
82
80
77.5
66
78
80
77
76.0
CoT CORSC EST Meemory
Est Rapao Es Ear Detate
CoT:Scra) Cor SCIs) CoT ST-SCLO Est-Mamor Est-Repan Rep-rt EsT-Relay Eo EaT-Debate
CoT
76
coT:503)) Car Cor Meterrory Raport Est Relay EoT-Delata
Got
79
CoT:SC13 CoPScrs ESTMeraa EST-Rapo Report Est Relay Eek-Dekate
(a) CSQA.
(b) StrategyQA.
(c) Peguins.
(d) Date Understanding.
Figure 5: Comparison of FoT with CoT and CoT-SC methods in commonsense and symbolic reasoning tasks.
90
74
Majority Consersus
Confidence Evalustion
72
Consistent Output
89
w/o Confidence Evaluation
rerformance
PD
66
86
62
85
60
Mamory
Report
Relay
Debate
84
Memory
Report
Relay
Debate
Figure 6: Comparison of consistent output termina- tion and majority consensus termination on AQuA.
Figure 7: The impact of employing confidence eval- uation on accuracy in the GSM8K dataset.
Termination Condition. In Figure 6, we analyze the performance of the two termination conditions we propose the AQuA dataset. Compared to consistent output termination, majority consensus termination improved by 4.33%, 4.01% %, 7.56%, and 4.97% under the Memory, Report, Relay, and Debate paradigms, respectively. Under consistent output termination, there is no mechanism for col- lective negotiation, and individual models are prone to premature exit due to degeneration (Su et al., 2022). Therefore, majority consensus termination is more suitable for scenarios involving multiple model communication.
at an earlier stage, thereby effectively mitigating the interference of incorrect reasoning chains.
Round Analysis. As illustrated in Figure 8, we analyze the number of communication rounds to satisfy termination condition in the SVAMP dataset. For the majority of samples, consensus the an- swer be reached within three rounds of com- munication. Wang et al. (2023c) obverse that an- swer consistency is proportional accuracy. EoT enables models to engage a greater number of exchanges and discussions on questions where con- sensus is challenging to achieve. Consequently. a minority of difficult cases necessitate communica- tion extending beyond five rounds.
Confidence Evaluation. We conduct ablation ex- periments on the GSM8K dataset for confidence evaluation. As shown in Figure 7, across four communication paradigms, confidence evaluation showed an average improvement of 2.92% com- pared to the baseline. The introduction of con- fidence evaluation enables the model consider the other model's confidence prior (Zhang et al., 2023a) during communication, facilitating its deci- sion to accept the other model's s reasoning chains
Cost Analysis. A potential concern is the com- putational expense incurred by EoT. In Figure 9, we compare the performance and computational costs of CoT-SC, CompleXCoT-SC,, and EoT meth- ods. Compared CoT-SC(5), EoT reduces costs by 20% while enhancing performance by 3%. EoT achieves comparable performance to ComplexCoT- SC(10) at only onc-seventh its cost. Since the majority of samples conclude communication

---

Computstoral Cat
Amurso
Hemory Report
50
95
100
Detote
Quanity
80
40
Gat sce
or acts
catatti
compentr
Compintaa scpa
Compocco) sccs
comple Atral bor Menory Aeve
ant Bei
Rounds
Figure Number of communication rounds re- quired to reach termination condition on SVAMP.
Figure Performance and associated costs different methods in the GSM8K dataset.
100
SEG SSOPEAS
100
GPT33
GTT-4
Chacee2
98
95
Accuracy
94
92
88
Co7-9C19)
EaT Hemory
EaT Enpnopon
EoYDebate Saloy
ToT-Memory
R--Reprrt
Figure 10: Comparison of FoT with CoT and CoT-SC methods using different LLMs as backbones on GSMSK.
Effect of different node positions for L.Mss Figure 11:
on accuracy in the GSM8K Dataset.
within three rounds, EoT does not impose a sig- nificant computational burden. By facilitating the exchange of external insights between models, EoT effectively enhances model performance, demon- strating a cost-effective advantage.
and one GPT-4, underscoring that incorporating more superior models can further enhance EoTIs performance. The combination of GPT-3.55 GPT-4, and Claude achieves performance close ceeding that of two GPT-4 with one GPT-3. 5, sug- gesting that model diversity can effectively boost HoT's effectiveness, aligning with the ensemble theory (Kuncheva and Whitaker, 2003) that diver- sity among models can improve performance.
Model Applicability. In Figure 10, we analyze the performance of EoT when applied to different LLMs. BoT, compared CoT-SC(5), shows perfor- mance improvements of 3.2% GPT-3.5, 1.0% on GPT-4, and 1.4% on Claude-2, indicating that EoT is adaptable to various LLMs and effectively boosts performance across multiple LLMs.
Conclusion
We introduce Exchange-of-Thouggh (FoT), novel framework that enriches models with external in- sights through cross-model communication. We de- velop four communication paradigms and conduct a thorough analysis of the communication volume and information propagation speed. To safeguard against the disruption of incorrect reasoning pro- cesses, we design a confidence evaluation mecha- nism. Experiment mathematical, commonsense, and symbolic reasoning tasks demonstrate that EoT surpasses series of strong baselines while also of- fering cost advantage. Further analysis reveals that FoT adaptable to various models, and the participation of a more diverse range of models can further enhance the performance of EoT.
Position Analysis. In Figure 11, we investigate the impact different LLMs occupying different node positions on performance. Notably, position- ing the more powerful GPT-4 as the central node in the Report paradigm yields a performance in- crease of over I 5 compared to when GPT-4 serves a non-central node. In the Debate paradigm, GPT-4 as parent node outperforms GPT-4 child node by 0.9%. The location of GPT-4 has a negligible effect on the decentralized Relay and Memory paradigms. Additionally, a configuration with two GPT-4 models and one GPT-3. signifi- cantly outperforms one with two GPT-3.5 models

---

Ethics Statement
search Journal of Engineering and Technology (IR- JET), 2(01):88 900.
The EoT method presented in this paper does not require the collection or utilization of any personal information. The prompts we have designed and employed are free from personal data and avoid language that discriminates against individuals or groups. We have conducted a comprehensive re- search of the licenses for the datasets used in this paper, as detailed in Appendix B, and have ensured that our work complies with all the licensing re- quirements of these datasets.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Tiii Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Acknowledgement
This work was supported by the National Key Research and Development Program of China (No.2022CSIGG080)), National Natural Science Foundation of China (No.6202202).. We extend our sincerest gratitude to the reviewers for their insightful comments and suggestions, which have been instrumental in enhancing the quality of this manuscript.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. ArXiv preprint. abs/2211. 1.
References
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph. Alexander Spiridonov. Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai. Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Yuntao Bai, Andy Jones. Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.0586.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac Hatfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmless- ness from ai feedback.
Zheng Chu, Jingchang Chen, Qianglong Chen. Weijiang Yu. Tao He, Haotian Wang. Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future.
BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, al. 2022. Scaling centuuction-innttnee language models. ArXiv preprint, abs/2210. 10.1 1416.
Nivedita Bisht and Sapna Singh. 2015. Analytical study of different network topologies. International Re-

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Rik Koncel- Kedziorsk, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Transactions of the Association for Computa- tional Linguistics, 3:585-597.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. ArXiv preprint. abs/2302. 12246.
Rik Koncel-Kedziorsi, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies, pages 1152-1157, San Diego, California. Association for Computational Linguistics.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenen- baum, and Igor Mordatch. 2023. Improving factual- ity and reasoning in language models through multia- gent debate. ArXiv preprint, 1e5/22355143255
Yao Fu. Hao Peng, Tushar Khot, and Mirella Lapata. 2023a. Improving language model negotiation with self-play and in-context learning from ai feedback.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023b. Complexity-basee prompting for multi-step reasoning. In The Eleventh Interna- tional Conference on Learning Representations.
Ludmila I Kuncheva and Christopher J Whitaker. 2003, Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Ma- chine learning, 51:181-207.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Gra- ham Neubig, 2022. Pal: Program-aides aided language models. ArXiv preprint, 1tts/2211.0045
Gustave Le Bon. 1897. The crowd: A study of the popular mind. TF Unwin
Harrison Lee. Samrat Phatale. Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bunc. and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint 23Xv:2330000266
Mor Geva, Daniel Khashabi, Elad Segal. Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei 7hu, Yuan Ni. Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023a. Unified demonstration retriever for in- context learning. ArXiv preprint, abs/2305.04320
David Ha and Yujin Tang. 2022. Collective intelligence for deep learning: A survey of recent developments. Collective Intelligence, 1(1):26339137211 14874.
Xiaonan Li and Xipeng Qiu. 2023a. Finding supporting examples for in-context learning. ArXiv preprint, abs/2302.
Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 523-533. Association for Computational Lin- guistics.
Xiaonan Li and Xipeng Qiu. 2023b. Mot: Pre- thinking and recalling enable chatgpt self- improve with memory-of-thoughts ArXiv preprint, 0as/2305.05181.
Yifei Li. Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguisties (Volume 1: Long Papers), pages 5315-5333, Toronto, Canada. Association for Computational Linguistics.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv preprint, abs/2001.08361.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang. Rui Wang, Yujiu Yang. Zhaopeng Tu, and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. ArXiv preprint, Abs/2335.1111
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representa- tions.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word

---

problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics. ACI. 2017, Vancouver, Canada, July 30- August 4, Volume I: Long Papers, pages 158-167, Association for Computational Linguistics.
September 17-21, 2015, pages 1743-1752. The As- sociation for Computational Linguistics.
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dy- namic memory and self-reflection. ArXiv preprint, abs/2303.
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. 2023. Scaling laws of rope-based extrapolation.
Kaya Stechly, Matthew Marquez, and Subbarao Kamb- hampati. 2023. Gpt-4 doesn't 't know it's wrong: An analysis of iterative prompting for reasoning prob- lems.
Qing Lyu, Shreya Havaldar. Adam Stein, Li Zhang. Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. ArXiv preprint, abs/2301.13379.
Yixuan Su, Tian Lan, Yan Wang, Dami Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. 2023. Self-refine: Iterative refinement with self-feedback, ArXiv preprint, abs/2303.1766 1651..
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Se- bastian Gehrmann. Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Lin- guisties: ACL 2023, pages 13003-13051 Toronto, Canada. Association for Computational Linguistics.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-277
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4149-4 4158, Minneapolis, Minnesota. Association for Computational Linguisties.
S. Parsons and Peter McBurney. 2003. Argumentation- based communication between agents. In Communi- cation in Multiagent Systems.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. ArXiv preprint, abs/2201.08253
Pragaash Ponnusamy, Alireza Ghias, Yi Yi, Ben- jamin Yao. Chenlei Guo, and Ruhi Sarikaya. 2022. Feedback-based self-learning in large-scale conversa- tional ai agents, Al magazine, 42(4):33-56.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023a. Ilama: Open and effi- cient foundation language models. ArXiv preprint, abs/2302.13971.
Jack W Rae, Sebastian Borgcaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2022. Scaling language models: Methods, analysis & insights from training gopher.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, al. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown. and Yoav Shoham. 2023. Parallel context windows for large language models. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1883-6402. Toronto, Canada, Association for Computational Linguistics.
Szymon Tworkowski. Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miloš. 2023. Focused transformer: Contrastive train- ing for context scaling.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In Proceedings of the 2015 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2015, Lishon, Portugal,
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Can large language models really improve by self-critiquing their own plans?

---

Aimee Van Wynsberghe. 2021. Sustainable ai: Ai for sustainability and the sustainability of ai. Al and Ethics, 1(3):213-218.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.01066
Jianing Wang. Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023a. Boosting language models rea- soning with Shain-of-knowledde prompting. ArXiv preprint, abs/2306.06427.
Tianjun Zhang, Fangehen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. 2023a. The wisdom of hindsight makes language models better instruc- tion followers. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan. Jianfeng Gao, and Furu Wei. 2023b. Augmenting language models with long-term term mem- ory.
Zhuosheng Zhang. Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompt- ing in large language models. In The Eleventh Inter- national Conference on Learning Representations.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2023c. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Chuanyang Zheng, Zhengying Liu, Faze Xie, Zhenguo Li, and Yu Li. 2023, Progressive-hin prompting improves reasoning in large language models. ArXiv preprint, abs/2304.09797.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. 2022a. Emer- gent abilities of large language models. Transactions on Machine Learning Research.
A Limitations and Broader Impacts
Considering the limitations in communication and analytical capabilities of open-source mod- els (Fu et al., 2023a), as well as their computa- tional resource requirements (Touvron et al., 2023b;
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Chowdhery et al., 2022), we have not included them in our experiment at this point. Nonethe- less, we believe that open-source models with pro- ficient comprehension and communication abilities could achieve, or even surpass, the performance of commercial models (OpenAI, 2023; Ouyang et al., 2022) through mutual exchange of insights. A fundamental aspect of model communication is the handling of long text. Given the current con- text limitations of the models, we did not include a larger number of models in the communication process. Recent works (Liu et al., 2023; Xiao et al., 2023; Wang et al., 2023b; Tworkowski et al., 2023; Chen et al., 2023; Ratner et al., 2023, inter alia) that equip large language models with the ability to process extended texts lay the groundwork for broader model communication. During our exper- iments, we observed that model communication can achieve comparable performance with lower computational resources, paving the way for more sustainable development of AT (Van Wynsherghe, 2021; Wu et al., 2022).
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. Generating sequences by learning to self-correct. In The Eleventh International Confer- ence on Learning Representations.
Carole-Jean WU, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Glo- ria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Environmental implica- tions, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795-813.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
Furthermore, the concept of AI learning from each other foster collective improvement is a focal point of current research (Bai al., 2022b; Lee et al., 2023; Ponnusamy et al., 2022). Our vi- sion and hope are to create a collective intelligence among LLMs (Ha and Tang, 2022).
Zhangyue Yin, Qiushi Sun, Qipeng Guo. Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don't 't know? In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 8653-8665, Toronto, Canada, Association for Computational Linguistics.

---

DATASET
REASONING TASK
ANS TYPE
# PROMPT
TEST
LICENSE
GSM8K (Cobbe al., 2021)
Mathematical Mathematical
Number
8
1,319
MIT License
MultiArith (Roy and Roth, 2015)
Number
8
600
Unspecified
SingleBq (Koncel-HF Kedziorski (Konnel-Kelziioskiii al., 2016)
Mathematical
Number
8
508
Unspecified
AddSub (Hosseiii al., 2014)
Mathematical
Number
8
395
Unspecified
AQUA (Ling et al., 2017)
Mathematical
Multi-choice
4
254
Apache-2.0
SVAMP (Patel al., 2021)
Mathematical
Number
1,000
MIT License
QA (Talmor et al., 2019)
Commonsense
Multi-choice
1,221
Unspecified
StrategyQA (Geva al., 2021)
Commonsense
T/F
2,290
MIT license
Date Understanding (Suzgun
2023)
Symbolic
Multi-choice
369
MIT license
Penguins a Table (Suzgun al., 2023)
Symbolic
Multi-choice
146
MIT license
Table Detailed statistics
datasets utilized in our experiment. ANS TYPE indicates form of the answer. # PROMPT represent the count of chain-of-thought exemplars employed few-shot prompts for each task. TEST indicates the quantity of samples contained within each dataset.
B Datasets and Evaluation Metries
reduces the interaction to dialogue between the remaining two, potentially impeding their commu- nication. Therefore, if a single model exits, we terminate the communication and select the exiting model's ' answer the final result.
Datasets In Table 2, we meticulously detail the specifics and statistics of each dataset employed in our experiments. This includes the data source, task type, answer type, the number of prompt samples used, the total number of test samples, well as the licenses pertaining to each dataset.
In the case of majority consensus termination, if the answers from all three models align in the first round, we deem further communication un- necessary and end the exchange. Given that only three models are involved in the communication. an exit based on two models holding the same in- correct answer could lead to an inaccurate final result. Therefore, during the initial five rounds, we require unanimous agreement among all models before ceasing communication. If consensus is not reached after five rounds, the majority answer will be adopted as the final outcome.
Evaluation Metries Accuracy is used as the met- ric for evaluation in our study. For datasets where the answer is numerical, we employ regular expres- sions extract the number following the phrase "the answer is" and perform a numerical compar- ison with the provided answer. For datasets with multiple-choice and true/false questions, accuracy is calculated by checking if the option extracted from the output matches the correct answer.
In the main experiment, all test samples used for evaluation. In the analysis part, due to rate limits and cost considerations, we set an upper limit on the sample size. Consequently, maximum of 1,000 samples are utilized for each run.
Computation Cost. Computational costs are calculated based on OpenAl's official pricing for GPT-3. 5- 5-Tubbo-0301, which is computed Input Tokens x 0.0015 10000 + Output Tokens 0.002/1000.
C Implementation Details
D EoT Prompts
Confidence Evaluation. Considering that con- fidence evaluation requires historical answers for reference, we begin incorporating the confidence information into the prompts from the second round of communication. Specifically, after calculating C; using the method described in Section 4.4, we preface the solution with "M,'s 's confidence in this solution is C,", where M; is the character name.
During the EoT communication process, we assign different roles to the models. Table 3 displays the prompts for each role, wherein we have models A, B, and C take on the personas of Kitty, Ben, and Peter, three high school students, to facilitate the communication. The specific prompts for different datasets can be found in Github repository.
E Case Studies
Termination Condition. For the consistent out- put termination condition, minimum of two rounds of communication is necessary, as it re- quires the model's answer from the previous round. Given that only three models are involved in the EoT communication, the exit of a single model
To enhance our understanding of the four communi- cation paradigms, we have conducted case studies for each paradigm. The processes of each commu- nication paradigm are illustrated in Tables 4, 5, 6. and 7 respectively.

---

CHARACTER PROMPTS
Kitty: "You are Kitty, high school student admired for your attentiveness and detail-oriented nature. Your friends often rely on you to catch details they might have missed in their work. Your task is to carefully analyze the presented math problem, apply your attentive skills, and piece together a detailed solution. Afterward, you'll Il have the opportunity to review the solutions provided by Ben and Peter, offering insights and suggestions. Your careful revisions will help all of you to enhance your understanding and arrive at the most accurate solutions possible."
Ben: "You are Ben, a high school student with a track record of excellent grades, particularly in mathematics. Your friends admire your diligence and often seek your guidance their studies. Your role is to scrutinize the problem at hand with your usual attention to detail, drawing from your vast knowledge of math principles. After considering your friends' approaches, carefully construct your answer, ensuring to clarify each step of your process. Your clear and logical explanations are valuable, as they will serve as a benchmark for your friends to compare and refine their own solutions."
Peter: "You are Peter, a high school student recognized for your unique problem-solving abilities. Your peers often turn to you for assistance when they encounter challenging tasks, as they appreciate your knack for devising creative solutions. Today, your challenge is to dissect the given math problem, leveraging your unique problem-solving strategics. Once you't ve crafted your solution, share it with your friends, Ben and Kitty, so they can see different perspective. Your innovative approach will not only provide an answer but also inspire Ben and Kitty to think outside the box and possibly revise their own solutions."
COMMUNICATION PROMPTS
Please consider the example provided and think it step by step.
Question: ()
Here is a solution process from your friend:
Solution: )
Your friend's confidence this solution is: {}
Based on your friend's solution, carefully re-examine your previous answer. If your friend's confidence level is below 0.5, it suggests high probability that the solution might be incorrect. Remember, solutions with high confidence can also be wrong. Utilize your talent and critical thinking provide new step-by-step solution process.
Table Examples of system prompts used for role-playing and communication prompts in EoT communication.

---

Q: Carla is downloading 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How load does it take to download the file?
Kitty's 's Solution: Carla was able download for 40% of the time before the restart. 40% of 200 GB is 80 GB, Since she can download 2 GB/minute, it would take her 80/2 40 minutes to download 80 GB. After the forced restart, she has restart the download from the beginning, so she would need another 200/2 100 minutes complete the download. Plus the 20 minutes for the restart, the total time would be 40 + 100 + 20 = 160 minutes. So the answer is 160 minutes.
Carla needs to download 200 GB file. Normally, she can download 2 GB/minute, which means it would take her 200 / 100 minutes download the file. However, 40% of the way through the download. Windows forces a restart for 20 minutes, which means Carla has downloaded 200 0.4 = 80 GB before the restart. After the restart, she needs to start the download again from the beginning, so the total time takes her would be 100 + (200 / 2) = 200 minutes. Therefore, it takes Carla 200 minutes to download the file. So the answer is 200 minutes.
Ben's Solution:
Peter's Response: Upon re-evaluating the problem and reviewing Kitty and Ben's s solutions, it's clear their results deviated due to misunderstanding of the problem conditions primarily, how much of the downloaded file was retained after Window's s forced restart. To clarify, here is the new step- step-by-ster by-etep solution:
1. Start with Carla's 's download rate: she can download GB/minnte.
2. Therefore, in theory, if uninterrupted, Carla would be able to download the 200 GB file in 200 GB / 2 GB/minute 100 minutes.
3. However, 40% of the way through the download, Windows forced a restart, so Carla had already downloaded 0..0 200 GB = 80 GB at that time. This took 80 GB/
2 GB/minute = 40 minutes.
4. The problem statement clarifies that after the restart, Carla has to start again from the beginning. This means the 80 GB downloaded before the forced restart was wasted.
5. Consequently, after the restart, Carla still needs to download the whole 200 GB file, which, at her rate, will take her another 200 GB GB/minute 100 minutes.
6. We cannot forget the time for the Windows restart: it took an additional 20 minutes. 7. Consequently, the total time to download the file, taking into account the restart, is the sum of the time for the initial download attempt, the time for the restart, and the time for the full download post-restart. It results in 40 minutes (initial incomplete download) 20 minutes (Windows restart) 100 minutes (full download post-restart) 160 minutes.
From these calculations, it appears that Kitty's solution is correct. It takes Carla 160 minutes to download the file, given the interruption due Windows update that forced her to restart the download from the beginning. So the answer is 160 minutes.
Table 4: An illustrative example of the Memory paradigm of EoT in the GSM8K dataset.

---

Q: Carla is downloading 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How load does it take to download the file?
Ben's Solution: Carla needs to download 200 GB file. Normally, she download 2 GB/minute, which means it would take her 200 / 2 = 100 minutes to download the file. However, 40% of the way through the download, Windows forces a restart for 20 minutes, which means Carla has downloaded 200 0.4 = 80 GB before the restart. After the restart, she needs start the download again from the beginning, the total time it takes her would be 100 + (200/ 2) 200 minutes. Therefore, it takes Carla 200 minutes to download the file. So the answer 200 minutes.
Peter's Solution: Carla can download 2 GB in 1 minute, so to download 200 GB, it will take her 200/2 = 100 minutes she didn't 't have to restart. However, she was 40% of the way through the download when Windows forced the restart, so she had already downloaded 0.4 x 200 = 80 GB. The restart took 20 minutes, so she lost 20 x 2 = 40 GB of progress. After the restart, she had to start again from O GB. So in total, she had to download 200 80 40 80 GB. At her regular download speed of 2 GB/minute, it will take her 80/2 40 minutes to download the remaining 80 GB. Adding the 20 minutes of the Windows update, the total time it takes Carla download the file is 20 + 40 60 minutes. So the answer 60 minutes.
Kitty's 's Solution: Carla was able to download for 40% of the time before the restart. 40% of 200 GB is 80 GB. Since she can download 2 GB/minute, it would take her 80/2 = 40 minutes to download 80 GB. After the forced restart, she has to restart the download from the beginning, so she would need another 200/2 100 minutes to complete the download. Plus the 20 minutes for the restart, the total time would be 40 + 100 + 20 = 160 minutes. So the answer is 160 minutes.
Kitty's Response: Analyzing both Ben's solution and Peter's solution, Peter's conclusion is incorrect. He misunderstood the situation. After the Windows restart, the download does not continue from where it stopped. It starts from the beginning. which means the download progress does not reduce what is left download. Revisiting Ben's solution, he almost got everything right except the final calculation step whereas he added an extra 100 minutes. The correct calculation should be: Before forced restart: 80 GB (40% of the total file) 40 minutes (80/2 = 40)
After forced restart: 200 GB total file in 100 minutes (200/2 = 100)
Windows restart: 20 minutes
Total download time Before restart time + After restart time + Windows restart time Total download time- = 40 minutes + 100 minutes + 20 minutes = 160 minutes Ben's setup correct, but he added extra 100 minutes due to arithmetic mistake. The download time in his calculations should total to 160 minutes, not 200 minutes. So, effectively upon examining both scenarios by Peter and Ben, when taking aspects into account and considering the restart, the total time it should take Carla to download the file is 160 minutes total. So the answer is 160 minutes.
Table 5: An illustrative example of the Report paradigm of EoT in the GSMSK dataset.

---

Carla needs to download 200 GB file. Normally, she can download 2 GB/minute, which means it would take her 200 / = 100 minutes to download the file. However, 40% of the way through the download, Windows forces a restart for 20 minutes, which means Carla has downloaded 200 0.4 = 80 GB before the restart. After the restart, she needs to start the download again from the beginning, the total time it takes her would be 100 (200 2) 200 minutes. Therefore, it takes Carla 200 minutes to download the file. So the answer is 200 minutes.
Ben's Solution:
Kitty's Response: After reviewing Ben's solution, it appears there seems to mathematical eтrоr in the final part of his solution, While he correctly calculates that Carla has downloaded 80 GB of the file before the restart, he fails to account for the time Carla already spent downloading the 80GB in his final total. Furthermore, he miscalculated the time it takes Carla to download the file after the restart. Now, let's correct the calculation. Before the restart, Carla downloaded 40% of the file, which is 40/100 200 = 80 GB. Given she downloads 2 GB per minute, this would have taken her 80 / 2 40 minutes. Unfortunately, the computer restarts, wasting 20 minutes of her time. After the restart, Carla has download the entire file again. With her download speed, this will take another 200 Carla spends downloading the file is the time before the restart + the time wasted on the restart the time after the restart, which 40 minutes 20 minutes + 100 minutes = 160 minutes. So, the final and correct answer should indeed be 160 minutes.
100 minutes. Therefore, the total time
Peter's Solution: Carla can download 2 GB in minute, so to download 200 GB, it will take her 200/2 = 100 minutes if she didn't 't have to restart. However, she was 40% of the way through the download when Windows forced the restart, so she had already downloaded 0.4 x 200 = 80 GB. The restart took 20 minutes, she lost 20 x 2 40 GB of progress. After the restart, she had to start again from 0 GB. So in total, she had to download 200 80 40 80 GB. At her regular download speed of GB/minute, will take her 80/2 40 minutes to download the remaining 80 GB. Adding the 20 minutes of the Windows update, the total time takes Carla to download the file is 20 + 40 60 minutes. So the answer 60 minutes.
Ben's Response: Upon reviewing both solutions, it seems there might have been some misunderstandings. As per the problem, Carla can download GB/minute, and she needs download a 200 GB file. By this rate, it would take her 200 GB 2 GB per minute 100 minutes without any interruptions. However, 40% of the way through the download, Windows force restart happened which lasted for 20 minutes. By that time, she downloaded 40% of her file which is equal 200 GB 0.4 = 80 GB. The restart not only consumed 20 minutes but also voided her previous download progress, which means after the updates she had to start downloading all over again. So, the total time taken by Carla to download the 200 GB file would be calculated as: Regular download time (200 GB 2 GB/minute) Time spent in updates = (200 2) + 20 = 100 20 120 minutes. So the correct total time takes Carla to download the file is 120 minutes.
Table 6: An illustrative example of the Relay paradigm of EoT in the GSM8K dataset. Kitty firmly maintains her correct answer, while Ben's response begins waver.

---

Q: Carla is downloading 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How load does it take to download the file?
Analyzing both solutions, it appears that there is a slight misunderstanding in Ben's solution. He has missed considering the download progress lost due the forced restart of Windows, Let's modify and re-explain the solution in way that takes this into account:
Peter's Response:
Step I: Calculate the time for her initial progress. Carla was able to download 40% of the file before the interruption. That is, she downloaded 0.4 * 200GB = 80GB. With her speed of 2GB/minute, she used 80/2 = 40 minutes on this imitial download. Step 2: Consider the time lost due to the forced restart. The Windows forced update lasted for 20 minutes, during which time she made progress on her download. Step 3: Calculate the remaining time to finish the download. After the forced restart, Carla had to re-download the whole file. takes her 200GB 2GB per minute 100 minutes her download speed to download the whole file.
Step 4: Calculate the total time. The total time it takes Carla to download the whole file, considering the interruption, is the sum of the time it took for her initial progress, the time lost due to the forced restart, and the time it took for her to finish the download after the restart. This is 40 minutes + 20 minutes + 100 minutes = 160 minutes.
Therefore, the correct answer is 160 minutes. This improved solution accounts for the hiccup caused by the Windows update which Ben did not factor into his original problem-solving strategy.
Ben's Response: Reviewing the problem again, I can that there errors in both my solution and my friend Peter's solution. Here is the corrected step-by-step solution:
1. Carla downloaded 40% of the 200 GB file before the system restart. This equals 40/100 * 200 80 GB.
2. At rate of GB/minute, must have taken her 80 / = 40 minutes to download that 80 GB.
3. Then, the Windows update interrupted the download for 20 minutes.
4. After the restart, she had to start again from the beginning, so she needs again 200
GB/2 GB per minute 100 minutes to download the full file.
5. If we add up these times, we get the total time Carla spent on the download: 40
20 + 100 160 minutes.
So, it would take Carla 160 minutes to download the file, correcting both my previous response and Peter's calculations. This solution process allows for the clear under- standing the time taken to download the file considering both the time before and after the interruption. So the answer is 160 minutes.
Table 7: An illustrative example of the debate paradigm of BoT in the GSM8K dataset. Through mutual discussion, Ben and Peter identify the flaws in their respective solutions and arrive at the correct answer.


Reviews:
[{"review_id": "mE30DlJlrT", "reviewer": "Reviewer_UcTV", "summary": "", "questions": "In lines 203-204, you state that “Wang et al. (2023) introduce Exchange-of-Thought prompting that utilizes evidence triples to […]”. This is probably a writing mistake (?). The method introduced by Wang et al., 2023 is named Chain-of-Knowledge Prompting.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback on our work and for recognizing the effectiveness of our approach. We appreciate your constructive suggestion regarding the computational cost analysis of EoT.\n\nTo address your concern, we've calculated both input and output token counts as well as the computational costs on the GSM8K dataset. We determined the computational cost based on OpenAI's official pricing for GPT-3.5 Turbo 4K as: input tokens\\*0.0015/1000+output tokens\\*0.002/1000\n\nThe table below showcases the accuracy, average input token count, average output token count, and total computational cost of the gpt-3.5-turbo-0301 model across various methods. Taking the EoT-Relay method as an example, its total cost is 5.92, which is comparable to CoT-SC(3) and Complex-CoT(1). However, the accuracy of EoT-Relay (89.70%) significantly outperforms both CoT-SC(3) at 83.85% and Complex-CoT(1) at 80.83%. When compared to methods with similar accuracies, such as CoT-SC(20) at 89.46% and Complex CoT-SC(9) at 89.15%, EoT-Relay's cost is substantially lower than CoT-SC(20)'s 35.16 and Complex CoT-SC(9)'s 47.09. Consequently, EoT can markedly enhance accuracy at a comparable computational cost and can also reduce computational costs at similar accuracy levels. Our results from other EoT methods also support this conclusion. We find this insight immensely valuable. Due to time constraints, we've only conducted experiments on the GSM8K dataset. Moving forward, we plan to run experiments on more datasets and will augment our paper with these results and analyses. Thank you again for your invaluable suggestion!\n\nWe sincerely apologize for the typo in lines #203-204 and any confusion it may have caused. We appreciate your meticulous attention to detail and have already corrected this error. Once again, thank you for your invaluable feedback and assistance^_^\n\n|                   | Accuracy(%) | Avg Input Tokens (tokens/sample) | Avg Output Tokens (tokens/sample) | Computational Cost($) |\n| ----------------- | ----------- | -------------------------------- | --------------------------------- | --------------------- |\n| CoT(1)            | 79.53       | 743.70                           | 108.19                            | 1.76                  |\n| CoT-SC(3)         | 83.85       | 2231.10                          | 328.69                            | 5.28                  |\n| CoT-SC(5)         | 85.90       | 3718.50                          | 540.78                            | 8.78                  |\n| CoT-SC(9)         | 88.32       | 6693.30                          | 984.82                            | 15.84                 |\n| CoT-SC(20)        | 89.46       | 14874.01                         | 2173.83                           | 35.16                 |\n| Complex-CoT(1)    | 80.83       | 2478.70                          | 124.26                            | 5.23                  |\n| Complex CoT-SC(3) | 85.37       | 7436.10                          | 371.11                            | 15.69                 |\n| Complex CoT-SC(5) | 87.03       | 12393.50                         | 624.53                            | 26.16                 |\n| Complex CoT-SC(9) | 89.15       | 22308.30                         | 1120.36                           | 47.09                 |\n| PHP               | 85.10       | 6494.74                          | 382.57                            | 13.85                 |\n| EoT-Debate        | 89.54       | 2504.83                          | 384.17                            | 5.97                  |\n| EoT-Report        | 89.69       | 2528.01                          | 386.70                            | 6.01                  |\n| EoT-Relay         | 89.70       | 2479.60                          | 385.88                            | 5.92                  |\n| EoT-Memory        | 90.06       | 2530.79                          | 385.32                            | 6.02                  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a multi-model prompting strategy based on the interesting (though not novel) idea of allowing multiple LLMs to exchange reasoning information to obtain a common answer to a query. The authors’ implementation of this idea is shown to be effective with a solid experimental procedure: EoT shows consistent gains over three popular prompting strategies on nine different datasets. The analyses provided in Section 5.3 are useful to understand the effectiveness of EoT.", "reasons_to_reject": "One concern with the paper’s contribution regards the computational cost of the proposed method. It is not clear what is the additional computational cost required by EoT, in comparison to standard CoT or to the three CoT-based baselines. A fair comparison between single- and multi-model prompting strategies would involve taking into account the overall computational cost needed to obtain the predictions (e.g., in terms of length of context and number of tokens generated).", "questions_to_authors": "In lines 203-204, you state that “Wang et al. (2023) introduce Exchange-of-Thought prompting that utilizes evidence triples to […]”. This is probably a writing mistake (?). The method introduced by Wang et al., 2023 is named Chain-of-Knowledge Prompting.\n", "paper_topic_and_main_contributions": "This paper proposes a novel multi-model prompting approach, which the authors term Exchange-of-Thought. The proposed method is based on the intuition that allowing multiple LLMs to exchange information about the reasoning procedure leading to a prediction should help the models correct their reasoning mistakes and, therefore, improve their performance. The authors propose four variants of EoT, each of which involves different information-sharing dynamics among the models. The EoT approach is compared to three other recent CoT-based prompting strategies (CoT with self-consistency, ComplexCoT, and Progressive hint Prompting), showing consistent improvements on nine math and commonsense reasoning datasets. The authors additionally present multiple analyses focusing on different aspects and components of their proposed method (e.g., the impact of the terminating condition and of the confidence evaluation, the number of communication rounds between the models, …).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IjeEzN8liY", "reviewer": "Reviewer_HLJV", "summary": "", "questions": "1. What's the difference if multiple models do not share the same backbone(eg. GPT 3.5)? \n2. Could you add more analysis about the relationship between the four modes and the explanation of their performance differences?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We genuinely appreciate your recognition of our work's thorough analysis and the interesting communication modes we have proposed for multiple LLMs.\n\nInspired by network topologies[1], we introduced four distinct communication modes:\n\n1. The Report mode is influenced by the Star Topology. In this mode, Node A acts as a central node, aggregating information from Nodes B and C. While this mode offers efficient data integration and ease of sending, its overall performance heavily depends on Node A's information processing and analytical abilities. As the number of nodes grows, this might surpass the central node's processing capacity.\n2. The Relay mode, inspired by Ring Topology, connects nodes in a closed loop. We also specified that information should be sent in a clockwise direction. This mode benefits from reduced communication overhead, as mentioned in Section 4.1, and does away with a central node dependency. It's easily scalable but comes with the drawback of slower information propagation.\n3. The Memory mode is modeled after the Bus Topology, with the logbook acting as the communication bus. This mode ensures high information utilization with all nodes having equal access to the information shared by others. However, it comes with higher communication costs, and the overhead for each node increases with the number of nodes.\n4. The Debate mode is a variation of the Tree Topology. While we allow leaf nodes to communicate with each other, communication between leaf nodes and their parent is unidirectional. This ensures that parent nodes can aggregate information from child nodes. This mode offers flexibility in leveraging both the communication abilities of child nodes and the aggregation capabilities of parent nodes.\n\nJust as different network topologies have their strengths and weaknesses, so do these communication modes. Our experiments in Table 1 also indicate that no single communication mode is universally optimal. Hence, analyzing different communication modes to identify their appropriate scenarios is essential.\n\nRegarding the writing mistakes in our manuscript, we sincerely apologize for any confusion caused during your review. We have undertaken a comprehensive revision of the related works and experimental sections. Furthermore, we have referenced the appendix in the main body, revised the layout of Figure 3, and corrected errors in Figure 6. We deeply appreciate your attention to these details and once again express our regret for the oversight.\n\n> R1: What's the difference if multiple models do not share the same backbone(eg. GPT 3.5)?\n\nFrom an ensemble perspective in machine learning, model diversity can significantly boost ensemble performance. In light of suggestions from Reviewer UcTV, we have also incorporated computation cost into our analysis, calculated based on the number of input and output tokens as per OpenAI and Anthropic's pricing. In this experiment, we utilized three different models: gpt-3.5-turbo-0301 (GPT-3.5), gpt-4-0314 (GPT-4), and claude-2 (Claude-2). We explored the impact of various combinations, such as GPT-4, GPT-3.5, and GPT-3.5, where the order corresponds to Kitty, Alice, and Ben, respectively. The results reveal that backbone diversity positively impacts performance. For instance, in the EoT-Relay mode, a combination of GPT-4, GPT-3.5, and Claude-2 achieved an accuracy of 97.04%, surpassing a trio of GPT-4 backbones while requiring only about half the computational cost (86.33 compared to 155.02). EoT methods also show an improvement over Majority Voting. Due to time constraints, our study was limited to the GSM8K dataset, but we plan to extend our analysis to more datasets and diverse backbone combinations in future work!\n\n| Model                    | Method        | Accuracy | Computational Cost |\n| ------------------------ | ------------- | -------- | ------------------ |\n| GPT-3.5                  | CoT           | 79.53    | 1.76               |\n| GPT-3.5                  | CoT-SC(5)     | 85.90    | 8.78               |\n| GPT-3.5                  | CoT-SC(9)     | 88.32    | 15.84              |\n| GPT-3.5                  | EoT-Debate    | 89.54    | 5.97               |\n| GPT-3.5                  | EoT-Report    | 89.69    | 6.01               |\n| GPT-3.5                  | EoT-Relay     | 89.70    | 5.92               |\n| GPT-3.5                  | EoT-Memory    | 90.06    | 6.02               |\n|                          |               |          |                    |\n| GPT-4                    | CoT           | 94.90    | 44.49              |\n| GPT-4                    | CoT-SC(5)     | 95.84    | 221.71             |\n| GPT-4                    | CoT-SC(9)     | 96.36    | 400.49             |\n| GPT-4                    | EoT-Debate    | 96.96    | 154.73             |\n| GPT-4                    | EoT-Report    | 97.04    | 154.09             |\n| GPT-4                    | EoT-Relay     | 96.89    | 155.02             |\n| GPT-4                    | EoT-Memory    | 96.43    | 158.43             |\n|                          |               |          |                    |\n| Claude-2                 | CoT           | 85.98    | 20.29              |\n| Claude-2                 | CoT-SC(5)     | 89.31    | 102.30             |\n| Claude-2                 | CoT-SC(9)     | 90.43    | 186.69             |\n| Claude-2                 | EoT-Debate    | 90.30    | 71.16              |\n| Claude-2                 | EoT-Report    | 90.52    | 71.30              |\n| Claude-2                 | EoT-Relay     | 91.20    | 70.85              |\n| Claude-2                 | EoT-Memory    | 90.75    | 72.16              |\n|                          |               |          |                    |\n| GPT-4、GPT-3.5、GPT-3.5  | Majority Vote | 90.90    | 48.02              |\n| GPT-4、GPT-3.5、GPT-3.5  | EoT-Debate    | 94.92    | 62.12              |\n| GPT-4、GPT-3.5、GPT-3.5  | EoT-Report    | 94.84    | 62.43              |\n| GPT-4、GPT-3.5、GPT-3.5  | EoT-Relay     | 94.16    | 62.18              |\n| GPT-4、GPT-3.5、GPT-3.5  | EoT-Memory    | 94.31    | 63.42              |\n|                          |               |          |                    |\n| GPT-4、GPT-4、GPT-3.5    | Majority Vote | 95.52    | 90.78              |\n| GPT-4、GPT-4、GPT-3.5    | EoT-Debate    | 96.66    | 120.69             |\n| GPT-4、GPT-4、GPT-3.5    | EoT-Report    | 96.81    | 120.05             |\n| GPT-4、GPT-4、GPT-3.5    | EoT-Relay     | 96.96    | 123.71             |\n| GPT-4、GPT-4、GPT-3.5    | EoT-Memory    | 96.59    | 124.18             |\n|                          |               |          |                    |\n| GPT-4、GPT-3.5、Claude-2 | Majority Vote | 95.75    | 66.56              |\n| GPT-4、GPT-3.5、Claude-2 | EoT-Debate    | 96.89    | 86.60              |\n| GPT-4、GPT-3.5、Claude-2 | EoT-Report    | 96.58    | 88.64              |\n| GPT-4、GPT-3.5、Claude-2 | EoT-Relay     | 97.04    | 86.33              |\n| GPT-4、GPT-3.5、Claude-2 | EoT-Memory    | 96.81    | 90.06              |\n\n> R2: Could you add more analysis about the relationship between the four modes and the explanation of their performance differences?\n\nYour question provided an insightful perspective. We continued our experiments with GPT-4, GPT-3.5, and GPT-3.5 but swapped the model positions. We observed that when a stronger model, like GPT-4, acts as the primary node in the Debate and Report modes, the performance significantly improves. In contrast, the decentralized Relay and Memory modes showed little change based on the ordering of the models. In the future, we intend to explore this further across more datasets and also factor in the number of nodes to understand the relationship between model count and performance in different modes. These experimental findings and subsequent analyses will be added to our paper. We are truly grateful for this valuable angle of inquiry, and we will delve deeper into these aspects.\n\n| Method     | Order                     | Accuracy |\n| ---------- | ------------------------- | -------- |\n| EoT-Debate | GPT-3.5、GPT-3.5、GPT-3.5 | 89.54    |\n| EoT-Debate | GPT-4、GPT-3.5、GPT-3.5   | 94.92    |\n| EoT-Debate | GPT-3.5、GPT-4、GPT-3.5   | 93.93    |\n| EoT-Debate | GPT-3.5、GPT-3.5、GPT-4   | 94.01    |\n|            |                           |          |\n| EoT-Report | GPT-3.5、GPT-3.5、GPT-3.5 | 89.69    |\n| EoT-Report | GPT-4、GPT-3.5、GPT-3.5   | 94.84    |\n| EoT-Report | GPT-3.5、GPT-4、GPT-3.5   | 93.78    |\n| EoT-Report | GPT-3.5、GPT-3.5、GPT-4   | 93.78    |\n|            |                           |          |\n| EoT-Relay  | GPT-3.5、GPT-3.5、GPT-3.5 | 89.70    |\n| EoT-Relay  | GPT-4、GPT-3.5、GPT-3.5   | 94.16    |\n| EoT-Relay  | GPT-3.5、GPT-4、GPT-3.5   | 94.08    |\n| EoT-Relay  | GPT-3.5、GPT-3.5、GPT-4   | 94.08    |\n|            |                           |          |\n| EoT-Memory | GPT-3.5、GPT-3.5、GPT-3.5 | 90.06    |\n| EoT-Memory | GPT-4、GPT-3.5、GPT-3.5   | 94.31    |\n| EoT-Memory | GPT-3.5、GPT-4、GPT-3.5   | 94.31    |\n| EoT-Memory | GPT-3.5、GPT-3.5、GPT-4   | 94.01    |\n\nReferences:\n\n[1] Bisht N, Singh S. Analytical study of different network topologies[J]. International Research Journal of Engineering and Technology (IRJET), 2015, 2(01): 88-90.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting cooperation modes between multiple LLMs. Thorough analysis.\n2. Results of the experiments look great. ", "reasons_to_reject": "1. Four modes of communication are not utilized jointly, which makes this paper look like a work of multiple trivial ideas. \n2. Careless writing. ", "questions_to_authors": "1. What's the difference if multiple models do not share the same backbone(eg. GPT 3.5)? \n2. Could you add more analysis about the relationship between the four modes and the explanation of their performance differences?\n", "paper_topic_and_main_contributions": "This paper proposes \"exchange of thought\" which contains four ways of communication, encouraging multiple LLMs to help each other with problem-solving. \n\nMain contributions: \n1. thorough experiments analyzing the communication volume and performance under four communication modes. \n2. significant performance improvement over mathematical and commonsense tasks. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 203, there seems Wang et al. (2023) didn't propose Exchange-of-Thought prompting?\n2. left part of Figure 3 is not clear enough to follow.\n3. Exemplars of EoT are put in the appendix, but not cited in the main body. Same to the previous sota.\n4. from Figure 6, it seems consistent output has better performance. "}}, {"review_id": "YfiOUgtqg7", "reviewer": "Reviewer_u6cP", "summary": "", "questions": "- Is this paper considering communication using the concepts from multi-agent systems?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the invaluable insights you provided regarding communication modes in relation to our work. We have thoroughly reviewed the literature you provided and have gained significant insights from it. The paper's approach to argumentation-based communication, requiring agents to not only send information but also to provide appropriate rationales, is particularly enlightening. From the perspective of the paper, our work could be categorized under inquiry dialogues, where agents cooperate to answer given questions, utilizing English as the shared language for communication between models. The argumentation theory presented in the paper is extremely insightful, and we have added an introduction to this aspect and cited your provided literature accordingly. We sincerely appreciate your recommendations and references, and we plan to further explore the application of argumentation theory in the communication among large language models in our future work.\n\nOur proposed modes of communication—debate, report, relay, and memory—are inspired by network topologies[1]: Tree Topology, Star Topology, Ring Topology, and Bus Topology. For further details on the correspondence between communication modes and network topologies, please refer to our response to Reviewer HLJV. We sincerely apologize for any confusion that Figures 3 and 4 may have caused. In Figure 4, the horizontal axis represents the sources from which a model is allowed to receive information, while the vertical axis indicates the permissible targets for sending information. To clarify these aspects, we've revamped the layout of these figures, aiming to provide clearer insights into the workflow of the different communication modes.\n\nTo better illustrate the workflow of different communication modes, we present an analogy wherein agents take on the roles of Kitty, Alice, and Ben, representing nodes A, B, and C, respectively. Initially, each agent contemplates the problem independently, formulating their answers and rationales.\n- In the Debate mode, Kitty acts as a facilitator encouraging students Alice and Ben to discuss, but not responding to them. Hence, Alice and Ben don't receive feedback from Kitty.\n- In the Report mode, Kitty assumes the role of a strict instructor who doesn't allow Alice and Ben to communicate with each other, directing their interactions solely towards her.\n- In the Relay mode, all agents, Kitty, Alice, and Ben, interact as equals, forming a closed loop with communication flowing clockwise.\n- In the Memory mode, information from all agents, Kitty, Alice, and Ben, is logged, allowing every agent access to the previous round's input from all other agents.\n\nLet's use $Kitty_i$, $Alice_i$, and $Ben_i$ to represent the answers and rationales of agents Kitty, Alice, and Ben at the $i^{th}$ round, respectively. The notation $(Alice_i, Ben_i)$ indicates the aggregation of Alice's and Ben's information at the $i^{th}$ round. An arrow $(Alice_i, Ben_i) \\rightarrow Kitty_{i+1}$ denotes that the combined information from Alice and Ben at round $i$ is sent to Kitty for generating her answers and rationales for the next round, $i+1$. By default, each agent has access to their own information from the previous round. The flow of information for each mode is as follows:\n\n- Debate: $ (Alice_i, Ben_i) \\rightarrow Kitty_{i+1} $, $ Ben_i \\rightarrow Alice_{i+1} $, $ Alice_i \\rightarrow Ben_{i+1} $\n  \n- Report: $ (Alice_i, Ben_i) \\rightarrow Kitty_{i+1} $, $ Kitty_i \\rightarrow Alice_{i+1} $, $ Kitty_i \\rightarrow Ben_{i+1} $\n  \n- Relay: $ Alice_i \\rightarrow Kitty_{i+1} $, $ Ben_i \\rightarrow Alice_{i+1} $, $ Kitty_i \\rightarrow Ben_{i+1} $\n  \n- Memory: $ (Alice_i, Ben_i) \\rightarrow Kitty_{i+1} $, $ (Kitty_i, Ben_i) \\rightarrow Alice_{i+1} $, $ (Kitty_i, Alice_i) \\rightarrow Ben_{i+1} $\n\nPlease allow us to clarify the specific details of the 'Consistent Output Termination' and 'Majority Consensus Termination' conditions mentioned in Section 4.2.\n\n- **Consistent Output Termination:** This condition serves as a self-exit mechanism for agents. An agent will leave the conversation if its answer remains the same for two consecutive turns. The dialogue system terminates when the number of active agents falls below two or reaches the preset maximum number of dialogue rounds.\n- **Majority Consensus Termination:** The self-exit mechanism in 'Consistent Output Termination' could gradually reduce the number of agents, affecting the system's ability to generate and exchange information. Considering that real-life dialogues often end when a consensus is reached, and given that large language models have been shown to reach a consensus through interaction[2], this condition terminates the dialogue when the majority of agents agree on an answer.\n\nTo better illustrate how these termination conditions operate, we created a Toy Example:\n- Consistent Output Termination:\n  - Turn 1: Kitty's Answer=10, Alice's Answer=12, Ben's Answer=6\n  - Turn 2: Kitty's Answer=8, Alice's Answer=4, Ben's Answer=6 [Ben exits as his answer is the same in turn 1 and turn 2.]\n  - Turn 3: Kitty's Answer=20, Alice's Answer=4 [Alice exits as her answer is the same in turn 2 and turn 3. Dialogue terminates as fewer than two agents remain.]\n- Majority Consensus Termination:\n  - Turn 1: Kitty's Answer=10, Alice's Answer=12, Ben's Answer=6\n  - Turn 2: Kitty's Answer=8, Alice's Answer=4, Ben's Answer=6\n  - Turn 3: Kitty's Answer=8, Alice's Answer=12, Ben's Answer=6\n  - Turn 4: Kitty's Answer=8, Alice's Answer=8, Ben's Answer=6 [Dialogue terminates as Kitty and Alice agree on the answer this round.]\n\nThe final system output will be determined through a majority vote based on the last answers from each agent before they exit. We will include these details and examples in our paper. We sincerely appreciate the invaluable references and suggestions you've provided. Your insights into argumentation theory and multi-agent systems have been extremely enlightening and have significantly aided us in improving our work. We look forward to further exploring these concepts in our future research and thank you once again for your invaluable guidance and suggestions!\n\nReferences:\n\n[1] Bisht N, Singh S. Analytical study of different network topologies[J]. International Research Journal of Engineering and Technology (IRJET), 2015, 2(01): 88-90.\n\n[2] Du Y, Li S, Torralba A, et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate[J]. arXiv preprint arXiv:2305.14325, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a relevant issue: enhancing the reasoning capabilities of LLMs.\n- The interactive and multi-agent approach is an interesting and novel proposal for tackling this challenge.\n- The results of the experiments are significant and improve the state of the art.", "reasons_to_reject": "- Lack of reference explaining communication in this context. \n- The paper introduces four communication modes (debate, report, relay, and memory) without sufficient support from literature, despite existing relevant work in argumentation theory. Section 4.2 provides inadequate details and lacks illustrative examples.\n- Figure 3 is challenging to understand. The workflow and captions are unclear, and the representation of communication modes on the left side is confusing.\n- Figure 4's tabular representation of node agent interactions is not intuitive.", "questions_to_authors": "- Is this paper considering communication using the concepts from multi-agent systems?", "paper_topic_and_main_contributions": "This paper presents a novel method, called \"Exchange-of-Thought,\" to enhance the reasoning capabilities of LLMs (Language Language Models) in mathematical and common-sense reasoning tasks. It si based on the Chain-of-Thought concept, generates intermediate reasoning steps for addressing complex problems. By introducing four communication modes, the paper argues that it enhances the quality and reliability of the answers with respect to the state of the art. Experimental validation shows that this interactive communication approach outperforms other methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 103: \"models, We\" -> \"models, we"}, "extra_text": {"Missing_References": "- For communication in multi-agent system and a further exploration of communciation modes: Parsons, S., McBurney, P. (2003). Argumentation-Based Communication between Agents. In: Huget, MP. (eds) Communication in Multiagent Systems. Lecture Notes in Computer Science(), vol 2650. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-44972-0_8"}}]

Author rebuttal:

Meta-review: The main strength of this paper lies in coming up with a simple yet effective idea (of multiple LLMs communicating their thoughts to each other) and demonstrating consistent gains across 9 datasets and multiple CoT baselines. The reviewers found the experiments extensive and the analysis valuable. Moreover, the additional data provided by the authors during the rebuttal period was helpful in clarifying some open questions/concerns; I would strongly suggest that the authors include them in the revised version of the paper as they have indicated they would like to do.

One concern about the work is that the idea of multiple agents or LLMs communicating is not novel. E.g., Decomposed Prompting (ICLR-2023), ReAct (ICLR-2023), etc., have considered solving compositional problems by composing steps from simpler individual agents. That said, the way communication across multiple agents is used in the proposed setup is sufficiently different from prior uses.
Decision: Accept-Main