LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation
Authors: Zhitao He
Pengfei Cao
Yubo Chen
Kang Liu
ruopeng li
Mengshu Sun
Jun Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Causality explanation generation; Commonsense reasoning; Large language model

Abstract: Causality Explanation Generation refers to generate an explanation in natural language given an initial cause-effect pair. It demands rigorous explicit rationales to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to be easily memorized, making it challenging for large language models since they are often suffering from spurious causal associations when they encounter the content that does not exist in their memory. In this work, we introduce LEGO, a Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for causality explanation generation. Specifically, we treat LLM as character malleable LEGO block and utilize role-playing to assign specific roles to five LLMs. We firstly devise a Fine-grained World Knowledge Integration Module to augment information about tasks for alleviating the phenomenon of spurious causal associations. Then, we leverage an Iterative Feedback and Refinement Module to improve the generated explanation by multi-aspect feedback. Extensive experiments on widely used WIKIWHY and e-CARE datasets show the superiority of our multi-agent framework in terms of reasoning about the causality among cause and effect.

LEGO: A Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for Causality Explanation Generation
Zhitao Hel Pengfei Cao Yubo Chen 1.2 Kang Liu
Ruopeng li Mengshu Sun", Jun Zhao 1.2
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Ant Group, Hangzhou, China
{zhitao. he, pengfei.cao, yubo.chen, kliu, jzhao]@nlpr. ia.ac.cn
Abstract
Imput (Cause- pair)
Cause The flett of the seventt voyage of the Ming Treesture Voyages
Causality Explanation Generation refers to gen- erate an explanation in natural language given initial cause-effect pair. It demands rigor- ous explicit rationales to demonstrate the ac- quisition of implicit commonsense knowledge, which is unlikely to be easily memorized, mak- ing it challenging for large language models since they are often suffering from spurious causal associations when they encounter the content that does not exist in their memory. In this work, introduce LEGO, Multi-agent Collaborative Framework with Role-playing and Iterative Feedback for causality explana- tion generation. Specifically. we exploit LLM as character malleable L.EGO block and uti- lize role-playing to assign specific roles to five LLMs. We firstly devise a Fine-grained World Knowledge Integration Module to augment in- formation about tasks for alleviating the phe- nomenon of spurious causal associations. Then, we leverage an Iterative Feedback and Refine- ment Module improve the generated expla- nation by multi-aspect feedback. Extensive experiments on widely used WIKIWHY and c-CARE datasets show the superiority of our multi-agent framework in terms of reasoning about the causality among cause and effect.
salling under favomble conditions and running before the southwest monsoon
their homse voyage. Effect e; The fleet of the seventh voyage of the Ming Treasure Voyages did not stop at Ceylon or southern India.
Output (Explanation)
Explaration X monszon was striking Ceylon or and the Dleet of the sevecth voyage of the Ming Tressue Voyage did not want to he cought the storm.
at time
Figure I: An example causality explanation genera- tion from the WIKIWHY dataset (Ho et al., 2023).
Despite the extensive applications of causality explanation generation, this task is highly challeng- ing as it demands rigorous and explicit rationales to demonstrate that model not only "knows" the causality in task but also "understands" the under- lying mechanies of why that is the case. Previous studies (Du et al., 2022: Ho et al., 2023) leverage single language model with in-context learning to tackle the task, however, language models of- ten generate spurious causal association based on the given cause-effect pair, leading to deviation from the correct reasoning path. As depicted in Figure 2(a), when observing the information that "The fleet was sailing under favorable conditions" in cause C, GPT-3 and GPT-3.5 both generate the spurious causal association without exception, i.c.. "under favorable condition" + "they were making good time", which leads to incorrect reasoning di- rections and hinders language models from taking a step further to explore the causal mechanism.
1 Introduction
Causality explanation generation generative task that aims to explain why the given cause-effect pair true using natural language. For example, given the cause clause C and effect clause in Figure 1, a corresponding explanation X needs to be generated is "A monsoon was striking Ceylon or southern India at the time and the fleet of the sev- enth voyage of the Ming Treasure Voyage did not want be caught in the storm". Causality explana- tion generation can facilitate various applications, including explainable question answering (Yang et al., 2018), complex reasoning (Dalvi et al., 2021) and future event prediction (Zhou et al., 2022).
Through in-depth analysis, we attribute this kind of errors to the limitation of unidirectional rea- soming of large language models (LLMs).. As il- lustrated in Figure 2(a), both GPT-3 and GPT-3.5 only conduct unidirectional reasoning based on the content of the cause. When condition, they directly associate it with making good time, which suggests that this spurious causal

---

sputious cansel asanciation
voyage" mentioned in cause side (pass through). Subsequently, from the regional climate data, we learn another key fine-grained world knowledge: "The southwest monsoon from June to September accounts for most of the rainfall in the region". Fi- nally, by linking these two pieces of fine-grained world knowledge obtained by the bidirectional rea- soning process, we discover that the return time of the seventh voyage coincides with the arrival of the southwest monsoon in southern India. Therefore, we can conclude = "A monsoon was striking Cey- lon or southern India at the time". Combining this with task-specific commonsense knowledge, such as the fleet needed to avoid storms at sea, we can deduce = "the fleet did not want to be caught in the storm". In summary, to sucessful complete this task, the system needs the abilities of bidirectional reasoning and knowledge retrieval to effectively integrate fine-grained world knowledge. Further- more, it necessitates the capacity for commonsense induction to augment task spec-spec commonsense knowledge.
Couse salling under averable conditions they were making good time
Output (GPT-3)
The fleet of the seventh voyage the Ming Tresure Voyages salling uader
favorable conditione cuminng before svutawest monsoon homeward
voyage, mrant tay wase making good and noed stop at Caylon
or southem Indi.
Output (GPT-3.5)
Since the Beet was seiling under Eavorible cordition. they were abie to make good tine and didntt need make any slogs for supplies rest. Caylon and southem Indla but since the feet didot need
typiod stogping points for theer
they simply sailed pastthem.
6a) The sparious assuciation and explanation of LLM
Cause- Ming Tr Preasure Voyages the sev eeeeetth voyage homeward vay age
be tressrm veen the oreee menttrn experitioe enderiiinn be betwere 14teland 14SS..
The depared Monb
1833.. On 33 juy
1a33 thy antrad cagizal Dejing
Taesosteest fran Juae to Suparberr aoouaa &ee east of the rasiall region...
Effect -e Ceylon or southeen India
C Climate
Output (Human)
A monsacn was stnling Ceyloz or souther Indis at the tme and the Elctt of the seveni: voyage the Ming Treasure Voyage
orught the storm.
The thought and explanation buman
The explanation generated by large language models and human for the task mentioned above.
Figure 2:
Based on the detailed analysis above, we can be observed that it demands multiple abilities, such as bidirectional reasoning, knowledge retrieval, and commonsense induction for this complex task. Al- though LLMs demonstrate wide range of capa- bilities, single language model is unable to si- multancously provide all of them. Therefore, we propose novel Multi-agent Collaborative Frame- work with Role-playing and Iterative Feedback (LEGO) to effectively combine the different abili- ties of multiple large language models for causality explanation generation. Specifically, we exploit LLMs as character malleable LEGO blocks and utilize role-playing to assign specific roles to the five LLMs. We firstly devise Fine- Tine-grained World Knowledge Integration Module to augment infor- mation about tasks to alleviate the phenomenon of spurious causal associations. The module con- sists of three LLMs, two LLMS are designated as Cause Analyst and Effect Analyst, reasoning around Cause and Effect respectively to simulate the process of bidirectional inference, and posing questions to another LLM which acts as Knowl- edge Master to autonomously mine fine-grained world knowledge. Then, we leverage an lterative Feedback and Refinement Module to improve expla- nation by multi-aspect feedback. The module uti- lizes one LLM as Explainer to generate an initial explanation, and iteratively receives Observation
association appears to be ingrained in the param- eters of large language models. As a result, they inevitably deviate from the correct reasoning path, overlook crucial information present in the effect, and fail to autonomously scarch for broader world knowledge beyond the given text.
Actually, it is widely accepted that human mem- ory is characterized by its bidirectional associa- tive and parallel processing capabilities (Kosko, 1988; Hattori and Hagiwara, 1995; Anderson and Bower, 2014). The ability of bidirectional reason- ing may be helpful in alleviating the phenomenon of spurious causal associations. We demonstrate the bidirectional thought process required to ar- rive at the gold explanation in Figure 2(b). On the one hand, we begin our reasoning from the infor- mation on cause side and progressively search for "Ming Treasure Voyages", "the seventh voyage", and "homeward voyage". Eventually, we obtain the key fine-grained world knowledge: "The fleet departed from Hormuz on 9 March 1433, ... On 22 July 1433, they arrived in the capital Beijing". On the other hand, we reason from the information on effect side and search for "Ceylon or southern In- dia", then associate it with the route of "the seventh

---

feedback and Commonsense feedback from Critic LLM refine its explanation.
Analyst. In addition, The third agent, serves the Knowledge Master for answering the questions of the reasoners, does not require specific prompt. Reasoning Towards Causality After the roles are assigned, the Cause Analyst M and Effect An- alyst A will collaborate in reasoning about the fine- grained world knowledge by thought and action (Yao et al., 2022). The Cause Analyst is responsi- ble for reasoning about the information in Cause and directing the conversation around the causality in task. Meanwhile, the Effect Analyst is designed to reason about the information Effect and fol- low the reasoning trace of the Cause Analyst. One example of Effect Analyst reasoning about the task presented in Figure is shown below:
Overall, the main contributions of this work can be summarized as follows: 1) We propose a novel multi-agent collaborative framework with role-playing and iterative feedback (LEGO) to ef- feetively combine different abilities of multiple LLMs for causality explanation generation: 2) We devise Fine-grained World Knowledge Integra- tion Module to augment information about tasks through the interaction three agent, i.e. Cause Analyst, Effect Analyst and Knowledge Master. We leverage an Iterative Feedback and Refinement Module to improve the generated explanation by multi-aspect feedback involving two LLMs, i.e. Ex- plainer and Critic; 3) Extensive experiments WIKIWHY and e-CARE show the superiority of our multi multi-agent framework in terms of reasoning about the causality among cause and effect.
Thought: need to analyze the about Ceylon, southern India and southwest monsoon. Ask: [Ceylon, southern India and southwest monsoon]
Effect. So need
2 Methodology
this section, we introduce our multi-agent col- laborative framework LEGO. As shown in Figure 2. our framework consists of two major compo- nents involving five LLMs: (1) Fine-grained World Knowledge Integration Module, which augments information about tasks through the interaction of three agents; (2) Iterative Feedback and Refine- ment Module, which utilizes LLM serve as Ex- plainer generate an initial output, and iteratively receives Observation and Commonsense feedback from Critic LLM to refine its explanation.
The Knowledge Master returns its Observation:
Observation: Ceylon, is an island country located in the Indian Ocean, off the southern coast of India. It experiences tropical climate and is greatly influenced by the southwest monsoon... . The southwest monsoon typically occurs between June and September each year...
Formally, we denote the Cause Analyst message obtained time by Mt and the corresponding observation from Knowledge Master is Om. the Effect Analyst message A₃ and the correspond- ing observation is Oat the historical messages:
2.1 Fine-grained World Knowledge Infegration
We devise this module to precisely augment infor- mation about task for alleviating the phenomenon of spurious causal associations.
H, {(M, Om, ; Ai, 0
(1)
Cause-Effect Analyst Role Assignment After receiving the task, the Cause Analyst role and the Effect Analyst role will be assigned to two LLMs respectively by inception prompt (Li et al., 2023). In practice, a system message prompts are passed to the LLMs before the conversations start to assign LLMs with corresponding roles. We refer to the Cause Analyst message prompt is Pc and that of the Effect Analyst is PR. Let L1 and L2 denote two large language models. when the system message is passed to those large language models respectively, we obtain M + Lffe and A + L''s which are referred to as the Cause Analyst role and the Effect
At the next time step t 1, the Cause Analyst takes the historical conversation message set and creates new message Mj 1. as shown:
M41 M(H))
(2)
The assistant reasoning will respond with A411:
A++1 = A(He, (M++1, Omez))
(3)
We focus on the Observations obtained by two reasoners during their interaction, which encom- pass fine-grained knowledge about the task. These

---

Carae: Tae flort the seventh vocage ot the Ming Trresure
Bocum Narto Novis Peplaration Condenit ty diffinit ohen put treedaie There wne aad to bolk pooda gosé sasis azé do so on usiet tree ladked. Thes bro decanns patitress ca Hisyali a'aasttiaa while workad
There tree conmiias
Grao: The flret the seveth of the
Elmngu Varuntra det possard watting
kyages saling Iavoritle cooditions and ruszing befoce snrtioes taansioss et hocreore vayagg. Edfict: The Bost the seventh vogagg of Ming Treeare Yogages did notitog Cegion
Vegaggs wos ing under favorible osrdil ons and nurningg before the souftveed mousoon on thei borraward suyage
Effect. T'ke fleet the serenth veyage of the bing Treesure Veyages did stop at Coylon or arrthorn India.
Hocsur Eflect Balayy llasal siosa.
by University oc 1964.
Explaration 1f the Univeentt Teoe is consevative thy woultht want parlo verking them who hawe lisend stews
R easoning
F Few-shot prompt
Initial Input
Cause Analyst
Theaghtl wed to wna you fhee Crrass So inee'ss alochout Doct of les oosggo Slug
Observathee The Eac the thesowsth Ming Trasure
Observation Feedback
Veengg nnd and is thanetty surmerr 1431. .. The aut and ins Chea and taniied to Seedhaud a. poddacss balas the Aesaa Gial, and vasi cuat Africa..
Troose Troose-Vorgge Verigns
soowrt eagage of Hinn Theosr Vaye yod
Common sense Feedback
haught to wajeu boss be Eilect, tumilz deaa Celkee scndass loda and surbbvod
Oereraati Cryloe. alo zoowa
lnased reger teperte trog gresly
Critic
Expleiner
Knowledge Master
Refined Explanation
adcchoot sygaee sevciih no
Dhough cad es free Clsic
trriighh Vegres pheasi-duy
Ming cagges
(Initial Explanation)
Ex
Explain
Effect Analyst
Fine-grained World Knowledge Integration
Iterative Feedback and Refinement
Figure 3: Overview of our multi-agent framework. It first augments information about current task through the interaction of three agents (left), then utilizes one LLM serve as Explainer to generate an initial output and iteratively refine its explanation by following the Observation and Commonsense feedback from Critic L.M (right).
Observations can aid the Explainer in alleviating the spurious causal associations. Moreover, the reasons why we use LLM as the Knowledge Mas- ter instead of searching on Knowledge base like Wikipedia are that 1) Wikipedia cannot accept free- form text queries from Analyst and the error anal-
Never forget you Cause A Aaalyst and I am Effect A nalyst. Never flip roles! You need reason ONLY in the following two ways:
1. T Thought with necessary Ask:
Thought: <Y OUR THOUGHT>
Ask: <Y <UO QUESTION>
ysis of ReAct (Yao et al., 2022) indicates 23% of the errors came from the search information re- turning empty or useless information; 2) Recently Yu et al. (2022) has demonstrated that the gener- ated contextual documents more frequently contain the correct answer compared to the retrieved docu- ments. Therefore, utilizing LLM to query large amount of fine-grained knowledge is more efficient than traditional search methods. We will further analyze this in the experiment section.
2. Thought without any Ask:
Thought: <Y OUR THOUGHT>>
Ask: None
Here are some examples: ...
Y ou must give me ONLY one Thought at a time!...
Here is the task: <TASK> Never forget our task!
Figure 4: Part of the Cause Analyst inception prompt.
2.2 Iterative Feedback and Refinement
Inception Prompting Following Li et al. (2023), we utilize the inception prompts to declare roles to each LLM before the conversation begins. After the inception prompt is delivered to the LLM as a system message, the agent automatically assumes the corresponding role and interacts in the conver- sation with the way of thinking first and then action. Our inception prompt consists of Cause Analyst prompt Po and Effect Analyst Pp. which encom- pass role definitions, action spaces, and guidelines. We present part of the Cause Analyst prompt Pc in Figure 4. The details of inception prompts are in Appendix E.
In this section, one LLM serve as Explainer to gen- erate output, and iteratively receives multi-aspect feedback from Critic LLM refine its explanation. Multi-aspect Feedback Although LLMs can generate coherent outputs with in-context Iearn- ing, they often fall short in addressing more intri- cate requirements. According to our error analy- sis' of GPT-3.5 on WIKIWHY dataset (Ho et al., 2023), the most prominent errors are the lack of common-sent sense knowledge and repetitive causal re- lationships (accounting for combined 54%) and
We random sample of 50 entries.

---

vation feedback that the current explanation does not repeat cause-effect relation but overlooks the key information that "the southwest monsoon typ- ically arrives in Ceylon around May or June and lasts until September." Furthermore, it raises the Commonsense feedback that "Flects need to pay close attention to the weather forecast and marine meteorological information... After receiving the feedback, the Explainer becomes aware that the reason why the fleet did not stop was the monsoon striking the southern India at that time, and the fleet needed to avoid getting caught the storm. can be observed that the Explainer corrects the error of spurious causal association and generates valid explanation by incorporating the mentioned commonsense.
Current Explanation (y:): The fleet suiling under favorable conditions running before the southwest est monscon on their fe homeward voyage. This meems that they were nying to make good time to get back home and did not want to make any unmeressary Explainer
stops that would slow them down.
The Explanation is not a simple concatenation of Cause and Effect, but igmores that the southwest monsoon typically amies Ceylon around May or Jnee and lasts until September.
Observation Feedback:
Critic
C Commonsense Feedbacks Fleets need to pay close attention to the weather forecast and marine metoorological information, ty to avoid being involved in the storm.
Reffined explanation (Vr=1): The monsoon season typically arrives in Ceylon around May or June and lasts andil Septamber, which could have created risky sailin conditions for the fleet. Therefore, the fleet did not stop at these places wood heing insolved in the storm and continued their joumey to get back.
Figure 5: The explanation refinement process.
Experiments
3.1 Datasets
there are many research studies (Bai al., 2022b; Yang et al., 2022) have demonstrated the success of multi-aspect feedback. Therefore, we decide to break critic feedback into observation and com- monsense. Specifically, we utilize a LLM to act Critic to provide multi-aspect feedback on the explamation. Critic receives the explanation Y and Observations from previous stage, then provides Observation and Commonsense feedback to im- prove the explanation. The Observation feedback covers two aspects: 1) report on whether the ex- planation is repeating the cause-effect relation; 2) supplementary information based on Observation. The Commonsense feedback aims to present the commonsense knowledge required to explain the causality of the task.
We conduct extensive experiments on two datasets without training. WIKIWHY (Ho et al., 2023) a large-scale QA dataset built around explaining why an answer is true in natural language, which con- tains over 9,000 "why" question-answerr-ttion triples, grounded Wikipedia facts across di- verse of topics. e-CARE (Du al.. 2022) a large human-annotated explainable causal reason- ing dataset, which contains over 21K causal rea- soning questions, together with natural language formed explanations of the causal questions. Since the test set of c-CARE is not public and our method needs to call the OpenAl API which is costly. conduct experiments the pub- lished validation set of e-CARE. We present data examples and other details in Appendix A.
Therefore,
Iterative Refinement The Explainer improves it output based on received feedback and previous generated output. The Critc provides feedback + Explainer refines explanation = Critc provides feedback loop can be applied multiple times. We set the number of iterations to fixed number due budget. One key aspect of Critic is the retention of history of past experiences. This achieved by appending the previous outputs to the prompt continuously. This allows Explainer to learn from past mistakes and avoid repeating them.
3.2 Baselines
To assess the adaptability of our method, we use text-davinci-002. text-davinci-00) and 3pt-3.5turbo the underlying language model, we also refer to these model GPT-3 (Brown al., 2020), InstructGPT (Ouyang et al., 2022), GPT-3.5. Furthermore, we conduct experiments with a wide range of LLMs and the ReAct (Yao et al., 2022) with Wikipedia search engine. For a fair compar- ison, we employ the same examples as Ho et al. (2023) to generate the initial explanation.
Figure 5 depicts the explanation refinement pro- cess. The current explanation exhibits the spuri- ous causal association that "under favorable con- dition" + "they were trying to make good time". The Critic, based on the Observations obtained by knowledge integration module, provides Obser-
3.3 Evaluation Metrics
Automatic Evaluation We provide numerous examples from both datasets in Table 6 in the Ap-

---

Unordered
Ordered
Experiments
Precision
Recall
F1
Precision
Recall
F1
GPT-2
0.249
0.196
0.220
0.239
0.179
0.204
GPT-3
0.347
0.388
0.366
0.307
0.355
0.329
InstructGPT
0,537
0,571
0,553
0.468
0,503
0,435
GPT-3.5
0.609
0.620
0.615
0.467
0.557
0.508
ReAct(GPT-3.5)
0,599
0.669
0.631
0.455
0.595
0.516
LEGO (InstructGPT)
Iteration1
0.565
0.586
0.575
0.486
0.521
0.503
Iteration2
0.579
0.606
0.592
0.515
0.526
0.520
Iteration3
0.556
0,583
0,569
0.479
0,515
0.496
LEGO (GPT-3.5)
Iteration1
0.611
0.708
0.656
0.476
0.618
0.538
Iteration2
0.624
0.714
0.666
0.464
0.634
0.536
Iteration3
0.598
0.705
0.647
0.427
0.627
0.508
Table 1: Performance on WIKIWHY dataset. We conduct experiments InstructGPT (text-davinci-003 and GPT-3.5 (gpt-3. 5- turbb as baselines and underlying language models respectively.
Fine Grained Human Evaluation
Setting
Correctness
Concision
Fluency
Validity
Win (t)
Tic
Lose (4)
GPT-2
0.100
0.880
0.860
0.520
0.040
0.040
0.920
GPT-3
0.660
0.680
1.00
0.960
0.080
0.360
0.580
InstructGPT
0.673
0,700
1.00
0.973
0.166
0.353
0,480
GPT-3.5
0.713
0,526
1.00
0.966
0.193
0.326
0.481
ReAct (GPT-3.5)
0.726
0.533
0.946
0.206
0.333
0.460
LEGO (InstrucIGPT)
0.706
0.660
0.986
0.973
0.186
0,340
0.474
LEGO (GPT-3.5)
0.753
0.506
1.00
0.953
0.226
0.313
0.448
Table Human evaluation. We show the results after using the our framework. Overall correctness is marked on binary scale to indicate an explanation is complete and satisfying or not.
Model
AVG-BLEU
ROUGE-I
CEQ
Human
and CEQ (Causal Explanation Quality) (Du et al., 2022) as metrics.
InstruciGPT
19.72
30.49
0.023
GPT-3.5
22.75
32.82
0.027
0.526
In contrast, explanations in WikiWhy exhibit two structures: multi-hop step sequences and rationale sets, rendering them more instantiated. According to statistics, the average length of explanations in this dataset is 1.5 sentences (Table 4), with some extending up 6 sentences. Additionally, there is fixed order among the sentences. Consequently, the paper of WikiWhy introduces both ordered and un- ordered evaluation to compare the ideas contained in the predictions and references. Specifically, we follow Ho et al. (2023) to use unordered evaluation and ordered evaluation. The former aims to com- pare the ideas contained in the predictions and ref- erences, and the latter tends to penalize incorrectly ordered explanations for the structure of multi-hop explanations. Details of the unordered and ordered evaluation can be found in Appendix B.
AcAct
23.26
33.05
0.028
0.513
32.54
0.028
LEGOXGPT:3.5)
25.23
36.18
0.032
0.560
Table 3: Performance e-CARE dataset. We used binary scale (correct/incorreet) for human evaluation and report the proportion of correct evaluations. comparison, the human-generaie CEQ score is 0.038.
pendix 5. It is worth noting that the explanations in e- c-CARE focus on elaborating the causal facts at conceptual level, typically encompassing only one conceptual sentence. For instance, "Emoticons combinations of characters used represent vari- ous emotions." (example 4 of e-CARE). Therefore. maintain consistency with previous works, we follow (Du et al., 2022) to leverage average-BLEU (n=4) (Papineni et al., 2002), ROUGE-I (Lin, 2004)
Human Evaluation
To ensure consistency with

---

Correctness
Win
GPT-2
GPT-2
G
1EGO
GPT-35
GPT-35
RoAd
ReAct
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Win Rate
Win Rate
Figure 6: LEGO (GPT-3.5) vs baselines on the human evaluation metric Correctness.
Figure LEGO (GPT-3.5) vs baselines on the human evaluation metric Win
prior works (Ho et al., 2023), we present a panel of three graduate students random sample of 50 entries from each setting and the following binary
Precision
Recall
True/False criteria guidelines: 1) Correctness: Is
0.68
the explanation both complete and satisfying? 2) Concision: Does the explanation say everything needs to say and nothing more? 3) Fluency: the
0.61
explanation writing fluent? 4) Validity: Does the explanation make logical sense? 5) Win/Tie/Lose: Compare the generated explanation against the pro- vided reference. The three annotators work inde- pendently on randomly selected samples. The hu- man cvaluations were conducted via spreadshects. We randomly shuffled the golden reference and the generated candidates, we don't reveal which column the golden reference and which is the generated candidate in the evaluation setup. Details human evaluation can be found in Appendix B.
0.56
052
LBGO (ter
LBGO dter 2)
LEGO dter 3)
Figure 8: Performance change of our framework without knowledge integration module.
Thus by using the explanations of LEGO (GPT- 3.5) as references, we compared different base- lines against LEGO (GPT-3.5) separately. This was done to visually demonstrate the superiority LEGO (GPT-3.5), as shown in Figure 6 and 7. The quality of the explanations we generate is signifi- cantly better than that generated by baselines under different metries. More detail can be found in Table in Appendix.
3.4 Main Results
Our main results of WIKIWHY dataset are pro- vided in Table 1. We can observe that our frame- work significantly improves the quality of expla- nations. For InstructGPT based setting, there is 3.9% improvement in unordered fl score by using our framework; similarly, for GPT-3.5 based ting, there is a 5.1% improvement. This establishes that our method is effective for different language models. The corresponding human evaluation ex- periments presented in Table 2. 75.3% of the expla- nations generated by our framework with GPT-3.5 are judged to be satisfactory, significantly higher than other baselines. It should be noted that the results Table just intend count the number of samples in the current generated explanations that meet these metrics (c.g. "Correctness"), which can- not intuitively reflect the advantages our model.
Furthermore, We show the complete cases Ap- pendix D. We demonstrate the experimental results of e-CARE in Table 3. The produced explanations by our framework with GPT-3.5 judged to be sat- isfactory is 56% and achieve a CEQ score close to that of human -eenarteee explanations, We consider this is due to the e-CARE dataset focusing elab- orating the causal facts conceptual level and there is no need to integrate complex fine-grained world knowledge. We further conducted ablation experiments on the WIKIWHY dataset.

---

Precision
Recall F1
the thinking process can accumulate significant amount of information. Relying solely on the mem- ory of a single agent can lead to the omission of cru- cial information during reasoning. Our approach of using multiple agents effectively mitigates this issue, as demonstrated.
0.72
0.88
0.61
0.6
Impact of Iterative Feedback Module This module aims provide multi-aspect feedback dur- ing the iterative process, allowing the Explainer to supplement fine-grained knowledge and task- specific commonsense knowledge. Take Figure 9 as an example, we observe significant improve- ment in Recall (8.8%) after one round of refine- ment, we attribute this improvement to the incor- poration of commonsense knowledge into the ex- planations. However, after one iteration, the im- provement Recall slows down or even decreases, which may be the modification commonsense knowledge in the subsequent iterations not obvi- ous and multiple iterations will increase the length of explanation and lead to the decrease in precision. Refinement ability of LLM By comparing the recall of ordered evaluation of our approaches based on different underlying models (Instruct- GPT and GPT-3.5), we observed that InstructGPT demonstrated a growth of 3.5%, while GPT-3.5 exhibited an increase of 9.4% after one round of iteration. We attribute this noticeable difference to the stronger task comprehension and common- sense induction abilities of GPT-3.5. Furthermore, we noticed that InstructGPT is more "conservative" during the iteration process as presented in Figure 10. Due to space limitations, we present the ex- planations generated by GPT-3.5 in Figure 11 Appendix. We can find that the explanations of InstructGPT remained almost identical, with only few words being replaced by synonyms. In contrast, the explanations generated by GPT-3.5 exhibited greater richness and diversity.
0.56
0.52
Base (GPT-35)
LEGO (ter
LEGO (Iter
LBGO Oter
Figure 9: Performance our framework with knowl- edge integration module (one round of interaction).
3.5 Discussion and Analysis
Impact of Knowledge Integration Module Our ablation experiments with GPT-3.5 as base LM are presented in Table 9. 1) Without knowledge inte- gration module, we plot the performance compar- ison of LEGO without the knowledge integration module in Figure 8. We can observe that the preci- sion of the model experiences continuous decline the iterative process. This indicates that with- out fine-gra mne-grained knowledge support, relying solely on iterative self-feedback is insufficient to effec- tively improve the generated explanations. 2) With knowledge integration module (one round of in- teraction), shown in Figure 9, the precision of LEGO gradually increases demonstrates that this module provides the necessary fine-grained knowl- edge. 3) With knowledge integration module (mul- tiple round of interaction), as illustrated in Table 9, in this setting, the performance of the model does not further improve. This could be attributed to the fine the-grained knowledge stored in Observations is sufficient after one round of interaction. The more times the two analysts interact, the more noisy the collected knowledge may become, resulting in in- efficient information in the feedback. Furthermore, Based on our analysis, the knowledge required for explanations often fine-grained, such as "The fleet of the seventh voyage of the Ming Treasure Voyages." Firstly, this type of question cannot be directly answered through an APT; it needs to be broken down into components like "Ming Treasure Voyages" and "the seventh voyage." Additionally, the information related to "the seventh voyage" is located on line 370 of the search page for "Ming Treasure Voyages," making knowledge localization a challenge. Moreover, as depicted in Figure 2,
Related Work
Causality Explanation Generation Under- standing causality is one of the most central cog- nitive abilities of human beings, different from the causality identification (Caselli and Vossen, 2017; Zuo et al., 2021; Cao et al., 2021; Tran Phu and Nguyen, 2021) which can only distinguish whether there is causal relationship, causality explanation generation especially worth explore since it not only test a model "knows" that causality but also if it "understands" the underlying mechanics

---

Cause similar nature of the railways and to foster cooperation and volunteer exchanges
and the ability address intricate problems that surpass the capabilities of individual agents. Li et al. (2023) enables two communicative agents to engage in a conversation and cooperate with each other to solve assigned tasks. However, designing effective cooperative AI systems is still an active area of research, as it requires addressing range of technical, ethical, and social challenges.
Effect Lyuton & B amstaple signing a twinning agreement with the W Walhalla Goldfilld Railway
Explanation Railways with similar goals improve their chance of success by creating a joint venture.
basd3 explaration
Iteration O: Lynton Barnstaple signed twinning agreement with the aalaalla Goldfields Railway in order to foster cooperation and volunteer exchanges due to the similar nature of the ways.
Learning from Feedback The utilization of nat- ural language feedback, generated by both humans and machines, has proven to be effective across various tasks. Reinforcement learning (RL) ap- proaches have been employed to optimize for hu- man preferences or task accuracy, resulting in the generation of valuable feedback. (Bai et al., 2022a; Lu al., 2022; Le et al., 2022). Recently LLMs have been used to generate feedback for general domain solution (Yang al., 2022; Fu et al., 2023; Peng et al., 2023). To make better use of feedback, pairs of feedback and revision have been employed to learn supervised refiners and correctors (Schick el al., 2022: Bai et al., 2022b; Yasunaga and Liang, 2020). However, gathering supervised data from humans is costly, to overcome this, Welleck et al. (2022) selected the best output by relying on know- ing the ground truth at test time. Madaan et al. (2023) proposes soliciting feedback from an LLM on its own output, refining the output with feed- back, and repeating this feodback-refine process.
Itaration 1: Lynton & Barastaple signed twinning agreement with the W alhaalaa Goldfields Railway to promote cooperation and exchange of ideas between the similar railways.
Heration 2 Lynton & Bamstaple signed twinning egreement with the W aalballa Coldfields Railway foster cooperation exchange of ideas between the two similar railways.
Figure 10: The explanations generated by our method with InstructGPT as the base model.
why that is the case. Du al. (2022) proposed a human-annotated explainable CAusal REasoning dataset (e-CARE) to explore, which contains over 21K causal reasoning questions, together with nat- ural language formed explanations of the causal questions. As large language models (LLMs) grow larger and more sophisticated, Ho et al. (2023) in- troduced WIKIWHY, which built around novel auxiliary task: explaining why an answer is true in natural language, while InstructGPT basedlines achieve only low nuaan-evaluated correctness in the end-to-end answer and explain condition.
5 Conclusion
We introduce LEGO, a Multi-agent Collaborative Framework with Rolc-playing and Iterative Feed- back for causality explanation generation. Specifi- cally, we treat LLM as character mallcable LEGO block and utilize role-playing to assign specific roles to five LLMs, i.e. Cause Analyst, Effect Ana- lyst, Knowledge Master, Critic and Explainer. We devise a Fine-grained World Knowledge Integra- tion Module to augment information about tasks for alleviating the phenomenon of spurious causal asso- ciations. Then, we leverage an Iterative Feedback and Refinement Module to improve the generated explanation by multi aspect feedback. Extensive experiments on WIKIWHY and c-CARE show the superiority of our multi-agent framework in terms of causality explanation generation.
Communicative Agents Large language models have exhibited remarkable multi-dimmenional capa- bilities in tackling complex tasks. However, their effectiveness greatly hinges on human guidance to steer the conversation, a process that can present challenges and consume significant time (Brown al., 2020; Yao et al., 2022; Yang et al., 2023). It is important to consider how to autonomously guide the communicative agents toward task com- pletion while maintaining consistency with human intentions. Communication between AI agents can occur in competitive setting or cooperative set- ting Tladfield-Menel et al., 2016; Silver et al., 2017; Bard et al., 2020; Dafoe et al., 2020; Meng et al., 2023; Kwan et al., 2023). Cooperative AI systems consider the requirements and capacities of other agents within the system, actively striv- ing to collaborate and synchronize their actions. This approach offers numerous advantages, such as enhanced efficiency, improved dacision-making
Limitations
Our method aims to explore the cooperation of multi-agents on causality explanation. In the exper-

---

iments, we found that GPT-3.5 is more powerful than InstructGPT in terms of task comprehension and commonsense induction abilities, but due to the high cost of API calls, we did not further test the performance of GPT-4 in this complex task. In ad- dition, Explanations come in various structures, as presented in the typology defined by Neves Ribeiro et al. (2022), multiple explanations may be valid, the datasets we based on covered large proportion of explanations to simple 'why' questions which minimizes the variability of explanations to some exient. The experimental results show that the ex- planations generated by LLMs are not ideal in the ordered evaluation. Moreover, although the auto- matic evaluation results show the effectiveness of our method, 44.8% of the explanations are still judged by human to be worse than the gold refer- ence. In future work, we intend to delve deeper into the underlying structure of causal explanations within LLMs.
new frontier for ai research. Artificial Intelligence. 280:103216.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-19-19
Pengfei Cao, Xinyu Zuo, Yubo Chen, Kang Liu, Jun Zhao, Yuguang Chen, and Weihua Peng. 2021. Knowledge-enrichee event causality identification via latent structure induction networks. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4862-4872, Online. Association for Computational Linguisties.
Tommaso Caselli and Piek Vossen. 2017. The event StoryLine corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77- 86. Vancouver, Canada. Association for Computa- tional Linguistics.
Acknowledgements
This work is supported by the National Key Re- search and Development Program of China (No. 2020AAA0106400), the National Natural Science Foundation of China (No. 62176257, 61976211). This work is also supported by the Strategic Pri- ority Research Program of Chinese Academy of Sciences (Grant No.XDDA7020100) ), the Youth In- novation Promotion Association CAS, and Yunnan Provincial Major Science and Technology Special
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tan- tum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. 2020. Open problems in cooperative ai. arXiv preprint 2rX.v:20/2.086300
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith. Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entail- ment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7358-7370, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Plan Projects (No. (N0.202202AD0800044
References
Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e e-CARE: able causal reasoning. Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432 446, Dublin, Ireland. Association for Computational Lin- guistics.
dataset for exploring explain- Proceedings of the 60th
John R Anderson and Gordon H Bower. 2014. Human associative memory. Psychology press.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.055: 005662.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate you desire. arXiv preprint wwXiv:2302.04166
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from feedhack. arXiv preprint arXiv:2212.08073.
Dylan Radfield-Menel Stuart Russell, Pieter Abbeel, and Anca Dragan. 2016. Cooperative inverse rein- forcement learning, Advances in neural information processing systems.
Motonobu Hattori and Masafumi Hagiwara. 1995. Quick learning for multidirectional associative mem- ories. In Proceedings of ICNN Interratttii Con- ference on Neural Networks, volume 4, pages 1949- 1954, IEEE
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Ed- ward Hughes, et al. 2020. The hanabi challenge: A

---

Pengcheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: brodding-enhance bert with disentangled attention. arXiv preprint arXiv:2006.03654
Roth. 2022. Entailment tree explanations via itera- tive retrieval- gevrrratgnnrrtionnnooo reasoner. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 465-475, Scattle, United States, Associ- ation for Computational Linguistics.
Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2023, Wikiwhy: Answering and explaining couse-and-effect questions. In International Confer- ence on Learning Representations.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 11027730-277744
Bart Kosko. 1988. Bidirectional associative memories. IEEE Transactions on Systems, man, and Cybernet- ics, 18(1):49-60 60.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Wai-Chung Kwan, Hong-Ru Ru Wang, Hui-Min Wang. and Kam-Fai Wong. 2023. A survey on recent advances and challenges in reinforcement learning methods for task-oriented dialogue policy learning. Machine Intelligence Research, 20(3)3118-33
Hung Le, Yue Wang, Akhilesh Deepak Gotmare. Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng. Yujia Xie. Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedhack. arXiv preprint arXiv:2302. 12813.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" explo- ration of large scale language model society. arXiv preprint arXiv:2303. 17760.
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. arXiv preprint arXiv:2208. 11663.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang. Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. 2017. Mastering the game of go without human knowledge. nature, 550(7676):354-355
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591- 27609.
Minh Tran Phu and Thien Huu Nguyen. 2021. Graph convolutional networks for event causality identifi- cation with rich document-level structures. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3480-3490, Online. Association for Computational Linguistics.
Zhiyi Luo, Yuchen Sha, Kenny Q Zhu, Seung-won Hwang, and Zhongyuan Wang- 2016. Commonsense causal reasoning between short texts. In Fifteenth international conference the principles of knowl- edge representation and reasoning.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinmment with self-feedback. arXiv preprint 176..
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053.
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210006774.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, et al. 2023, Offline pre-trained multi-agent decision transformer. Ma- chine Intelligence Research, 20(2):233-248.
Zhengyuan Yang. Linjie Li. Jianfeng Wang. Kevin Lin, Ebsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. Mm- react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv: 1 11881.
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong. Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhibeng Huang, Andrew Arnold, and Dan

---

Zhilinn Yang, Peng Qi. Saizheng Zhang, Yoshua Ben- gio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint 17..iv:100.090000
WikiWhy Statistics
# of Train
7397
# of Dev
1004
# of Test
1005
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXv:22210.03629.
# of Rationale
9406
# of Rationale Elements
14238
Avg. # Rationale Length
1.5137
Michihiro Yasunaga and Percy Liang. 2020. Graph- based, self-supervised program repair from diagnos- tic feedback. In International Conference on Ma- chine Learning, pages 10799 10808. PMLR.
Avg. # Tokens per Element
16.697
Table 4: WikiWhy Summary Statistics
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Gen- erate rather than retrieve: Large language mod- els are strong context generators. arXiv preprint arXiv:2209. 10063.
which contains over 21K causal reasoning ques- tions, together with natural language formed expla- nations of the causal questions, The statistics are shown in Table 5.
Bo Zhou, Chenhao Wang, Yubo Chen, Kang Liu, Jun Zhao, Jiexin Xu. Xiaojian Jiang, and Qiuxia Li. 2022. Augmentation, retrieval, generation: Event sequence prediction with three-stage sequence-to- sequence approach. In Proceedings of the 29th Inter- national Conference on Computational Linguisties, pages 1865-1874, Gyeongju, Republic of Korea. In- ternational Committee Computational Linguistics.
e- C-CARE Statisties
Causal Questions
Uniq. Explanations
Train
14928
10491
Dev
2132
2102
Test
4264
3814
Total
21324
13048
Xinyu 7uo, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Weihua Peng, and Yuguang Chen. 2021. LeamDA: Learnable knowledge-guider data augmen- tation for event causality identification. In Proceed- ings of the 59th Annual Meeting the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3558-3571, Online. Association for Computational Linguistics.
Table 5: Corpus level statistics of the e-CARE dataset. Uniq. Explamations refer to the explanations that only correspond to single causal fact.
A.3 Examples from Datasets
Table 6 contains examples from WikiWhy and e- CARE. e denotes cause, e effect, and x is the ex- planation of the cause-effect pair.
Dateset Details
B Evaluation Details
A.1 WIKIWHY
WIKIWHY (Ho al., 2023) large-scale QA dataset built around explaining why an answer is true in natural language, which contains over 9,000 triples, grounded on Wikipedia facts across a diverse set of topics. Each entry contains cause-effect pair and ratio- nale explaining the pair's causal relation. On aver- age, each rationale contains 1.5137 elements. The statistics for the reasoning component shown in Table 4.
B.1 Metrics for WIKIWHY
While the still developing area of text generation has measures and proxies for similarity that help with simple sequences, comparing reasoning se- quences or rationale sets requires more involved measures. Ho et al. (2023) proposed two related metrics, unordered and ordered, to handle sets and sequences, respectively.
"why" question-answer-raae
Unordered Evaluation This first approach com- pares the ideas contained in the predictions and ref- erences. First, split predicted and reference expla- nations into "ideas" or "steps" by sentence. Then compute a matrix of pairwise similarity scores be- fore using a threshold to classify "matches". Since
A.2 &CAREE
c-CARE (Du et al., 2022) is a large human- annotated explainable causal reasoning dataset,

---

single prediction sentence may contain multiple reference ideas, so keep separate counts of pre- cise prediction steps and covered reference steps. These counts are then avero-averaaed for the test set's overall precision, recall, and F1 scores.
WIKIWHY Example
The thermal stress at dawn and dusk.
The houlders on Ceres are hrittle and degrade rapidly.
The thermal temperatures change so drastically the rocks expand and contract. This process weakens the structural integrity of the rocks.
Ordered Evaluation To respect the structure of multi-hop explanations, the method penalizes incorrectly ordered explanations. Here, use the previously generated pairwise score matrix and its alignments to generate all possible assignments of prediction sequence elements to reference el- ements. Then compute the length of the longest common subsequence (LCS) between a prediction alignment against the reference labels for each can- didate assignment. This length becomes the count of correctly incorporated structural elements- true positives. Note that under this scheme, the repeated ideas in the prediction are discounted by the LCS- style alignment process.
The duration of Hotel Califomia was longer than songs generally played by radio stations.
Don Felder had douhts about the 1997 Eagles song Hotel California.
Most songs are only 3-4 minutes long. Hotel Cali- fomia is over minutes. People would not want to listen to sumee song on radio for that long.
Seeing the Castle of Cagliostro entrenched in Ya- mazaki that Japan can make high quality films. Director Takashi Yamazaki modeled his 2019 film Lupin III: The First after The Castle of Cagliostro.
Viewing The Castle of Cagliostro inspired Takashi Yamazaki. Out of national pride, Takashi Yamazaki followed model that he believed would produce quality films.
We employ the BERTSscore metric to measure sentence similarity (not evaluated directly using BERTScore). Taking the unordered evaluation al- gorithm proposed by (Ho et al., 2023) as an exam- ple, Precision and Recall are obtained using the following formulas:
The goographic isolation of the Hapa homeland.
The Hupa had few interactions with early European explorers up the 19th century.
The Hupa's homeland was sepurated hy bodies of water or mountains. Not many people could get to the Hupu's 's homeland.
The use coal power Turkey.
Burning coal leads to air pollution. Air pollution causes sickness and early death. Sick and dead peo- ple cannot work.
Precision precise
covered
prediction Recoll
relevant
1.4 million working days were lost across the popu- lation of Turkey in 2019.
Where "predictions" represents the number of sen- tences in the generated explanations, "relevant" denotes the number of sentences in the reference explanations, "precise" indicates how many of the generated sentences correspond to sentences in the explanations, and "covered" represents the number of sentences in the reference explanations that were successfully predicted.
C-CARE
Example
He was infocted with gram-positive bacteria. The doctor raised the lysozyme in his body.
Lysozyme destroys cell wall of bacteria.
The reseurcher investigated the premature death in these pet birds.
He found they died of Malnourishment.
For example. consider the following examples of reference explanations and generated explanations:
Malnourishment is leading cause premature death in pet birds.
Reference Explanation: Opening the highway brought in an influx of unsafe people. With the higher traffic from a highway it would be hard to police the unsafe people. With them being harder to police it would become safer for them and they could drug deal and prostitute in the open.
It quite cold here in winter, and the temperature can reach as low as minus degrees.
In winter here, people wear clothes with very good thermal insulation when they go out.
Clothing provides protection from the elements by increasing the insuluting capucity of the bodl.
Generated Explanation: When Interstate 5 was opened, it diverted traffic away from these areas and caused decline in economic activity. With less commerce and people around, it became easier for illegal activities and establishments to operate without attracting attention from law enforcement or regular citizens.
Mary sent a emoticon Cryingio her boyfricnd on her cell phone.
Her frifnd immediately called to comfort her.
Emoticons combinations of characters used to represeut various emotions.
Table 6: Examples from datasets

---

Score: precise=2. predictions=2, covered=3, rel- evant=3.
between word wi and wj, which is estimated from large corpus as:
The proposed algorithm keeps separate counts of precise prediction steps and covered refer- ence steps, which effectively addresses the situa- tion where a single prediction sentence may con- tain multiple reference ideas. For further details of the algorithms, please refer to the evaluation methods provided by (Ho et al., 2023).
Count( uP, myy
cs(ws, my) = Count ux (wy)
where a is penalty coefficient and empirically set = 0,66 as same Luo et al. (2016). We use the average CEQ score as the final score.
B.3 Human Evaluation
For fair comparison, we follow Ho et al. (2023) to select BERTScore using large De- BERTa (He et al., 2020) model (microsoftddebrrta xlarge-mnli)* at a threshold of 0.64.
Automatic metrics may not reliably evaluate results produced by models with few-shot capabilities like GPT-3. In light of this, we select the highest scor- ing explanations for each set of experiments for additional fine-grained evaluation and measure the agreement among the annotators.
B.2 Metric for e-CARE
Causal Explanation Quality (CEQ) Du et al. (2022) proposed a novel causal explanation quality evaluation metric (namely, CEQ score) as step towards directly measuring the quality of generated explanations. Specifically, let C, E and X denote the cause, the effect and the generated explanation, respectively. Formally, the CEQ score defined:
(1) For each human evaluation task, present panel of three undergraduate students a random sample of 50 entries from each setting and the fol- lowing binary True/False criteria guidelines:
Correctness: Mark true if and only if the explanation is both complete and satisfying.
CEQ 8 cC,C, E|x) cs(C, E)
Concision: Mark true if the explanation says everything it needs to say and nothing more. Mark false if extra information is included.
where Cs(C, C F)) is the original causal strength be- tween C and E: essCs: X) is the causal strength after involvement of the additional explanation information. The explanation enhanced causal strength cs, X) is defined as:
Fluency: Is the explanation writing fluent? Mark false if there are any mechanical mis- takes.
cs E|X) marlcs(C + E),
Validity: Does the explanation make logical sense? Ignore whether or not the explanation successfully explains the cause/effect relation. Mark false the explanation contains any il- logical or untrue conclusions.
where "+" denotes the string concatenate opera- tion. Therefore, the CEQ score is positively related to the increase of causal strength between C and after the involvement the explanation X. we employ a widely-adopted model-agnostic method proposed by Luo et al. (2016) to calcu- late the causal strength. The model-agnostic nature enable to avoid reliance certain models and keep the fairness of evaluation. Specifically, the phrase-level causal strength is derived through syn- thesizing the word-level causality.
Compare the generated expla- nation against the provided reference. Mark Win if you prefer the generated explanation, Tie if you have no preference, and Lose if you prefer the reference explanation.
Win/Tie/Lose:
(2) Using the explanations of LEGO (GPT-3.5) as references, we compared different baselines against LEGO (GPT-3.5) separately. This was done to vi- sually demonstrate the superiority of LEGO (GPT-
cs(Ca, EB) = No. NEA
Σ EE
ca(ux, my)
3.5), as shown in Table 8.
where (CA, EB) is an arbitrary causal fact; NoA and NEp are the number of words within CA and FB, respectively; cs(w;, wy) is the causal strength
(3) We measured the agreement among three an- notators, and the table below displays our Fleiss' Kappa scores for the annotation agreements of Win/Tie/Lose (Compare the generated explanation against the provided reference).
'tttps: hhigss///uugingffae www.hugggggggn....ooogoco wwt.a/.......cooo.eeeee..ee..coo crosoft// deberta- xaartta-xxaggeeell

---

Model
K
GPT-3.5
0.84
ReAct
0.79
LEGO (InstructGPT)
0.76
LEGO (GPT-3.5)
0.78
Table 7: The agreement among three annotators
Cause similar nature of the railways and to foster conperation and volunteer exchanges
Effectt Lynton Bamstaple signing twinning agreement with the aalballa Goldfields Railway
Explanation Railways with similar goals Imprvve their chance
success by creating joint venture.
GPT-3.5 hased explanation
Itoratin O Lynton & Barnstale signed twinning agreement with Walhalla Goldfields Railway order foster cooperation and volunteer exchanges due the similar nature of the wailways.
Iteration 1: The similar nature the railways, as well as the desire foster cooperation and volunteer exchanges, led. Lynton Bamstople to sign twinning agrvement with Walhalla Goldfields Iaallwa...
Iteration Organizations often form partnerships twinning agreements with similar groups orpanizations order enbance collaboration and muhual exchange. The Lynton & Bamstaple W a'lballaa Goldfields railways signed a twinning agreement due their similar nature and desire foster cooperation and volunteer exchanges.
Figure 11: The explanations generated by method with GPT-3.5 as the base model.
C Details of Ablation Experiments
As shown Table 9, we conduct extensive ablation experiments on WIKIWHY dataset.
D Case Analysis
The cases show are in Figure 12 and 13
E Prompts
We include all the prompts used in the experiments in Figures 14-17.

---

Cause: The fleet of the seventh voyage of the Ming Treasure Voyages sailing under favorable conditions and running before the southwest monsoon their homeward voyage. Effect: The fleet of the seventh voyage of the Ming Treasure Voyages did not stop at Ceylon or southern India.
Initial Explanation (G PT-3.5): The fleet was sailing under favorable conditions and running before the southwest monsoon on their homeward voyage. This means that they were trying to make good time to get back home and did not: want to make any unnocessary stops that would slow them down.
The flest of the seventh vorage of the Ming Tresure Voyages: The fleet of the seventh Ming Treasure
ossevvation
Voyage set sail the early summer of 1430, taking advantage of the favorable southwest monsoon Ceylon, southern India and southwest monsoon: Ceylon, also known as Sri Lanka, is an island country
located in Indian Ocean, off the southem coast of India. It experiences troical climate and is greatly influenced by the southwest monsoon...
Observation Feedback: The Explanation is not a simple concatenation of Cause and Effect, but ignores that, the southwest monsoon typically arrives in Ceylon around May or June and lasts until September: Commonsense Feedback: Fleets need to pay close attention to the weather forecast and marine meteorological information, ty to avoid being involved in the storm.
LEGO Explanation: The monsoon season typically arrives in Ceylon around May or June and lasts until September; which could have created risky sailing conditions for the fleet. Therefore, the fleet did not stop at these places to avoid being involved the storm and continued their journey get back home as quickly and safely as possible.
O Obervation Feedback: None.
Commonsense Feedback: Sailors and ships tend to avoid dangerous weather conditions in order to minimize the risk of damage to the ship and loss life.
Gold Explanation: A monsoon was striking Caylon or southern India at the time and the fleet of the seventh voyage of the Ming Treasure Voyage did not want to be caught in the storm.
Figure 12: Case study.

---

Cause: The Irish Republican Amy had not resumed its ceasefire or agreed to disarm. Effect: Sinn Fein had been elected to take part in the multi-party negotiations in Belfast for the 1 1996 Manchester bombing but were barred.
Initial Explanation (G PT-3.5):: Sinn Fein was elected to take part in the negotiations because they were considered a key player in Irish politics at the time. However, they were barred because the Irish Republican my had not resumed its ceasefire or agreed disarm.
O Oservvaion:
The Trish Republican A myy, What is the significance of ceasefire and disarmament in the context of the conflict in Northern Ireland?: ... Ceasefires allowed the two sides to engage in political negotiations and attempt to find a long- term solution to the conflict. Disamament was seen peace process, as it removed the potential for violent acts and reduced the potential for future conflic.... What was Sinn Fein's position on ceasefire and disarnament during the time of the 1996 Manchester bombing? At the time of the 1996 Manchester bombing, Sinn Fein's position on ceasefire and disarnament was that they wished for an end to violence on all sides but would only support disarmament once they were satisfied that the British govermment was committed to addressing the underlying causes of the conflict...
essential part of the
O Observation Feedback: The current Explanation is a concatenation Cause and Effect content. The explanation the Task needs to be modified reflect the fact that Sinn Fein was barred from the negotiations due to the IRA 's failure to resume a ceasefire or agree to disarm, rather than their mere association with the IRA.
Commonsense Feedback: Political negotiations require trust and security. A ceasefire and disarmament often prerequisites for pmoress in peace talks.
LEGO Explanation: Sinn Fein was elected to take part in the negotiations due to their importance in Irish politics. However, they were ultimately barred from the negotiations because the IRA had not resumed its ceasefire or agreed to disarm, which was necessary for progress in peace talks. Political negotiations require trust and security for progress, and a ceasefire and disarmament are often prerequisites.
The Explanation is not simple concatenation of Cause and Effect, which is correct. Commonsense Feedback: Political negotiations require trust and security for progmss, and a ceasefire and disarmament are often prerequisites. The IRA involvement in peace talks was crucial to the success of the process, and failure to disarm was sigmificant obstacle.
Observation Feedback:
Gold Explanation: For negotiations to occur there needs to be a ceasefire or disarming for safety, Since the Irish Republican A my did not resume the ceasefire or disarm it was unsafe to take part in the negotiations so they were barred.
Figure 13: Case study.

---

Cause There were time constraints to writing "Boruto: Naruto the Movie"
Effect: Hiroyuki Yamashita felt pressured writing "Boruto: Naruto the Movic"
Explanation Creativity difficult when put strict timetable. There was need both produce a good movie and do strict time budget. These two demands put stress Hiroyuki Yamashita while he worked.
Cause: Homer P. Rainey had liberal views.
Effect: Homer Rainey was fired by the University of Texas in 1944.
Explanation: If the University of Texas is conservative, they wouldn't want people working there who have liberal views.
Cause: the large size and reddish tint of red maple buds
Effect: Red maple buds which form in fall and winter are often visible from a distance.
Explanation: The color red stands out from a distance, so the buds are red in the fall and winter, you'd be able to see them from a distance.
Cause: Ther were advances technology, lower energy prices, a favorable exchange rate of United States dollar, and lower alumina prices.
Effect: Productions costs aluminum changed in the late 20th century.
Explanation: With advances in technology, prices of manufacturing change usually because they easier and cheaper make. In this case is aluminum that the price changed on because the technology improved the process.
Figure 14: Initial generation prompt for WIKIWHY
Cause: The child ran towards hippos.
Effect: His parents stopped him.
Explanation:
Hippos are among the megafaunal species dangerous to humans.
Cause: Jack keeps the goats alone.
Effect:
The goat has poor appetite.
Explaration:
Goats are herd animals depend on the companionship of other goats.
Cause: The photogmahhr has to edit digital images.
Effect: He understood the pixels basic building blocks.
Explanation: Pixels are the basic building blocks of all digital images.
Cause: The researcher separated integral membrane proteins from the tonoplast.
Effect: They are TIPs.
Explamation: TIPs are integral membrane proteins in the tonoplast.
Cause: Tom eats of eggs every day.
Effect: The cholesteml content his body is extremely high, so he suffers from gallstones.
Explaration:
Gallstones lumps composed mainly of cholesterol.
Caus:: The shamans were start their sacred ceremonies.
Effect: They moved cave.
Explanation: Caves sacred ceremonies sites utilized by the shamans.
Figure 15: Initial generation prompt for e-CARE

---

Why Orville Babcock unpopular: Orville Babcock (1835-1884) was U.S. A A officer and personal secretary to President Ulysses S. Grant. Babcock was ... Why President Grant of the United States dismiss Orville E. Babcock: President Ulysses S. Grant dismissed his personal secretary, Orville E. Babcock, in 1876 due to
Observation:
allegations of his involvement in major corruption scandal known as the W hiskey Ring.The Whiskey Ring...
Task: Cause: Orville E. Babcock was unpopular: Effect: In 1876, President Grant of the United States dismissed Orville E. Babcock from the White House under public pressure.
Explanation: Orville E. Babcock's unpopularity led to public pressure on President Grant to remove him from the W White House.
Check whether the Explanation is just a concatenation Cause and Effect content, so, please point and give feedback based on the information the Observation and Task.
What common sense knowledge is required explain the causal relationship in the task? If the current explanation is perfect, return None.
O bservation Feedback: The Explanation simple concatenation of Cause and Effect, which is incorrect Based Observation, Orville Babcock was unpopular due to his involvement in the W Whiskey Ring scandal.
Commonsense Feed Feedback: Politicians dismiss unpopular figures around them. Politicians typically try to remain popular with the people.
Figure 16: Feedback prompt of Critic LLM (truncated).
Task: Cause: Orville E. Babcock was unpopular. Effect: In 1876, President Grant of the United States dismissed Orville E. Babcock from the White House under public pressure. Explanation: Orville E. Babcock's unpopularity resulted in public pressure on President Grant, which led to his dismissal.
Check whether the Explanation is just a concatenation Cause and Effect content, so, please point out and give feedback based on the information the Observation and Task.
Observation Feedback: The Explanation simple concatenation of Cause and Effect, which is incorrect Based Observation, Orville Babcock unpopular due to his involvement in the W hiskey Ring scandal.
What common sense knowledge is required explain the causality in the task? Commonsense Feedback: Politicians dismiss unpopular figures around them. Politicians typically try remain popular with the people.
Okay, impove the Explanation using these feedback (Please keep Explanations concise rather than redundant): Explanation: Politicians dismiss unpopular figures around them to remain popular with the people. Orville E. Babcock's scandal resulted public pressure on President Grant, which led to his dismissal.
Check whether the Explanation is just. a concatenation of Cause and Effect content, if please point and give feedback based on the information in the Observation and Task.
O Observation Feedback: None
What common sense knowledge is required to explain the causality in the task? Commonsense Feedback: None
Figure 17: Refine prompt of Explainer LLM (truncated).

---

Never forget you are a Cause A nalyst and I am Effect nayyst. Never flip roles!
We share common interest collaborating to complete task that reasoning about given DofuctEffect,, pair. must help you to complete the task. You need to reason ONLY the following two ways:
1. Thought with necessary Ask: Thought: <Y OUR THOUGHT> Ask: <Y OUR QUESTION>
2. Thought without any Ask: Thought: <Y OUR THOUGHT>
Ask: None
The "Thought" describes your reasoning process which includes entities or questions that need to ask knowledge base, and always starts with "I need to analyze from the Cause, Related entities in Cause are:". The paired presents entities or questions the Thought.
Here are some examples:
Task: Cause: Orville E. Babcock was unpopular: Effect: In 1876, President Grant of the United States dismissed Orville E.
Babcock from the W hite House under public pressure.
Thought: need to analyze from the Cause. Related entities in Cause are: E. Babcock). So I need to ask about
Orville E. Babcock, find why he was unpopular:
Ask: Orville E. Babcock, why Orville E. Babcock was unpopular
You must give ONLY one Thought at time!
Never forget are Cause nalyst!
Do not output anything else other than Thought and the optional corresponding Ak! Here is the task: SK> Never forget our task!
Your Thoughh must prioritize what is mentioned in the Task.
Now you must start reason using the two ways described above.
Figure 18; Inception prompt Cause Analyst LLM (truncated).

---

Never forget you are an Effect nalyst and am a Cause nalyst. Never flip roles!
We share common intersst in collaborating complete task that reasoning about the given Cause-Effect pair. You must help to complete the task.
You need to reason ONLY in the following way:
1. Thought with necessary Ask: Thought: <Y OUR THOUGHT> Ask: <Y OUR QUESTION>
2. Thought without any Ask: Thought: <Y <YOUR_THH, THOUGHT> Ask: None
The "Thought" describes your reasoning process which includes entities or questions that need ask knowledge base, and always start. with 'I need analyze from the Effect. Related entities in Effect are:". The paired "As presents the entities or questions in the Thought.
Here are some examples:
Task: Cause: Orville E. Babcock was unpopular: Effect: In 1876, President Grant of the United States dismissed Orville E.
Babcock from the W hite House under public pressure.
Thought: need to analyze from the Effect. Related entities Effect are::Preeeee President Grant of the United States, Orville E. Babcock). As stated in the main reasoning process, Orville E. Babcock was unpopular due to his involvementin in the Whiskey Ring scandal and Babcock was a personal secretary and close confidant President Ulysses Grant. So President Ulysses Grant may criticized by the public and may cause his approval rating to drop. need find more information about why President Grant of the United States dismiss Orville E. Babcock. Ask: why President Grant the United States dismiss Orville E. Babcock
must give me one Thought at time!
Never forget you are an Effect nalyst!
Do not add anything else other than your Thought and the optional corresponding
Here is the task: <TA SK>, Never forget our task!
Your Thought must prioritize what mentioned in the Task.
Now you must start reason using the two ways described above.
Figure 19: Inception prompt Effect Analyst LLM (truncated).

---

Fine Grained Human Evaluation
Setting
Correctness
Concision
Fluency
Validity
Win (1)
Tie
Lose (L)
GPT-2 LEGO
0.113
0.593
0.146
0.100
0.073
0.086
0.841
GPT-3 LEGO
0.226
0.433
0.253
0.213
0.133
0.333
0.534
GPT:3.5 LEGO
0.293
0.406
0.326
0.280
0.206
0.340
0.454
ReAct (GPT-3.5) vs LEGO
0.306
0.393
0,313
0.326
0.213
0.346
0.441
LEGO (InstructGPT) vs LEGO
0,360
0.426
0,453
0.393
0.226
0.373
0.401
Table 8: Other baselines vs LEGO (GPT-3.5).
Unordered
Ordered
Experiments
Precision
Recall
F1
Precision
Recall
F1
LEGO
0.624
0.714
0.666
0.464
0.634
0.536
w/ single Analyst
0.596
0.705
0.646
0.455
0.617
0.523
wo Knowledge Master
0.578
0.698
0.632
0.447
0.598
0.511
w/o Critic
0.584
0.685
0.630
0.449
0.608
0.516
interaction
0.593
0.703
0.643
0.451
0.609
0.518
interaction (2 rounds)
0.619
0.701
0.657
0.448
0.625
0.521
Table 9: Ablation experiments. We use GPT-3.5 as the base model.


Reviews:
[{"review_id": "dD6hNjR4lB", "reviewer": "Reviewer_8kHz", "summary": "", "questions": "A: Given just the input cause-effect pair, it's challenging to assess the accuracy of a causality explanation. External knowledge, such as the monsoon example provided, can heavily influence the explanation. Different external knowledge might validate other seemingly spurious explanations. For example, If different external knowledge is available, e.g. the fleet was behind their schedule for homeward voyage, then the spurious explanation given in the paper might be appropriate as well. To address this variability, would it be viable to augment the dataset with multiple explanations for a single cause-effect pair?\n\nB: Why were different evaluation metrics chosen for the datasets? Could you provide comprehensive results?\n\nC: What informed the decision to break critic feedback into observation and commonsense? Was this grounded in theory or based on preliminary findings such as prompt engineering?\n\nD: Do you have any insights on the optimal number of interactions/iterations and potential termination criterias?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews.\n\n### Respone to Question\n>**Answer for Question 1**: Would it be viable to augment the dataset with multiple explanations for a single cause-effect pair? \n\nWe think your insights into this task are highly valuable. Evaluating generated explanations indeed poses specific challenges. To clarify, all the data is **sourced from Wikipedia** articles, which means that the explanations in the dataset are **based on established world knowledge**. External knowledge may not form a correct causal explanation. We agree with your point about \"the variability of explanations.\" However, as seen in the typology defined by Ribeiro et al [1], explanations come in various structures. It's not an easy task to perfectly organize all possible explanations. The WikiWhy dataset includes two types of explanations (step sequence and sets) which may not comprehensive, as a novel effort in QA explanations and reasoning, they probably cover a large proportion of explanations to **simple ‘why’ questions and widely accepted** by the NLP community. This minimizes the variability of explanations to some extent, facilitating the fair assessment of complex reasoning capabilities of language models. We appreciate your suggestions and expect to explore more intricate structures and variability in explanations for complex \"why\" questions in our future work.\n\n>**Answer for Question 2**: Why were different evaluation metrics chosen for the datasets? Could you provide comprehensive results?\n\nThe e-CARE and WikiWhy datasets have distinct annotation standards [2,3]. In Table 6, we have presented numerous examples from both datasets. The explanations in e-CARE focus on elaborating the causal facts at a conceptual level, typically encompassing **only one conceptual sentence**. For instance, \"Emoticons are combinations of characters used to represent various emotions.\" (example 4 of e-CARE). Therefore, to maintain consistency with previous works, we are following the evaluation metrics proposed in the e-CARE paper. In contrast, explanations in WikiWhy exhibit **two structures**: multi-hop step sequences and rationale sets, rendering them more instantiated. According to statistics, the average length of explanations in this dataset is 1.5 sentences (Table 4), with some extending up to 6 sentences. Additionally, there is a fixed order among the sentences. Consequently, the WikiWhy paper introduces both ordered and unordered evaluation to compare the ideas contained in the predictions and references. We apologize for not providing a detailed description of the chosen evaluation metrics. We will add it in the revised version.\n\n>**Answer for Question 3**: What informed the decision to break critic feedback into observation and commonsense? Was this grounded in theory or based on preliminary findings such as prompt engineering?\n\nWe are based on **preliminary findings from error analysis and relevant research papers**. Firstly, according to our statistics, the most prominent errors are the lack of common-sense knowledge and repetitive causal relationships (accounting for a combined **54%**). Secondly, many research studies [4,5] have indicated the success of multi-aspect feedback. To sum up, we decide to break critic feedback into observation and commonsense.\n\n>**Answer for Question 4**: Do you have any insights on the optimal number of interactions/iterations and potential termination criterias?\n\nFirstly, increasing the number of interactions might allow the Critic to **observe more relevant information**. However, simultaneously, this increase in information could **introduce noise and redundancy**. We validated this conclusion through the experiments presented in Table 7. The results from experiments with two interaction rounds were better than those without any interactions, but they were inferior to the setup with only one interaction. The more information Critic observes, the longer the feedback it provides, leading to more redundancy within the explanations provided by the explainer. While this might **be beneficial for improving Recall** (increasing the number of correct sentences in the generated explanations), evaluation metrics penalize excessive sentences. Secondly, increasing the number of iterations can also lead to **excessive explanation lengths, harming precision**. Based on our analysis, the generated common-sense feedback does not exhibit significant changes during iterations (as illustrated in Figure 10). However, the observed feedback accumulates within explanations (much of which could be informational noise), resulting in a decline in precision.\n\n### Respone to Reject\n>**A. The concerns about evaluation metrics.** (Please refer to Answer2)\n\n>**B. Baseline updates**\n\nWe appreciate your suggestion on experiment. Following your suggestion, we have added this experiment with a single ReAct agent and the updated paper revision contains the results (Table 1).\n\n| Model     |  | unordered| | \\ | |ordered| |\n| -------- |:----:| :----:|:----: | :----: | :----: |:----: |:----: |\n| GPT-3.5 | 0.609 | 0.620 | 0.615 | \\ | **0.467** | 0.557 | 0.508 |\n| ReAct (GPT-3.5) | 0.597 | 0.669 | 0.631 | \\ | 0.455 | 0.595 | 0.516 |\n| LEGO (GPT-3.5) | **0.624** | **0.714** | **0.666** | \\ | 0.464 | **0.634** | **0.536** |\n\nBased on our analysis, the knowledge required for explanations is often fine-grained, such as \"The fleet of the seventh voyage of the Ming Treasure Voyages.\" Firstly, this type of question **cannot be directly answered** through an API; it needs to be broken down into components like \"Ming Treasure Voyages\" and \"the seventh voyage.\" Additionally, the information related to \"the seventh voyage\" is located on **line 370 of the search page** for \"Ming Treasure Voyages,\" making knowledge localization a challenge. Moreover, as depicted in Figure 2, the thinking process can accumulate a significant amount of information. Relying solely on the memory of a single agent can lead to **the omission of crucial information** during reasoning. Our approach of using multiple agents effectively mitigates this issue, as demonstrated.\n\n>**C. About the human evaluation**\n\nTo ensure consistency with prior works, we strictly adhere to the organizational approach of Ho et al [2]. The three annotators work **independently** on **randomly selected samples**. The human evaluations were conducted via spreadsheets. We randomly shuffled the golden reference and the generated candidates, we don't reveal which column is the golden reference and which is the generated candidate in the evaluation setup.\n\n>**D. Differences in human evaluation results**\n\nIn fact, we had three annotators independently evaluate 50 randomly selected samples, with a maximum score of 150 for each metric. However, due to the evaluation metrics set by Ho et al, which are intended to count the number of samples in the current results that meet the “Correctness” metric, this **cannot serve as an intuitive measure** for comparing differences between different outcomes. We appreciate your suggestion and have conducted additional human evaluation. Using the explanations of **LEGO (GPT-3.5) as references**, we compared different baselines against LEGO (GPT-3.5) separately. This was done to visually demonstrate **the relative superiority of other baselines over LEGO (GPT-3.5)**.\n\n| Model|Correctness  | Concision|Fluency | Validity |Win |Tie|Lose |\n| -------- |:----:| :----:|:----: | :----: | :----: |:----: |:----: |\n| **GPT-2** vs LEGO (GPT-3.5) | 0.113 | 0.593 | 0.146 | 0.100 | 0.073 | 0.086 |0.841 | \n| **GPT-3** vs LEGO (GPT-3.5)  | 0.226 | 0.433 | 0.253 | 0.213 | 0.133 | 0.333 |0.534 | \n| **GPT-3.5** vs LEGO (GPT-3.5)  | 0.293 | 0.406 | 0.326 |0.280 | 0.206 | 0.340 | 0.454 | \n| **ReAct (GPT-3.5)** vs LEGO (GPT-3.5)  | 0.306 | 0.393 | 0.313 | 0.326 | 0.213 | 0.346 |0.441 | \n| **LEGO (InstructGPT)** vs LEGO (GPT-3.5)  | 0.360 | 0.426 | 0.453 | 0.393 | 0.226 | 0.373 |0.401 | \n\n>**E. About the Ablation experiments**\n\nDue to the limited space, in Section 4, we focus on the primary conclusions (impact of knowledge integration and impact of iterative feedback). We appreciate your suggestion on the suggestion. In the revised version, we will incorporate a comparison between LEGO (with only the knowledge integration module), LEGO (with only the iterative feedback module), and GPT-3.5.\n\n### Other Additions\n\nAs you have correctly understood, \"interaction\" relates to fine-grained world knowledge integration and \"iteration\" involves iterative feedback and refinement. We will clarify these terms in the revised version.\n\n### Reference\n[1] Ribeiro D, Wang S, Ma X, et al. Entailment tree explanations via iterative retrieval-generation reasoner. NAACL 2022\n\n[2] Ho M, Sharma A, Chang J, et al. Wikiwhy: Answering and explaining cause-and-effect questions. ICLR 2023 \n\n[3] Du L, Ding X, Xiong K, et al. e-CARE: a new dataset for exploring explainable causal reasoning. ACL 2022 \n\n[4] Bai Y, Kadavath S, Kundu S, et al. Constitutional ai: Harmlessness from ai feedback. arXiv:2212.08073, 2022.\n\n[5] Yang K, Peng N, Tian Y, et al. Re3: Generating longer stories with recursive reprompting and revision. EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written with clear presentations, complemented by motivating examples and illustrations.\n\nThe proposed role-playing and iterative feedback framework for LLM is innovative. Similar collaborative frameworks have potential applications beyond causality explanation.\n\nThe authors undertook comprehensive experiments on two renowned datasets using various base LLMs. Ablation experiments were astutely constructed to isolate the impacts of distinct modules.\n\nInsights on the optimal number of interactions between LLM agents in collaborative tasks are beneficial for the research community.", "reasons_to_reject": "Different evaluation metrics were used for the datasets without a defined rationale, which undermines the results.\n\nThe majority of the automatic evaluation metrics in the paper gauge similarity between predicted explanations and the reference standard. It seems to me that accurate knowledge retrieval is more important than making casual reasoning in those evaluations. Given that knowledge retrieval is primarily executed by the ReAct framework with Internet access, the paper might benefit from an added baseline to assess a single ReAct agent's performance on the explanation task.\n\nThe human evaluation section needs elaboration. Did the three annotators work independently or engage in panel discussions? How were the 50 samples presented: randomly or in clusters?\n\nIn the results section, line 379, the author asserts significance without performing statistical analysis. A 4% improvement, given the human evaluation method, indicates that only 2 out of 50 samples were labeled differently. Without supporting evidence, this difference could be attributed to noise or evaluation bias.\n\nThe presentation of the Ablation experiments is ambiguous. Section 4 only addresses how the number of iterations and interactions affect LEGO's performance. However, a crucial comparison between LEGO (with only the knowledge integration module), LEGO (with only the iterative feedback module), and GPT-3.5 is absent. While some details are in the appendix (Table 7), an in-depth discussion in the main text would be beneficial.", "questions_to_authors": "A: Given just the input cause-effect pair, it's challenging to assess the accuracy of a causality explanation. External knowledge, such as the monsoon example provided, can heavily influence the explanation. Different external knowledge might validate other seemingly spurious explanations. For example, If different external knowledge is available, e.g. the fleet was behind their schedule for homeward voyage, then the spurious explanation given in the paper might be appropriate as well. To address this variability, would it be viable to augment the dataset with multiple explanations for a single cause-effect pair?\n\nB: Why were different evaluation metrics chosen for the datasets? Could you provide comprehensive results?\n\nC: What informed the decision to break critic feedback into observation and commonsense? Was this grounded in theory or based on preliminary findings such as prompt engineering?\n\nD: Do you have any insights on the optimal number of interactions/iterations and potential termination criterias?", "paper_topic_and_main_contributions": "This paper introduces a novel framework to enhance the performance of LLMs in the Causality Explanation Generation task. LLMs assume five distinct roles, encompassing input processing, external knowledge retrieval from the Internet, and iterative answer refinement. LLMs in these roles communicate using a static protocol with specific prompts to collaboratively complete the explanation task. The method was evaluated on two datasets: WIKIWHY and e-CARE. The primary contribution is the multi-agent collaborative framework and the communication protocol, which breaks the primary causality explanation task into several sub-tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The terms \"iteration\" and \"interaction\" are used without clarification in the results section. I presume \"iteration\" pertains to Iterative Feedback and Refinement, and \"interaction\" relates to Fine-grained World Knowledge Integration. Clarifying this distinction would be helpful."}}, {"review_id": "gPvLV93m8E", "reviewer": "Reviewer_3dKC", "summary": "", "questions": "A) How did you determine significance?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback.\n### Respone to Question\n>**Answer for Question A**:  How did you determine significance?\n\nThe task of causal explanation generation as well as its evaluation are both challenging. *To clarify, the datasets [1,2] and evaluation metrics mentioned in the paper are not the contributions of our study*. As they have been widely accepted by NLP community, and in order to maintain consistency with related works, we strictly adhered to the evaluation methods used in previous research for our experiments, including both automated and human evaluation. The results of *automated evaluation* are presented in Table 1 and Table 3. In comparison to the baselines of a single agent (including the updated baseline ReAct), our multi-agent framework effectively enhances the quality of generated explanations. Furthermore, we tested our framework with different underlying language models (InstructGPT, GPT-3.5), and observed an **overall improvement of 8.5% and 2.8%** respectively (based on automated ordered evaluation). Additionally, the results of fine-grained *human evaluation* are displayed in Table 2. Explanations generated by the LEGO framework with GPT-3.5 received a **recognition rate of 75.3%** from human annotators. In response to the concern raised by Reviewer 8kHz about the challenge of intuitively comparing diverse baselines, we have conducted *additional human evaluation*, which can provide a more intuitive reflection of the significance of our results.  Using the explanations of LEGO (GPT-3.5) as references, we compared different baselines against LEGO (GPT-3.5) separately. This was done to visually demonstrate the relative superiority of other baselines over LEGO (GPT-3.5). \n\n| Model|Correctness  | Concision|Fluency | Validity |Win |Tie|Lose |\n| -------- |:----:| :----:|:----: | :----: | :----: |:----: |:----: |\n| **GPT-2** vs LEGO (GPT-3.5) | 0.113 | 0.593 | 0.146 | 0.100 | 0.073 | 0.086 |0.841 | \n| **GPT-3** vs LEGO (GPT-3.5)  | 0.226 | 0.433 | 0.253 | 0.213 | 0.133 | 0.333 |0.534 | \n| **GPT-3.5** vs LEGO (GPT-3.5)  | 0.293 | 0.406 | 0.326 |0.280 | 0.206 | 0.340 | 0.454 | \n| **ReAct (GPT-3.5)** vs LEGO (GPT-3.5)  | 0.306 | 0.393 | 0.313 | 0.326 | 0.213 | 0.346 |0.441 | \n| **LEGO (InstructGPT)** vs LEGO (GPT-3.5)  | 0.360 | 0.426 | 0.453 | 0.393 | 0.226 | 0.373 |0.401 | \n\nThe results indicate that explanations generated by LEGO (GPT-3.5) are considered superior to GPT-3.5 in **70.7%** of cases and outperform the strongest baseline, ReAct, in **69.4%** of cases.\n\n### Respone to Reject\n>**A. About datasets and evaluation metrics** \n\nWe agree that some complex causal relationships may have multiple explanations. To clarify, **these widely used datasets and evaluation metrics mentioned in the paper are not the contributions of our study**. We have provided a detailed description of the datasets and evaluation metrics in the appendix. Explanations come in various structures, as seen in the typology defined by Ribeiro et al [3]. It's not an easy task to perfectly organize all possible explanations. The WikiWhy dataset includes two types of explanations (step sequence and sets) are likely not comprehensive, as a novel effort in QA explanations and reasoning, they probably cover a large proportion of explanations to simple ‘why’ questions. This minimizes the variability of explanations to some extent, facilitating the fair assessment of complex reasoning capabilities of language models. Given the widespread acceptance of these two datasets and their corresponding evaluation metrics within the community, there is no reason for us to reject the utilization of these datasets and forgo our exploration of the causal explanation task.\n\nIn the paper, we extensively elaborate on the metrics for automated evaluation, emphasizing that evaluation is not based solely on a single metric (Appendix B.1). For a comprehensive understanding, please refer to Reviewer 3. Specifically, the concepts of \"completeness\" and \"satisfying\" within the \"Correctness\" are not contradictory. Considering that a gold explanation involves multiple fine-grained knowledge, it's reasonable to require the generated explanation to exhibit \"completeness.\" Simultaneously, to prevent the model from arbitrarily combining these fine-grained knowledge or generating excessive unrelated information, the demand for the generated explanation to be \"satisfying\" to humans is also justifiable. To ensure the validity of human evaluations, we enlisted three graduate students with a certain level of expertise to conduct independent assessments, guaranteeing the credibility of the human evaluation results.\n\n>**B. The impact of our framework** \n\nIn Answer A, we illustrate the **overall impact** of the role-playing framework. For **the impact of different modules** within the framework, such as interactions/iterations, please refer to Answer 4 in our response to Reviewer 8kHz. Additionally, in Figure 7, we present the performance of our framework with knowledge integration module (one round of interaction). As stated in section 4, through iterative refinement, the explainer incorporates multi-aspect feedback from the critic, ensuring that the generated explanations encompass essential information (i.e., worldly knowledge and common sense), thereby enhancing precision. However, irrelevant information is also inevitably present in the feedback. Consequently, after multiple iterations, this irrelevant information contributes to redundancy and disorder within the explanations. As the evaluation algorithm penalizes instances of sentence redundancy and disorder in the explanations, this leads to a decline in accuracy after multiple iterations. Therefore, the findings from our ablation experiments indicate that the **optimal round of iterations** is 2.\n\n### Reference\n[1] Ho M, Sharma A, Chang J, et al. Wikiwhy: Answering and explaining cause-and-effect questions. ICLR 2023\n[2] Du L, Ding X, Xiong K, et al. e-CARE: a new dataset for exploring explainable causal reasoning. ACL 2022\n[3] Ribeiro D, Wang S, Ma X, et al. Entailment tree explanations via iterative retrieval-generation reasoner. NAACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea to augment the input before generating the explanation is interesting.", "reasons_to_reject": "- Multiple explanations are valid, while only a single ground-truth explanation is available in the evaluation dataset. This situation is not well captured by automated metrics and hence evaluation by a single metric (BERT-score) is not sufficient. In addition, the human evaluation guidelines are ambiguous \"Correctness: Mark true if and only if the explanation is both complete and satisfying.\". Correctness (the truth and nothing but the truth) is different from completeness (the whole truth) and far different from satisfying (which is subject to interpretation).\n- The impact of the role-playing framework remains unclear. Unfortunately, section 4 sheds little light as the single-metric issues persists, differences are subtle and not always conclusive (e.g., while the authors state that precision gradually increases in Figure 7, whereas it actually decreases in iteration 3).", "questions_to_authors": "A) How did you determine significance?", "paper_topic_and_main_contributions": "The paper presents a role-playing framework with multiple LLMs for causality explanation generation. According to the presented evaluation, the method improves over single LLM baselines on two datasets.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figure 6 shows differences, Figure 7 absolute values. Please unify.\n- Table 7 is in the appendix and should be referred to as such."}, "extra_text": {}}, {"review_id": "0wLx5UKggL", "reviewer": "Reviewer_dYrk", "summary": "", "questions": "Did you measure the agreement among the annotators?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank your detailed notes and feedback.\n\n### Respone to Question\n>**Answer for Question 1**: Did you measure the agreement among the annotators?\n\nWe apologize that we don’t give detail description about human evaluation. In order to ensure the consistency of the human evaluation with previous works, we follow Matthew Ho [1] to conduct human evaluations via spreadsheets. We randomly shuffled the golden reference and the generated candidates, we don't reveal which column is the golden reference and which is the generated candidate in the evaluation setup. We measured the agreement among three annotators, and the table below displays our **Fleiss' Kappa** scores for the annotation agreements of Win/Tie/Lose (Compare the generated explanation against the provided reference).\n\n| Model |  K |\n| -------- |:----:|\n| GPT-3.5 | 0.84 |\n| ReAct | 0.79 | \n| LEGO (InstructGPT)| 0.76 |\n| LEGO (GPT-3.5) |0.78 | \n\nIt can be observed that all scores exceed 0.7, indicating that the annotation results achieved substantial agreement. We appreciate your suggestions, and we will add it in the revised version.\n\n### Respone to Reject\n\nWe apologize that we don’t give detail description about the **automated evaluation algorithm proposed by Matthew Ho [1]**. To clarify, we employed the BERTScore metric to measure sentence similarity, and \"BERT-f1\" is derived from both Precision and Recall. Taking the unordered evaluation algorithm as an example, Precision and Recall are obtained using the following formulas: \n\n$Precision=\\frac{precise}{predictions}$,  $Recall=\\frac{coverd}{relevant}$\n\nWhere \"predictions\" represents the number of sentences in the generated explanations, \"relevant\" denotes the number of sentences in the reference explanations, \"**precise**\" indicates how many of the generated sentences correspond to sentences in the explanations, and \"**covered**\" represents the number of sentences in the reference explanations that were successfully predicted. \n\nFor example, consider the following examples of reference explanations and generated explanations:\n\n**Reference Explanation**: Opening the highway brought in an influx of unsafe people. With the higher traffic from a highway it would be hard to police the unsafe people. With them being harder to police it would become safer for them and they could drug deal and prostitute in the open.\n\n**Generated Explanation**: When Interstate 5 was opened, it diverted traffic away from these areas and caused a decline in economic activity. With less commerce and people around, it became easier for illegal activities and establishments to operate without attracting attention from law enforcement or regular citizens.\n\n*Score*: precise=2, predictions=2, covered=3, relevant=3.\n\nThe proposed algorithm keeps **separate counts of precise prediction steps and covered reference steps, which effectively addresses the situation where a single prediction sentence may contain multiple reference ideas** (Appendix B). For further details on the algorithm, please refer to the evaluation methods provided by Ho et al. We intend to make our code publicly available in the future to facilitate upcoming research efforts. We appreciate your feedback and will add detail information about metrics in the revised version.\n\n### Other Additions\n\nWe appreciate your feedback. We will add detail information about evaluation algorithm and report The results of automatic evaluations and human evaluations in separate tables in the revised version.\n\n### References\n\n[1] Ho M, Sharma A, Chang J, et al. Wikiwhy: Answering and explaining cause-and-effect questions. ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed bi-directional reasoning mechanism in the cause analyst and effect analyst is novel and interesting. It is also helpful in mitigating the problem of spurious causal association in LLMs. \n\nThe proposed framework is well-motivated and the intuition behind each agent is clearly explained. \n\nThe experiments in this paper are comprehensive. The ablation studies analyze the impact of each major component in the framework. It conducts human evaluation using both absolute-scale metrics (Correctness, Fluency, Validity, etc.) and ranking-based metrics (win, lose, or tie). \n", "reasons_to_reject": "The presentation of the evaluation metrics and experiment results should be improved. It is not clear how to compute the precision, recall, and BERT-f1 scores results in Table 1. The results of automatic evaluations and human evaluations should be reported in separate tables. Please check the detailed comments in Presentation Improvements. ", "questions_to_authors": "Did you measure the agreement among the annotators?", "paper_topic_and_main_contributions": "This paper proposes a multi-agent collaborative framework for causality explanation generation. The framework consists of five agents: cause analyst, effect analyst, knowledge master, explainer, and critic. The cause analyst and effect analyst conduct bi-directional reasoning between the cause and the effect while enquiring world knowledge from the knowledge master. After that, the explainer and critic agents generate the causality explanation via iterative feedback and refinement. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In appendix B.1, it mentions that it uses BERTScore for similarity measurements. Therefore, it seems that all the precision, recall, and F1 scores in Table 1 are computed by BERT score. However, the heading in Table 1 is “Precision, Recall, BERT-f1”, which is quite confusing. Does it mean the precision and recall scores are not computed by BERTScore?\nMoreover, the results of automatic evaluations and human evaluations should be reported in separate tables. This is because automatic evaluations are conducted on the whole test set, while human evaluations are only conducted on 50 random samples. Presenting both automatic and human evaluation results in the same table (e.g., Table 1) may cause confusion. "}}]

Author rebuttal:

Meta-review: This paper introduces a collaborative framework for causality explanation generation where LLMs assume five roles: input processing, knowledge retrieval, and iterative answer refinement. The primary strength of the paper is the novelty of the framework, including bi-directional reasoning mechanism in the cause analyst and effect analyst. Indeed, such a collaborative framework can hold applications beyond causality in the future.

Several concerns related to empirical evaluation were addressed during the rebuttal phase. Please include the new empirical results in the camera ready. Additional ablations would further strengthen the paper, e.g., if cause and effect analysts do not communicate, what will be the impact. 

In summary, this is an interesting and novel framework, with good experiments and analysis (incl. those conducted during the rebuttal).
Decision: Accept-Findings