CReTIHC: Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings
Authors: Changwoo Chun
SongEun Lee
Jaehyung Seo
Heuiseok Lim
Venue: EMNLP 2023 Findings (2023)
Keywords: Causality
Commonsense reasoning
Large Language Models
Temporal Interventions
Hallucinated Confoundings
TL;DR: CReTIHC is a novel dataset that enhances the causal reasoning of Large Language Models by emphasizing the distinction between true causal relationships and the noise introduced by temporal interventions and hallucinated confoundings.

Abstract: Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their ability to establish causal relationships, particularly in the context of temporal interventions and language hallucinations, remains challenging. This paper presents \textbf{CReTIHC}, a novel dataset designed to test and enhance the causal reasoning abilities of LLMs. The dataset is constructed using a unique approach that incorporates elements of verbal hallucinations and temporal interventions through the reengineering of existing causal inference datasets. This transformation creates complex scenarios that push LLMs to critically evaluate the information presented and identify cause-and-effect relationships. The CReTIHC dataset serves as a pioneering tool for improving LLM's causal inference capabilities, paving the way for a more nuanced understanding of causal relationships in natural language processing (NLP) tasks. The whole dataset is publicly accessible at: (https://github.com/ChangwooChun/CReTIHC)

CReTIHC:
Designing Causal Reasoning Tasks about Temporal
Interventions and Hallucinated Confoundings
Changwoo Chun', Songeun Lee', Jaehyung Seoè and Heuiseok Lim
Hyyndai Motor Company
Department of Computer Science and Engineering, Korea University [[wwchun, songeun.
sonncuu , 1ee J@hyundai. com {[seojae777, limhseok eekorea ac.
Abstract
In this study, introduce novel set of tasks and an accompanying dataset, CReTIHC (Causal Reasoning tasks about Temporal Interventions and Hallucinated Confoundings), specifically designed to assess and enhance the causal reasoning capa- bilities of LLMs. Our tasks are rooted in common- sense reasoning and involve a sequence of events that must be analyzed temporally to ascertain cause- and-effect relationships. By integrating elements of language hallucinations and temporal interven- tions, we present LLMs with challenging scenarios that require critical evaluation of information.
Large language models (LL.Ms) have demon- strated impressive capabilities in natural language processing, However, their ability to establish causal relationships, particularly the context of temporal interventions and language hallucinations, remains challenging. This paper presents CReTIHC, novel dataset designed to test and enhance causal reasoning abilities of LLMs. The dataset is constructed using a unique ap- proach that incorporates elements of verbal hallucinations and temporal interventions through the reengineering of existing causal inference datasets. This transformation creates complex scenarios that push LLMs to critically evaluate the information presented and identify cause-and-effect relationships. The CReTIHC dataset serves as pioneering tool for improving LLM's causal inference capabilities, paving the way for more nuanced understanding of causal relationships in natural language processing (NLP) tasks. The whole dataset is publicly accessible at: (https://github.com/ ChanngaoCCCunn//CCRIIIe
The CRETIHC dataset was derived by redesign- ing an existing natural language commonsense causal reasoning dataset, adding complexity by exploiting the difference between temporal and causal order and correlation and causation of word co-occurrences. This made the task of causal rea- soning easy for humans, but LLMs based on word co-occurrence probabilities had difficulty clearly understanding causality from noise caused by tem- poral intervention and confounders in the form of chains, forks, and colliders commonly addressed in existing causality studies (Guo et al., 2020).
1 Introduction
In our evaluation, we delve into the dynamic na- ture of causal relationships text-based languages and their susceptibility to influences from tempo- ral cues and sequences. Our findings indicate that LLMs require further refinement in their causal in- ference capabilities, particularly when utilizing the CReTIHC dataset. This underscores the necessity for a more fundamental approach to understanding causality within natural language.
LLMs have emerged as powerful tool in natu- ral language processing, with the ability to gener- ate human-like text by exploiting extensive prior knowledge. Despite these advancements, LLMs are based on the Transformer structure (Vaswani al., 2017) and have exhibited shortcomings in establishing causal relationships their outputs. This stems from their reliance on empirical reason- ing and statistical patterns from training data (Zhao et al., 2023). Specifically, LLMs face challenges in disentangling true causal relationships from con- founding biases introduced by language halluci- nations and temporal interventions (Zhang al., 2022).
The integration of LLMs into the dataset con- struction process presents a significant advance- ment. This not only enables the generation of con- sistent data based on given instructions but also duces the cost and labor associated with traditional crowdsoureing methods. This innovative methodol- ogy paves the way for the creation of larger, more intricate datasets, demonstrating that the thought-
i Corresponding author

---

ful motivations and innovative ideas of NLP re- searchers can lead to the generation of diverse and complex NLP task datasets.
ROCK (Zhang et al., 2022), and e-CARE (Du et al., 2022). These aim at assessing causal in- ference abilities independently of the sheer vol- ume of knowledge. However, existing datasets and tasks have limitations in measuring and evaluating the dynamic nature of causal relationships, espe- cially in scenarios involving sequential events lead- ing to particular outcome. Furthermore, these datasets don't address confounding biases intro- duced through temporal signals or language hallu- cinations stemming from word co-occurrences
This research is instrumental in addressing the limitations of LLMs, thereby contributing to the de- velopment of more robust and causal-aware natural language processing systems.
2
Related Works
Causal reasoning, the process of identifying and understanding the cause of certain outcomes and the influence of changes in these causes, has been a long-standing focus in various fields (RA, 1958; Cochran and Chambers, 1965; Rosenbaum, 2022). In the domain of deep learning-based natural lan- guage processing, efforts have been made to in- corporate and interpret causal reasoning to discern how causal factors contribute to the outcomes gen- erated by language models (Wood-Dov) boughDyyg et al., 2018; Sridhar and Getoor, 2019; Veitch et al., 2020; Keith et al., 2020; Feder et al., 2021).
To address these limitations, our study restruc- tures the e-CARE? dataset to present tasks cen- tered around temporal interventions and halluci- nated confoundings. These tasks are designed to be more challenging and representative of real-world scenarios. This approach aims to provide foun- dation for more robust and fine-grained analysis of causality in natural language processing, particu- larly in the era following the advent of LLMs.
3 Proposed Method
In the era of deep learning, causal reasoning has emerged crucial capability to ensure that mod- els generalize well and produce accurate outcomes when faced with new data that may not conform to the training data distribution (Schölkopf et al., 2021). Although the scaling laws have facilitated the emergence of models with human-like gener- alization capabilities (Radford et al., 2018; Brown et al., 2020; Chowdhery et al., 2022), such as GPT- 3, GPT-4", and LLaMA (Touvron et al., 2023), by leveraging extensive data and large architectures, these models often struggle establish causality. Pre-training on extensive data does not suffice in ensuring causal inference as causal relationships are often obscured by interventions, confounding biases, and language hallucinations (Feder et al.. 2022). Recently, to enhance causal reasoning in models, methods such as adversarial training and data augmentation techniques rooted in linguis- tic theories of causal connectives have been intro- duced (Staliunaité al., 2021). Notably, these methods have achieved statistically significant im- provements on the COPA dataset, including its bal- anced version designed to avoid surface-level cues. Recently, there have been efforts in creating tasks, frameworks, and datasets focusing com- monsense causal reasoning, such COPA (Gor- don et al., 2012), oNLI (Bhagavatula al., 2020),
Our key idea is investigate changes in the causal relationship between two events when a new event appears. We designed the CReTIHC dataset by adding temporal interventions and hallucinated confoundings to the e-CARE, an existing natural language commonsense causal reasoning dataset. Grounding prior studies regarding causality (Im- bens and Rubin, 2015; Keith et al., 2020; Zhang et al.. 2022: Feder al., 2022), we have no- ticed that unseen, unobserved interventions can heavily impact the causal relationship between events. Also, previous causality research shows that causal relationships guarantee antecedent rela- tionships, but antecedent relationships do not guar- antee causal relationships (Russell, 1912; Bunge, 2017: Zhang al., 2022).
To examine whether LLMs can identify temporal order and actual causal relationships, both tempo- ral order relationships confused with actual causal relationships are needed. However, the original e- C-CARE dataset, which is not arranged chronolog- ically, consists of premises, hypotheses, and ex- planations that make the hypotheses plausible. As described in Figure to create chronologically ordered sequence of events. When the premise and hypothesis are in an effect relationship, we reordered them so that the premise is the event that occurred first and the
restructured this dataset
OpeeAA. (2023). CharGPT & GPT-4 https://openaii
com
Pht t/wp://b..ttut..eee thub. www/ htt////ttteweeee woodeee-wod/e-aar

---

[Original c. CARE corpua]
on the timing of the causal event S1. If S1 and S2 have plausible causal relationship. we intervene with an event that could occur between S and S2 to weaken the causal relationship. Si and S2 do not have a causal relationship with each other, an intervention can strengthen the causal relationship between the two events. The event Z is positioned between the two events, forming chain that al- ters the relationship between the causally related events. Depending on its temporal placement, an intervention can relatively transform the strength of causality between the two events Si and S2.
Premise: There is light rain today
Ask-for: Effect
Hypothesis The roots many plants are not moistened by rain.
Hypothesis 2: Tourists have seen many ripples.
Explanation: Precipitation reaches scil surfaces.
chronological order
[Modified: e- e-CARE corpus]
There a light rain today.
scil Precipitation
S The roots of many plants are not mo istened by rain.
& Tourists have seen many ipples.
Gonerate intervenionn santances
IModified: e-CARE corpus]
There is light rain today
Clempo-leemooral intervention
Sp.s The roots of many plants are not mo istened by rain. S Toursts have seen many ipples.
Prompts of Task 1 As described in Table 1, we consistently used the directive in all prompts: "Never use words or phrases that have already been used, and keep them short and concise, no more than 10 words." " This was implemented to mitigate the tendency of the LLMs to generate wordy sentences and repetitively use previously mentioned words or phrases. The aim was to guide the LLMs towards generating more concise and diverse expressions. Then, two verbs were used to create each sentence with the causal relationship changing in the same direction (e.g. 'make" and 'enhance" to strengthen causal relation- ships). This reason is based on the experimental observation that is an ideal way to choose the one that produces better sentences between mak- ing causal relationships and strengthening them according to the sample.
Inteerventioss Z]
Enhancing Spai The plants' roots deep, beyond the reach of iight
Weakening Sei Most grasses have shallow roots the soil.
Enhancing Syip: Raindrops cause disturbances in the pond,
forming ripples.
Weakening Sus
have seen many ripples caused by
the wind.
Construct Reasoning
(CReTIHC Reasoning tasks]
light today
5y The roots of many plants are not moistemed by rain.
Ac: The plants' rocts are dees. beyend the reach of light rain.
Most grusses shallow reots the soil.
gardener's watering schedule adjusted according to the weather farecast.
Figure I: CReTIHC: Changing the causal relationship of the e-CARE dataset
hypothesis is the event that occurred second. Con- versely, when the premise and hypothesis are in a cause relationship, we reordered them so that the hypothesis is the event that occurred first and the premise is the event that occurred second. The pre- ceding event is denoted S1, and the succeeding event is denoted as S2.
Direction Weaken
Prompt
Write sentence that weakens the causal relationship between S and S2.
Eliminate
Write sentence that completely eliminates causal relationship between S₁ and Sg.
In the next phase, we employed LLMs to ma- nipulate the causal relationships in the restructured dataset, presented We used temporal interventions (Task 1) and intro- duced confounders through linguistic manipulation (Task 2). with the LLM generating sentences that vary the causal relationship between S] and S2. Hallucinated confounders, adding new elements to make spurious causal relationships, were utilized to create confusion between word co-occurrence and actual causality. An event that changes the causal relationship between S and S2 are denoted by Z.
Make
Write sentence that makes the causal relationship hetween Si and Sz.
series of consecutive events.
Enhance
Write sentence that enhances the causal relationship between Si
and S2.
Table I: A prompt create temporal intervention that transforms the causal relationship between S1 and Sy
Task 2: Hallucinated Confoundings Task 2 ex- amines cases where no causal relationship exists between event Sj and Sg, but confounding event z obscures this fact. The task involves two structures: a Collider and a Fork. In the collider structure, the confusing event z is causally related to both S2 and
Task 1: Temporal Interventions Task uses a temporal intervention to manipulate the causal re- lationship between event S, and outcome S2 based

---

are found, in the self-refining phase, LLM au- tonomously rectifies the sentences to suit the di- rection of the task better. At this time, more fine- grained sentences can be obtained using the previ- ously generated results as input to the LLM, along with an additional prompt and few-shot examples. St. Here, no direct causal relationship exists be- tween S2 and St, but event S1 is changed to imply a causal relationship. In the fork structure, both S2 and S1 are the result of the confusion event z, and event S1 is changed to imply a causal relation- ship that does not exist. This creates confusion that there is a causal relationship between S1 and S2. even though there is no direct causal relationship between them.
This self-refining process is iterative and LLM continually evaluates and improves its results un- til it reaches satisfactory level of accuracy. The reason for not providing all information as input from the beginning is based on our experimental ex- perience that sentences obtained through repeated operations of inputting the output of LLM are of higher quality than sentences obtained by providing lot of information from the beginning.
Prompts Task 2 As shown in Table 2, to inte- grate Hallucinated Confounding events, we crafted sentences encompassing confounders, specifically Forks and Colliders, drawing upon the principles of Structural Causal Models (SCMs) (Guo et al., 2020). These SCMs exert influence causality, thereby enabling to seamlessly introduce con- founding elements into narrative. Notably, to ensure the effectiveness of the confounding, we permitted the reuse of words.
Incorporating these self-refining techniques into your data collection methods will improve the qual- ity of the text produced. On the other hand, due to lack of time and cost considerations, we only repeated this iteration a maximum of three times. If the final results did not show sufficiently sat- isfactory results in the self-feedhack phase even after three iterations, they were excluded from the CReTIHC dataset.
Confounding Fork
Prompt
Write a sentence with new factor that affects both SI and S2 and confounds the causal relationship.
Collider
Write a sentence with new factor that is influenced by both S1 and S2 and confounds the causal relationship.
Prompts of Self-refining Self-refining process is divided into two steps: Self-feedback and self- refining. In the self-feedbacl step, the model eval- uates a generated sentence that follows a sequence of events, S1 and S2. In the self-refining step, the model is prompted to rewrite the sentence by intro- ducing
Table 2: A prompt that creates confounding factors in the causal relationship between S1 and $2
confounding factor that either climi- nates establishes causal relationship between S1 and S2. This step is executed when the quality of the sentence generated through the previous self- feedback low. Sentences that have gone through self-refining are used as input to the self-feedback step again iteratively. An example of a prompt used for self-refining is in Appendix B.
Task 1 and Task highlight the importance of context in interpreting causality, demonstrat- ing how linguistically introduced confounders and temporal interventions alter the perception of causal relationships. The fundamental premise of the CReTIHC is maintaining the arrangement of sentences according to the temporal order of events, providing more challenging and realistic task for evaluating the causal reasoning abilities of LLMs.
4 Experiments
3.1 Self-refining Technique
Our proposed CReTIHC dataset introduces the task of identifying all three factors (Enhance, Weaken, Irrelevance) involved in a causal rela- tionship. When appearing an event between two chronologically ordered events, S1 and $2, it is a matter of determining how that event affects the existing causal relationship between S1 and S2. A total of 5 lines of sentences are given in one sample; the SI and S2 events are fixed, and the task is to de- termine how each of the three sentences, Al to A3, affects the causal relationship between the previous
We needed a process to critically evaluate and re- vise sentences generated by LLM to obtain a high- quality causal reasoming dataset. Inspired by the self-refinement method (Madaan et al., 2023), we integrated self-refining the sentences generated by LLM and rewriting inappropriate sentences into the data collection process.
The self-feedback step determines whether the generated sentences the given instructions and guidelines. If conditions are not met or errors

---

Accuracy
F1
results from a sampled set of 100 problems (a total of 300 causal reasoning assessments):
True
False
None
GPT-3
0.671
0.736
0.733
0.501
ChatGPT
0.527
0.664
0.539
0.331
Evaluator
TRUE
FALSE
None
Human A
0.96
0.87
0.81
Table 3: Quantitative evaluation results of LLMs
Human B
0.92
0.91
0.89
Human C
0.92
0.85
0.88
Human D
0.97
0.89
0.88
two events. To exclude the influence of sentence or- der prompts, the positions of Enhance, Weaken, and Irrelvvncce were randomly mixed.
Average 0.94
0.88
0.863
Table 4: Experiments with human evaluators
True: Enhancing causal relationship be- tween S1 and S2.
Post-experiment feedback emphasizes that exter- nal knowledge or personal experience sometimes influences judgment. There was some confusion among some evaluators about the distinction be- tween weakened and nonexistent causal relation- ships, resulting in 2.3% of the sentences being jected. To measure the consistency of agreement between raters, Fleiss' kappa coefficient was cal- culated, resulting in a score of 0.7579. This sug- gests that there was considerable agreement be- tween the raters.
False: Weakening causal relationship be- tween S1 and S2-
None: Irrelevance Not changing a causal relationship between S and S2.
LLM Evaluation In our experiments using 1,000 sentences from the CReTIHC dataset, even ad- vanced LLMs like GPT-3 and GPT-3.5 struggled to accurately identify causal relationships, shown in Table 3 In particular. the accuracy of determin- ing whether the causal relationship has been weak- ened or has no effect on the causal relationship is lower than the accuracy of determining whether the causal relationship has been strengthened. These results support the hypothesis that language models trained on word occurrence probabilities have diffi- culties distinguishing whether co co-occurring words have a causal relationship, especially in the pres- ence of temporal interventions and hallucinated confounding. These results highlight the potential of the CReTIHC dataset in causal inference. Our dataset allows us measure the strength of causal relationships using ranking methods, opening new avenues for research in this field. The detailed version and parameters of the model used in the evaluation are in the Appendix A.
Conclusions
The significance of causal reasoning in language understanding cannot be overstated. Our research has sought to reflect the complexity of real-world causal reasoning by enhancing an existing com- monsense causal reasoming dataset with temporal interventions and hallucinated confoundings. The CReTIHC dataset, rich with continuous event quences and confounding variables, can offer a more rigorous and holistic exploration of causal relationships in language. Our approach mirrors real-world causal reasoning scenarios and has lever- aged LLMs to generate comprehensive datascts. This underscores the potential of LLMs in enhanc- ing various tasks in NLP and highlights the need for models that can efficiently learn from small amounts of data and understand causal relation- ships adeptly humans do. While acknowl- edge that language model trained on our dataset may not fully resolve causal inferences, we believe that our work with the CReTIHC benchmark per- formance measurements contributes to the ongoing journey toward full causal understanding by ma- chines. In conclusion, our research underscores the importance of causal reasoning in language under- standing and paves the way for future rescarch in this critical area.
Human Evaluation We conducted human eval- uation of 100 samples from the CReTIHC dataset. Four evaluators were provided with the same prompts as those used in the LLM, and they were tasked with assessing causality between sentences, with labels True, False, and None. The evaluators weren't 't informed about the actual labels but were asked to identify samples with causal inference errors. The human evaluators achieved aver- age accuracy of 0.895 on the CReTIHC dataset. They excelled identifying strengthened causal relationships but were less accurate with weakened or irrelevant ones. The Table 4 summarizes the
hhttps://en.wwrr wwwipedia.acioommmm hwtp:iaawwwg/iii/eeieieeesseeees kappa

---

Limitations
the responsible use of our research findings and datasets.
While our study presents an innovative approach to augmenting the causal reasoning capabilities of LLMs, it is important to acknowledge its inherent limitations.
A Acknowledgments
This research was supported by Basic Science Re- search Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R A6AI A03045425).
Firstly, our dataset is derived from the e-CARE dataset, which may limit scenario diversity and potentially contain biases or typos from the source data.
Secondly, the self-refinement technique embed- ded in our methodology, although beneficial in en- hancing sentence quality, is not infallible and may not consistently yield optimal refinements. Due to resource constraints, we limited the iterative pro- cess to maximum of three rounds.
References
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reason- ing.
Thirdly, we assessed GPT-4's performance and achieved a high accuracy of 0.92. However, we sus- pect this result may stem from circular evaluation, where the model assessed using data it generated rather than truly reflecting GPT-4's inference capa- bilitics. This raises questions about the suitability of our dataset for evaluating GPT-4.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Lastly, our study primarily focuses tasks in the English language. The generalizability of our findings to other languages remains an open ques- tion.
Going forward, our objectives include expanding the CReTIHC dataset by incorporating broader spectrum of causal scenarios. We also plan to develop more sophisticated self-refinement tech- niques capable of iteratively adjusting sentence quality and difficulty beyond current limitations. Furthermore, we remain open post-release col- laboration with human annotators to address any ad- ditional errors or issues that may arise after dataset publication.
Mario Bunge. 2017. Causality and modern science. Routledge.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov. Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Ethics Statement
Our research uses LLMs and does not directly in- volve human subjects or personal data. The devel- opment and use of LLMs can have serious social implications, including the potential for misuse to create misleading or harmful content. To alleviate these concerns, we have focused and worked on im- proving the causal inference capabilities of L.LM, which can contribute more accurate and reliable results. We also inspected the data for any social bias or harmful expressions that may have been included. Our dataset, CReTTHC, is publicly avail- able and can be used by other researchers to further enhance the capabilities of LLM. We encourage
W. G. Cochran and S. Paul Chambers. 1965. The plan- ning of observational studies of human populations. Journal of the Royal Statistical Society, Series A (Gen- eral), 128(2):234-266

---

Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a new dataset for exploring explain- able causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 432-446, Dublin, Ireland. Association for Computational Lin- guistics.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021. Towards causal representation learning.
Dhanya Sridhar and Lise Getoor. 2019. Estimating causal effects of tone in online debates.
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Dowe bood-Doughty, Ja- cob Eisenstein, Justin Grimmer, Roi Reichart, Mar- garet Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2022. Causal inference in natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Transactions of the Association for Computational Linguistics, 10:1 138-1158.
Ieva Staliünaité, Philip John Gorinski, and Ignacio Ia- cobacci. 2021. Improving commonsense causal rea- soning by adversarial training and data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13834-13842.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anas Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. Causalm: Causal model explanation through counterfactual language models. Computational Lin- guistics, 47(2))333-3))
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEEvll-2013 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In "SEM 2012: The First Joint Conference on Lexical and Computational Seman- tics Volume Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Victor Veitch, Dhanya Sridhar, and David M. Blei. 2020. Adapting text embeddings for causal inference.
Zach Wood-Doughty, lly Shpitser, and Mark Dredze. 2018. Challenges of using text classifiers for causal inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 4586-4598, Brussels, Belgium. Association for Computational Linguistics.
Ruocheng Guo, Lu Cheng, Jundong Li, P. Richard Hahn, and Huan Liu. 2020. A survey of learning causality with data. ACM Computing Surveys, 53(4):1-37.
Jiayao Zhang, Hongming Zhang, Weijie J. Su, and Dan Roth. 2022. Rock: Causal inference principles for reasoning about commonsense causality.
Guido W Imbens and Donald Rubin. 2015, Causal in- ference in statistics, social, and biomedical sciences. Cambridge University Press.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models.
Katherine A. Keith, David Jensen, and Brendan O'Connor. 2020. Text and causal inference: re- view of using text to remove confounding from causal estimates.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisatiwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. reff-re Iterative refinement with self-feedback.
Details of the Language Models
For creating sentences involving temporal inter- ventions and hallucinated confoundings, we uti- lized GPT-4 due to its advanced capabilities in un- derstanding and manipulating causal relationships. However, for self-checking processes, we used ChatGPT. This combination of models allowed us to effectively generate and refine the dataset, lever- aging the strengths of each model.
Fisher RA. 1958. Cancer and smoking.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018 Language models are unsupervised multitask learners.
Paul R. Rosenhaum. 2022. Springer New York, NY.
Observational Studies.
To evaluate the causal inference performance of the LLM,, zero-shot performance was measured on the dataset. Specifically, the text-davinci-003 model based on GPT-3, ChatGPT (gpt-3.5 -.5tur.,,
Bertrand Russell. 1912. On the notion of cause. In Proceedings of the Aristotelian society, volume 13, pages 1-26. ISTOR.

---

and GPT-4 model were used for this purpose. The API and all parameters used default values except those shown the Table
S2, and which one strengthens it. Remember that
A1, A2, or A3 might be unassociated with S1 and
S2. You should pinpoint one sentence from Al
A3 that amplifies the link between S1 and S2, and another sentence from A to A3 that impedes this connection.
Type
Version & Value
GPT-3 4
text-davinci-003
ChatGPT 5
gpt-3.5-turbo-0301
Answer "True" if A# strengthens the causal re- lation between S1 and S2. Answer 'False' if A# weakens causality between S1 and S2. Answer 'None' if A# clearly does not affect the causation between S1 and S2.
GPT-4
gpt-4-0314
temperature
0.0
top_p
null
Table 5: Details the ChatGPT Model
Please assess A1, A2, and A3 individually based on their relevance to S1 and S2. Don't be ver- bose. Only return results among "True", "False" or "None". Do not generate any further explanation." This prompt was designed to encourage the LLM to critically evaluate the relationship between the given sentences and make a determination about the nature of the causal relationship. We ve used some paraphrasing so that the prompts don't overlap too closely with the sentences for creating the dataset.
B Prompts of Self-refining
Step Self-Feedback
Prompt
Answer the sentence given af- ter S2 based on the conditions below. Response 'True' if all of the conditions below are met, and 'False" if any of the following condition not met: Condition 1. Does a sentence enhance/weaken the causal relationship between S1 and S2? Condition 2. Is sentence event located be- tween S1 and S2?
D Details of CReTIHC dataset
From the e- e-CARE training set, total of 14.9K sets of initial datasets were collected for causal- ity evaluation. Of these, about 47% were filtered through the self-refining process. and result of additional filtering through the final manual work of annotators, the CReTIHC dataset consisted of 2,638 causal inference test sets. The criteria pro- vided as a filtering guide for annotators are:
Self-Ree ReffnReinnng
For the given sentence after S2, rewrite the sen- tence by adding a new confounding factor to make/eliminntt//cnnass the causal relationship between S1 and S2. Never use words or phrases that have already been used, and keep them short no more than 10 words.
1. In the case of a sentence that refers the causal relationship by directly using the sentence number 'S1" or 'S2' (c.g "S1 and S2 do not have direct relationship.")
2. In of multiple sentences instead of sin- gle sentence
3. case one instance set is incompletely cre- ated (if any of $1, S2, Enhance, Weaken, or Ir- relevance missing): Most these cases are caused of OpenAl API error or usage quota limit.
Table Prompt to self- self-feedbuck whether the generated sentence meets the appropriate conditions and correct and select the sentence
4. In case of sentences rejected by human anno- tators due to broken text encoding or inappro- priate.
C Prompts for Experiments
In our experiments, we utilized the following prompt for the LLMs:
Statistics of the total data in Table 7. We also rectified errors the e-CARE dataset. Although LLM is robust to small typos, in subtle tasks such as causal inference, even minor differ- ences in words could have a large impact on causal
"Here are five sentences: S1, $2, A1, A2, and A3. SI and S2 depict sequential events. Your task is to determine which among A1, A2, and A3 undermines the causal relationship between SI and

---

Basic Statistics
CReTIHC
# of Test-case
2,638
# of Total Sentences
14,565
# of Unique Sentences
13,324
Average length
52.6
# of Words
126K
# of Unique Words
11K
Table Statistics for CReTIHC dataset
relationships. For example, an event that has an effect on Tomy may not have an effect on Tony. Even errors that could be overlooked by humans were major drawback of the CReTIHC dataset generated from LLM. Therefore, original data ensure accuracy and consistency. Of the 14,028 sentences in our dataset that directly corresponded to e-CARE, 976 were meticulously edited to address spelling errors.
modified the
E Samples of CReTIHC dataset
The CReTIHC dataset consists of six columns. The first column, 'IDX', denotes the individual instance index within the CReTIHC dataset. The second and third columns, 'S." and 'S²", represent two sequential events from e-CARE. The fourth col- umn, 'TRUE', contains events that strengthen the causal relationship between S and Sz. The fifth column, 'FALSE', includes events that weaken this causal relationship. The sixth column, NNONE,, introduces confounding factors that add ambiguity the causal relationship between and S2.

---

IDX
S1
S2
TRUE
FALSE
None
There is a The roots of many plants are not moistened by rain.
The plants' roots are deep, beyond the reach of light rain.
The roots of the majority of plants very shallow.
The gardener's watering sched- ule is adjusted according the weather forecast.
light rain today. Precipitation reaches soil surfaces.
2
Susan wants to buy a restricted pesticide.
She bought rotenone.
She was granted permission to purchase restricted-uss pesticides.
Rotenone's use restrictions were recently lifted.
Local farming communities rig- orously review permits pur- chase rotenone pesticides. sudden wave crashed, startling both the man and the
Rotenone is restricted-use pesticide.
3 He greeted the orcas the wa- ter. Orcas are very social ani- mals.
The oreas
TThe man good at imitat- ing the sounds of killer whales.
The orcas were simply expelling water as part of their breathing.
spouted water to respond to him.
orcas.
4 Tom eats a lot of eggs every day.
His hatchery can produce egg consump- hundreds of tion fuels his unrelated to his hatchery's output.
Tom's high
Tom's egg consumption
Gallstones are lumps com- posed mainly of cholesterol.
millions of fertilized eggs every year.
hatchery's production.
5 Tom eats lot of eggs every day. Gallstones are lumps com- posed mainly of cholesterol.
The cholesterol content in his body is ex- tremely high, so he suffers from gallstones.
He's gallstones are direct sult of his di- etary choices.
Tom is also taking medica- tion lower his cholesterol levels.
Jack's doctor pointed out his habit of cating lot of eggs.
Table 8: Examples of CReTIHC dataset


Reviews:
[{"review_id": "v5xnyLnxnw", "reviewer": "Reviewer_EU9a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We appreciate the time and effort you've invested in reviewing our work. We would like to address the concerns you raised:\n\n> R1: *‘The modification of the e-CARE dataset is based on instruction to the language models. Though it’s a valid method, I do have concerns over the quality of the generated dataset (as acknowledged in the limitations section), and worry about the circular evaluation of the model generating the task and being evaluated on it.’*\n\n**[Concerns over Dataset Quality]**\n\nOur decision to employ LLMs for dataset generation is rooted in their ability to produce diverse and coherent sentences within specific guidelines, surpassing the constraints of solely relying on human annotators. This approach is supported by a technical report which emphasizes the efficiency of LLMs in data labeling, suggesting they can label data as efficiently as humans but at a faster pace [1].\n\nFurthermore, there are inherent advantages of LLM evaluation over human evaluation. As highlighted in a recent study, LLM evaluations are more reproducible. The challenges of human evaluations, such as the difficulty in hiring the same group of evaluators for repeated assessments, make them less consistent in comparison [2]. \n\nTo ensure quality, we implemented a self-feedback mechanism where the LLM iteratively assessed and refined its generated sentences. Only 44.1% of GPT-4 generated sentences met our quality criteria for inclusion in the CReTIHC dataset. Additionally, we conducted a human evaluation. (The results have been added below.)\n\n[1] *LLMs can label data as efficiently as humans, but at a faster pace (https://www.refuel.ai/blog-posts/llm-labeling-technical-report)*\n\n[2] *Chiang, C. H., & Lee, H. Y. Can Large Language Models Be an Alternative to Human Evaluations?. ACL 2023.*\n\n**[Circular Evaluation Concern]**\n\nModel Separation: As authors, we were interested in investigating whether there are Circular Evaluation Concerns when using the same model for data set creation and evaluation. Another study conducted by Wang et al. 2023 [3] confirmed the validity of our approach by simultaneously using one LLM for both generation and evaluation. Nonetheless, we have separated the model used for dataset generation (GPT-4) from the model used for evaluation (GPT-3.5). This further alleviates the circularity problem.\n\nContext-specific differences: Different contexts are used to generate and evaluate data, including prompts. By differentiating the prompt sentences used for data generation and evaluation, contextual conditions change within the LLM environment, similar to the difference between \"CoT reasoning\" and \"without CoT reasoning.\"\n\n[3] *Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. ACL 2023.*\n\n> R2: *‘I would love to see at least a small section of sampling the generated data and doing some kind of human quality eval/control.’*\n\n**[Human Evaluation]**\n\nWe performed a human evaluation of 100 samples from the CReTIHC dataset. Three subjects were given the same prompts as in the LLM and evaluated causality between sentences. 100 samples were selected from the CReTIHC dataset, and one sample consists of 5 sentences. Subjects marked a sentence that strengthens the causal relationship between $\\mathcal{S}1$ and $\\mathcal{S}2$ as True, a sentence that weakens the causal relationship as False, and a sentence that does not change the causal relationship as None. All $\\mathcal{A}1$ to $\\mathcal{A}3$ were generated with one True/False/None target, but this information was not disclosed to the subjects. Therefore, the subjects solved three causal reasoning problems per sample, and the following figures were calculated from 300 test results.\n\nTo provide a more detailed breakdown:\n\n|Evaluator|TRUE|FALSE|NONE|\n|--- |--- |--- |--- |\n|Human A|0.96|0.87|0.81|\n|Human B |0.92|0.91|0.89|\n|Human C |0.92|0.85|0.88|\n\nIn this Table, TRUE refers to *Strengthened Causality*, FALSE indicates *Weakened Causality*, and NONE means *No Causality*.\n\nTo assess inter-annotator agreements, we employed Fleiss' kappa. The **Fleiss' kappa coefficient was calculated to be 0.74**. Fleiss' kappa is a statistical measure used to determine the consistency of agreement between multiple raters. A kappa value above 0.7 typically indicates substantial agreement, suggesting that our human evaluators had a high level of consensus in their assessments. Only 2.3% of the sentences were rejected. Details including these results will be reflected in the final manuscript.\n\nWe hope this format is more suitable for your needs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Models causal reasoning capabilities require more work and this paper adds to that literature and dataset.\n- Making the current dataset more challenging to probe the models and provide opportunities to improve is a valuable contribution\n", "reasons_to_reject": "- The modification of the e-CARE dataset is based on instruction to the language models. Though it’s a valid method, I do have concerns over the quality of the generated dataset (as acknowledged in the limitations section), and worry about the circular evaluation of the model generating the task and being evaluated on it.\n- I would love to see at least a small section of sampling the generated data and doing some kind of human quality eval/control.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a causal reasoning dataset with temporal interventions (a modification of e-CARE dataset). The paper further evaluated two models on the dataset and showed the models have limited accuracy.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "534L7loRxJ", "reviewer": "Reviewer_NqUz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments.We appreciate the time and effort you've invested in reviewing our work. We would like to address the concerns you raised:\n\n> R1 : *‘1. Even for a short paper, this article lacks some necessary work, such as evaluating state-of-the-art models like GPT-4; at the same time, it also lacks necessary analysis of the results.’*\n\n**[Lack of Analysis with State-of-the-Art Models]**\n\nWe understand the importance of evaluating datasets using state-of-the-art models such as GPT-4. During the internal review, there were concerns about **“circular evaluation”** because the datasets were created using the same model. \n\nHowever, based on your comments, we conducted further experiments with GPT-4. GPT-4 was tested on 100 samples like human evaluation, and showed slightly higher results than the results of human subjects with an accuracy of 0.92. We believe this is related to the \"circular evaluation\" concern.\n\n**[Insufficient Analysis of Results]**\n\nOur main focus is designing a new dataset to test the LLM's causal reasoning ability. The LLM's performance results on the data set presented in the \"Experimental\" section are intended to demonstrate the usefulness of the data set and new causal inference challenges rather than to provide an exhaustive analysis of causal understanding.\n\nWe performed a human evaluation of 100 samples from the CReTIHC dataset in terms of result analysis. Three subjects were given the same prompts as in the LLM and evaluated causality between sentences. 100 samples were selected from the CReTIHC dataset, and one sample consists of 5 sentences. Subjects marked a sentence that strengthens the causal relationship between $\\mathcal{S}1$ and $\\mathcal{S}2$ as True, a sentence that weakens the causal relationship as False, and a sentence that does not change the causal relationship as None. All $\\mathcal{A}1$ to $\\mathcal{A}3$ were generated with one True/False/None target, but this information was not disclosed to the subjects. Therefore, the subjects solved three causal reasoning problems per sample, and the following figures were calculated from 300 test results.\n\nTo provide a more detailed breakdown:\n\n|Evaluator|TRUE|FALSE|NONE|\n|--- |--- |--- |--- |\n|Human A|0.96|0.87|0.81|\n|Human B |0.92|0.91|0.89|\n|Human C |0.92|0.85|0.88|\n\n\nIn this Table, TRUE refers to *Strengthened Causality*, FALSE indicates *Weakened Causality*, and NONE means *No Causality*.\n\nTo assess inter-annotator agreements, we employed Fleiss' kappa. **The Fleiss' kappa coefficient was calculated to be 0.74**. Fleiss' kappa is a statistical measure used to determine the consistency of agreement between multiple raters. A kappa value above 0.7 typically indicates substantial agreement, suggesting that our human evaluators had a high level of consensus in their assessments. Only 2.3% of the sentences were rejected. Details including these results will be reflected in the final manuscript.\n\n> ‘R2: *More in-depth analysis of the newly obtained data (beyond statistical data), including data distribution, comparison with other existing datasets, etc., has not been discussed.’*\n\n**[Lack of In-depth Dataset Analysis]**\n\nBased on the E-CARE paper that started this study, we provided the same level of data set distribution information. Nonetheless, recognizing the importance of more in-depth data set analysis, the final version will include additional details such as basic statistics (word count, sentence length distribution) and examples in an appendix.\n\nWe believe that our work makes a significant contribution to the field of causal reasoning in NLP. We hope that our clarifications address your concerns, and we are committed to making the necessary revisions to enhance the quality of our manuscript.\n\nThank you for considering our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  \"Cause-and-effect\" is a long and interesting research topic.\n\n2.  The idea of redesigning existing natural language commonsense causal reasoning datasets will be promising in future Causal-aware tests.", "reasons_to_reject": "1. Even for a short paper, this article lacks some necessary work, such as evaluating state-of-the-art models like GPT-4; at the same time, it also lacks necessary analysis of the results.\n\n2. More in-depth analysis of the newly obtained data (beyond statistical data), including data distribution, comparison with other existing datasets, etc., has not been discussed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents CReTIHC, a novel dataset designed to test and enhance the causal reasoning abilities of LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H1l7EasUju", "reviewer": "Reviewer_eRnj", "summary": "", "questions": "Line 257: The method described in this paragraph sounds like adversarial training, is it different in some way or are there other reasons why you do not use this term to describe it? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We appreciate the time and effort you've invested in reviewing our work. We would like to address the concerns you raised:\n\n> R1 : *‘Line 169: The main part of the data collection is placed in the appendix. This should have definitely been in the main body of the paper.’*\n\n**[Data Collection in the Appendix]**\n\nWe acknowledge your concern regarding the placement of the main part of the data collection in the appendix. Given the page limit of 4 for short papers, we made this decision. However, if our paper is accepted, we will move this section to the main body in the camera-ready version, which allows up to 5 pages.\n\n> R2 : *‘Line 257: The method described in this paragraph sounds like adversarial training, is it different in some way or are there other reasons why you do not use this term to describe it?’*\n\n**[Differences from adversarial training]**\n\nIn response to your question about the similarity of our method to adversarial training: While both approaches aim to challenge and improve model performance, they differ fundamentally in their objectives and implementations.\n\nOur method, as outlined in the CReTIHC dataset, emphasizes designing causal reasoning tasks that mirror real-world scenarios. The core objective is for the Large Language Model (LLM) to critically assess causality, distinguishing genuine causal relationships from biases induced by language hallucinations and temporal events.\n\nLet's take an example :\n\n1. **copper block was heated from fire.**\n\n2. **pick the copper block by bare finger.**\n\n3. **The skin burns.** \n\nNote: *The causal relationship between events 2 and 3 changes depending on which sentence is put between events 2 and 3.*\n\nOption 1 > **Enhanced : About 5 seconds have passed.**\n\nOption 2 > **Weaken : About 5 days have passed.** \n\nThe former can strengthen the causal relationship, but the latter weakens the causal relationship because we know that the copper block cools down over the 5-day period.\n\nThe key point lies in understanding how an new intervening event, occurring between two chronologically ordered events, impacts their existing causal relationship. This nuanced approach sets our method apart from adversarial training, which primarily confuses models by altering specific words to produce similar sentences.\n\nOn the other hand, adversarial training typically involves generating perturbed or \"adversarial\" examples to challenge the model, ensuring it doesn't overfit to specific patterns in the training data and generalizes better to unseen data.\n\nIn essence, while adversarial training seeks to improve model robustness against slight perturbations, our method aims to enhance a model's causal reasoning abilities by presenting it with complex scenarios that require a deeper understanding of cause-and-effect relationships.\n\nWe hope this clarifies the distinction between our approach and adversarial training. We will ensure to make this differentiation more explicit in the revised manuscript to avoid any confusion.\n\n> R3: *'Improving commonsense causal reasoning by adversarial training and data augmentation\" by I Staliunaite, PJ Gorinski, I Iacobacci (2021) have presented a very similar method of generating confounders as well as generating adversarial examples to improve commonsense causal reasoning models.’*\n\n**[Missing References]**\n\nWe sincerely thank you for letting us know about valuable existing research. We recognize the importance of distinguishing our work from existing research. Our method focuses on constructing a new causal inference task using the e-CARE dataset, incorporating temporal interventions and hallucinated confoundings. Our approach requires discerning which sentence strengthens or weakens the causal relationship between two given sentences. In contrast, Staliunaite et al. [1] utilize adversarial training and data-augmentation techniques as a means to increase the data size for training causal inference models, given the limited size of the COPA dataset. We will clarify these distinctions in our revised manuscript and ensure that the mentioned paper is appropriately cited.\n\n[1] *Staliūnaitė, I., Gorinski, P. J., & Iacobacci, I. (2021, May). Improving commonsense causal reasoning by adversarial training and data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 15, pp. 13834-13842).*\n\n> R4: *‘Line 204: Very repetitive explanation.’* & *‘Line 300: What does ‘reversely weaken’ mean? Is this meant to be described in contrast to strengthening? Better phrasing could help make it more clear.’*\n\n**[Typos and Presentation Improvements]**\n\nI saw what could be seen as repetitive explanations and unclear wording. It was intended to emphasize the meaning, but in the direction of the reviewer's advice, I will edit the mentioned line for clarity and conciseness. The term 'reversely weaken' is replaced with the explicit expression '(just) weaken' to avoid confusion.\n\n> R5: *‘Would be hard pressed to reproduce the results. The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.’*\n\n**[Reproducibility Concerns]**\n\nAll prompts necessary to generate and evaluate the data have been clarified. And we followed the standard procedure as described in the OpenAPI User Guide. For clarity and to ensure reproducibility, we will state all parameters, including default values, in the final version.\n\nWe believe that our work makes a significant contribution to the field of causal inference in NLP. We hope that our explanations address your concerns, and we promise to make necessary corrections to improve the quality of the manuscript.\n\nThank you for considering our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A very interesting way of combining causal inference theory on forks, chains and colliders for training better commonsense reasoning models. \n", "reasons_to_reject": "Line 169: The main part of the data collection is placed in the appendix. This should have definitely been in the main body of the paper. \n", "questions_to_authors": "Line 257: The method described in this paragraph sounds like adversarial training, is it different in some way or are there other reasons why you do not use this term to describe it? \n", "paper_topic_and_main_contributions": "The paper presents a causal reasoning dataset, which has been constructed from already existing causal inference datasets, by using verbal hallucinations and temporal interventions on the existing data points. The new dataset presents the task of not only predicting the presence of a causal link between clauses, but also the prediction of whether another clause strengthens or weakens this relationship. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 204: Very repetitive explanation. \n\nLine 300: What does ‘reversely weaken’ mean? Is this meant to be described in contrast to strengthening? Better phrasing could help make it more clear. \n"}, "extra_text": {"Missing_References": "\"Improving commonsense causal reasoning by adversarial training and data augmentation\" by I Staliunaite, PJ Gorinski, I Iacobacci (2021) have presented a very similar method of generating confounders as well as generating adversarial examples to improve commonsense causal reasoning models. "}}]

Author rebuttal:

Meta-review: The work introduces a dataset for enhancing the causal reasoning of Large Language Models. 

Among the major concerns expressed during the reviewing period, the lack of human evaluation of the contents produced by language models stands out (even though it is becoming a current practice supported by some investigations) and the circular evaluation of the model generating the task and being evaluated on it. The authors provided additional materials that tackle this issues and that should be included in the final version of the paper if accepted - together with all other comments.
Decision: Accept-Findings