Causal-driven Large Language Models with Faithful Reasoning for Knowledge Question Answering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:31:54

Abstract: In Large Language Models (LLMs), text generation that involves knowledge representation is often fraught with the risk of ''hallucinations'', where models confidently produce erroneous or fabricated content. These inaccuracies often stem from intrinsic biases in the pre-training stage or from the incorporation of human preference biases during the fine-tuning process. To mitigate these issues, we take inspiration from Goldman's causal theory of knowledge, which asserts that knowledge is not merely about having a true belief but also involves a causal connection between the belief and the truth of the proposition. We instantiate this theory within the context of Knowledge Question Answering (KQA) by constructing a causal graph that delineates the pathways between the candidate knowledge and belief. Through the application of the do-calculus rules from structural causal models, we devise an unbiased estimation framework based on this causal graph, thereby establishing a methodology for knowledge modeling grounded in causal inference. The resulting CORE framework (short for ``Causal knOwledge REasoning'') is comprised of four essential components: question answering, causal reasoning, belief scoring, and refinement. Together, they synergistically improve the KQA system by fostering faithful reasoning and introspection. Extensive experiments are conducted on ScienceQA and HotpotQA datasets, which demonstrate the effectiveness and rationality of the CORE framework.

Main paper: 
 introduction: 
 With the development of instruction tuning techniques, large language models (LLMs) have demonstrated impressive performance across tasks in natural language understanding and generation. These models have shown great potential in human-like capabilities of common sense, logical reasoning and tool manipulation, which can understand and harness knowledge to improve problem-solving and decision-making tasks. However, many studies [3, 5, 8, 11, 18] have found that LLMs may produce some hallucinations that are neither accurate nor factual. Hallucinations in LLMs occur when they produce incorrect or misleading information, often linked to shortcomings in two types of knowledge: intrinsic and extrinsic. Intrinsic knowledge refers to common sense, and logical reasoning abilities, which LLMs may sometimes fail to utilize, leading to nonsensical or factually wrong outputs. Extrinsic knowledge is about grasping new, contextual, or specific tool-based information, which LLMs may not always interpret accurately. Influenced by biases in their training data, both knowledge types can lead to LLMs delivering coherent, yet factually incorrect or misleading responses.To address these issues, there are mainly two types of research works proposed to reinforce LLMs, i.e. reasoning-based method and confidence-based method. Reasoning-based methods [6, 12, 28, 32, 40, 45] are proposed to enhance the capability of LLM not by retraining models but through reasoning with knowledge of LLMs, allowing LLMs to improve decision-making on a continual thought process. Confidence-based methods [14, 17, 21, 23, 31, 34, 35] are designed to explore various reasoning paths and employ a verifier that checks the correctness or self-consistency of each reasoning step to leading a unique answer throughout the multi-path reasoning. While both types of methods have been found to be effective, there is a notable absence of a rational explanation regarding the necessity of incorporating all these designs. In addition, whether these two approaches are intrinsically related still needs to be further explored.In fact, there is a specific philosophical definition of "knowledge", which has been discussed comprehensively in Goldman's causal theory of knowledge [10] . The causal theory posits that true knowledge is derived from a causal connection between a proposition and a belief. In the LLM context, the "Proposition" represents the input and output information from LLMs. The "Causal Connection" is the critical step, wherein the LLM must establish a logical and factual link between the proposition and the potential belief. This involves not only recalling information from its vast training dataset but also applying logical reasoning to ensure that the connections made are rational and grounded in causality. Finally, the "Belief" is the confidence to the proposition of whether it is a faithful knowledge, which should be a true reflection of the causal and logical processing undertaken in the previous step.Drawing from Goldman's causal theory of knowledge, we propose a CORE framework (short for "Causal knOwledge REasoning") to process Knowledge Question Answering (KQA) tasks, in which a typical KQA process is as illustrated in Figure 1 . The CORE framework suggests a reasoning structure where each decision or output follows a clear path: Proposition → Causal Connection → Belief. To this end, we translate the reasoning structure into a practical application by creating a causal graph that outlines the connections among knowledge theory and training biases of LLMs. Meanwhile, we apply do-calculus rules from structural causal models [27] to derive an unbiased estimation framework for this causal graph. Based on the estimation formulation, the CORE framework consists of four key components, each playing a crucial role in enhancing KQA systems with robust and reliable reasoning capabilities: 1) Question Answering. This module aims to provide a preliminary answer hypotheses to a KQA question as illustrated in Figure 1 .2) Causal Reasoning. This component focuses on establishing a logical and factual link between a given question-answering response (the Proposition) and the potential confidence (the Belief).3) Belief Scoring. This module assesses the logical consistency and faithful accuracy of KQA output, aligning with the principles of causal reasoning. 4) Refinement. This last component fine-tunes the outputs, ensuring that the reasoning process aligns with the causal mechanisms outlined in the previous causal reasoning process.By incorporating the CORE framework, LLMs can be guided to process information in a manner that mimics human-like causal reasoning to produce knowledge, termed as "Faithful Reasoning" [6, 12, 13] , potentially reducing instances of hallucinations and enhancing the model's ability to generate more accurate and reliable responses. This approach aligns with the intrinsic and extrinsic knowledge frameworks, ensuring that the LLM's reasoning is not only based on the data it has been trained on but also the logical coherence and factual accuracy of the information. Further, this "Faithful Reasoning" method can synergize with existing reasoningbased and confidence-based methods. By integrating causal theory into these approaches, LLMs can be better equipped to navigate the complex reasoning paths, assess the self-consistency of their outputs, and arrive at conclusions that are not only confident but also causally and logically sound. By conducting experiments on two real-world datasets, we have illustrated the superiority of our proposed framework on both overall performance comparison and micro-scope studies.The main contributions of this work are summarized as follows:• We introduce a causal knowledge reasoning framework, which is a novel approach integrating Goldman's causal theory of knowledge into large language models, particularly in the KQA context. This framework is designed to enhance the reliability and accuracy of knowledge understanding in LLMs by establishing a clear pathway from proposition to belief through causal connections.  
related works 2.1 knowledge in llms: 
 The exploration of knowledge in pre-trained LLMs has been a central theme in recent research, reflecting the advances and challenges in harnessing LLMs for enhancing decision-making and problemsolving in multimodal contexts. Yang et al. [38] examine the capabilities of GPT-4V, a model that augments LLMs with multi-sensory skills, including visual understanding. This research demonstrates the potential of LLMs in harnessing both intrinsic and extrinsic knowledge, highlighting their utility in diverse applications. Similarly, Yin et al. [42] provide an extensive survey on multimodal large language models, elucidating their surprising emergent capabilities and potential paths toward artificial general intelligence. These studies collectively showcase the significant strides made in LLMs, especially in terms of multimodal understanding and logical reasoning. On the other hand, Davis and Aaronson [8] bring to light the limitations of LLMs in interfacing with external tools like Wolfram Alpha, particularly in solving math and science problems. Their findings resonate with the challenges LLMs face in processing intrinsic and extrinsic knowledge accurately, often leading to incorrect or misleading outputs. Complementing this perspective, Arkoudas [3] critically evaluates GPT-4's reasoning abilities, underlining its shortcomings in logical reasoning and problem-solving, which are core aspects of intrinsic knowledge.These studies collectively underscore the dual nature of LLMs' advancements. On one hand, they demonstrate significant progress in multimodal understanding and logical reasoning. On the other hand, they reveal the persistent challenges of hallucinations and biases, emphasizing the need for improved models that can accurately harness intrinsic and extrinsic knowledge without succumbing to inaccuracies and misleading information. To this end, we aim to enhance the reasoning capabilities of LLMs, ensuring more reliable and accurate knowledge processing in line with the principles of "Faithful Reasoning".Advancements in LLMs have been centered around enhancing "Faithful Reasoning", mainly categorized into reasoning-based and confidence-based methods.Reasoning-Based Methods. This category includes methods that improve intrinsic reasoning capabilities of LLMs. Chain-ofthought prompting [32] , least-to-most prompting [45] , and the rethinking with retrieval approach [12] exemplify this. These methods break down complex problems into simpler steps, enhancing logical processing abilities of LLMs. The tree of thoughts framework [40] and Reflexion [28] also fall under this category, promoting adaptive learning and strategic decision-making in LLMs.Confidence-Based Methods. Focused on verifying the reasoning processes, this category includes DIVERSE [23] , discriminatorguided reasoning [14] , and self-evaluation guided beam search [35] . These methods ensure the reliability and accuracy of LLM outputs by verifying each reasoning step and employing feedback mechanisms. Self-consistency methods [1, 31, 34] further exemplify this approach by marginalizing out multiple reasoning paths to select the most consistent answer.In light of these developments, we provide a causal-based perspective grounded in Goldman's causal theory [10] and Pearl's structural causal model [27] , elucidating why these faithful reasoning methods are effective. Based on a theoretical unbiased formula, we propose a CORE framework with four main components, which synergistically combines these two methodological approaches to enhance the reliability and interpretability of LLMs in processing and generating knowledge.The KQA task, particularly in its multimodal form, presents a unique challenge and opportunity for language modeling systems. It combines objective knowledge extraction with reasoning across multiple modalities, making it an ideal testbed for advanced multimodal LLMs.Current studies [2, 7, 9, 16, 19, 20, 24, 43, 44] mainly focus on enhancing multimodal reasoning capabilities in KQA task. Innovations such as BLIP models [7, 19, 20] , ScienceQA [24] , and Multimodal-CoT [44] exemplify this trend, showcasing significant advancements in the LLM ability to process and reason within text-rich visual contexts and using multimodal information for more accurate and comprehensive question-answering.In contrast to these methods, our causal-driven approach in the CORE framework potentially offers a unified solution for VQA tasks. By grounding LLM reasoning in causal relationships as per Goldman's causal theory, our approach aims to validate and generate faithful knowledge, which seamlessly integrate and reason across diverse knowledge forms, from text to visual content, in a causally faithful manner. 
methods: 
 This section describes the formulation of an interventional causal model for the KQA task within a visual-question answering (VQA) context, grounded in Goldman's causal theory [10] and Pearl's structural causal model [27] . Based on this theoretical method, we build a CORE framework to facilitate faithful reasoning by implicitly accounting for confounders without direct modeling, which mitigates dataset biases and foster the acquisition of faithful knowledge.V Q T B U A (a) Original graph V Q T B U A (b) Interventional graph3.1.1 Causal Graph. As Goldman's causal theory of knowledge posits, true knowledge is derived from a causal connection between a proposition and a belief. Since the knowledge in LLMs is acquired during the pre-training stage, this may introduce dataset bias in the models, leading to inaccurate knowledge understanding. To confront these problems, we employ Pearl's structural causal model to obtain a causal graph between the causal theory and the learning process of LLMs, which is illustrated in Figure 2a , where nodes and edges represent endogenous variables and causal relationships, respectively. To justify the rationality of the proposed causal graph, we give detailed interpretations of the causal graph as follows.{𝑉 , 𝑄, 𝐴} → 𝑇 → 𝐵. The sequence of nodes 𝑉 , 𝑄, 𝐴 represents the input variables: the Visual information of images (V), the Textual Question (Q), and the Answer (A), in which these variables combine to form a VQA example. Causal Thoughts (T) are the reasoning process about how the answer causally relates to the question within the given multimodal context. The flow from T to Belief (B) represents the evaluation of the causal thoughts, leading to a belief about the accuracy of answer. This progression from question and visual context, through causal reasoning, to belief is central to our CORE approach to understanding and reasoning knowledge.𝑈 → {𝑉 , 𝑄, 𝐴, 𝐵}. The Unobservable Confounder (U), often representative of Dataset Bias, is an external factor that can influence the belief in the truth of a VQA proposition as knowledge. It is important to recognize the potential of U to introduce training bias into the understanding of knowledge, which can lead to incorrect beliefs. Our CORE framework aims to address this by isolating and mitigating the influence of U on the causal pathway from inputs to belief, thus striving for a more accurate and unbiased reasoning process.Given images and questions, LLM-based VQA models can perform natural language generation to answer the questions. However, the generated answers may be incorrect because of the training bias of the LLMs. To further verify the beliefs of the generated knowledge based on the causal model, we aim to estimate the true belief of knowledge using LLMs without the impact of the training biases. Therefore, we need to estimate the interventional conditional probability of the beliefs of whether the answers correctly answer the questions, which is not equal to the observational conditional probability:EQUATIONwhere the 𝑑𝑜 (•) operation for input variables show that we want to pursue the true causal effect of knowledge {𝑉 , 𝑄, 𝐴} to beliefs 𝐵 by intervention [27] . This circumvents hallucinations from biases derived from 𝑈 and ensures the reliability of the model outputs within the KQA tasks.To this end, we employ causal intervention to disentangle the influence of dataset bias (i.e., the unobservable confounder 𝑈 ) from the knowledge reasoning process, as depicted in Figure 2b . We adopt do-calculus rules [27] to derive an unbiased belief estimation, focusing on a causal-driven process that considers only the variables of interest while controlling the confounding influences. Let • be the intervention operator 𝑑𝑜 (•) for blocking the dataset bias, the causal intervention process can be described mathematically as follows:EQUATIONThis derivation process is similar to the methods [30, 33, 37] based on the front-door criterion [27] , which transforms the intervention probability estimation task with the 𝑑𝑜 (•) operator into an association probability estimation task. Since 𝑃 (𝑡 |𝑣, 𝑞, 𝑎) is an encoder for mapping multimodal question-answering inputs to causal thoughts, then the probability of 𝑃 (𝑡 𝑣,𝑞,𝑎 |𝑣, 𝑞, 𝑎) equals to one if this causal reasoning model is assumed to have an deterministic thought process 𝑡 𝑣,𝑞,𝑎 to explain how well the answer response to the question. This means that Equation 2 can be further transformed as follows:EQUATIONThis formula indicates that the probability of the true belief 𝑏 of a proposition {𝑣, 𝑞, 𝑎} can be unbiasedly calculated by averaging the expectations over supporting samples that follow the same causal thought process 𝑡 𝑣,𝑞,𝑎 . Therefore, identifying different proposition {𝑣 ′ , 𝑞 ′ , 𝑎 ′ } that support the causal explanation 𝑡 𝑣,𝑞,𝑎 is the key to the estimation of the true belief. In practical question-answering scenarios, the visual 𝑣 ′ and question 𝑞 ′ generally remain constant within a given context, meaning 𝑣 ′ = 𝑣 and 𝑞 ′ = 𝑞. This implies that our estimation should primarily focus on exploring diverse answers 𝑎 ′ that coincide with the established causal thought process 𝑡 𝑣,𝑞,𝑎 . This recognition plays an important role in estimating the confidence in knowledge generated by the model in response to a visual-question inquiry.Based on the unbiased estimation of the interventional target of knowledge as outlined in Equation 3, we aim to estimate it within the context of the VQA task. Given textual inquiry 𝑞 and the corresponding visual information 𝑣, we first generate an answer 𝑎 to the question, and then propose the causal thought process 𝑡 to support or repel the proposition {𝑣, 𝑞, 𝑎}. After that, our goal is to refine the same question {𝑣, 𝑞} using different perspective responses 𝑎 ′ to eventually improve the expected accuracy of beliefs 𝑏.To operationalize this theory in VQA, we develop the CORE framework, which integrates four principal components: Question Answering 𝑃 (𝑎|𝑣, 𝑞), Causal Reasoning 𝑃 (𝑡 |𝑣, 𝑞, 𝑎), Belief Scoring 𝑃 (𝑏 |𝑣, 𝑞, 𝑎, 𝑡), and Refinement 𝑃 (𝑎 ′ |𝑣, 𝑞, 𝑎, 𝑡). As illustrated in Figure 3 , each component serves a unique function in the faithful reasoning process, working collaboratively to ensure a reliable estimation of the interventional target. Below, we detail the specific functions of each component and outline their implementation within the prompting strategy.Answering. The Question Answering 𝑃 (𝑎|𝑣, 𝑞) module is the foundational component of our framework, tasked with generating preliminary answer hypotheses based on visual inputs 𝑣 and questions 𝑞. Leveraging large language models, this module synthesizes information across modalities to produce contextually relevant answer.Similar to the previous works [24] , we first employ the state-ofthe-art visual-language model InstructBLIP [7] to represent the visual context of image as text, where the model has been instructiontuned on a wide variety of tasks based on the pretrained BLIP-2 model [19, 20] . And then, the visual context is concatenated to the question context to form a visual-question input. To provide a preliminary answer hypotheses, we follow the approach of Sci-enceQA [24] , which perform multi-option question answering task based on a few-shot learning prompt technique. Specifically, we build instructions using two in-context examples with components of the question text, options, and the correct answer text. This style of prompt enables the LLMs to answer specific questions without parameter updates.Reasoning. To verify the faithful accuracy of the answer, our CORE framework emphasizes establishing a causal connection between a given question-answering response and the corresponding belief. Central to our CORE framework, the Causal Reasoning 𝑃 (𝑡 |𝑣, 𝑞, 𝑎) constructs causal thoughts 𝑡 that serve as explanatory narratives that connects questions and visual cues to potential answers. Figure 3 : Comprehensive schematic of the CORE framework, which details the sequential interaction of the four primary components. 2In this work, we embodies the essence of causal reasoning processes as illustrated in Figure 3 , enabling the model to observe information and discern logical pathways by prompting the LLMs. By integrating chain-of-thought prompting techniques to facilitate causal reasoning, our Causal Reasoning module perform introspection and evaluate answer hypotheses through a causally-aware process.The Belief Scoring module 𝑃 (𝑏 |𝑣, 𝑞, 𝑎, 𝑡) evaluates the faithful accuracy and logical consistency of the generated answers. It assigns belief scores 𝑏 to the proposition and the causal thought process, reflecting the degree of confidence in the correctness and relevance of candidate knowledge.To estimate the belief score effectively, our approach employs a verbalized method for estimating model confidence. This method is inspired by recent observations [29, 36] that LLMs have been shown to offer more accurate measure of confidence using output tokens, typically better-calibrated than methods using traditional conditional probabilities. As shown in the prompt text in Figure 3 , we adopt a grading scale akin to the US five letter grades to quantify the confidence of the initial answer being correct. Specifically, "A" indicates Confidently Correct 100%, "B" indicates Correct with Moderate Confidence [75%, 1), "C" indicates Incorrect with Some Confidence [50%, 75%), "D" indicates Confidently Incorrect. [25%, 50%), and "F" indicates No Direct Answer [0%, 25%).As the final stage of our reasoning process, the Refinement 𝑃 (𝑎 ′ |𝑣, 𝑞, 𝑎, 𝑡) will polish the initial response. It harmonizes initial answer 𝑎 with the causal thought 𝑡 to generate a new answer 𝑎 ′ that conform the most plausible causal explanations.To implement this, we repurpose the prompting method of Question Answering module. By appending the original VQA response and associated causal thoughts to the prompt of the Refinement model, we guide the LLMs to correct their answers according to the established reasoning path. Refine the answer 𝑎 to 𝑎 𝑟 based on the causal thought 𝑡.Re-evaluate 𝑎 𝑟 using causal reasoning and belief scoring module to get belief score 𝑏 𝑟 . answer. Following this, the Causal Reasoning module analyzes the answer to identify the underlying causal rationale. After that, the Belief Scoring module evaluates the confidence level of the model in the initial answer, based on how accurate and consistent it aligns with the causal explanations. If any answers lack sufficient confidence, the Refinement module helps by adjust them to make more sense based on the initial answer and the causal explanations. Note that we can perform refinement process when the belief score of the answers is lower than a pre-defined belief threshold 𝜏, so as to make a trade off between accuracy and efficiency. And we set 𝜏 = 𝐴 for better accuracy by default. In addition, when an answer undergoes refinement, the refined answer is not simply accepted; it undergoes reevaluation through the causal reasoning and belief scoring processes, consistent with the formulation 𝑃 (𝑏 |𝑡 𝑣,𝑞,𝑎 , 𝑣 ′ , 𝑞 ′ , 𝑎 ′ ) established in Equation 3. The revised answer, denoted as 𝑎 ′ , is adopted only if it achieves a belief score superior to its initial counterpart. Otherwise, the initial response is retained. This iterative refinement process ensures that the framework outputs are both consistent and deeply rooted in a robust causal understanding, thus enhancing the overall reliability of the knowledge they represents. Overall, all these processes are inspired by the unbiased knowledge evaluation process as shown in Equation 3. For more examples and details on how each part works, please check out the case study (section 4.6) and appendix. 
experiments 4.1 experimental settings: 
 4.1.1 Datasets. Our methodology is evaluated on two prominent datasets, each with its unique structure and challenges. The first, ScienceQA [24] , is a comprehensive multimodal science question dataset with annotations that include detailed lectures and explanations. It encompasses a broad domain diversity with 21k multiplechoice questions spanning 3 subjects, 26 topics, 127 categories, and 379 skills. The dataset is organized into distinct training, validation, and test sets containing 12,726, 4,241, and 4,241 examples, respectively.The second dataset, HotpotQA [39] , consists of 113k Wikipediabased question-answer pairs. Its defining characteristics are as follows: (1) it necessitates the retrieval and synthesis of information across multiple documents to construct answers, and (2) the questions are varied, avoiding restrictions from any predefined knowledge bases or schemas. This dataset provides an additional level of complexity to our evaluation, allowing us to test more advanced prompting techniques on our proposed method. In the following experiments, we randomly select a subset of 1,000 samples for performance evaluation.For the ScienceQA dataset, we establish two-shot in-context learning baselines, following the format of "A", "AE", and "ALE". This involves concatenating question text, context of text and visual, and multiple options as input, with predictions for answers (A), lectures (L), and explanations (E) as output, aligning with the previous works [24, 44] . These prompting formats are employed in our Question Answering module and the output formatting of the Refinement module for a fair comparison. Following the ScienceQA settings [24] , our baselines include VQA models such as ViLT [16] , Patch-TRM [25] , and VisualBERT [22] ; Text-to-text Language Models (LMs) like Uni-fiedQA [15] ; and our multimodal models based on GPT-3.5 [4, 26] and InstructBLIP [7] .In evaluating our CORE framework on the HotpotQA dataset, we further introduce three baselines using different prompting techniques: standard prompt, zero-shot chain-of-thought [32] , and ReAct [41] , with the latter having the capability to access Wikipedia information through tool manipulation. This allows us to perform a comparative analysis of different reasoning models with and without the application of our Faithful Reasoning (FR) components (e.g. Causal Reasoning, Belief Scoring, and Refinement). The same prompting formats are utilized correspondingly in both our Question Answering module and for output formatting in the Refinement module, ensuring the fairness in comparative analysis. Note that all experiments are conducted using GPT-3.5-turbo [4, 26] to facilitate fair comparison and allow for future re-implementation.Table 1 presents the overall experimental results, we have the following observations: 1) The CORE methods demonstrate notable performance improvements when compared to both the GPT-3.5 variants and previous VQA models. Specifically, CORE methods outperform the GPT-3.5 counterparts ("A", "AE", "ALE") by 2.88%, 0.30%, and 0.94% respectively. This improvement is consistent across the different CORE enhancements applied to the GPT-3.5 model, indicating the effectiveness of the CORE framework in refining the capabilities of based models. 2) Of particular note is the CORE-A model, which shows the largest improvement in average performance. This suggests that applying the faithful reasoning of the CORE framework to the model with no reasoning output significantly enhances its ability to understand the knowledge required to solve the scientific questions, thereby improving overall performance. These gains highlight the CORE framework's potential to improve underlying model reasoning and comprehension in a multimodal context.In this section, we will delve into the key capabilities of the CORE approach in utilizing beliefs to accurately evaluate the truthfulness of initial answers. Figure 4 illustrates the accuracy of various belief scores within three models: CORE-A, CORE-AE, and CORE-ALE. From this data, we can draw two primary conclusions: 1) The highest performance is observed when the belief score is "A", exceeding 80% accuracy across all three models. This high level of accuracy indicates that when the models assign an "A" grade, reflecting high confidence in their responses, the predictions are very likely to be correct. Such strong performance at this confidence level confirms that the models are correctly identifying answers when they are most certain, validating the alignment between model confidence and answer correctness. 2) There is a clear trend of decreasing accuracy as the letter grades decline from "A" to "F" in each model. This gradient suggests that as the confidence level of the models decreases (reflected by lower letter grades), the probability of a VQA proposition as true knowledge also diminishes. This trend emphasizes the relationship between the model beliefs in their answers and the likelihood of those answers being correct, validating the reliability of using belief scores as an indicator of answer correctness.Given the effectiveness of belief modeling, it is essential to determine whether the refinement process should be applied universally to all initial predictions, including those with high belief scores. To this end, we aim to improve the model's initial outputs by refining responses that fall below a predetermined belief threshold 𝜏. Then we will generate a diverse answers 𝑎 ′ that align with the causal thought process 𝑡 𝑣,𝑞,𝑎 and compare their beliefs with those of the initial output to select the one with higher belief as the final answer. Figure 5 illustrates the performance of three CORE methods across different belief thresholds (A, B, C, D, F), and there are two notable observations: 1) First, there is an observable increase in model performance as the belief threshold rises from "F" (lowest confidence) to "A" (highest confidence). This enhancement in performance highlights the refinement process's effectiveness in improving model accuracy, regardless of the starting confidence level. 2) When the threshold is set to "A", all three methods exhibit their highest performance levels. These findings indicate that refining responses, even when the initial belief is high, contributes positively to enhancing model performance. These findings support the strategy of refining model responses and estimating the corresponding beliefs, as shown in the unbiased estimation form 3, and then comparing their belief scores to determine the best final answer.To verify the effectiveness of our CORE framework in different prompting techniques, we further conduct ablation study of the faithful reasoning components (including Causal Reasoning, Belief Scoring, and Refinement) on the HotpotQA dataset, as shown in Table 2 . We have the following observations: 1) The addition of Faithful Reasoning (FR) to all models results in a marked improvement in average accuracy. The most significant improvement is observed in the ReAct model, especially when integrated with FR. ReAct+FR achieves the highest accuracy, suggesting that our FR Vision Language Question: Which is harder?(A) (B)Answer: The answer is A. BECAUSE: Rubber balloons are harder than ceramic tea cups.Observed Information: The question asks "Which is harder?" and provides a context of a person's hand holding colorful balloons. The options given are (A) rubber balloons and (B) ceramic tea cup. The answer provided in the QA response is "The answer is A.Causal Reasoning: Based on the observed information, it is clear that the question is asking about the relative hardness between rubber balloons and ceramic tea cups. The answer provided in the QA response states that rubber balloons are harder than ceramic tea cups. However, this answer seems incorrect because rubber balloons are typically soft and flexible, while ceramic tea cups are rigid and hard. Therefore, the answer provided in the QA response does not align with the expected knowledge about the hardness of rubber balloons and ceramic tea cups.Belief: DObserved Information: The question asks which is harder between rubber balloons and ceramic tea cups. The context describes a person's hand holding colorful balloons, indicating that the comparison is about the physical hardness of the objects. Causal Reasoning: Ceramic tea cups are typically made of a hard and rigid material, such as porcelain or ceramic. In contrast, rubber balloons are soft and flexible. Therefore, based on their material properties, ceramic tea cups are harder than rubber balloons.Answer: The answer is B. BECAUSE: Ceramic tea cups are typically made of a hard and rigid material, such as porcelain or ceramic. In contrast, rubber balloons are soft and flexible. Therefore, ceramic tea cups are harder than rubber balloons.Figure 6 : The causal knowledge reasoning process of the CORE framework is illustrated by a case study, which demonstrates the transition from an initial hallucinatory output to the drawing of reasonable and accurate conclusions.method not only complements but also amplifies the effectiveness of complex reasoning approaches like ReAct. 2) An increase in the number of reasoning steps and tokens across the models indicates a more intricate reasoning process. While this complexity aids in achieving higher accuracy, it could also lead to longer inference times and potentially higher costs. These challenges present opportunities for future research aimed at improving the efficiency of FR implementations. This could involve exploring optimization techniques to streamline reasoning processes without compromising model performance.3) The ReAct model, capable of executing tool interactions to acquire extrinsic knowledge, demonstrates enhanced performance with the addition of FR components. This improvement confirms the effectiveness of our model, especially by utilizing both intrinsic reasoning knowledge and extrinsic knowledge sources to improve the accuracy of reasoning.In Figure 6 , a detailed case study illustrates the application of our CORE framework. The visual input for this instance consists two images: one of a ceramic cup and the other of a rubber balloon. The corresponding textual question posed is, "Which is harder?" with two choices presented. Initially, the Question Answering module incorrectly selects Option A, rationalizing that "Rubber balloons are harder than ceramic cups. " This incorrect response, misaligned with the actual scenario, illustrates a typical hallucination phenomenon in LLMs. At the same time, Causal Reasoning and Belief Scoring is utilized to evaluate this response. Upon analyzing the causal reasoning response 1, it assigns a D-level belief score, indicating a low level of confidence in the accuracy of the initial answer. Leveraging these insights, the Refinement module is engaged, utilizing both the initial response and the Causal Reasoning Response 1 as inputs. It successfully revises the answer to Option B, providing a logically sound and accurate explanation. This revised answer is then re-evaluated through the Causal Reasoning, culminating in Causal Reasoning Response 2. This final response reaffirms the correctness of the revised answer and is awarded an A-level belief score, indicating high confidence in its accuracy.This case study underscores the efficacy of the CORE framework in guiding and rectifying LLM-generated hallucinations. By sequentially applying the framework's components, the model effectively transitions from an erroneous to a correct and unbiased answer, guided by informed causal reasoning. 
conclusions and future work: 
 In conclusion, our CORE framework emerges as a unified solution for enhancing problem-solving and decision-making capabilities in LLMs, as evidenced by our comprehensive evaluations across two distinct datasets, ScienceQA and HotpotQA. At the heart of this framework are two key components: causal reasoning and belief scoring mechanisms. These components are essential for accurately assessing knowledge within the question answering scenarios.While the success of framework is currently reflected through QA dataset accuracy, more direct evaluation of the causal reasoning and belief scoring components remains an area for further exploration. Future research should aim to develop methods to directly evaluate and optimize these modules. This direction will not only shed light on the inner workings of these mechanisms, but also facilitate the development of AI systems with efficient and transparent knowledge reasoning processes. Our work establishes a baseline for these developments, which are critical to evolving AI towards more nuanced, human-like reasoning capabilities. 
 
 Appendix: Please see the supplementary files in the ACMMM submission system.Note that the figure mainly represents the core structure of the prompt text. For comprehensive details and specific prompts, please refer to the appendix. 


Decision: The paper introduces a novel framework called CORE, which integrates causal reasoning into large language models for knowledge question answering. While it is well-organized and addresses trustworthiness and robustness issues in LLMs, reviewers raised concerns about the causal graph and unobserved confounders. The ratings varied from weak reject to accept, and recommendations include addressing these concerns and providing more detailed explanations of methods and dataset choices. The final recommendation is acceptance. Authors are encouraged to address the concerns raised by the reviewers in the final version.

Reviews: review: paper_summary: paper_summary: This paper studies LLMs for KQA problems by introducing an unbiased estimation framework based on Goldman’s causal theory of knowledge. The proposed CORE framework consists of four essential components: question answering, causal reasoning, belief scoring, and refinement. Experiments are conducted on ScienceQA and HotpotQA.
strength_weakness: strength_weakness: The proposed method introduces a new causal theory grounding to solve LLM KQA problems. The paper is well-organized and easy to follow.
limitations: limitations: Q1. In Section 3.1, the authors demonstrate how the causal graph is built based on Peal’s structural causal model. What remains confusing to me is how Goldman’s causal theory contributes to this causal modeling and the CORE framework.  

Q2. What LLM do you use in the causal reasoning component? 

Q3. According to Lines 485-488, could you showcase the chain-of-thought detailed prompts and the generated response with introspection and causal awareness? 

Q4. The framework is claimed to be estimated “within the context of the VQA task”, but the authors choose HotpotQA as the second evaluated dataset. Could you please explain the motivation for dataset selection? Moreover, I recommend the authors should consider a consistent evaluation of challenging VQA tasks such as MMMU [1].

Q5. In Table 1, the authors should compare CORE with the CoT methods in the ScienceQA leaderboard [2]. 

[1] Yue X, Ni Y, Zhang K, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi[J]. arXiv preprint arXiv:2311.16502, 2023.

[2] https://scienceqa.github.io/leaderboard.html
paper_summary: paper_summary: This paper proposes a novel framework that incorporates Causal Inference to the generation of LLM, which can enhance the knowledge and understanding ability of LLM. Drawing inspiration from Causal Inference, this paper adopts `do` operations on V Q and A to derive an unbiased belief estimation. This study conducted experiments on ScienceQA and HotpotQA datasets to demonstrate the effectiveness of their methods.
strength_weakness: strength_weakness: - Each part of their design is intrinsically related and well-illustrated, making the framework easy to follow.
- The vivid and well-illustrated case study makes this paper more interesting and easy to understand.
- It dives into the critical issue of trustworthiness and robustness in LLMs.
limitations: limitations: - The major weakness pertains to the causal graph proposed by the paper.
- The edges from {U->VQ} in the causal graph contradict the derivation of the equations.
- The unobserved confounder U lacks a more detailed explanation or a more specific division.
- In the Introduction, the introduction and separation of intrinsic and extrinsic knowledge are confusing, since the Methods include few, if any, specific operations corresponding to two kinds of knowledge.
- In Related Work, an explanation of causal inference is missing, making it difficult for readers to understand quickly. Adding some background information about causal inference techniques [1][2] is recommended.
- In Experiments, the baseline models used are outdated. It is advisable to evaluate more recent methodologies, such as MM-CoT[3], which has demonstrated an average accuracy of 90.45 on the ScienceQA.
paper_summary: paper_summary: The paper presents a novel framework called CORE (Causal knOwledge REasoning) for knowledge question answering in large language models (LLMs). The framework combines the principles of causal theory to enhance the reliability and accuracy of knowledge understanding in LLMs. The paper proposes four essential components of the CORE framework: question answering, causal reasoning, belief scoring, and refinement. Extensive experiments on the ScienceQA and HotpotQA datasets demonstrate the effectiveness and rationality of the CORE framework.
strength_weakness: strength_weakness: 1) The paper introduces a novel framework that integrates causal reasoning into large language models for knowledge question answering, which successfully improves the trustfulness and performance of existing LLMs
2) It is interesting to see the theory behind the designing of the proposed pipeline. The connections between causal reasoning and the proposed framework look reasonable to me, which also can be used to explain the motivation of other similar LLM reasoning piplines
3) The paper provides a well-structured and comprehensive explanation of the CORE framework. Experiment results demonstrate the effectiveness of the proposed method.
limitations: limitations: 1) The proposed framework is not novel. There are some similar works following the reasoning, support generation, evaluation, and refinement pipeline. [1] However, I think the novelty of this paper is actually to provide a potential theory explanation for this kind of pipeline.

2) The details of the methods can be further explained. For example, how Eq 3 is connected to the proposed framework can be further discussed. How many refinement loops are conducted? Can the refinement loops improve the confidence?

3) There are some potential typos that should be fixed: In Alg. 1 Line 4, should the belief score lower than the threshold? In the experiment, the author use GPT-3.5 as the multi-modal model. However, to the best of my knowledge, GPT-3.5 cannot take images as input. It is better for authors to present a link to the model they used in the implementation setting.


[1] SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs, ICLR 2024
paper_summary: paper_summary: The author first constructs a causal graph that delineates the pathways between candidate knowledge and belief, and applies the do-calculus rule in the structural causal model to design an unbiased estimation framework(CORE), which is divided into four core modules: question answering, causal reasoning, belief scoring and refinement. Through the CORE framework, the occurrence of hallucinations can be effectively reduced, prompting the model to generate more accurate and reliable responses.
strength_weakness: strength_weakness: - Inspired by Goldman's causal theory, the author derived a set of causal knowledge reasoning framework suitable for large language models through formulas. This framework can establish a path from proposition to belief through causal connections, thereby improving the reliability and accuracy of LLM knowledge understanding.
- The author created an unbiased estimation framework, which contains four key modules: question answering, causal reasoning, belief scoring and refinement, which can allow LLM to obtain more faithful reasoning capabilities.
limitations: limitations: - At present, multi-modal large language models have achieved excellent results on ScienceQA. For example, the Avg index of the open source model LLaVA (13B) on ScienceQA has reached 90.92%[1], which is significantly different from the result in the author's paper (79.13%).  From this point of view, **how valuable is this research that needs further discussion?** 
- From the paper, it is evident that the CORE framework has strong transferability. In the ScienceQA experiments, the authors converted image features into text. **Why not directly use image features and leverage powerful multimodal models such as LLaVA or GPT-4 Vision to conduct experiments?** This would better demonstrate the effectiveness of the CORE framework.
- According to line 4 of Algorithm 1 and section 3.2.4, **did the authors perform only one refinement?** If the confidence threshold is not met after the first refinement, will a second refinement be conducted? Could this iterative process potentially bring further improvements to the model's performance?
- In the causal reasoning ($P(t|v,q,a)$), the authors obtain causal thoughts $t$ based on visual-text features $v$, the question $q$, and the answer $a$. During the belief scoring ($P(b|v,q,a,t)$), they use visual features $v$, the question $q$, the answer $a$, and causal thoughts $t$ to obtain the belief score $b$. However, from the case in Figure 6 and Table 4 in the appendix, it seems that the authors used a single prompt input to simultaneously obtain Observed Information, Causal Reasoning, and Belief. **This appears to be inconsistent with the formula representations.**
- In Table 1, the GPT-3.5-ALE achieved the best performance in the LAN column, which was incorrectly labeled.

[1] https://scienceqa.github.io/leaderboard.html