Inference to the Best Explanation in Large Language Models (2024)

Abstract: While Large Language Models (LLMs) have found success in real-world applications, their underlying explanatory process is still poorly understood. This paper proposes IBE-Eval, a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLM explanations. IBE-Eval estimates the plausibility of natural language explanations through a combination of explicit logical and linguistic features including: consistency, parsimony, coherence, and uncertainty. Extensive experiments are conducted on Causal Question Answering (CQA), where IBE-Eval is tasked to select the most plausible causal explanation amongst competing ones generated by the LLM (e.g. GPT 3.5 or LLaMA 2). The experiments reveal that IBE-Eval can successfully identify the best explanation with up to 77% accuracy (≈ 27% above random), improving upon a GPT 3.5-as-a-judge baseline (≈ +17%) while being intrinsically more efficient and interpretable. Additional analysis suggests that, despite LLM-specific variances, generated explanations tend to conform to IBE criteria and that IBE-Eval is significantly correlated with human judgment, opening up opportunities for future development of automated explanation verification tools.

heading: 1 Introduction
text: Large Language Models (LLMs) such as OpenAI’s GPT (Brown et al., 2020) and LLaMA (Touvron et al., 2023) have been highly effective across a diverse range of language understanding and reasoning tasks (Liang et al., 2023). While LLM performances have been thoroughly investigated across various benchmarks (Wang et al., 2019; Srivastava et al., 2023; Gao et al., 2023; Touvron et al., 2023), the principles and properties behind their step-wise reasoning process are still poorly understood (Valentino et al., 2021). LLMs are no-
toriously black-box and can be difficult to interpret (Chakraborty et al., 2017; Danilevsky et al., 2020). Moreover, the commercialization of LLMs has led to strategic secrecy around model architectures and training details (Xiang, 2023; Knight, 2023). Finally, LLMs are susceptible to hallucinations and adversarial perturbations (Geirhos et al., 2020; Camburu et al., 2020), often producing plausible but factually incorrect answers (Ji et al., 2023; Huang et al., 2023). As the size and complexity of LLM architectures increase, the systematic study of generated explanations becomes crucial to better interpret and validate the LLM’s internal inference and reasoning processes (Wei et al., 2022b; Lampinen et al., 2022; Huang and Chang, 2022).
The automatic evaluation of natural language explanations presents several challenges (Atanasova et al., 2023; Camburu et al., 2020). Without resource-intensive annotation (Wiegreffe and Marasovic, 2021; Thayaparan et al., 2020; Dalvi et al., 2021; Camburu et al., 2018), explanation quality methods tend to rely on either weak supervision, where the identification of the correct answer is taken as evidence of explanation quality, or require the injection of domain-specific knowledge (Quan et al., 2024). In this paper, we seek to better understand the LLM explanatory process through the investigation of explicit linguistic and logical properties. While explanations are hard to formalize due to their open-ended nature, we hypothesize that they can be analyzed as linguistic objects, with measurable features that can serve to define criteria for assessing their quality.
Specifically, this paper investigates the following overarching research question: “Can the linguistic and logical properties associated with LLMgenerated explanations be used to qualify the models’ reasoning process?”. To this end, we propose an interpretable framework inspired by philosophical accounts of abductive inference, also known as Inference to the Best Explanation (IBE) - i.e. the
217
process of selecting among competing explanatory theories (Lipton, 2017). In particular, we aim to measure the extent to which LLM-generated explanations conform to IBE expectations when attempting to identify the most plausible explanation. To this end, we present IBE-Eval, a framework designed to estimate the plausibility of natural language explanations through a set of explicit logical and linguistic features, namely: logical consistency, parsimony, coherence, and linguistic uncertainty.
To evaluate the efficacy of IBE-Eval, we conduct extensive experiments in the multiple-choice Causal Question Answering (CQA) setting. The overall results and contributions of the paper can be summarized as follows:
1. To the best of our knowledge, we are the first to propose an interpretable framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to automatically assess the quality of natural language explanations.
2. We propose IBE-Eval, a framework that can be instantiated with external tools for the automatic evaluation of LLM-generated explanations and the identification of the best explanation in a multiple-choice CQA setting.
3. We provide empirical evidence that LLMgenerated explanations tend to conform to IBE expectations with varying levels of statistical significance correlated to the LLM’s size.
4. We additionally find that uncertainty, parsimony, and coherence are the best predictors of plausibility and explanation quality across all LLMs. However, we also find that the LLMs tend to be strong rationalizers and can produce logically consistent explanations even for less plausible candidates, making the consistency metric less effective in practice.
5. IBE-Eval can successfully identify the best explanation supporting the correct answers with up to 77% accuracy (+≈ 27% above random and +≈ 17% over GPT 3.5-as-a-Judge baselines)
6. IBE-Eval is significantly correlated with human judgment, outperforming a GPT3.5-asa-Judge baseline in terms of alignment with human preferences.
For reproducibility, our code is made available on Github1 to encourage future research in the field.

heading: 2 Inference to the Best Explanation (IBE)
text: Explanatory reasoning is a distinctive feature of human rationality underpinning problem-solving and knowledge creation in both science and everyday scenarios (Lombrozo, 2012; Deutsch, 2011). Accepted epistemological accounts characterize the creation of an explanation as composed of two distinct phases: conjecturing and criticism (Popper,
1https://github.com/dhairyadalal/ IBE-eval
2014). The explanatory process always involves a conflict between plausible explanations, which is typically resolved through the criticism phase via a selection process, where competing explanations are assessed according to a set of criteria such as parsimony, coherence, unification power, and hardness to variation (Lipton, 2017; Harman, 1965; Mackonis, 2013; Thagard, 1978, 1989; Kitcher, 1989; Valentino and Freitas, 2022).
As LLMs become interfaces for natural language explanations, epistemological frameworks offer an opportunity for developing criticism mechanisms to understand the explanatory process underlying state-of-the-art models. To this end, this paper considers an LLM as a conjecture device producing linguistic objects that can be subject to criticism. In particular, we focus on a subset of criteria that can be computed on explicit linguistic and logical features, namely: consistency, parsimony, coherence, and uncertainty.
To assess the LLM’s alignment to such criteria, we focus on the task of selecting among competing explanations in a multiple-choice CQA setting (Figure 1). Specifically, given a set of competing hypotheses (i.e. the multiple-choice options), H = {h1, h2, . . . , hn}, we prompt the LLM to generate plausible explanations supporting each hypothesis (Section 3). Subsequently, we adopt the proposed IBE selection criteria to assess the quality of the generated explanations (Section 4). IBE-Eval computes an explanation plausibility score derived from the linear combination of the computed selection criteria. The explanation with the highest score is selected as the predicted answer and additionally assessed as to the extent to which the observable IBE features are correlated with QA accuracy. We hypothesize that IBE-Eval will produce higher scores for the explanation associated with the correct answer and that the IBE criteria should meaningfully differentiate between competing explanations.

heading: 3 Explanation Generation
text: For the first stage, the LLM is prompted to generate competing explanations for the hypotheses using a modified Chain-of-Thought (CoT) prompt (Wei et al., 2022a). Specifically, the COT prompt is modified to instruct the LLM to produce an explanation for each competing hypothesis (see Figure 1). We adopt a methodology similar to Valentino et al. (2021), where the generated explanation is
constrained into an entailment form for the downstream IBE evaluation. In particular, we posit that a valid explanation should demonstrate an entailment relationship between the premise and conclusion which are derived from the question-answer pair.
To elicit logical connections between explanation steps and facilitate subsequent analysis, the LLM is constrained to use weak syllogisms expressed as If-Then statements. Additionally, the LLM is instructed to produce the associated causal or commonsense assumption underlying each explanation step. This output is then post-processed to extract the explanation steps and supporting knowledge for evaluation via the IBE selection criteria. Additional details and examples of prompts are reported in Appendix A.2.

heading: 4 Linguistic & Inference Criteria
text: To perform IBE, we investigate a set of criteria that can be automatically computed on explicit logical and linguistic features, namely: consistency, parsimony, coherence, and uncertainty.
Consistency. Consistency aims to verify whether the explanation is logically valid. Given a hypothesis, comprised of a premise pi, a conclusion ci, and an explanation consisting of a set of If-Then statements E = s1, ..., si, we define E to be logically consistent if pi ∪E ⊨ ci. Specifically, an explanation is logically consistent if it is possible to build a deductive proof linking premise and conclusion.
To evaluate logical consistency, we leverage external symbolic solvers along with autoformalization - i.e., the translation of natural language into a formal language (Wu et al., 2022). Specifically, the hypotheses and explanations are formalized into a Prolog program which will attempt to generate a deductive proof via backward chaining (Weber et al., 2019).
To perform autoformalization, we leverage the translation capabilities of GPT 3.5. Specifically, we instruct GPT 3.5 to convert each IF-Then explanation step from the generated explanation into an implication rule and the premise statement into grounding atoms. On the other end, the entailment condition and the conclusion are used to create a Prolog query. The query instructs the Prolog solver to attempt to find a path through the implications rules such that the conclusion be directly connected to the premise. Further details about the autoformalization process can be found in Appendix A.3.
After autoformalization, following recent work on neuro-symbolic integration for LLM explanations (Quan et al., 2024), we adopt an external Prolog solver for entailment verification2. The explanation is considered consistent if the Prolog solver can satisfy the query and successfully build a deductive proof. Technical details can be found in Appendix A.5.
Parsimony. The parsimony principle, also known as Ockham’s razor, favors the selection of the simplest explanation consisting of the fewest elements and assumptions (Sober, 1981). Epistemological accounts posit that an explanation with fewer assumptions tends to leave fewer statements unexplained, improving specificity and alleviating the infinite regress (Thagard, 1978). Further, parsimony is an essential feature of causal interpretability, as only parsimonious solutions are guaranteed to reflect causation in comparative analysis (Baumgartner, 2015). In this paper, we adopt two metrics as a proxy of parsimony, namely proof depth, and concept drift. Proof depth, denoted as Depth, is defined as the cardinality of the set of rules, R, required by the Prolog solver to connect the conclusion to the premise via backward chaining. Let h be a hypothesis candidate composed of a premise p and a conclusion c, and let E be a formalized explanation represented as a set of rules R′. The proof depth is the number of rules |R|, with R ⊆ R′, traversed during backward chaining to connect the conclusion c to the premise p:
Depth(h) = |R|
Concept drift, denoted as Drift, is defined as the number of additional concepts and entities, outside the ones appearing in the hypothesis (i.e., premise and conclusion), that are introduced by the LLM to support the entailment. For simplicity, we consider nouns as concepts. Let N = {Nounp, Nounc, NounE} be the unique nouns found in the premise, conclusion, and explanation steps. Concept drift is the cardinality of the set difference between the nouns found in the explanation and the nouns in the hypothesis:
Drift(h) = |NounE − (Nounp ∪Nounc)|
Intuitively, the parsimony principle would predict the most plausible hypothesis as the one supported
2https://github.com/neuro-symbolic-ai/ explanation_based_ethical_reasoning
by an explanation with the smallest observed proof depth and concept drift. Implementation details can be found in Appendix A.6.
Coherence. Coherence attempts to measure the logical validity at the level of the specific explanation steps. An explanation can be formally consistent on the surface while still including implausible or ungrounded intermediate assumptions. Coherence evaluates the quality of each intermediate If-Then implication by measuring the entailment strength between the If and Then clauses. To this end, we employ a fine-tuned natural language inference (NLI) model. Formally, let S be a set of explanation steps, where each step s consists of an If-Then statement, s = (Ifs, Thens). For a given step si, let ES(si) denote the entailment score obtained via the NLI model between Ifs and Thens clauses. The step-wise entailment score SWE(S) is then calculated as the averaged sum of the entailment scores across all explanation steps |S|:
SWE(S) = 1
|S|
|S|∑
i=1
ES(si)
We hypothesize that the LLM should generate a higher coherence score for more plausible hypotheses, as such explanations should exhibit stronger step-wise entailment. Additional details can be found in Appendix A.7.
Uncertainty. Finally, we consider the linguistic certainty expressed in the generated explanation as a proxy for plausibility. Hedging words such as probably, might be, could be, etc typically signal ambiguity and are often used when the truth condition of a statement is unknown or improbable. Pei and Jurgens (2021) found that the strength of scientific claims in research papers is strongly correlated with the use of direct language. In contrast, they found that the use of hedging language suggested that the veracity of the claim was weaker or highly contextualized.
To measure the linguistic uncertainty (UC) of an explanation, we consider the explanation’s underlying assumptions (Ai) and the overall explanation summary (S). The linguistic uncertainty score is extracted using the fine-tuned sentencelevel RoBERTa model from Pei and Jurgens (2021). The overall linguistic uncertainty score (UCoverall) is the sum of the assumption and explanation summary scores:
UCoverall = UC(A) + UC(S)
Where UC(A) is the sum of the linguistic uncertainty scores (UC(A)) across all the assumptions |A| associated with each explanation step i:
UC(A) =
|A|∑
i=1
UC(ai)
and linguistic uncertainty of the explanation summary UC(S). We hypothesize that the LLM will use more hedging language when explaining the weaker hypothesis resulting in a higher uncertainty score. Further details can be found in Appendix A.8.

heading: 4.1 Inference to Best Explanation
text: After the IBE criteria are computed for each competing hypothesis, they are used to generate the final explanation plausibility score. We define a simple linear regression model θ(·), which was fitted on a small set of training examples consisting of extracted IBE features to predict the probability that an explanation Ei corresponds to the correct answer. Specifically, we employ IBE-Eval to score each generated explanation independently and then select the final answer a via argmax:
a = argmax i [θ(E1), . . . , θ(En)]
Additional details can be found in Appendix A.9.

heading: 5 Experimental Setting
text: Causal Question-Answering (CQA) requires reasoning about the causes and effects given an event description. We specifically consider the task of cause and effect prediction in a multiple-choice setting, where given a question and two candidate answers, the LLM must decide which is the most plausible cause or effect. Causal reasoning is a challenging task as the model must both possess commonsense knowledge about causal relationships and consider the event context which would make one option more plausible than the other. For our experiments, we use the Choice of Plausible Alternatives (COPA) (Gordon et al., 2012) and E-CARE (Du et al., 2022) datasets.
COPA. COPA is a multiple-choice commonsense causal QA dataset consisting of 500 train and test examples that were manually generated. Each multiple-choice example consists of a question
premise and a set of answer candidates which are potential causes or effects of the premise. COPA is a well-established causal reasoning benchmark that is both a part of SuperGlue (Wang et al., 2019) and the CALM-Bench (Dalal et al., 2023).
E-CARE. E-CARE is a large-scale multiplechoice causal crowd-sourced QA dataset consisting of 15K train and 2k test examples. Similar to COPA, the task requires the selection of the most likely cause or effect provided an event description. We randomly sample 500 examples from the E-CARE test set for our experiments.
LLMs. We consider GPT-Turbo-3.5, LLaMA 2 13B, and LLaMA 2 7B for all experiments. GPT 3.5 is a proprietary model (Brown et al., 2020) and is highly effective across a wide range of natural language reasoning tasks (Laskar et al., 2023). We additionally evaluate the open-source LLaMA 2 model (Touvron et al., 2023). We consider both the 13B and 7B variants of Llama 2 as both are seen as viable commodity GPT alternatives and have been widely adopted by the research community for LLM benchmarking and evaluation.
Baselines. We employ LLM-as-a-Judge (Zheng et al., 2023) and human evaluators as baseline methods for the selection of the best explanation in the CQA setting. (Zheng et al., 2023) found LLMs can align with human judgment and be utilized for automated evaluation and judgment. We specifically uses GPT 3.5 as the LLM judge. For each CQA example, we present the judges with two competing explanations generated by the target LLM. The judge is asked to identify the best and most plausible explanation. Additional details about the baselines can be found in Appendix A.4.

heading: 6 Preliminary Analysis
text: We conduct a preliminary analysis as a sanity check to measure the extent to which LLMs generate selfevident or tautological explanations - i.e., explanations that simply restate the premises and conclusions. Tautological explanations present a risk for IBE-Eval as the metrics would be theoretically uninformative if the LLM adopts the tested causal relation as the explanation itself (e.g. A → B) without providing additional supporting statements.
We consider the parsimony metric to compute the percentage of explanations with proof depth equal to 1 (i.e, explanations containing only one inference step) and concept drift equal to 0 (i.e. no
additional concepts other than the ones stated in premises and conclusions appear in the explanation). In such cases, the LLM is effectively generating a self-evident or tautological explanation.
We found that about 2% of the cases consist of self-evident explanations. For GPT 3.5, LLaMA 2 13B, and LLaMA 2 7B, 2% of the generated explanations exhibit a concept drift of 0, and on average 1.5% of the explanations have a proof depth of 1. We then conducted an error analysis to evaluate the cases where IBE-Eval selected a self-evident explanation as the best one. Across all LLMs, less than 0.1% of the errors were caused by the selection of such explanations. Our analysis suggests that the impact of self-evident explanations is not sig-
nificant and that the IBE framework can be robustly applied to identify such cases.

heading: 7 Results
text: To assess the LLM’s alignment with the proposed IBE framework and evaluate the efficacy of IBEEval, we run a regression analysis and conduct a set of ablation studies to evaluate the relationship between IBE and question accuracy. The main results are presented in Figure 2 and Table 1.
Our regression analysis finds that the IBE criteria are generally consistent across the LLMs as demonstrated by similar correlation patterns found on both the COPA and E-CARE tasks (Figure 2). GPT 3.5 exhibits the strongest alignment with IBE
expectations as we observe nearly all the IBE criteria have statistically significant and directionally aligned correlations across both tasks. Thus our proposed IBE criteria can serve as promising build blocks for future work on automated explanation evaluation.
In Table 1 we evaluate the accuracy of the IBE criteria and IBE-Eval in selecting the most plausible explanation in the CQA setting. We find that though independently the IBE criteria are generally limited in their ability to identify the more plausible explanation - they still outperform the GPT-3.5-asa-judge baseline. IBE-Eval, which considers all IBE criteria, improves the ability to select the best explanation by 17% over both the GPT 3.5-as-a-judge and random baselines. We can achieve up to 77% accuracy utilizing just the extracted IBE criteria demonstrating IBE’s potential value for automatic explanation evaluation.
Next, we explore each explanation feature in further detail to better understand the variances across the IBE criteria and LLMs.
Consistency. We find that the LLMs are surprisingly strong conjecture models. The LLMs can generate logically consistent explanations for any hypothesis as observed by similar consistency scores for correct and incorrect (Figure 3) explanations. Moreover, we observe that consistency tends to be a statistically insignificant predictor for the LLaMA models. Therefore, we conclude that evidence of logical consistency provides a limited signal for plausibility and is better understood in the context of other IBE criteria. For the incorrect candidate explanations, we find that LLMs overrationalize and introduce additional premises to demonstrate entailment in their explanations.
Parsimony. The results suggest that parsimony has a more consistent effect and is a better predictor of explanation quality. We observe negative correlations between proof depth, concept drift, and question-answering accuracy, suggesting that LLMs tend to introduce more concepts and explanation steps when explaining less plausible hypotheses. On average, we found the depth and drift to be 6% and 10% greater for the incorrect option across all LLMs (Figure 4). Moreover, the results suggest that as the LLM parameter size increases, the tendency to over-rationalize increases as well. This is attested by the fact that the average difference in depth and drift is the great-
est in GPT 3.5, suggesting that the model tends to find the most efficient explanations for stronger hypotheses and articulates explanations for weaker candidates. Finally, we found that the LLaMA models tend to generate more complex explanations overall, with LLaMA 2 13B exhibiting the largest concept drift for less plausible hypotheses. The parsimony criterion supports the IBE predictive power with an average of 14% improvement over consistency.
Coherence. Similarly to parsimony, we found coherence to be a better indicator of explanation quality being statistically significant for both GPT 3.5 and Llama 2 13B on COPA and both Llama 2 models on E-Care. We found that the average coherence score is consistently greater for the
stronger hypothesis across all LLMs and datasets (see Figure 5). Both GPT and Llama 2 13B exhibit a higher relative difference between the correct and incorrect hypotheses in contrast to Llama 2 7B.
Uncertainty. The results reveal that linguistic uncertainty is the strongest predictor of explanation quality and is a statistically significant feature for all LLMs. This suggests that LLMs use more qualifying language when explaining weaker hypotheses (see Figure 6). We found that uncertainty can improve accuracy by 13pp on COPA and 4pp on E-CARE. We also examine the uncertainty cues expressed by LLMs by analyzing both the frequency of hedge words and the types of hedge cues employed in incorrect explanations. We find the distribution of hedge cues across LLMs tends to be similar, with only minor differences be-
tween LLMs (Figure 6). Epistemic cues were most frequently used by all three models, with LLaMA 2 7B being more likely to use conditional cues. See Appendix A.8 for further details..

heading: 7.1 Correlation with Human Judgement.
text: We first sample 100 generated explanation pairs across both the COPA and E-CARE tasks and evaluated LLMs. Two human evaluators are instructed to evaluate the pair of explanations and to select which explanation is most plausible. No additional information about the original question nor the correct answer is provided to prevent biasing the judge.
The human evaluators on average were able to identify the explanation associated with the correct answer 96% (COPA) and 91% (E-Care) of time. We compute the inter-evaluator agreement score between two human evaluators and find that there is Cohen’s Kappa score of .68 suggesting there is a strong agreement between the two evaluators.
To evaluate if IBE-Eval is correlated with human judgement, we compute the Spearman’s rank correlation between GPT-3.5-as-a-judge, IBE-Eval and human judgment. We find that GPT-3.5-as-a-judge exhibits a weak and statistically insignificant correlation with human judgment (0.31). In contrast, we find that the IBE-Eval is significantly aligned with human preferences (with a Spearman’s correlation of 0.64 and p < 0.01) further suggesting the IBE’s potential for automatic explanation evaluation.

heading: 8 Related Work
text: Explorations of LLM reasoning capabilities across various domains (e.g. arithmetic, commonsense, planning, symbolic, etc) are an emerging area of interest (Xu et al., 2023; Huang and Chang, 2023). Prompt-based methods (Wei et al., 2022b; Zhou et al., 2023; Wang et al., 2023), such as CoT, investigate strategies to elicit specific types of reasoning behavior through direct LLM interaction. Olausson et al. (2023) investigate automatic proof generation and propose a neurosymbolic framework with an LLM semantic parser and external solver. Creswell et al. (2022) propose an inference framework where the LLM acts as both a selection and inference module to produce explanations consisting of causal reasoning steps in entailment tasks. Research on LLM faithfulness (Atanasova et al., 2023) investigates if LLM explanations are robust to spurious input alterations. Parcalabescu and Frank (2024) propose
a self-consistency measure CC-SHAP which measures how specific alterations to a model’s input contribute to the generated explanation. This paper primarily draws inspiration from recent work on the evaluation of natural language explanations (Quan et al., 2024; Valentino et al., 2021; Wiegreffe and Marasovic, 2021; Thayaparan et al., 2020; Dalvi et al., 2021; Camburu et al., 2018). However, differently from previous methods that require extensive human annotations or specific domain knowledge, we are the first to propose a set of criteria that can be automatically computed on explicit linguistic and logical features.

heading: 9 Conclusion
text: This paper proposed IBE-Eval, an interpretable framework for LLM explanation evaluation inspired by philosophical accounts of Inference to the Best Explanation (IBE). IBE-Eval can identify the best explanation supporting the correct answer with up to 77% accuracy in CQA scenarios, improving upon a GPT 3.5 Judge baselines by +17%. Our regression study suggests that LLM explanations tend to conform to IBE expectations and that IBEEval is strongly correlated with human judgment. Linguistic uncertainty is the stronger IBE predictor for explanation quality closely followed by parsimony and coherence. However, we also found that LLMs tend to be strong conjecture models able to generate logically consistent explanations for less plausible hypotheses, suggesting limited applicability for the logical consistency criterion in isolation. We believe our findings can open new lines of research on external evaluation methods for LLMs as well as interpretability tools for understanding the LLM’s underlying explanatory process.

heading: 10 Limitations
text: IBE-Eval offers an interpretable explanation evaluation framework utilizing logical and linguistic features. Our current instantiation of the framework is primarily limited in that it does not consider grounded knowledge for factuality. We observe that LLMs can generate factually incorrect but logically consistent explanations. In some cases, the coherence metric can identify those factual errors when the step-wise entailment score is comparatively lower. However, our reliance on aggregated metrics can hide weaker internal entailment especially when the explanation is longer or the entailment strength of the surrounding explanation
steps is stronger. Future work can introduce metrics to evaluate grounded knowledge or perform more granular evaluations of explanations to better weight factual inaccuracies.
Additionally, IBE-Eval currently does not support single natural language explanations and was evaluated in the limited domain of causal commonsense reasoning. Future work will explore globally calibrating IBE-Eval plausibility scores to extend evaluation to more diverse explanation generation and QA settings. Calibration efforts would allow for IBE-Eval to generate comparable scores across unrelated explanations and could be used to produce global thresholds explanation classification.
Finally, the list of criteria considered in this work is not exhaustive and can be extended in future work. However, additional criteria for IBE might not be straightforward to implement (e.g., unification power, hardness to variation) and would probably require further progress in both epistemological accounts and existing NLP technology.

heading: 11 Ethics Statement
text: The human annotators for computing the human judgment baseline are all authors of the papers and as such were not further compensated for the annotation task.

heading: Acknowledgements
text: This work was partially funded by the Swiss National Science Foundation (SNSF) project NeuMath (200021_204617), by the EPSRC grant EP/T026995/1 entitled “EnnCore: End-to-End Conceptual Guarding of Neural Architectures” under Security for all in an AI-enabled society, by the CRUK National Biomarker Centre, and supported by the Manchester Experimental Cancer Medicine Centre, the Science Foundation Ireland under grants SFI/18/CRT/6223 (Centre for Research Training in Artificial Intelligence), SFI/12/RC/2289_P2 (Insight), co-funded by the European Regional Development Fund, and the NIHR Manchester Biomedical Research Centre.

heading: A Appendix

heading: A.1 Reproducibility
text: All experimental code is available online3 to encourage future research in the field. We additionally summarize all the model implementations and technical resources used for the computation of the proposed IBE criteria below:
• We adopt the Prolog solver for neurosymbolic integration from (Quan et al., 2024).
• We use Spacy (Honnibal and Montani, 2017) to tokenize and extract part-of-speech (POS) tags.
• To compute coherence, we employ the RoBERTa-based NLI model (Nie et al., 2020) that has been finetuned on a range of NLI and fact verification datasets consisting of SNLI (Bowman et al., 2015), aNLI (Nie et al., 2020), multilingual NLI (Williams et al., 2018)), and FEVER-NLI (Nie et al., 2019).
• To measure sentence-level uncertainty, we employ a finetuned RoBERTa model provided by (Pei and Jurgens, 2021).
• We use a fine-tuned BERT-based token classification model to classify all the words in the generated explanation with uncertainty categories introduced in the 2010 CoNLL shared task on Hedge Detection (Farkas et al., 2010).

heading: A.2 Explanation Prompting
text: 3https://github.com/dhairyadalal/ IBE-eval
A modified CoT prompt is used to instruct the LLM to generate explanations. The prompt includes a set of instructions for explanation generation and an in-context example. Appended to the end of the prompt are the CQA context, causal question, and answer candidates. The LLM is instructed to first convert the options into the EEV format consisting of a premise and conclusion. The EEV format will differ depending on the directionality of the causal question (see Figure 7). Cause prediction questions will treat the answer candidate as the premise and the context as the conclusion. In contrast, effect prediction reverses the relationship treating the context as the premise and the answer options as the conclusion. After the EEV conversion, the model is instructed to generate a step-by-step explanation consisting of IF-THEN statements and the associated causal or commonsense assumptions. For ease of post-processing, the LLM is instructed to use headers and enumerate steps using the Step # format. A full example of the prompt template is exhibited in Figure 8.

heading: A.3 Autoformalization
text: Autoformalization is the process of translating natural language descriptions into formal specifications (Wu et al., 2022). We adopt the translational capabilities of GPT-3.5-Turbo to convert the expla-
nation into a formal entailment hypothesis. The IF-THEN explanation steps are converted into a set of Prolog rules, the entailment description is used to generate Prolog atoms, and the conclusion statement is translated into a Prolog query. We provide an example of the autoformalization prompt in Figure 9 and an example of the formalized output in Figure 11. After autoformalization, we deploy a post-processing script to extract the formalized rules, atoms, and query and generate a Prolog program for entailment verification.

heading: A.4 LLM-as-a-Judge Baseline
text: GPT 3.5 is used as the LLM for the LLM-as-aJudge baseline. Similar to the human evaluators, GPT is presented with both generated explanations and asked to identify which explanation is more plausible.

heading: A.5 Logical Consistency
text: An explanation hypothesis is considered logically consistent if the external solver can build a deductive proof connecting the conclusion to the premise. We use NLProlog (Weber et al., 2019), a neuro-symbolic Prolog solver integrating backward chaining with word embedding models via a weak unification mechanism. NLProlog allows for a level of flexibility and robustness that is necessary for NLP use cases (e.g. unification applied to synonyms). We provide the autoformalized query, atoms, and rules to NLProlog. If NLProlog can satisfy the entailment query, it will return the proof
consisting of the set of rules traversed, the weak unification score, and the proof depth. For simplicity, we assign a score of one if the entailment query is satisfied and zero if it is not. The proof depth score is evaluated as part of the parsimony analysis. An end-to-end example of consistency evaluation can be found in Figure 11.
Algorithm 1: Neuro-symbolic Solver Input :Symbolic KB kb, Goal goal,
Glove embedding model e(·) Output :proof chain chain, proof depth
depth
1 threshold← 0.13; 2 depth← 1 ; 3 chain← emptylist ; 4 foreach step t in backward_chaining(kb,
goal) do 5 foreach max_unification(q, qt) do 6 unification_score←
CosineSimilarity(e(q,ms), e(qt,ms));
7 depth← depth × unification_score ; 8 end 9 chain← backward_chaining(kb, goal);
10 end
11 if chain is not empty and depth > threshold then 12 chain← current_proof_chain[0]; 13 end 14 else 15 depth← 0 ; 16 end
17 return chain, depth;

heading: A.6 Parsimony
text: Parsimony measures the complexity of an explanation and is represented by the proof depth and concept drift metrics. Proof depth is automatically calculated by NLProlog and reflects the number of rules traversed by the solver to satisfy the entailment query. If the hypothesis is not logically consistent, depth is set to zero. The concept drift metric measures the entropy of novel concepts introduced to bridge the premise and conclusion. To compute the drift of an explanation, we consider the nouns found in the premise, conclusion, and explanation steps. We use Spacy (Honnibal and Mon-
tani, 2017) to tokenize and extract part-of-speech (POS) tags. All tokens with the ’NOUN’ POS tag extracted. For normalization purposes, we consider the lemma of the tokens. Concept drift then is calculated as the set difference between the unique nouns found across all explanation steps and those found in the premise and conclusion.
Algorithm 2: Concept Drift Input :Premise, Conclusion, Explanation,
Spacy model spacy(·) Output :Drift Score drift
1 Nounp ← spacy(Premise); 2 Nounc ← spacy(Conclusion); 3 NounE ← spacy(Explanation); 4 N ← {Nounp, Nounc, NounE}; 5 drift← length(set(NounE)−
set(Nounp ∪Nounc)); 6 return drift;

heading: A.7 Coherence
text: Coherence evaluates the plausibility of the intermediate explanation. We propose stepwise entailment as a metric to measure the entailment strength of the If-then implications. We employ a RoBERTabased NLI model (Nie et al., 2020) that has been finetuned on a range of NLI and fact verification datasets consisting of SNLI (Bowman et al., 2015), aNLI (Nie et al., 2020), multilingual NLI (Williams et al., 2018)), and FEVER-NLI (Nie et al., 2019). To compute the stepwise entailment score, we first measure the entailment strength between the If and Then propositions. For example, to calculate the score of the statement “IF a balloon is pricked, THEN the balloon may deflate” we consider “a balloon is pricked” and “the balloon may deflate” as input sentences for the NLI model. The NLI will produce independent scores for the entailment and contradiction labels. We compute the entailment strength by subtracting the contraction label score from the entailment label score. An entailment strength of one indicates the If-then implication is strongly plausible whereas a score of zero suggests that it is likely implausible. The overall stepwise entailment score is the average of entailment strength measures across all explanation steps.

heading: A.8 Linguistic Uncertainty
text: Linguistic uncertainty measures the confidence of a statement where hedging cues and indirect
Algorithm 3: Stepwise Entailment Input :Explanation E, NLI Model nli(·) Output :Average Entailment Strength
strength
1 EntailmentStrengthScores← empty list;
2 foreach Step (Ifs, Thens) in E do
3 EntailmentScore← nli(Ifs, Thens); 4 ContradictionScore← nli(Ifs, Thens); 5 EntailmentStrength← EntailmentScore− ContradictionScore; 6 Append EntailmentStrength to EntailmentStrengthScores; 7 end
8 strength← Avg(EntailmentStrengthScores);
9 return strength;
language suggest ambiguity around the proposition. To measure sentence-level uncertainty, we employ a finetuned RoBERTa model provided by (Pei and Jurgens, 2021). The model was trained on a sentence-level dataset consisting of findings and statements extracted from new articles and scientific publications and human annotated evaluation of sentence certainty. A scale from one to six was used to annotate sentences where one corresponds to the lowest degree of certainty and six is the highest expressed by the sentence. We invert the scale to retrieve the uncertainty scores. To compute the overall linguistic uncertainty of an explanation, we first compute the uncertainty for each assumption and the explanation summary and then average all the scores.
We use a fine-tuned BERT-based token classification model to classify all the words in the generated explanation with uncertainty categories introduced in the 2010 CoNLL shared task on Hedge Detection (Farkas et al., 2010). Farkas et al. (2010) classify hedge cues into three categories: epistemic, doxatic, and conditional. Epistemic cues refer to hedging scenarios where the truth value of a proposition can be determined but is unknown in the present (e.g. the blocks may fall). Doxatic cues refer to beliefs and hypotheses that can be held to
be true or false by others (e.g. the child believed the blocks would fall). Finally, conditional cues refer to propositions whose truth value is dependent on another proposition’s truth value (e.g. if the balloon is pricked it may deflate).
Algorithm 4: Linguistic Uncertainty Input :Assumptions, Explanation
Summary, Uncertainty Estimator Model uc(·)
Output :Overall Uncertainty
1 AssumptionUncertaintyList← empty list;
2 foreach Assumption in Assumptions do
3 UncertaintyScore← uc(UncertaintyModel, Assumption); 4 Append UncertaintyScore to AssumptionUncertaintyList; 5 end
6 AverageAssumptionUncertainty ← Avg(AssumptionUncertaintyList);
7 ExplanationUncertainty ← uc(UncertaintyModel, ExplanationSummary);
8 OverallExplanationUncertainty ← AverageAssumptionUncertainty + ExplanationUncertainty;
9 return OverallExplanationUncertainty;
A.9 Inference to Best Explanation To perform IBE, we first fit a linear regression model over the extracted explanation features from the COPA train set and 500 random sample train examples from the E-CARE train set. We consider all explanations independently and annotate each explanation with a 1 if it corresponds to a correct answer or 0 if corresponds to an incorrect answer. After the linear model is fitted, we evaluate the COPA and E-CARE test sets. For each example, we use the trained linear model to score each answer candidate explanation and then select a candidate with the highest score. We use the linear regression implementation from scikit-learn (Buitinck et al., 2013) for the IBE model. We additionally use the R stats package (R Core Team, 2013) for conducting our regression analysis.

heading: A.10 E-CARE Results

heading: A.10.1 E-CARE Consistency
text: See Figure 12.

heading: A.10.2 E-CARE Proof Depth
text: See Figure 13.

heading: A.10.3 E-CARE Concept Drift
text: See Figure 13.

heading: A.10.4 E-CARE Coherence
text: See Figure 15.

heading: A.10.5 E-CARE Uncertainty
text: See Figure 16.

heading: A.10.6 E-CARE Hedge Ratio
text: See Figure 17.

heading: A.10.7 E-CARE Hedge Distribution
text: See Figure 18.

heading: A.11 Causal Directionality
text: When considering the causal directionality (i.e. cause vs effect), we observed that accuracy tended to differ between LLMs on COPA. In particular, we found both GPT and LLaMA 2 7B to be more accurate in predicting the effects in causal scenarios (see Figure 19). We hypothesize that LLMs may suffer the challenge of causal sufficiency as the space of potential causal explanations can be far greater than the range of effects once an event has been observed. This hypothesis is partly supported by the fact that GPT and LLaMA 2 7B express greater linguistic uncertainty and produce more complex explanations when predicting causes rather than effects.

heading: A.12 Dataset Details
text: COPA is released under a BSD-2 license and made available for broad research usage with copyright notification restrictions 4. We do not modify or use COPA outside of its intended use which is primarily open-domain commonsense causal reasoning. E-CARE is released under the MIT license and can be used for broad purposes with copyright notification restrictions 5. We do not modify or use E-CARE outside of its intended use which is causal reasoning evaluation of language models.
4people.ict.usc.edu/~gordon/copa.html 5github.com/Waste-Wood/e-CARE?tab=
MIT-1-ov-file#readme

heading: Future Work
text: All experimental code is available online3 to encourage future research in the field. We additionally summarize all the model implementations and technical resources used for the computation of the proposed IBE criteria below:
• We adopt the Prolog solver for neurosymbolic integration from (Quan et al. , 2024). • We use Spacy (Honnibal and Montani, 2017) to tokenize and extract part-of-speech (POS) tags. • To compute coherence, we employ the RoBERTa-based NLI model (Nie et al. , 2020) that has been finetuned on a range of NLI and fact verification datasets consisting of SNLI (Bowman et al. , 2015), aNLI (Nie et al. , 2020), multilingual NLI (Williams et al. , 2018)), and FEVER-NLI (Nie et al. , 2019). • To measure sentence-level uncertainty, we employ a finetuned RoBERTa model provided by (Pei and Jurgens, 2021). • We use a fine-tuned BERT-based token classification model to classify all the words in the generated explanation with uncertainty categories introduced in the 2010 CoNLL shared task on Hedge Detection (Farkas et al. , 2010)