Venue: NAACL 2025

# Evaluating Input Feature Explanations through a Unified Diagnostic

Evaluation Framework

 Jingyi Sun  Pepa Atanasova Isabelle Augenstein

University of Copenhagen

{jisu, pepa, augenstein}@di.ku.dk

###### Abstract

Explaining the decision-making process of machine learning models is crucial for ensuring their reliability and transparency for end users. One popular explanation form highlights key input features, such as i) tokens (e.g., Shapley Values and Integrated Gradients), ii) interactions between tokens (e.g., Bivariate Shapley and Attention-based methods), or iii) interactions between spans of the input (e.g., Louvain Span Interactions). However, these explanation types have only been studied in isolation, making it difficult to judge their respective applicability. To bridge this gap, we develop a unified framework that facilitates an automated and direct comparison between highlight and interactive explanations comprised of four diagnostic properties1. We conduct an extensive analysis across these three types of input feature explanations-each utilizing three different explanation techniques-across two datasets and two models, and reveal that each explanation has distinct strengths across the different diagnostic properties. Nevertheless, interactive span explanations outperform other types of input feature explanations across most diagnostic properties. Despite being relatively understudied, our analysis underscores the need for further research to improve methods generating these explanation types. Additionally, integrating them with other explanation types that perform better in certain characteristics could further enhance their overall effectiveness.

Footnote 1: Our code can be found at [https://github.com/copenlu/A-unified-framework-for-input-feature-explanations](https://github.com/copenlu/A-unified-framework-for-input-feature-explanations)

0
Footnote 2: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1059–10577

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Input feature explanations reveal how a model makes decisions based on a specific input. Among these, the most widely used explanation type is _Token Explanations_ (TokenEx), which for Natural language Understanding tasks provide importance scores for the tokens of the input, using methods such as Shapley Values Lundberg and Lee (2017) and Integrated Gradients Sundararajan et al. (2017). However, for complex reasoning tasks that require reasoning across multiple pieces of text, e.g., Fact Checking is performed given a claim and evidence, Natural Language Inference is performed given a premise and a hypothesis, TokenEx can be insufficient to capture the relations employed between the different parts of the input. To this end, _Token Interactive Explanations_ (TokenIntEx) are proposed as another explanation type that provides importance scores for interactions between two tokens of the input with methods such as Bivariate Shapley Masoomi et al. (2022) and Layer-wise Attention Attribution Ye et al. (2021). Further, to enhance the semantic coherence of these interactive explanations, _Span Interactive Explanations_ (SpanIntEx) is an explanation type that provides importance scores for interactions across tuples of spans, e.g., generated by Louvain community detection Ray Choudhury et al. (2023). Figure 1 showcases these three types of input feature explanations.

It is crucial to develop automated, rigorous and

Figure 1: An example of the three types of Input Feature Explanations on an instance from the SNLI dataset, with their two most important pieces of explanation (token for TokenEx, token tuple for TokenIntEx, span tuple for SpanIntEx, correspondingly).



comprehensive evaluation frameworks to ensure the principled selection of the most suitable explanation in a practical application (Yu et al., 2024) and systematic progress in the development of different types of explanations (Atanasova et al., 2022; Jolly et al., 2022). However, these three types of input feature explanations have typically been studied in isolation, where explanation methods of the same type are compared (Atanasova et al., 2020; DeYoung et al., 2020; Janizek et al., 2021; Ray Choudhury et al., 2023). Moreover, evaluations of interactive explanations have been restricted to one property. To address these gaps, we develop _a unified framework that facilitates an automated and direct comparison between highlight and interactive input feature explanations on a suite of four diagnostic properties_. The framework allows for a rapid, scalable comparison across explainability methods, essential for the growing number of new techniques. Using the framework, we then perform an _extensive empirical analysis of the properties of existing explanation methods across all three explanation types_.

* Faithfulness, Agreement with Human Annotation, Simulatability, and Complexity. They are the most widely used for TokenEx(Nauta et al., 2023) and include the only property used for interactive explanations
- Faithfulness. _Faithfulness_(Chen et al., 2020, 2021; Ray Choudhury et al., 2023) measures the extent to which explanations accurately reflect the reasons used by the model in its predictions. _Agreement with Human Annotation_(Atanasova et al., 2020) evaluates whether explanations exhibit an inductive bias akin to human reasoning, potentially enhancing their plausibility to end users. _Simulatablity_(Pruthi et al., 2022) estimates whether the explanations are useful to an agent for replicating the model's decisions. Finally, _Complexity_(Bhatt et al., 2021) evaluates whether the explanations are easy to comprehend. In the unified evaluation framework, we further extend the four properties to facilitate their application and comparison across all three explanation types (SS2).

* Sufficiency. Additionally, SpanIntEx and TokenIntEx align more closely with human annotations at the token level than TokenEx. Moreover, SpanIntEx demonstrate the highest interaction overlap with human annotations. Further, SpanIntEx are found to be most beneficial for Simulatability. Finally, our results suggest that SpanIntEx and TokenEx are easier to comprehend.

Overall, our analysis highlights the strengths of each explanation type across various diagnostic properties, with SpanIntEx generally outperforming others on most measures. However, we observe a trade-off between Comprehensiveness and plausibility, particularly with SpanIntEx, underscoring the need for methods that enhance both. Future research could explore integrated approaches that combine explanation types to optimize performance across all diagnostic properties.

## 2 A Unified Automated Evaluation Framework for Highlight Explanations

According to the established taxonomy of evaluation approaches for explainability methods, evaluations can be functionally grounded, human-grounded, or application-grounded (Doshi-Velez and Kim, 2017). Our work focuses on automated functionally-grounded evaluation, which serves as a foundation for the other evaluation types as it enables the rapid, scalable comparison across explainability methods, essential for the growing number of new techniques. To this end, we present a unified framework comprised of four widely employed diagnostic properties: Faithfulness, Agreement with Human Annotations, Simulatability, and Complexity. This section formally introduces them and outlines the extensions that allow for their application across different input feature explanation types.

### Preliminaries

We start with a dataset \(D\), and a model \(M\) fine-tuned on \(D\). An instance \(x\in D\) comprises two parts, e.g., a claim and an evidence, the first consisting of \(m\) tokens, and the second - of \(n\) tokens. We apply an explanation attribution method \(A_{E}\) of type \(E\in\{\texttt{TokenEx},\texttt{TokenIntEx},\texttt{SpanIntEx}\}\) to \(M\), and each \(x\in D\): \(A_{E}(M,x)=\{e^{x}_{k},a^{x}_{k}|k\in[0,s-1]\}\), where \(e^{x}_{k}\) is a token/pair of tokens/pair of token spans from the input and \(a^{x}_{k}\) denotes its importance score. \(k\) is the importance ranking 

of the corresponding piece of explanation. \(s\) is an upper limit for the number of most important pieces of explanation for an instance, such that: \(s\in[1,m+n]\) for TokenEx, \(s\in[1,m\cdot n]\) for TokenIntEx, and for SpanIntEx \(s\) varies for each instance with \(s\in[1,f],f<m!\cdot n!\). Depending on the explanation type \(E\), \(e_{k}^{x}\) can consist of: one token \(x_{i}\) for TokenEx, \(i\in[0,m+n-1]\), one token pair \((x_{p},x_{q})\) for TokenIntEx, where \(p\in[0,m-1]\) and \(q\in[m,m+n-1]\), one span pair \(((x_{s},\dots,x_{s+l_{1}}),(x_{t},\dots,x_{t+l_{2}}))\) for SpanIntEx, where \(s,s+l_{1}\in[0,m-1]\) and \(t,t+l_{2}\in[m,m+n-1]\).

Considering a particular threshold for the number of most important explanation pieces \(s\), we compute the total set of input tokens involved in the presented explanation for \(x\):

\[T_{A_{E},M}(x,s)=\{x_{i}|x_{i}\in e_{k}^{x},k\in[0,s-1]\} \tag{1}\]

However, as noted above, the upper bound for \(s\) can vary across input feature explanations. Additionally, the number of tokens included in the top-\(k\) important explanations can differ substantially among explanation types - in top-1 we can have: 1 token for TokenIntEx, 2 tokens for TokenIntExt, and more than 2 tokens for SpanIntEx. This variability makes it difficult to compare results across different explanation types. To this end, we propose extensions for each of the studied diagnostic properties that result in _unified diagnostic properties_ applicable for a direct comparison of the different types of input feature explanations.

### Faithfulness

Faithfulness DeYoung et al. (2020) assesses whether explanations accurately reflect the model's decision-making process. It involves two aspects - Sufficiency and Comprehensiveness - measured as the number of the model's prediction changes after keeping (SP) or omitting (CP) \(k\) most important portions of the input (see omission details in SSA):

\[CP(x,A_{E},k)=\left\{\begin{array}{ll}0,&\text{if $f(x-T_{A_{E},M}(x,k))= f(x)$}\\ 1,&\text{otherwise}\end{array}\right\} \tag{2}\]

\[SP(X,A_{E},k)=\left\{\begin{array}{ll}1,&\text{if $f(T_{A_{E},M}(x,k))= f(x)$}\\ 0,&\text{otherwise}\end{array}\right\} \tag{3}\]

To take different thresholds \(k\), we average over \(k\in[0,s=m+n-1]\). We then also average the results across instances within \(D\) to compute the final Comprehensiveness and Sufficiency scores:

\[Comp_{A_{E}}=\frac{\sum_{x\in D}^{|D|}\sum_{k=0}^{s}CP(x,A_{E},k)}{s*|D|} \tag{4}\]

\[Suff_{A_{E}}=\frac{\sum_{x\in D}^{|D|}\sum_{k=0}^{s}SP(x,A_{E},k)}{s*|D|} \tag{5}\]

**Unified Faithfulness.** To facilitate the comparison of faithfulness scores, we introduce a _dynamic threshold_\(\theta_{x,k}\). It represents the number of unique tokens used for a perturbation on \(x\), same across explanation methods of all explanation types. Since top-\(k\) explanations for \(A_{E}\) of type SpanInterEx typically contain more tokens than for TokenExc or TokenIntEx, we set \(\theta_{x,k}\) across the explanation methods of all types to:

\[\theta_{x,k}=|T_{A_{\texttt{SpanIntEx}},M}(x,k)| \tag{6}\]

Thus, \(\theta_{x,k}\) becomes a dynamic threshold that adapts based on each instance's top-\(k\) explanation tokens from \(A_{\texttt{SpanIntEx}}\). We then adjust the number of top-\(k\) explanations selected from \(A_{\texttt{TokenEx}}\) and \(A_{\texttt{TokenIntEx}}\), \(kA_{\texttt{TokenEx}}\) and \(kA_{\texttt{TokenIntEx}}\), correspondingly, to result in the same number of perturbed tokens \(\theta_{x,k}\):

\[|T_{A_{\texttt{TokenEx}},M}(x,k_{A_{\texttt{TokenEx}}}(x))|=\theta_{x,k} \tag{7}\]

\[|T_{A_{\texttt{TokenIntEx}},M}(x,k_{A_{\texttt{TokenIntEx}}}(x))|=\theta_{ x,k} \tag{8}\]

Furthermore, a _Random baseline_ is established, where the tokens for perturbation are selected randomly to match the average \(\theta_{x,k}\) across \(D\).

### Agreement with Human Annotations

Agreement with Human Annotations has been used to assess the overlap between generated and human-annotated explanations, which can indicate the plausibility of the generated explanations to end users. For \(E=\texttt{TokenEx}\), the measure is computed by calculating a precision score for the top-\(k\) most important explanations compared to the gold human annotations Atanasova et al. (2020).

For \(E=\texttt{TokenEx}\), \(a_{i}^{x},i\in[1,m+n]\) is the attribution score of \(i\)th most important explanation for \(x\). \(s=m+n\) is the number of explanations extracted from \(x\). Corresponding to each explanation's attribution score, \(s\) thresholds are set, forming the threshold list \(\omega_{AE}(x)=[a_{0}^{x},...,a_{s}^{x}]\). By selecting explanations with higher attribution scores than each threshold in \(\omega_{A_{E}}(x)\), \(s\) targeted explanation sets are obtained, where \(C_{A_{E}}(x,i)\{e_{j}^{x}:a_{j}^{x}<=a_{i}^{x}\}\) represents the set for the \(i^{th}\) threshold, \(a_{j}^{x}\) is the attribution score of token \(e_{j}^{x}\) for \(E=\texttt{TokenEx}\). Comparing these sets with the golden explanation set \(e^{G}\), \(s\) precision-recall pairs 

\(P_{i}/R_{i}(x,e^{G},A_{E})\) can be derived. Average Precision (AP) is then obtained by weighting \(P_{i}\) with the corresponding \(R_{i}\) increase:

\[P_{i}/R_{i}(x,e^{G},A_{E})=Pre/Rec(C_{A_{E}}(x,i),e^{G}) \tag{9}\]

\[AP_{A_{E}}(x,e^{G})=\sum_{i=0}^{s}(R_{i}-R_{i-1})*P_{i} \tag{10}\]

Mean AP (MAP) is calculated for all \(x\in D\):

\[MAP_{A_{E}}=\frac{\sum_{x\in D}^{|D|}AP_{AE}(x,e^{G})}{|D|} \tag{11}\]

**Unified Agreement with Human Annotation Measure.** For a fair comparison between the different types of explanations, the thresholds \(\omega_{A_{E}}(x)\) for including the same number of tokens across the explanation methods follows the procedure set for the Unified Faithfulness (SS2.2). Furthermore, we measure Agreement with Human Annotations at the **interaction level** for the gold SpanIntEx/TokenIntEx explanations and at the **token level** for gold TokenExt explanations.

**Interaction-level Agreement with Human Annotations.** For a fair comparison between TokenIntEx and SpanIntEx methods, we adapt \(MAP_{\texttt{TokenExt}}\) to the interaction level. Specifically, we compute the mean average precision (Eq. 11) w.r.t. the human-annotated TokenIntEx/SpanIntEx sets.

**Token-level Agreement with Human Annotations.** For a fair comparison between TokenEx and TokenIntEx/SpanIntEx methods, we extract tokens from TokenIntEx/SpanIntEx and compare them with tokens from golden TokenIntEx/SpanIntEx sets. To compute MAP at token level, we follow the similar procedure set for \(E=\texttt{TokenEx}\) (Eq. 11) with threshold lists \(\omega_{A_{\texttt{TokenIntEx/SpanIntEx}}}(x)\), but the targeted sets \(C_{A_{E_{\texttt{token}}}}(x,i)\) contain tokens extracted from TokenIntEx/SpanIntEx methods. The golden set \(S_{A_{E_{\texttt{token}}}}(x)\) aggregates tokens from golden TokenIntEx/SpanIntEx sets.

We also set a _Random baseline_, where the number of randomly selected span pairs, token pairs, or tokens for each instance matches the average number of tokens per instance extracted with a SpanIntEx explanation method.

### Simulatability

Simulatability was initially proposed to measure how accurately humans can predict a model's outputs based on its explanations Chen et al. (2024); Hase et al. (2020). Previous studies demonstrated that Simulatability can be approximated using an automated agent model as a surrogate for human understanding Pruthi et al. (2022). _Given the established positive correlation between Simulatability and human evaluation of explanation utility_, we integrate the Simulatability scores obtained from an agent model with different explanation types to approximate their utility for humans.

Following existing work Hase et al. (2020), we train an agent model \(AM\), sharing the same architecture as the original model \(M\), to simulate \(M\)'s predictions \(Y^{\prime}\) using produced explanations. During \(AM\)'s training phase, we extract the top-\(k\) explanations and incorporate them in the input. In comparison, another agent model, \(AM_{O}\), is trained without explanation guidance as a baseline on the same training set. During the testing phase, the simulation accuracy of \(AM\) and \(AM_{O}\) over the shared dataset \(D\) is calculated.2 The difference between the accuracies is interpreted as the explanation's effect in enhancing the simulatability of \(M\):

Footnote 2: While existing work Hase et al. (2020) notes that incorporating natural language explanations in the testing phase could leak the predicted label, we use only input feature explanations that do not contain additional information.

\[Sim=ACC(AM(D),Y^{\prime})-ACC(AM_{O}(D),Y^{\prime}) \tag{12}\]

**Unified Simulatability.** To compare the simulation utility of different explanation types, we train a separate agent model \(AM_{E}\) for each explanation method \(A_{E}\) and calculate the corresponding simulation performance on the common test set. For a fair comparison across the different explanation method types \(A_{E}\), we first ensure top-\(k_{E}\) explanations are presented for assisting the agent's training for \(A_{E}\), following Section 2.2. This ensures each model is exposed to the same quantity of unique tokens from different explanation types.

During the training of \(AM\), we introduce the explanations from \(A_{E}\) into the learning of \(AM_{Ag}\); we supplement \(x\) with top-\(k_{A_{E}}\) explanations instead so that the agent model is trained with the same mechanism whether the explanations are provided or not, and each training instance will contain the input sequence \(x_{A_{E}}\) and golden label \(Y^{\prime}\) which is predicted by the original model \(M\). Specifically, we examine two different ways of presenting the explanations as part of the original input sequence, \(I_{Symbol}\) and \(I_{Text}\) (see SSB.); all aim to ensure the explanations of different types are inserted similarly.

At test time, the F1 scores of agent models 

and \(AM_{O}\) over \(D\) are calculated:

\[SF_{E}=F1(AM_{E}(x_{E}),Y^{\prime}),x\in D \tag{13}\]

\[SF_{O}=F1(AM_{O}(x),Y^{\prime}),x\in D \tag{14}\]

Note that the explanations from \(A_{E}\) are also provided to the input for the unseen instances for agent model \(AM_{E}\). The final simulation metric then indicates how much this specific type of input feature explanation enhances the model's simulatability:

\[RSF_{E}=SF_{E}-SF_{O} \tag{15}\]

### Complexity

Feature attribution explanations are designed to aid human understanding of a model's reasoning over specific instances. However, since humans have a limited capacity to process large amounts of information simultaneously, these explanations need to be easy to comprehend. Even if we select only the top-\(k\) attributions with the highest importance scores, they need to be distinctive as opposed to the attribution scores having a uniform distribution. Bhatt et al. (2021) propose to measure the complexity of a produced explanation with entropy Renyi (1961) over the attribution scores of all the produced explanations by \(A_{\texttt{TokenEx}}\) method:

\[P(x,p)=\left|a_{p}^{x}\right|/\sum_{q=1}^{m+n}\left|a_{q}^{x}\right| \tag{16}\]

\[CL(x)=-\sum_{p=1}^{m+n}P(x,p)\ln(P(x,p)) \tag{17}\]

\(m+n\) is the total number of generated \(A_{\texttt{TokenEx}}\) explanations, and all explanations are considered for the complexity score computation. Higher entropy means different features have similar attribution scores, where the simplest explanation, with low entropy, would be concentrated on one feature.

**Unified Complexity.** To ensure a fair comparison across different types of explanation methods \(A_{E}\), we maintain consistency in the size of the chosen explanation lists across all \(A_{E}\) for the same instance, denoted as \(k_{x}\), as the number of generated \(A_{\texttt{TokenEx}}\)/\(A_{\texttt{TokenIntEx}}\)/\(A_{\texttt{SpanIntEx}}\) explanations originally vary for the same \(x\). The complexity score \(CL_{A_{E}}(x,k_{x})\) of the top-\(k_{x}\) explanation list under method \(A_{E}\) is calculated as:

\[P_{A_{E}}(x,k_{x},i)=\left|a_{i}^{x}\right|/\sum_{j=1}^{k_{x}}\left|a_{j}^{x}\right| \tag{18}\]

\[CL_{A_{E}}(x,k_{x})=-\sum_{i=1}^{k_{x}}P_{A_{E}}(x,k_{x},i)\ln(P_{A_{E}}(x,k_{ x},i)) \tag{19}\]

where \(a_{i}^{x}\)/\(a_{j}^{x}\) represent the \(i\)/\(j\)th highest attribution score from the explanation set for \(x\).

The final complexity score is an average of \(CL_{A_{E}}(x,k_{x})\) across all \(x\in D\):

\[CL_{E}=\sum_{x\in D}^{\left|D\right|}CL_{E}(x,k_{x})/\left|D\right| \tag{20}\]

Notably, \(k_{x}\) is calculated from the number of explanations produced by method \(A_{\texttt{SpanIntEx}}\) for \(x\), which varies based on the span interaction extraction method and is known only after generation.

## 3 Experimental Setup

### Datasets

We select the natural language inference dataset SNLI Bowman et al. (2015), where instances consist of a premise, hypothesis, and a label \(y\in\{ entailment,neutral,contradiction\}\). Additionally, we select the fact checking dataset FEVER Thorne et al. (2018); Atanasova et al. (2020), where instances consist of a claim, evidence, and a label \(y\in\{ entailment,neutral,contradiction\}\).3 We generate input feature explanations by sampling 4,000 instances from each train, dev, and test set, due to the high computational cost, particularly for Shapley-based explanations Atanasova et al. (2020). For Agreement with Human Annotations property, we use existing human explanation annotations for SNLI and FEVER (see SSC).

Footnote 3: [https://huggingface.co/datasets/cope](https://huggingface.co/datasets/cope) nlu/fever_gold_evidence

### Input Feature Explanation Methods

To generate importance scores, we first select three common TokenEx techniques - Shapley Lundberg and Lee (2017), Attention DeYoung et al. (2020), and Integrated Gradients (IG, Sundararajan et al. (2017)). For TokenIntEx, we employ Bivariate Shapley Masoomi et al. (2022), Attention Clark et al. (2019), and Layer-wise Attention Attribution Ye et al. (2021). Notably, the TokenIntEx techniques are the bivariate version of the techniques used for generating TokenEx; e.g. Layer-wise Attention Attribution uses IG.

Following Ray Choudhury et al. (2023), we apply the Louvain algorithm Blondel et al. (2008) for each of the three selected TokenIntEx to generate importance scores for SpanInterExt methods, where the importance score of each span interaction is averaged over the importance scores of the token interactions within it. We will refer to Shapley, Attention, and IG as the explanation base types 

used for generating all types of input feature explanations for brevity. See SSE for more details.

### Models

We use the BERT-base-uncased model Devlin et al. (2019) with 12 encoder layers, and the BART-base-uncased model Lewis et al. (2020) with 6 encoder and 6 decoder layers, as common representatives of the encoder and the encoder-decoder Transformer architecture Vaswani et al. (2017). Our choice is particularly influenced by the substantial computational requirements of the input feature explanations, especially pronounced for Shapley Atanasova et al. (2020). Additionally, our choice is guided by the need to directly access the models' internals for generating IG and Attention-based explanations. Furthermore, while our framework currently utilizes the said models, it is designed to be easily adaptable to other models or newly developed explainability techniques, provided that there are more robust computational resources available.

We fine-tune the base models on SNLI and FEVER and use them to generate explanations. Their performance on the test sets is shown in SSD. For the Simulation property, we follow existing work Fernandes et al. (2022); Pruthi et al. (2022) and train simulator agent models (SS2.4) with the same architectures as the base ones. Following Fernandes et al. (2022), we split the test set into train/dev/test for the training of the agent model.

## 4 Results and Discussion

We now present the results of our unified evaluation framework (SS2) illustrated in Fig. 2. They include explanation methods of types SpanIntEx, TokenIntEx, and TokenExc (SS3.2), two models (SS3.3), two datasets (SS3.1), and three base explanation techniques per explanation type (SS3.2). For Simulatability, we select the results of \(I_{sym}\), as this form avoids repeating the input text and increasing the input size substantially. For Agreement with Human Annotations, we select the Token-level results as they are present for all explanation types. SSF lists detailed results per property.

### Faithfulness

**Unified Comprehensiveness**. Across both datasets and models, TokenEx and TokenIntEx are identified as the most comprehensive explanation types, achieving the highest scores in 7/12 and 5/12 cases, respectively. SpanIntEx, designed to enhance the semantic coherence of interactive explanations by including additional context, often incorporates tokens that do not directly contribute to the model's prediction, thus explaining its lower comprehensiveness scores. Compared to the random baseline, TokenEx and SpanIntEx always outperform it, while TokenIntEx mostly underperform it when based on IG. Across the base explanation techniques, TokenEx performs best when based on Attention for BERT and on Shapley for BART, indicating that **different base explanation techniques can perform better for different architectures**. Both TokenIntEx and SpanIntEx show optimal performance when based on Shapley and Attention. Overall, the results indicate a _stronger performance of Attention and Shapley over IG_ across all explanation types.

**Unified Sufficiency.**_SpanIntEx ranks as the most sufficient explanation type in 7/12 cases_, surpassing TokenEx, which performs well in only 3/12 cases. While contrary to SpanIntEx Comprehensiveness performance, we attribute this to the semantic coherence of the extracted top spans, which provide more meaningful information. Note that while Sufficiency is highly desirable, Comprehensiveness is not required in all downstream applications as end-users prefer simpler, more general explanations with fewer causes Thagard (1989). Unlike TokenEx, which consistently outperforms the random baseline, TokenIntEx and SpanIntEx struggle to outperform it on FEVER, likely due to the longer input, posing challenges for the explanations to accurately unveil the model's internal processes. The results from different base explanation techniques show no clear trends, indicating a **significant variability stemming from the specific dataset and model architecture**.

### Agreement with Human Annotation

SpanIntEx and TokenIntEx show higher agreement scores with human annotations than TokenEx. Similarly, SpanIntEx consistently achieve higher agreement with human interaction-level annotators, especially when based on Attention scores (see SSF). This indicates that **SpanIntEx are more plausible to humans due to their enhanced semantic coherence**. In contrast, TokenEx often score lower than the random baseline. Moreover, considering SpanIntEx's lower performance in Comprehensiveness, there emerges **a distinct trade-off between Compre 

hensiveness and Agreement with Human Annotations**. Across base explanation techniques, IG performs best for FEVER; IG and Attention - for SNLI. In addition, we find that for the interaction-level agreement, TokenIntEx and SpanIntEx perform worst when based on Shapley. The lower agreement results for Shapley compared to its better results on Comprehensiveness again indicate an existing trade-off between the two properties.

### Simulatability

Our results show that SpanIntEx achieve the highest simulatability in 9/12 cases, helping the agent model accurately reproduce the original model's prediction. This again underscores the **critical role of contextual information and enhanced semantic coherence provided by SpanIntEx**. Notably, providing SpanIntEx to agents improves their ability to simulate the original model by up to 7.9 F1 points compared to without explanations. Among base explanation techniques, IG consistently performs best for SpanIntEx; other techniques do not exhibit a clear trend.

### Complexity

SpanIntEx and TokenEx generally achieve similar complexity, which consistently remains lower than those of TokenIntEx. This suggests that **TokenEx and SpanIntExt generate more distinctive attribution scores, potentially making them easier for humans to understand**. Regarding the base explanation techniques, Attention consistently yields the best complexity scores for BERT across all explanation types. There is no

Figure 2: Unified evaluation framework (§2) results for all feature attribution methods (§3.2).

 

clear trend for BART. Additionally, TokenIntEx frequently underperform the random baseline, highlighting its complexity (see SSF).

### Overall

In summary, we find that while TokenEx and TokenIntEx generally provide more comprehensive insights, SpanIntEx performs better in Sufficiency due to its enhanced semantic coherence (SS4.1). This calls for _better methods for generating SpanIntEx that are both comprehensive and sufficient_. Additionally, there is a trade-off between Comprehensiveness and Agreement with Human Annotations (SS4.2), suggesting that the most faithful explanations might be less plausible to end users. This highlights the _need for advanced methods to boost both the Comprehensionness and plausibility_ of SpanIntEx possibly leveraging the advantage of TokenEx. Furthermore, SpanIntEx significantly improves simulatability by allowing agents to accurately replicate model decisions (SS4.3), which is crucial in practice. Finally, the complexity analysis (SS4.4) shows that SpanIntEx and TokenEx are potentially easier to comprehend than TokenIntEx when considering the importance score distribution.

Overall, our results highlight the differences between the different types of input feature explanations, with SpanIntEx outperforming others on most measures. As no one type performs best on all properties, we _call for the development of combined methods that can leverage the strength of the different explanation types and potentially lead to an overall improvement of the explanation utility_.

## 5 Related Work

**Input Feature Explanations.** Considerable research exists on extracting explanations for input data. Methods like perturbation-based attribution (e.g., Shapley Lundberg and Lee (2017)), attention-based methods (e.g., Attention Jain and Wallace (2019); Serrano and Smith (2019)), and gradient-based methods (e.g., Integrated Gradients Sundararajan et al. (2017); Serrano and Smith (2019)) are prevalent for highlighting individual tokens Atanasova et al. (2020a). As individual tokens might be insufficient to explain the model, many attribution methods have been extended to bivariate forms Masoomi et al. (2022); Janizek et al. (2021); Sundararajan et al. (2017); Ye et al. (2021) to capture input token interactions. More recent work has explored how interactions between groups of tokens collectively contribute to model reasoning Ray Choudhury et al. (2023); Chen et al. (2021). Unlike other work where token groups might consist of tokens from arbitrary positions, Ray Choudhury et al. (2023) explicitly capture span interactions, enhancing the comprehensiveness of explanations by containing the entire spans.

**Automated Explanation Evaluation.** For evaluating TokenEx, DeYoung et al. (2020); Atanasova et al. (2020a) propose metrics to measure how faithful explanations are to the model's inner reasoning. They also assess the plausibility of explanations to humans by measuring the agreement of TokenIntEx with human annotations. To assess the utility of explanations to humans, Pruthi et al. (2022) propose to use an agent model as a proxy for humans and evaluate whether explanations aid in model simulatability. Complexity Bhatt et al. (2021) measures the distribution of attribution scores of TokenEx and assesses whether the key tokens in token explanations are easily comprehensible to humans. To evaluate TokenIntEx most works adopt the faithfulness or axiomatic/theoretical path Tsang et al. (2020); Sundararajan et al. (2020); Janizek et al. (2021). Current work on evaluating SpanIntEx has primarily focused on faithfulness Ray Choudhury et al. (2023). However, since SpanIntEx, TokenIntEx, and TokenExt contain varying amounts of tokens, which, e.g., affects the faithfulness test, this makes direct comparisons between different explanation types using existing metrics challenging. To our knowledge, no prior paper has involved all types of input feature explanations within a unified evaluation framework.

## 6 Conclusion

We introduced a unified evaluation framework for input feature attribution analysis to guide the principled selection of the most suitable explainability technique in practical applications. Our analysis outlines the diverse strengths and trade-offs among TokenEx, TokenIntEx, and SpanIntEx. Our findings particularly underscore SpanIntEx's superior performance in Sufficiency, agreement with human inductive biases, its enhancement of Simulatability, and Complexity, compared to TokenEx and TokenIntEx. Future efforts should focus on developing combined methods that enhance all explanation properties 



### Limitations

Our work introduces a unified framework to evaluate input feature explanations across four key properties. We generated three types of explanations using three attribution methods on two Transformer models (BERT-base and BART-base) for two NLU tasks (NLI and fact-checking). Thereby, we can assess and compare the properties of each explanation type. Due to computational resource limitations, we did not include larger decoder-only models in our evaluation. Future research could _explore other models_ to provide additional insights.

We note that our work considered the FEVER and SNLI datasets as they are the only available datasets with annotations of human interactive explanations, required for the Agreement with Human Annotations property. In future work, given the availability of other datasets, examining the properties of different explanations in various _tasks beyond NLI and fact checking_ would be valuable, especially for simpler tasks that consist of only one input part or more complex tasks that consist of more than two parts with possible relationships between them. Additionally, tasks with longer textual inputs, which are known to pose greater challenges for current explainability techniques Atanasova et al. (2020), could also be analyzed.

Furthermore, while we consider four widely used explanation properties in this automatic evaluation framework, future works should consider verifying, potentially with supplementary human studies, that the properties are well aligned with the downstream utility of the explanations in different application tasks Miller (2019). We note that manual evaluation, while valuable, is time-consuming and costly. Automated evaluation, with our proposed framework, allows for quicker insights, helping prioritize methods that may benefit from human-centered evaluation. Additionally, the properties we evaluate demonstrate why human evaluation is not necessarily required at this stage. Faithfulness measures whether explanations reflect the model's internal reasoning, a task humans cannot assess (see Faithfulness evaluation guidelines in Jacovi and Goldberg (2020)). Explanations that fail this test should not be considered for further human evaluation as they can be harmful, e.g. by hiding a model's flaws and biases. Agreement with Human Annotation already captures alignment with human reasoning, ensuring explanations are plausible. Automated Simulatability correlates strongly with human studies, providing a reliable proxy for replicating the model's behaviour without the need for expensive human experiments Pruthi et al. (2022). Finally, automated methods ensure consistency and objectivity, while human annotations can introduce subjectivity and variability. Studies have even shown conflicting results from human evaluations Poursabzi-Sangdeh et al. (2021); Ribeiro et al. (2016). Automated evaluation provides an objective, reproducible baseline, which can later be supplemented by human evaluations where needed.

We have also employed three base representative explainability methods for each of the three types of input feature explanations. However, _more existing and newly emerged base explainability methods_ could be explored in future work. Additionally, our study focuses solely on post-hoc explainability techniques, while other supervised feature extraction methods could also be investigated Yu et al. (2021); Liu et al. (2024). These methods typically treat human-annotated important fragments within the input as gold causal features, akin to our Agreement with Human Annotation measurement. The broader set of properties introduced in our framework could be leveraged to evaluate such explanations more comprehensively. Apart from that, future work could adapt and extend our framework to other forms of explanations such as free-text explanations produced by self-rationalization models Liu et al. (2022, 2023, 2023, 2024, 2025). All said potential future studies are well facilitated by the efficient automated evaluation proposed with our framework.

Our findings indicate that span interactive explanations (SpanIntEx) have a notable advantage over other explanation types in terms of Agreement with Human Annotation, Simulatability, and Complexity, suggesting they are easier for humans to understand. This insight could inspire future work to leverage SpanIntEx as the input feature explanation in HCI models. However, SpanIntEx shows low comprehensiveness in faithfulness evaluations. The Louvain algorithm, used for SpanIntEx generation, may limit its comprehensiveness despite using different attribution methods for TokenIntEx. Future work should explore better methods for capturing span interactions and possibly combine SpanIntEx and TokenEx for higher faithfulness, as TokenExt demonstrates a stable advantage in comprehensiveness.



Another core finding is that the attribution method significantly affects most diagnostic properties of all explanation types, such as sufficiency. No single attribution method consistently excels across all properties, highlighting the need for continuous evaluation and improvement in attribution methods, particularly for SpanIntEx.

To ensure a fair comparison, our unified evaluation framework currently considers only the token count differences among various input feature explanations, with interactive explanations flattened. Future work could involve a human-in-the-loop approach to account for the effects of interactive explanations beyond just token count differences. For example, a display system could visually present highlighted tokens and interactions to gauge human preferences. Our work provides a starting point for comparing input feature explanations from an automated evaluation perspective, and future research could explore additional factors, such as psychological elements and visual aspects, from a human perspective, which would benefit more non-expert users.

Another limitation of this work is that we focus solely on the automatic evaluation of input feature explanations without examining the potential biases they may exhibit. For instance, these explanations might favor certain words or phrases from the input sequence, for example, sometimes emphasizing some prepositions that might bear less meaning, raising questions about whether such biases stem from the models themselves or from the explainability techniques used. Also, it is worth checking, especially in sensitive domains such as healthcare and law, how contradictory explanations for different model decisions differed, which might sway the decision-making.

## Acknowledgements

This research was co-funded by the European Union (ERC, ExplainYourself, 101077481), by the Pioneer Centre for AI, DNRF grant number P1, as well as by The Villum Synergy Programme (grant number 40543). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

## References

* Atanasova et al. (2020a) Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020a. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Atanasova et al. (2022) Pepa Atanasova, Justin Wright, and Isabelle Augenstein. 2020b. Generating label cohesive and well-formed adversarial claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Bhatt et al. (2021) Umang Bhatt, Adrian Weller, and Jose M. F. Moura. 2021. Evaluating and Aggregating Feature-based Model Explanations. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20.
* Blondel et al. (2008) Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktaschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. E-SNLI: Natural Language Inference with Natural Language Explanations. Advances in Neural Information Processing Systems, 31.
* Chen et al. (2021) Hanjie Chen, Song Feng, Jatin Ganhotra, Hui Wan, Chulaka Gunasekara, Sachindra Joshi, and Yangfeng Ji. 2021. Explaining neural network predictions on sentence pairs via learning word-group masks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online. Association for Computational Linguistics.
* Chen et al. (2020) Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. 2020. Generating hierarchical explanations on text classification via feature interaction detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 5578-5593, Online. Association for Computational Linguistics 

Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. 2024. Do models explain themselves? counterfactual simulatability of natural language explanations. In _Forty-first International Conference on Machine Learning_.
* Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 276-286, Florence, Italy. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4443-4458, Online. Association for Computational Linguistics.
* Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. _arXiv preprint arXiv:1702.08608_.
* Fernandes et al. (2022) Patrick Fernandes, Marcos Treviso, Danish Pruthis, Andre Martins, and Graham Neubig. 2022. Learning to Scaffold: Optimizing Model Explanations for Teaching. _Advances in Neural Information Processing Systems_, 35:36108-36122.
* Hase et al. (2020) Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4351-4367, Online. Association for Computational Linguistics.
* Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4198-4205.
* Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics.
* Janizek et al. (2021) Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2021. Explaining Explanations: Axiomatic Feature Interactions for Deep Networks. _Journal of Machine Learning Research_, 22(104):1-54.
* Jolly et al. (2022) Shailza Jolly, Pepa Atanasova, and Isabelle Augenstein. 2022. Generating Fluent Fact Checking Explanations with Unsupervised Post-editing. _Information_, 13(10):500.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahaman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Liu et al. (2024a) Wei Liu, Zhiying Deng, Zhongyu Niu, Jun Wang, Haozhao Wang, YuanKai Zhang, and Ruixuan Li. 2024a. Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalization. _arXiv preprint arXiv:2410.06003_.
* Liu et al. (2024b) Wei Liu, Haozhao Wang, Jun Wang, Zhiying Deng, Yunanki Zhang, Cheng Wang, and Ruixuan Li. 2024b. Enhancing the Rationale-Input Alignment for Self-explaining Rationalization. In _2024 IEEE 40th International Conference on Data Engineering (ICDE)_, pages 2218-2230. IEEE.
* Liu et al. (2023a) Wei Liu, Haozhao Wang, Jun Wang, Ruizvan Li, Xinyang Li, YuanKai Zhang, and Yang Qiu. 2023a. MGR: Multi-generator Based Rationalization. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12771-12787.
* Liu et al. (2022) Wei Liu, Haozhao Wang, Jun Wang, Raixuan Li, Chao Yue, and YuanKai Zhang. 2022. FR: Folded Rationalization with a Unified Encoder. _Advances in Neural Information Processing Systems_, 35:6954-6966.
* Liu et al. (2024c) Wei Liu, Jun Wang, Haozhao Wang, Ruxuan Li, Zhiying Deng, Yunankai Zhang, and Yang Qiu. 2024c. D-Separation for Causal Self-Explanation. _Advances in Neural Information Processing Systems_, 36.
* Liu et al. (2023b) Wei Liu, Jun Wang, Haozhao Wang, Ruzuuan Li, Yang Qiu, Yunankai Zhang, Jie Han, and Yixiong Zou. 2023b. Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 1535-1547.
* Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. _Advances in neural information processing systems_, 30.
* Masoomi et al. (2022) Aria Masoomi, Davin Hill, Zhonghui Xu, Craig P Hersh, Edwin K. Silverman, Peter J. Castaldi, Stratis Ioannidis, and Jennifer Dy. 2022. Explanations of blackbox models based on directional feature interactions.

 

In _International Conference on Learning Representations_.
* Miller (2019) Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. _Artificial intelligence_, 267:1-38.
* Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jorg Schlotterer, Maurice van Keulen, and Christin Seifert. 2023. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. _ACM Computing Surveys_, 55(13s):1-42.
* Poursabzi-Sangdeh et al. (2021) Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Waughan, and Hanna Wallach. 2021. Manipulating and Measuring Model Interpretability. In _Proceedings of the 2021 CHI conference on human factors in computing systems_, pages 1-52.
* Pruthi et al. (2022) Danish Pruthi, Rachit Bansal, Zhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. 2022. Evaluating explanations: How much do explanations from the teacher aid students? _Transactions of the Association for Computational Linguistics_, 10:359-375.
* Choudhury et al. (2023) Sagnik Ray Choudhury, Pepa Atanasova, and Isabelle Augenstein. 2023. Explaining interactions between text spans. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12709-12730, Singapore. Association for Computational Linguistics.
* Renyi (1961) Alfred Renyi. 1961. On Measures of Entropy and Information. In _Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics_, volume 4, pages 547-562. University of California Press.
* Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?" Explaining the Predictions of Any Classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1135-1144.
* Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
* Sundararajan et al. (2020) Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. 2020. The Shapley Taylor Interaction Index. In _International conference on machine learning_, pages 9259-9268. PMLR.
* Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In _International conference on machine learning_, pages 3319-3328. PMLR.
* Thagard (1989) Paul Thagard. 1989. Explanatory Coherence. _Behavioral and brain sciences_, 12(3):435-467.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Tsang et al. (2020) Michael Tsang, Sirisha Rambhatla, and Yan Liu. 2020. How Does This Interaction Affect Me? Interpretable Attribution for Feature Interactions. _Advances in neural information processing systems_, 33:6147-6159.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Ye et al. (2021) Xi Ye, Rohan Nair, and Greg Durrett. 2021. Connecting attributions and QA model behavior on realistic counterfactuals. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5496-5512, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Yu et al. (2024) Haeun Yu, Pepa Atanasova, and Isabelle Augenstein. 2024. Revealing the parametric knowledge of language models: A unified framework for attribution methods. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8173-8186, Bangkok, Thailand. Association for Computational Linguistics.
* Yu et al. (2021) Mo Yu, Yang Zhang, Shiyu Chang, and Tommi Jaakkola. 2021. Understanding Interlocking Dynamics of Cooperative Rationalization. _Advances in Neural Information Processing Systems_, 34:12822-12835.

 

Unified Faithfulness Evaluation:

Explanation Masking Details

As discussed in Section SS2.2, we introduce the dynamic threshold \(\theta_{x,k}\) to ensure an identical number of tokens from different types of top input feature explanations for the same instance in Unified Faithfulness evaluation.

For the Unified Comprehensiveness evaluation, we conduct a similar process for all three explanation types separately. Using TokenEx as an example, we first calculate the maximum number of top TokenEx for disturbing each instance \(x\) as \(k_{\texttt{TokenExt}}(x)\) following Eq. 7. To omit the TokenEx tokens from the original input, we replace them with [MASK] tokens, while keeping the rest unchanged. The specific [MASK] token used depends on the model architecture. We then gradually increase the number of TokenEx tokens masked out until it reaches \(k_{\texttt{TokenEx}}(x)\) and record the corresponding changes in model predictions. The average prediction change across the dynamic threshold and all instances gives the Unified Comprehensiveness score for TokenEx. For TokenIntEx and SpanIntEx, the only difference is that we mask out token pairs for TokenIntEx and span pairs for SpanIntEx, with the maximum explanations masked out as \(k_{\texttt{TokenIntEx}}(x)\) and \(k_{\texttt{SpanIntEx}}(x)\) for each instance \(x\), calculated using Eq. 6 and Eq. 8 respectively.

For the Unified Sufficiency evaluation, we conduct experiments for the three explanation types separately. Unlike Unified Comprehensiveness, we retain only the tokens/token pairs/span pairs for the input while masking out all other tokens with [MASK] tokens for each instance, depending on the model architecture used. We first calculate the maximum number of top explanations involved in disturbance for each explanation type for instance \(x\) using Eq. 6, Eq. 7, and Eq. 8. Then, we keep the token/token pairs/span pairs in the model input by masking out all other tokens, starting with one explanation and adding one more explanation for each subsequent disturbance until the total number of explanations reaches \(k_{\texttt{TokenExt}}(x)\), \(k_{\texttt{TokenIntEx}}(x)\), or \(k_{\texttt{SpanInterlex}}(x)\). Meanwhile, we record the model predictions for each disturbance. The Unified Sufficiency score for each explanation type is then calculated by averaging the prediction changes across the dynamic threshold for that explanation type, considering all instances.

## Appendix B Detailed Explanation Insertion Method

To enable a fair comparison among different input feature explanations in terms of simulatability (SS2.4), we applied consistent insertion formats to combine the explanations with the original input for training the agent models. This design aims to minimize noise from insertion format differences. We tested two ways, each applicable to all types of input feature explanations, to construct input sequences with inserted explanations of type \(E\). These input sequences are denoted \(x_{E}\) in SS2.4, omitting specific insertion format details for brevity.

For Symbol-Insertion \(I_{Symbol}\), we preserve the original input sequence but insert special symbols \(<\) and \(>\) to quote the tokens (for TokenEx and TokenIntEx) or spans (for SpanIntEx) within the input. Additionally, for TokenEx, we append a ranking mark after each quoted token based on their attribution scores, ranked in descending order. For TokenEx and SpanIntEx, each quoted token/span is also assigned a ranking mark indicating the rank of their respective interactions by attribution score, ensuring tokens/spans from the same interaction share the same mark. This method allows us to generate input sequences combined with different input feature explanations in a consistent symbol insertion format.

For Text-Insertion \(I_{Text}\), we append tokens, token tuples, or span tuples to the end of the original input sequence for each explanation type. They are added in the order ranked by descending attribution score. Specifically, for TokenEx, tokens from different TokenExexplanations are separated by semicolons. For TokenIntEx and SpanIntEx, tokens/spans within each interaction are connected by a comma, and different interactions are separated by semicolons. This approach constructs input sequences combined with each type of input feature explanation in a consistent text insertion format.

## Appendix C Agreement Dataset Details

To assess how different types of input feature explanations overlap with human annotations, we collected golden explanations of various types from e-SNLI and SpanEx for instances within SNLI and FEVER, respectively. Detailed information about the annotated explanation types and the number of instances with labeled explanations for these datasets is shown in Table 1. For the SNLI dataset, 

e-SNLI provides TokenEx explanations, while _SpanEx-SNLI_ includes SpanIntEx explanations and token-level interactions (TokenIntEx explanations). We selected 3,865 overlapping instances and evaluated the human agreement score for different types of input feature explanations. For FEVER, _SpanEx-FEVER_ includes SpanIntEx and token-level interactions (TokenIntEx explanations). Since no TokenEx explanations are provided, we extracted tokens from the golden TokenIntEx explanations in _SpanEx-FEVER_ as an approximation. These selected instances are also used when evaluating other properties of input feature explanations.

## Appendix D Base Model Performance.

As shown in Table 2, we report the performance of fine-tuned BERT-base and BART-base models on SNLI and FEVER, respectively. These models, fine-tuned for their specific tasks, are used to generate various input feature explanations through different explainability techniques. Importantly, these are the original models that the agent models, as described in SS2.4, learn to simulate.

## Appendix E Explainability Techniques

In this section, we detail the explainability techniques employed to generate various types of input feature explanations. As outlined in SS3.2, we categorize these techniques based on the method used for generating TokenEx, while TokenIntEx explanations stem from their bivariate variants, forming the basis for SpanIntEx explanations.

As denoted in Section SS2.1, \(x_{i}\) represents the \(i\)th token with instance \(x\). To better illustrate the explainability techniques below, we use \(F\) as the set of all tokens within this instance and \(S\) as the subset of \(F\). All explanations are obtained using model \(M\), which is omitted in the following notions for brevity. We use \(A_{\texttt{TokenExt}}(x_{i})\) to denote the attribution score generated by explainability technique \(A\) for the \(i\)th token \(x_{i}\), \(A_{\texttt{TokenIntEx}}(x_{i},x_{j})\) as the attribution score for token interaction \((x_{i},x_{j})\), \(A_{\texttt{TokenIntEx}}(x_{i}~{}|~{}x_{j})\) as the importance score of token \(x_{i}\) conditioned on \(x_{j}\) is present when the directed importance between tokens within \((x_{i},x_{j})\) is considered in some attribution techniques, \(A_{\texttt{SpanIntEx}}(span^{0}_{i},span^{1}_{i})\) as the attribution score for corresponding span interaction, where \(span^{0}_{i}=(x_{s},...,x_{s+l_{1}})\) is a span from part1 and \(span^{1}_{i}=(x_{t},...,x_{t+l_{2}})\) is a span from part2 of the input. Note that in Section SS2.1, we use \(a^{x}_{k}\) to denote the importance score of the \(k\)th most important explanation of instance \(x\); here, we only focus on the attribution scores of explanations without ranking them.

**Shapley**. For TokenEx, we employ the SHAP method to assign importance scores to each token within the input by removing each token separately and computing its removal effect on the model prediction with different subsets of other tokens presented to the model, following Lundberg and Lee (2017).

\[\begin{split} Shap_{\texttt{TokenEx}}(x_{i})&= \sum_{S\subseteq F\setminus\{x_{i}\}}\frac{|S|!(|F|-|S|-1)!}{|F|!}\\ &[f(S\cup\{x_{i}\})-f(S)]\end{split} \tag{21}\]

\(Shape_{\texttt{TokenEx}}(x_{i})\) denotes the importance score of token \(x_{i}\). As the calculation of \(Shap_{\texttt{TokenEx}}(x_{i})\) is computationally expensive, we utilize Kernel SHAP to approximate these Shapley values.

\begin{table}
\begin{tabular}{p{99.6pt} p{142.3pt} p{142.3pt}} \hline \hline \(D\) & \multicolumn{1}{c}{\(E\)} & **Size** \\ \hline  & & 549367 Train \\ SNLI & - & 9842 Dev \\  & & 9824 Test \\ e-SNLI & TokenEx & 549367 Train \\ _SpanEx-SNLI_ & TokenExt & 9842 Dev \\ \multirow{2}{*}{
\begin{tabular}{} \end{tabular} } & SpanIntEx \\  & TokenIntEx & 3865 Test \\ \hline FEVER & - & 145449 Train \\  & - & 9999 Dev \\  & SpanIntEx \\ _SpanEx-FEVER_ & TokenIntEx & 3206 Test \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overview of datasets SNLI Bowman et al. (2015), FEVER Thorne et al. (2018), _SpanEx_Ray Choudhury et al. (2023) and e-SNLI Camburu et al. (2018). _SpanEx_ contains instances from SNLI and FEVER, annotated with SpanIntEx explanations including token-level explanations (TokenIntEx explanations). e-SNLI contains instances from SNLI dataset, annotated with TokenEx explanations.

\begin{table}
\begin{tabular}{p{99.6pt} p{142.3pt} p{142.3pt}} \hline \hline \multirow{2}{*}{\(Model\)} & \multicolumn{2}{c}{**F1 score**} \\  & Dev & Test \\ \hline BERT-SNLI & 87.21 & 88.43 \\ BART-SNLI & 86.81 & 85.40 \\ \hline BERT-FFVER & 86.21 & 89.49 \\ BART-FEVER & 85.19 & 84.88 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The performance of our BERT-base and BART-base models fine-tuned on SNLI and FEVER, respectively, regarding F1 score(%).

 

For TokenIntEx, we first apply Bivariate Shapley (Masoomi et al., 2022) to assess the mutual importance scores between two tokens, which are from different parts of the input, within a token interaction, and then average these two mutual importance scores as importance score of this token interaction. Specifically, to compute the importance score of a token \(x_{i}\) conditioned on the presence of token \(x_{j}\), the sets of tokens \(S\) considered are limited to those containing token \(x_{j}\), while the impact of other sets of tokens influences the importance of \(x_{i}\) is ignored in this case. Thus, \(Shap_{\texttt{TokenIntEx}}(x_{i}\mid x_{j})\) can be calculated by:

\[\begin{split} Shap_{\texttt{TokenIntEx}}(x_{i}\mid x_{j})=& \sum_{s_{j}\in S\subseteq F\setminus\{x_{i}\}}\frac{|S|!(|F|-|S|-1)!}{|F|!}\\ &[f(S\cup\{x_{i}\})-f(S)]\end{split} \tag{22}\]

The importance score \(Shap_{\texttt{TokenIntEx}}(x_{i},x_{j})\) for token interaction \((x_{i},x_{j})\) are calculated by averaging \(Shap_{\texttt{TokenIntEx}}(x_{i}\mid x_{j})\) and \(Shap_{\texttt{TokenIntExt}}(x_{j}\mid x_{i})\). We also use Kernel Shapley to approximate the calculation of Bivariate Shapley value.

For SpanIntEx, we first apply the Louvain Community Detection algorithm Blondel et al. (2008) to extract span interactions and then average the importance scores of token interactions comprised in each span interaction as its importance score, following Ray Choudhury et al. (2023).

To extract span interactions, we first construct a directed bipartite graph for instance \(x\), by taking each token \(x_{i}\) from the input as node \(i\) and the mutual importance scores between each two tokens from different parts obtained above as the weights of directed edges connecting them. Louvain Community Detection algorithm is then applied to search for communities of nodes with dense intra-cluster and sparse inter-cluster relationships. With each community of nodes(tokens) \(S_{p}\) obtained, we can get one span interaction \((span^{0}_{p},span^{1}_{p})\), where the two spans consist of neighboring tokens from the part1 subset of this community \(S^{0}_{p}\), and the part2 subset of it \(S^{1}_{p}\) respectively.

Then we calculate the importance score of this span interaction by averaging the importance scores of all token interactions it comprises.

\[\begin{split} Shap_{\texttt{quadratic}}(span^{0}_{p},span^{1}_{ p})=&\\ &\xrightarrow[k_{j}S^{0}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{ j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[ k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{ p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j}S^{1}_{p]{ \xrightarrow[k_{j}S^{1}_{p]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{jS^{1}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{jS^{1}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{jS^{1}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{jS^{1}S_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{jS^{1}S_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p ]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{ \xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1_{p}]{ \xrightarrow[k_{j}S^{1}_{p}]{\xrightarrow[k_{j}S^{1_{p}]{\xrightarrow[k_{j 



## Appendix F Detailed Experiment Results

### Faithfulness

### Agreement with Human Annotation

There is a notable gap between interaction-level and token-level agreement scores. For example, in Table 3, the highest interaction-level agreement score for SpanIntEx explanations is 57.40%, while the highest token-level agreement score for SpanIntEx is 78.26%. A similar pattern is observed for TokenIntEx. This suggests that although SpanIntEx and TokenIntEx explanations align more with human reasoning than TokenEx explanations, pairing important tokens or spans into interactions that are plausible to humans remains challenging.

### Simulatability

Regarding insertion formats, for BERT models, text insertion (\(I_{Text}\)), which adds explanation text to the end of the input sequence, consistently outperforms symbol insertion (\(I_{Sym}\)), where symbols are added to the original input sequence, as shown in Figure 19. The reason why we can learn from the context of the sentence is that it is not possible to use the word representation instead of the word representation. In this case, the word representation is used to predict the correct answer



\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**E** & **Shap** & **Att** & **IG** & **Rand** \\ \hline  & \multicolumn{4}{c}{Interaction level agreement} \\ SpanIntEx & **19.92** & **28.12** & **27.45** & **19.33** \\ TokenIntEx & 3.96 & 10.27 & 21.30 & 10.23 \\ TokenEx & **-** & **-** & **-** & **-** \\  & \multicolumn{4}{c}{Token level agreement} \\ SpanIntEx & **66.95** & **68.71** & **72.24** & **67.5** \\ TokenIntEx & 66.90 & 67.29 & 70.50 & 65.86 \\ TokenEx & 58.07 & 56.88 & 61.07 & 63.10 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Human Annotation Agreement Results (see §2.3) on the FEVER dataset when explanations are generated based on BERT. The rest of the settings are the same as Table 3.

Figure 4: Unified Comprehensiveness and Sufficiency of three types of feature attribution explanations on SNLI and FEVER datasets using the BART model. Subfigures (a) and (c) show Unified Comprehensiveness results, while (b) and (d) show Unified Sufficiency results. Explanations are generated by Shapley (Shap), Attention (Att), and Integrated Gradients (IG). Randomly selected span pairs, token pairs, and tokens are baselines corresponding to explanation type SpanIntEx TokenIntEx and TokenEx and form the group Random baseline (Rand). We set \(k=3\) for top span interactions and adjust token counts as per section SS2.2, ensuring the random baseline matches the average token count of the top \(k\) span interactions.



in Tables 7 and 8. However, the opposite effect is observed for BART models, as shown in Tables 9 and 10. This indicates that simulatability results are sensitive to the explanation insertion form, highlighting the need for consistency in insertion form when comparing different explanation types.

### Complexity

\begin{table}
\begin{tabular}{l c r r r} \hline \hline
**E** & \multicolumn{2}{c}{**Shape**} & \multicolumn{2}{c}{**Att**} & \multicolumn{2}{c}{**IG**} \\  & **SF** & **RSF** & **SF** & **RF** & **SRF** & **SF** & **RSF** \\ \hline SNLI & SpanIntEx & **87.9** & **3.2** & **86.7** & **2.0** & **88.9** & **4.2** \\  & TokenIntExt & 86.6 & 1.9 & 85.3 & 0.6 & 85.3 & 1.1 \\  & TokenExt & 87.4 & 2.7 & 85.7 & 1.0 & 86.0 & 1.3 \\ \hline FEVER & SpanIntEx & 83.9 & 3.8 & **85.3** & **5.2** & **85.2** & **5.1** \\  & TokenIntExt & 82.7 & 2.6 & 84.0 & **3.9** & 84.4 & 4.3 \\  & TokenExt & **84.0** & **3.9** & 82.3 & 2.2 & 81.8 & 1.7 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Simulatability results on SNLI and FEVER with BERT as the model used for all explanations \(E\in\texttt{SpanIntEx,TokenIntEx,TokenExt generation with attribution method **Shapley**, **Attention**, and **Integrated Gradients** respectively. Note that insertion form \(I_{Sym}\) is adopted for combining the explanations and the original input sequence for agent model \(AM_{E}\), as depicted in §2.4. The agent models used for baseline \(AM_{O}\), trained without explanations, have simulation F1 scores, as denoted in §2.4, of 84.7% and 80.0% on test set shared with other agent models \(AM_{E}\), as denoted in §2.4. We set \(k=1\) for top SpanIntEx and calculated the number of top TokenIntEx and TokenEx accordingly as stated in §2.2. The largest increases are highlighted in bold for each dataset with the identical attribution method.

\begin{table}
\begin{tabular}{l l r r r r r r} \hline \hline
**D** & **E** & \multicolumn{2}{c}{**Shape**} & \multicolumn{2}{c}{**Att**} & \multicolumn{2}{c}{**IG**} \\  & **SF** & **RSF** & **SF** & **RSF** & **SF** & **RF** \\ \hline SNLI & SpanIntEx & **86.8** & **6.9** & **85.0** & **5.1** & 84.3 & **4.4** \\  & TokenIntExt & 81.2 & 1.3 & 82.6 & 2.7 & **81.6** & 1.7 \\  & TokenResx & 83.3 & 3.4 & 83.5 & 3.9 & 82.2 & 2.3 \\ \hline FEVER & SpanIntEx & **78.2** & **4.8** & **75.3** & **1.9** & **74.6** & **1.2** \\  & TokenInExt & 74.8 & 1.4 & 74.1 & 0.7 & 73.9 & 0.5 \\  & TokenExt & 77.6 & 4.2 & 74.9 & 1.5 & 73.6 & 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Simultability results on SNLI and FEVER with BART as the model used for all input feature explanation generation respectively. Note that insertion form \(I_{Text}\) is adopted for combining the explanations and the original input sequence for agent model \(AM_{E}\), as depicted in §2.4. The base agent models \(AM_{O}\) trained without explanations have the simulation f1 scores of 79.9% and 73.4%, respectively. The other setting is the same as Table 7

\begin{table}
\begin{tabular}{l l r r r r r} \hline \hline
**D** & **E** & \multicolumn{2}{c}{**Shape**} & \multicolumn{2}{c}{**Att**} & \multicolumn{2}{c}{**IG**} \\  & **SF** & **RSF** & **SF** & **RSF** & **SF** & **RSF** \\ \hline SNLI & SpanIntEx & **86.8** & **6.9** & **85.0** & **5.1** & 84.3 & **4.4** \\  & TokenIntExt & 81.2 & 1.3 & 82.6 & 2.7 & **81.6** & 1.7 \\  & TokenResx & 83.3 & 3.4 & 83.5 & 3.9 & 82.2 & 2.3 \\ \hline FEVER & SpanIntEx & **78.2** & **4.8** & **75.3** & **1.9** & **74.6** & **1.2** \\  & TokenIntExt & 74.8 & 1.4 & 74.1 & 0.7 & 73.9 & 0.5 \\  & TokenExt & 77.6 & 4.2 & 74.9 & 1.5 & 73.6 & 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Semulatability results on SNLI and FEVER with BERT as the model used for all explanations \(E\in\texttt{SpanIntEx,TokenIntEx,TokenExt generation with attribution method **Shapley**, **Attention**, and **Integrated Gradients** respectively. Note that insertion form \(I_{Sym}\) is adopted for combining the explanations and the original input sequence for agent model \(AM_{E}\), as depicted in §2.4. The agent models used for baseline \(AM_{O}\), trained without explanations, have simulation F1 scores, as denoted in §2.4. We set \(k=1\) for top SpanIntEx and calculated the number of top TokenIntEx and TokenEx accordingly as stated in §2.2. The largest increases are highlighted in bold for each dataset with the identical attribution method.

\begin{table}
\begin{tabular}{l l l r r r r r} \hline \hline \multirow{2}{*}{**D**} & \multirow{2}{*}{**E**} & \multicolumn{2}{c}{**Shape**} & \multicolumn{2}{c}{**Att**} & \multicolumn{2}{c}{**IG**} \\  & **SF** & **RSF** & **SF** & **RSF** & **SF** & **RSF** \\ \hline SNLI & SpanIntEx & **87.8** & **7.9** & **87.3** & **7.4** & **86.5** & **6.6** \\  & TokenIntExt & 83.5 & 3.6 & 84.2 & 4.3 & 84.6 & 4.7 \\  & TokenExt & **88.2** & **8.3** & **81.4** & 1.5 & 85.8 & 5.9 \\ \hline FEVER & SpanIntEx & **80.6** & 7.2 & **76.1** & **2.7** & **75.2** & **1.8** \\  & TokenIntEx & 78.9 & 5.5 & 75.9 & 2.5 & 74.7 & 1.3 \\  & TokenExt & 80.1 & **6.7** & 75.0 & 1.6 & 74.1 & 0.7 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Simulatability results on SNLI and FEVER with BART as the model used for all input feature explanation generation. Note that insertion form \(I_{Sym}\) is adopted for combining the explanations and the original input sequence for agent model \(AM_{E}\), as depicted in §2.4. The agent models used for baseline \(AM_{O}\), which are trained without explanations, have simulation F1 scores of 84.7% and 80.0% on the test sets shared with agent model \(AM_{E}\). The other setting is the same as Table 7 



\begin{table}
\begin{tabular}{l l c c c c c} \hline

- \\  & TokenExt & 2.08 & 1.64 & **1.91** &
- \\ \hline FEVER & SpanIntEx & 2.78 & **2.22** & 2.90 & 2.98 & 3.18 \\  & TokenIntEx & 3.12 & 3.07 & 3.15 &
- \\  & TokenExc & **2.76** & **2.22** & **2.57** &
- \\ \hline \end{tabular}
\end{table}
Table 11: Complexity results on SNLI and FEVER datasets for three types of explanations generated by different attribution methods based on BERT model. The **Random** baseline represents the complexity score obtained by randomly generated scores in the range [0,1], ensuring the same number of scores as the number of explanations used. The **Upperbound** is calculated by setting all the attribution scores to the same value while ensuring the same number of scores as the number of explanations used. The lowest complexity score for each specific explanation type compared is highlighted in bold when the explanations are generated by each attribution method.

\begin{table}
\begin{tabular}{c l c c c c} \hline

- \\  & TokenExt & **1.86** & 1.95 & 1.94 &
- \\ \hline FEVER & SpanIntEx & **2.73** & **3.03** & **2.38** & 3.13 & 3.38 \\  & TokenIntEx & 3.30 & 3.36 & 2.93 &
- \\  & TokenEx & 2.82 & 3.07 & 3.19 &
- \\ \hline \end{tabular}
\end{table}
Table 12: Complexity results on SNLI and FEVER datasets for three types of explanations generated by different attribution methods based on BART model. The other settings are the same as Table 11.