IvRA: A Framework to Enhance Attention-Based Explanations for Language Models with Interpretability-Driven Training

Authors: sean  xie
soroush  vosoughi
saeed  hassanpour
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-20 10:57:37

Abstract: Attention has long served as a foundational technique for generating explanations. With the recent developments made in Explainable AI (XAI), the multi-faceted nature of interpretability has become more apparent.  Can attention, as an explanation method, be adapted to meet the diverse needs that our expanded understanding of interpretability demands?  In this work, we aim to address this question by introducing \texttt{IvRA}, a framework designed to directly train a language model's attention distribution through regularization to produce attribution explanations that align with interpretability criteria such as simulatability, faithfulness, and consistency. Our extensive experimental analysis demonstrates that \texttt{IvRA} outperforms existing methods in guiding language models to generate explanations that are simulatable, faithful, and consistent, in tandem with their predictions. Furthermore, we perform ablation studies to verify the robustness of \texttt{IvRA} across various experimental settings and to shed light on the interactions among different interpretability criteria.

Main paper: 
 introduction: 
 The rapid adoption of language models (Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019; Achiam et al., 2023) in recent years has sparked an escalating interest in enhancing model interpretability. This has given rise to the burgeoning field of Explainable AI (XAI), which has devised various methods to increase model interpretability (Shrikumar et al., 2016; Ribeiro et al., 2016; Shrikumar et al., 2017) . However, an universal definition for the term "interpretability" remains elusive in the research community (Lipton, 2016) . Interpretability assessment has primarily leaned on criteria tailored for different purposes that fall under the broad umbrella of the term "Interpretability". Some of the most popular criteria are simulatability (Doshi-Velez and Kim, 2017) , faithfulness (Jacovi and Goldberg, 2020; Ribeiro et al., 2016) , and consistency (Serrano and Smith, 2019; Jain and Wallace, 2019) . Simulatability measures whether a model's behavior is comprehensible enough for a human or another ML model to predict its outputs on unseen data, aligning with the objective of conveying the model's underlying mechanics to humans. Faithfulness measures the extent to which an explanation reflects the actual decision-making process of the model. Consistency assesses the explanation method's stability across varying input data, favoring explanations that remain similar for similar inputs and reflect input changes that lead to inconsistent outputs.The utility of attention for generating saliency explanations is widely recognized (Deng et al., 2017; Wiegreffe and Pinter, 2019; Vashishth et al., 2019; Martins et al., 2020) , notwithstanding initial doubts regarding the faithfulness and consistency of attention mechanisms (Serrano and Smith, 2019; Jain and Wallace, 2019) . Past works (Atanasova et al., 2020; Sun et al., 2024) that have benchmarked existing attention-based text attribution methods along interpretability criteria such as simulatability, faithfulness and consistency do not explore the possibility of directly training attention distributions to become more interpretable with regard to a criterion. On the other hand, works that do train their explanations to become more interpretable via some criterion either only focus on a small subset of criteria (Pruthi et al., 2022; Neely et al., 2021; Fernandes et al., 2022) and/or do not use attention as a technique (Chan et al., 2022b) , instead relying on a separate model as rationale extractor. In this work, we focus on developing an attentionbased explanation framework that enables a language model (LM) to produce explanations that align more closely with interpretability criteria. We summarize our contribution below:This paper introduces a novel framework-Interpretability via Regularized Attention along with saliency explanations ri. Different loss functions (L) corresponding to different criteria then take ri and ŷi as input to propagate loss back to the model. For simulatability ( §2.2.1), S denotes a student model, with ŷ * i and r * i denoting the output logit and explanations of S, respectively. For comprehensiveness and sufficiency ( §2.2.2), t k i denotes tokens recognized by ri with attention scores in the top k% of tokens. For consistency ( §2.2.3), xj represents another example within the dataset.(IvRA) that parameterizes attention distributions in LMs to produce attention-based attribution explanations (r i ) alongside their outputs. The operation of our framework is illustrated in Fig. 1 . During training, IvRA uses specialized loss functions for each criterion to propagate losses to a set of weights within the interpretable attention modules of the LM (Fig. 2 ) in order to optimize the attention-based explanations for each criterion. We empirically verify IvRA's effectiveness in terms of simulatability, faithfulness (comprehensiveness and sufficiency), and consistency 1 on three NLP tasks: Text Classification, Entailment Inference, and Question-Answering. Our results demonstrate that IvRA effectively enhances model interpretability, guiding LMs to generate simulatable, faithful, and consistent explanations for their decisions.2 Background and Methodology 2 Since our work seeks to integrate various criteria of interpretability for training, the amount of related literature needed to detail our methodology for each criterion is extensive. To conserve space, we included only key works that we think are crucial to understanding our contribution. See §A for additional related works. 1 We include a discussion as well as a study on the criterion of plausibility using human annotated rationales in §F 2 We share our source code at github.com/yx131/IvRA-Interpretability-Driven-Training 
interpretable attention module: 
 Given an input sequence x i of length L, an attention head h processes x i through linear projections to yield Vaswani et al., 2017) . Recent research has highlighted the effectiveness of attention-based interpretation methods in enhancing the interpretability of language models (Treviso and Martins, 2020; Kobayashi et al., 2020) . Furthermore, because the attention mechanism is instrinsic to the LM, attention-based explanations possess the advantage of not requiring a separate procedure that is decoupled from the decision-making process, in contrast to post-hoc methods (Shrikumar et al., 2016; Du et al., 2019) . Building on this foundation, our work seeks to cultivate more interpretable attention-based explanations by parameterizing the multi-head attention layers within a LM and optimizing the parameterized attention distribution in accordance with specified interpretability objectives. In more detail, for the query and key projections Q h i and K h i of head h, we compute normalized feature-wise distributions as shown in equations 1 and 2:Q h i and K h i , thereby comput- ing a normalized distribution Att h i ∈ ∆ L L-1 = softmax{Q h i (K h i ) T } (EQUATIONEQUATIONFor each layer ℓ, we compute the distribution Ψ ℓ over the attention heads using equation 3, where qh (i,n) represents the portion of normalized query projection in Qh i corresponding to token n in x i and λ h is a trainable coefficient for each head.Ψ ℓ = NORM λ h ψ h ℓ H ℓ h=1 (3)whereEQUATIONLastly, to determine the aggregated attention distribution r i , we sum the normalized distribution of all Ψ ℓ 's, as defined in equation 5, where η ℓ is a coefficient for each layer:r i = SUM NORM η ℓ Ψ ℓ L ℓ=1 ∈ ∆ L-1(5) The design of our interpretable attention module, as outlined above, serves dual purposes:1) Aggregation for Salience: In order to derive r i ∈ ∆ L-1 , it's necessary to aggregate the attention distributions across layers and heads. This is because the multi-head attention distribution is a matrix of dimension L × L. Common interpretability measures such as faithfulness and consistency are only applicable to 1-dimensional saliency scores. In the absence of IvRA, it's common to either use the attention heads in the final layer or the mean attention distribution across all layers in the model for layer aggregation (Fomicheva et al., 2020) . 2) Optimization for Interpretability: Our attention module facilitates systematic aggregation through learnable parameters and allows for hyperparameter experimentation, such as the normalizing function for NORM.Our approach to regularizing attention is similar to the attention-based explainer used in Fernandes et al. (2022) to elicit explanations for a studentteacher setup (SMaT). However, the SMaT explainer is relatively coarse, as it only learns weights for head selection. Fernandes et al. (2022) did not explore the effectiveness of feature and layer selection and confined their interpretability evaluation to just one criterion (simulatability). In contrast, our framework not only seeks to employ an attentionbased explainer that integrates four criteria, but also employs the parameterization of attention at the feature, head, and layer levels. Our detailed ablation study in §C, demonstrates that parameterization at all levels is the most effective strategy.We formulate our interpretability objectives as distinct loss functions: simulatability, faithfulness (comprehensiveness and sufficiency), and consistency. Given a classification task with C classes, we denote a dataset as D = {(x i , y i )} N i=1 consisting of N samples, with each x i as an input sequence of length L and y i representing the ground truth label. We denote the output logits of model M for input x i as M(x i ) ∈ R C , and the predicted class as ỹi = argmax(ŷ i ).Simulatability refers to the capacity of a model to generate decisions replicable by a human observer (Doshi-Velez and Kim, 2017; Lipton, 2016) . This interpretability measure proves beneficial by quantifying the efficacy of model behavior communication (Treviso and Martins, 2020) . Simulatability is evaluated both through manual annotations (Hase and Bansal, 2020) and automated methods (Pruthi et al., 2022) . In this work, we adopt the automated approach outlined by Pruthi et al. (2022) and extended by Fernandes et al. (2022) . Here, simulatability is gauged as the extent to which a student model can replicate the teacher model's predictions given a saliency explanation of the teacher's input. We employ this simulatability evaluation construct to enhance the simulatability of our primary (i.e., teacher) model M. To this end, we train M to generate an explanation r i , which we use in the training of a student model S to replicate ỹi . Let the output logits ŷ * i = S(x i ) of S for x i , and let r * i be the attention module output of S for y * i , we define simulatability accuracy on a dataset D as shown in equation 6.SIM(M, S, D) = 1 |D| (x i ,y i )∈D 1{ỹ i = argmax(ŷ * i )} (6)Considering ỹi,c and ŷ * i,c as values of ỹi and ŷ * i for class c ∈ C, respectively, we define simulatability loss for a single instance as the sum of crossentropy loss between M 's predictions and S's predictions and the Kullback-Leibler divergence loss between M ′ s and S's attention outputs (eq. 7).L SIM = c∈C ỹi,c log(ŷ * i,c ) + KLDiv(r i , r * i ) (7)It's crucial to note that simulatability should be evaluated under a constrained setting, wherein the student's learning capability is intentionally limited. Two frequently employed strategies are: 1) simplifying the student model architecture, or 2) utilizing a distinct data subset for simulatability evaluation, different from that used to train the teacher (Fernandes et al., 2022) . We adopt the second strategy in our experiments. For additional information on simulatability, please refer to §B.1 in the appendix.Faithfulness represents the extent to which an explanation accurately captures the underlying reasoning process of model M in predicting ỹi (Jacovi and Goldberg, 2020) . To gauge the faithfulness of our explanations for M, we examine the impact of salient tokens identified by our extracted explanation r i on ŷi using comprehensiveness and sufficiency (DeYoung et al., 2019) . We define t i as the sequence of tokens obtained by binarizing r i over a k% threshold, i.e., t k i ∈ {0, 1} L = 1 if r l i is in in the top k% of salient scores 0 else L l=1 Given p ỹi (x i ) as M's confidence probability for ỹi with input x i , we compute comprehensiveness as the difference in p ỹi with t k i removed from the input (eq. 8). In essence, if tokens identified by t k i are comprehensive, their exclusion from the input should decrease the predicted probability of M for ỹi . Similarly, we determine sufficiency by calculating the difference in p ỹi when only retaining the identified tokens in t k i (eq. 9). In this case, tokens in t k i are deemed sufficient if keeping them as the sole input elements does not reduce M's predicted probability for ỹi .EQUATIONIn our experiments, we compute COMP and SUFF for each individual k ∈ {1, 5, 10, 20, 50}. We calculate the final COMP and SUFF values as the area-over-precision curve (AOPC) for all k values in the set (DeYoung et al., 2019; Chan et al., 2022b) . Furthermore, we define comprehensiveness loss for a single instance x i as the difference between cross-entropy losses when using x i as input versus x i \t k i as input. This is lower-bounded by a margin µ comp to prevent exceedingly high negative losses (eq. 10). Likewise, we define sufficiency loss for a single instance as the difference between cross-entropy losses when using t k i as input and x i as input (eq. 11), lower-bounded by µ suff . For additional details on faithfulness loss, see B.3.EQUATIONConsistency refers to the ability of explanation methods to produce similar reasoning paths for similar instances of data (Robnik-Šikonja and Bohanec, 2018; Serrano and Smith, 2019; Jain and Wallace, 2019) . Consequently, if two instances x i and x j are perceived as similar by M, then r i and r j , the salient scores provided by IvRA, should also exhibit similarity. We note that our focus is on the similarity of interpretations in r i and r j , not on the similarity of outcomes. Identical predictions do not necessarily imply analogous model reasoning, which is the essence of our interest in consistency. We derive H i , the aggregate hidden state for x i , by averaging the hidden states in M for x i across all layers. This approach for obtaining input representation for consistency calculation has been effectively demonstrated by Atanasova et al. (2020) . Let Dist be a distance function; we compute consistency for a dataset D and model M by measuring Spearman's ρ between similarities in aggregate hidden states (H i and H j ) and similarities in attention explanations (r i and r j ) as detailed in eq. 12. We further define our loss function for consistency as the Kullback-Leibler divergence loss between explanations for two samples, weighted by the similarity between the samples' aggregate hidden states. (eq. 13). For additional information the consistency loss function, see §B.4.EQUATIONL CONS = 1 Dist(H i , H j ) + ϵ 0 • KLDiv r i , r j (13)The performance of our proposed method is primarily assessed on specific datasets and tasks. 2020) benchmarked the consistency of explainer on several datasets and found that attention-based explainer generally outperformed gradient-based explainer in terms of consistency. Neely et al. (2021) shows that without comparison against ground-truth explanations (often provided by human-labeled rationales (DeYoung et al., 2019) ), it is difficult to establish an objective better/worse explainer. Even more recent work (Joshi et al., 2022; Chan et al., 2022a) show that it is difficult to align attention networks' output with human rationales (plausibiltiy). In terms of the architecture used to evaluate interpretability, Guerreiro and Martins (2021), Jacovi and Goldberg (2021) and Ismail et al. (2021) take the approach of building model decisions upon aligned rationales, but focus on task performance and evaluate their work only on a subset of the interpretability objectives. 
experiments: 
 In order to evaluate our framework's effectiveness at producing simulatable, faithful and consistent explanations, we train three transformer-based language models with IvRA: Electra (Clark, 2020), Llama-2-7b (Touvron et al., 2023) , and GPT-2 (medium) (Radford et al., 2019) ) on three NLP Tasks: Sentiment Classification, Entailment Inference and Question-Answering, with the following datasets, respectively: IMDb (Maas et al., 2011 ), SNLI(Bowman et al., 2015 ), and SQuAD (Rajpurkar et al., 2016) . In the main paper, we present results of IvRA using ELECTRA as the base language model, with further results using Llama-2 and GPT-2 provided in I. In §3.1, 3.2, 3.3 we report results obtained when training models for each of the interpretability criteria separately. We then delve into mixed-criteria training in §3.4 and examine the IvRA's effect on downstream accuracy in §3.5.In order to assess the relative effectiveness of IvRA compared to other explanation methods, we conduct experiments using other methods on the same datasets and compare the extent to which each interpretability objective is achieved. We report the mean and standard error values from 5 runs for each experiment setting. The explanation methods that were employed in our experiments are: In Table 1 we show the simulatability accuracy (eq. 6) of our experiments. We observe that, overall, IvRA is more capable of producing simulatable explanations than other methods. We found that the gradient-based explanation methods and LIME did not consistently outperform the common attentionpooling techniques in terms of to simulatability.In addition, we see that using Sparsemax as the normalizing function leads to more simulatable explanations than Softmax. When normalizing with Softmax, all elements are guaranteed a representation in the distribution, however minute it may be. This leads to all tokens always having some weight in the explanation. Normalizing with Sparsemax leads to tokens having no weight at all in the explanation, thus producing more sparse and more concise explanations. We show example explanations from LIME (Ribeiro et al., 2016) , and IvRA when normalizing with Softmax and Sparsemax in Fig. 3 , where sparser parameters is observable at all levels when training with Sparsemax than training with Softmax. In addition, we observe IvRA is able to produce much concise explanations than LIME. This is intuitive when considered from a human standpoint, as simple and concise explanations are easier to follow along than long-winded explanations.We show the comprehensiveness obtained in our experiments in Table 2 and sufficiency scores obtained in our experiments in Table 3 . We observe that IvRA is able to produce more faithful explanations than methods. We further note that comprehensiveness is the only interpretability criterion in our experiments for which IvRA-Softmax consistently outperformed IvRA-Sparsemax. We hypothesize that this may be due to the fact that generating explanations with weights distributed over a large number of tokens proves advantageous when assessing comprehensiveness-typically, the more words included in an explanation, the more comprehensive it is. By design, Softmax excels in pro-ducing explanations that highlight a greater number of tokens. On the other hand, we note that Softmax tends to underperform when used for training aimed at sufficiency. For additional discussion and analysis on the number of words identified and its relationship with faithfulness, please see §D. We present our consistency results in Table 4 . While IvRA clearly outperforms other explainabil- We can observe that IvRA-sparsemax is able to produce explanations that are more consistent (highlightin words that are common to both instances of data with similar degrees of emphasis) than IG. For example, the word "instruments" is not highlighted in both instances by IG, whereas IvRA highlights the word with similar emphases in both cases. ity methods, we do not observe a clear winner between IvRA-Softmax and IvRA-Sparsemax. In particular, we observe overlapping IQR's between SMaT, IvRA-Softmax and IvRA-Sparsemax. In Fig. 4 , we show the explanations produced by Integrated Gradients and IvRA-sparsemax's for two instances of data from SQuAD with similar semantics. In the example, both questions inquire about collaborators with whom Kanye West has previously worked on his album. While IG's explanation is sporadic and pattern-less (e.g. 'inspired' having completely opposite color/weight of contribution in the two examples), we observe consistent highlighting of keywords by IvRA-Sparsemax in both examples.To what extent does the simulatability training of a model correlate with its comprehensiveness? Can a model trained to produce sufficient explanations also be consistent? These questions arise from the multifaceted nature of IvRA, which aims to accommodate various interpretability criteria. While earlier sections demonstrate IvRA's superiority over existing methods when trained individually for each criterion, this section explores the efficacy of mixed-criteria training. We train IvRA with dif-ferent combinations of interpretability losses (Eqn. 7, 10, 11, 13) enabled. We then compare the results of these models against models that were trained solely using each individual criterion. against a model trained solely on a. Our findings are presented in a heatmap shown in Fig. 5 . We observe underperformance (negative ARG) acrossthe-board i.e., we observe that models trained with multiple criteria losses enabled achieve each criterion less than models trained with a sole focus on the same criterion. While this may initially seem discouraging, we find the results to be intuitive-as we demonstrate, models trained to be more simulatable do not naturally exhibit greater comprehensiveness than models originally trained with comprehensiveness as the primary goal. Moreover, our results suggest that different interpretability criteria's parameters are at odds with each other and satisfying one criterion may not satisfy others. We observe that, particularly, training for consistency has the greatest adverse effect on all other criteria. We also observe the least amount of decrease in performance between models trained for simulatability and sufficiency and vice versa. We attribute this to our earlier discussions in §3 where we find that conciser explanations are helpful for both simulatability and sufficiency.In this section, we examine IvRA's influence on model performance. We specifically aim to assess how each of the four interpretability criteria  
conclusion: 
 We introduce IvRA, a paramerterized attention module for directly training a LM's attention distribution to produce explanations that align with interpretability criteria. We test IvRA's effectiveness at producing explanations that are simulatable, faithful (comprehensive and sufficient) and consistent using multiple LMs and on multiple NLP tasks. We perform ablation experiments to reveal insights on the interplay between different interpretability criteria and to assess IvRA's influence on downstream accuracy. Our findings demonstrate that IvRA's attention-based explanations is robust under various settings and empowers LMs to generate explanations that better align with interpretability criteria.We summarize the main limitations of our work below. While we acknowledge the potential shortcomings of this work in these areas, we also hope to inspire future works of research in these areas to address and improve upon our deficiencies.1. Reliance on Existing Interpretability Metrics: Our method builds upon existing interpretability metrics like faithfulness, consistency, and simulatability. Despite their widespread use, these metrics may not fully capture the complexity of interpretability in machine learning models. Developing more comprehensive and robust metrics could potentially enhance our approach and lead to better results 
 
 Appendix: IvRA as a an explanation technique that utilizes regularized attention has the advantage over gradient and perturbation-based methods (Ribeiro et al., 2016; Shrikumar et al., 2017) in that the process of explaining the output is intrinsic to the model and not decoupled from the prediction process. 2021) have only be shown to be effective at enhancing model interpretability for a subset of criteria. Finally, our work's scope is similar to that of Sun et al. (2024) , as both attempt to create a comprehensive framework for evaluating different interpretability criteria. However, while their work emphasizes diagnosing the properties of existing interpretability techniques, ours is focused on methods to train models to acquire these properties. Furthermore, we affirm the validity of our approach by highlighting that several of our interpretability criteria are closely aligned with those defined by Sun et al. (2024) .We use a fine-tuned model for M trained on the dataset for the task and an unfine-tuned model for S that has not been exposed to the dataset. Additionally, M's explanations (r i 's) are withheld from the student S during testing to prevent information leakage (Pruthi et al., 2022) .Optimizing the M's attention parameters θ(M) for the simulatability of a separate, student model S is a non-trivial process. We in this work take the scaffoled approach for optimizing the parameters of introduced in by Fernandes et al. (2022) . Specifically, let d ∈ D be a batch of data, we frame optimizing IvRA's attention weights as a bi-level optimization problem where eq. 14 updates θ(S) (S's parameters) based on model outputs (see eq. 7) and eq. 15 updates M's parameters based on how well S can simulate M, with newly updated parameters. We take a single optimization step to calculate the gradient for S (eq. 14). After updating S with the gradient, we take an additional gradient step but only use this gradient to update parameters of M, and not S (eq. 15). For even more specific details on scaffolded simulatability, we refer the reader to pilot updates by Zhou et al. (2021) .EQUATIONIn addition to limiting the model weights of M and S as described in §2.2.1, we further constrain the amount of data the student S is trained on. Specifically, while the full training set for IMDb, SNLI and SQuAD were used to finetune and train M, we only use 20% of the available testing set to train the student, which yields 5000 samples, 2000 samples and 2000 samples for IMDb, SNLI and SQuAD, respectively.Intuitively, the entropy (with respect to the output class ỹi ) can be higher when calculated with t k i removed than when calculated with the entirety of x i , i.e. -ỹ i log(M(x i \t k i )) ≫ -ỹ i log(M(x i )). Without bounding by µ COMP , eq. 10 can yield large, negative losses. While this can analogously happen for eq. 11, there exists alternative loss functions (see below). We experimented with µ COMP ∈ {0.1, 0.2, ..., 1} and µ SUFF ∈ {0.1, 0.2, ...1} and the reported results in Table 2 and Table 3 are for µ COMP = 1 and µ SUFF = 0.1, respectively.Apart from the loss function outlined in §2.2.2, we experiment with two additional sufficiency loss functions for sufficiency. Critically, we note that eq. 16 relies on the assumption that M is not able to make more accurate predictions when using only a subset of the sequence (t k i ) as its input. We also note here that while 11 and 16 are computed with respect to the output class ỹi , 17 computes the KL divergence loss over distributions. Similar to (Chan et al., 2022b) , we found that all three loss functions can be used to train IvRA for sufficiency, although we decided to report 11 in the main paper as it's more general and in conformity with eq. 10.EQUATIONEQUATIONFor simplicity and clarity, we defined eq. 12 and eq. 13 in the main paper for two examples x i and x j .In practice, both CONS and L CONS are calculated for every pair of samples within each batch during training. We report the consistency for a dataset by averaging the consistency across batches. i.e. CONS we calculate:EQUATIONFor batch loss L CONS during training, we calculate the following:EQUATIONAs a result, we note here that training for pair-wise consistency can be costly in terms of time. For more analysis on computational cost of IvRA see §G.We report results for using L2 distance as our Dist function. Although we experimented with L1 distance function, we found using L2 distance in general led to better performance. We note here that the Dist function in eq. 12 calculates pair-wise distance at the token level, whereas the Dist function for loss calculate (eq. 13) is the p-2 norm of the difference between H i and H j .Training with a focus on consistent reasoning shares similarities with the process of clustering similar examples together. To that end, we also experimented with a clustering loss for L CONS that is similar to the loss function for learning associations between examples in Chen et al. ( 2019) and Das et al. (2022) . Specifically, let χ and be the set of samples in d that belong belong to the same output class as x i , let γ be the set of samples in d that do not belong to the same output class as x i i.e. χ = {x j s.t. y j = y i | ∀x j ∈ d} and γ = {x j s.t. y j ̸ = y i | ∀x j ∈ d}, we define an alternative clustering loss for consistency training as:EQUATIONIntuitively, we try to train for consistency via minimizing the distance of r i and r j 's that are explanations of examples with the same class as x i and maximizing the distance between r i and r j 's that are explanations of examples with a different class than x i . In practice, we found this loss function to perform worse than eq. 13 both in terms of consistency as well as time.C Effect of Feature/Head/Layer Selection D Important Tokens IdentifiedIn Fig. 7 , we conduct an analysis of the number of important tokens in the output of different explainers. Every token receives some weight in the saliency outputs by Integrated Gradients (IG), LIME, and Softmax, although often minute. To find impactful tokens, we perform min-max normalizing on the saliency outputs of these explainers and find the number of tokens (as a percentage of the input's length) that score above thresholds in the set Z = {0.1, 0.2, ..., 0.9}. i.e. a token is important if its normalized saliency is higher than z ∈ Z.We then calculate the area-over-precision curve (DeYoung et al., 2019; Xie et al., 2023) ∀z ∈ Z to obtain the AOPC of important words identified. We find that, while the number of important remains roughly the same for IvRA-Softmax when trained on both COMP. and SUFF, IvRA-Sparsemax, in general, identifies fewer tokens when trained for SUFF than when trained for COMP.We hypothesize that the parameters learned by IvRA are transferable between datasets for the same task. To verify our hypothesis, we take models that were trained on IMDb and SNLI, denoted as M I and M S , respectively, and apply them on SST2 and MNLI from GLUE (Wang et al., 2018) . In order to gauge the transferability, we directly train another set of models on SST2 and MNLI, denoted as M * I and M * S , respectively. We then compare the results of M * I and M * S against the results of M I and M S using ARG. We report the ARG of I and M * S over M I and M S in percentages in Fig. 8 . A higher ARG means a greater difference in scores for each interpretability criterion between the directly-trained models and the models with transferred parameters. We observe that the parameters trained for SIM transferred the best between datasets, followed by SUFF and COMP. We also note that parameters trained for CONS did not transfer well, relatively speaking. We conjecture that, although the task for both datasets are the same, the difference in the semantics of samples between two datasets can vary widely, thus making it difficult for CONS parameters learned on one dataset to be applied to another.Plausibility is defined as how convincing are explanations to humans (Jacovi and Goldberg, 2020) . Recent studies have assessed plausibility by measuring the overlap between generated rationales (a set of tokens) and groundtruth labels (Sun et al., 2024; Xie et al., 2022a) . However, this approach not only highlights that plausibility is inherently human judgement-based and challenging to train for-requiring a distinct set of labeled groundtruth data for each dataset (Chan et al., 2022b) -but also that recent research suggests attributions might be ineffective in producing plausible outputs altogether (Ferreira et al., 2024) . Consequently, in this work, we have chosen not to use plausibility as a training objective. Nonetheless, we include a study on the plausibility of explanations generated by IvRA models trained under alternative criteria here.Similarly to simulatability, plausibility derives its advantage and utility as an evaluation metric from its alignment with human intuition. Therefore, we in this study conduct a plausibility study exploring which of IvRA models provides the most plausible explanations. We utilize the annotated MovieReviews dataset (DeYoung et al., 2019) which consists of human-labeled rationales for movie review sentiment classification. The rationales are in the form of tokens that have binary labels 0 and 1 that indicate their presence in the rationale. For each of the explanation method in Table 6, we calculate the plausibility score as the AUC ROC of tokens identified against salient tokens labeled by human annotators. We found the explanations generated by IvRA to be the most plausible i.e., aligning the most with human-generated rationales in terms of tokens identified. More specifically, we find that the explanations learned for the criterion of simulatability are the most plausible overall, followed by sufficiency, comprehensiveness and consistency. This study, in conjunction with our findings in §3.1, show us that sparser explanations that can better target keywords are deemed more intuitive and practical by both models and humans alike. Additionally, we observe , apart from IvRA(Cons.), explanations produced by models incorporating learnable interpretable attention modules (IvRA(SIM., COMP, SUFF) & SMaT) outperformed perturbation and gradient-based methods such as LIME and IG in generating more plausible explanations.This section explores the computational overhead associated with training and deploying IvRA to generate explanations. To assess the time complexity of IvRA during training, we employ an IvRA model (utilizing Llama-2 as the base) for each interpretability criterion using varying quantities of input data. Specifically, we conduct training for 10 epochs with N ∈ 10, 100, 1000, 10000, 100000 input samples from the SNLI dataset and measure the elapsed time in minutes. The outcomes of these experiments, depicted in Figure. 9, reveal that IvRA introduces only a marginal increase in training time complexity compared to the baseline model3 . It is important to note that training IvRA for all criteria except consistency proves to be feasible in terms of time. Furthermore, even in the case of consistency, the training time only becomes computationally challenging for input samples of very large sizes (N ≥ 100000).In terms of explanation generation time, IvRA presents a distinct advantage over existing post-hoc explanation methods like Ribeiro et al. (2016) and Shrikumar et al. (2016 Shrikumar et al. ( , 2017)) reducing computational complexity during its application. We conducted experiments comparing the time required for popular post-hoc methods and IvRA to generate explanations across different input sample sizes, as depicted in Figure . 10. Our results indicate that, for all versions of IvRA, the time needed to generate explanations is significantly shorter compared to post-hoc methods. Overall, while deploying a IvRA model may involve additional time complexity during the training phase, we found this to be manageable in implementation. Furthermore, IvRA offers the added benefit of producing superior (more simulatable, faithful, and consistent) explanations at faster speeds during application. Our compute resources consist of 4× RTX 6000, 4× RTX 4500 and 2× RTX 3090. For running Integrated Gradients in our experiments, we use 50 iterations for calculating the integral. For running LIME in our experiments, we use 500 perturbations to approximate the neighborhood in which the surrogate models are learned. For baseline embeddings, we use zero tensors (Atanasova et al., 2020) . Saliency scores (for each individual word) in all settings are the sum of saliency scores of its word pieces (DeYoung et al., 2019) . We use AdamW (Loshchilov and Hutter, 2017) as our optimizer for all our models, with the exception of training the student for simulatability, in which case we use SGD ( §B.1). In all the experiments detailed in Tables 1, 2, 3, and 4, we trained all models for 20 epochs at a batch size of 64, applying decay factor every two epochs, and the reported results are from the best iteration of each model. To determine the optimal learning rate, we explored a broad range of learning rates and decay factors. The outcomes of our hyperparameter search for the IMDb task are presented in Table 7 . Our investigation reveals that while IvRA can acquire the necessary information for producing interpretable explanations across most settings, achieving the optimal performance metrics requires specific learning rates, highlighting IvRA's sensitivity to variations in learning rates.In the experiments documented in Tables 1, 2 , We observe that training for simulatability achieves optimal results with smaller learning rates and higher decay factors (larger γ). Both comprehensiveness and sufficiency training benefit from a moderate learning rate and decay factor. Training for consistency performs best with a higher initial learning rate and a lower decay factor. The interpretable attention module of IvRA involves the selection of salient input elements at three levels: feature, head, and layer. What is the effectiveness of the selection process at each level in terms of achieving simulatable, comprehensive, sufficient, and consistent explanations? In this section, we conduct experiments aimed at answering this question. Specifically, we experiment with IvRA by enabling feature-level selection, head-level selection, and layer-level selection separately to observe their individual effects during training. The loss curve for each criterion during training is illustrated in Figure.6. We observe in our experiments, that, across all four interpretability criteria, layer-level selection exhibits the least reduction in loss during training. While head-level selection is shown to be more effective than layerlevel selection, its loss curve stabilizes at a higher level compared to feature-level selection. Notably, feature-level selection proves to be the most effective (out of the three levels) in identifying information that aligns with each of the interpretability criteria, leading to the lowest level of losses during training, relatively to head and layer-level selection. Finally, training with selection at all levels enabled proves to be the optimal solution to produce explanations that align with each of the criteria, albeit with only marginal improvements over feature-selection-only in certain cases.Details regarding our computational hardware are outlined in §H 


Decision: Reviewers agree that this paper is impactful, relevant, and well-written. There were minor suggestions regarding presentation and framing, but this is already a strong contribution that fits the theme of the workshop well.

Reviews: review: paper_summary: paper_summary: The paper is about exploring whether attention-based explanations can be adapted to meet the demands of interpretability paradigms (simulatability, faithfulness, consistency). The authors present IvRA which regularizes a language model's attention distribution s.t. attributions are guided to align more with these desiderata.
strength_weakness: strength_weakness: Almost none, but I'd suggest that the authors look into polishing the figures and explanations of the mathematical formalization (see Comments).
questions: questions: * l. 019: "consistent~~i~~"
* Figure 2 should explain what the variables represent. There are many labels missing that are hard to match between full text and figure.
* Figure 4 caption: "highlightin**g**"
* l. 470: "three ~~three~~"
* Depiction of $t^k_i$ in l.244: Format like Eq. 8 etc.
* Apply same highlighting of criteria to §2.2 as in Table 5
paper_summary: paper_summary: This paper adds an attention module, called IvRA, to an LM that is optimized with respect to four common interpretability measures: simulatability, comprehensiveness, sufficiency and consistency. The experiments cover three LMs and three text classification and answer span detection tasks and demonstrate that their IvRA method improves on all of them as compared to standard attention and other extractive explainability methods such as LIME and IG. Interestingly, the consistency optimization has a strong negative effect on the performance while the effect of sufficiency and simulatability is comparably small.
strength_weakness: strength_weakness: One question that comes up to me relates to Goodhart's law: Do we consider these four measures that are used as the *definition* of interpretability, that are suitable as training objectives to increase interpretability, or are they rather *indicators*? This is a philosophical rather than a practical problem for your paper, but I'd like to see it discussed.
questions: questions: l.019 "consistenti"