Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods
Year: 2023

Abstract:
Saliency maps can explain a neural model’s predictions by identifying important input features. They are difficult to interpret for laypeople, especially for instances with many features. In order to make them more accessible, we formalize the underexplored task of translating saliency maps into natural language and compare methods that address two key challenges of this approach – what and how to verbalize. In both automatic and human evaluation setups, using token-level attributions from text classification tasks, we compare two novel methods (search-based and instruction-based verbalizations) against conventional feature importance representations (heatmap visualizations and extractive rationales), measuring simulatability, faithfulness, helpfulness and ease of understanding. Instructing GPT-3.5 to generate saliency map verbalizations yields plausible explanations which include associations, abstractive summarization and commonsense reasoning, achieving by far the highest human ratings, but they are not faithfully capturing numeric information and are inconsistent in their interpretation of the task. In comparison, our searchbased, model-free verbalization approach efficiently completes templated verbalizations, is faithful by design, but falls short in helpfulness and simulatability. Our results suggest that saliency map verbalization makes feature attribution explanations more comprehensible and less cognitively challenging to humans than conventional representations. 1

heading: 1 Introduction
text: Feature attribution methods, or (input) saliency methods, such as attention- or gradient-based attribution, are the most prominent class of methods for generating explanations of NLP model behavior (Wallace et al., 2020; Madsen et al., 2022) and can be used to produce word-level importance scores
1Code and data at https://github.com/DFKI-NLP/SMV.
without human supervision (Wallace et al., 2019; Sarti et al., 2023). A major limitation of saliency maps is that they require expert knowledge to interpret (Alvarez-Melis et al., 2019; Colin et al., 2022). Furthermore, Schuff et al. (2022) revealed visual perception and belief biases which may influence the recipient’s interpretation.
Natural language explanations (NLEs), on the other hand, exceed other explainability methods in plausibility (Lei et al., 2016; Wiegreffe and Pinter, 2019; Jacovi and Goldberg, 2020), accessibility (Ehsan and Riedl, 2020), and flexibility (Brahman et al., 2021; Chen et al., 2023), i.e. they can be adapted to both different target tasks and different audiences. Most previous approaches in generating NLEs depend on datasets of humanannotated text highlights (Zaidan et al., 2007; Lei et al., 2016; Wiegreffe and Marasović, 2021) or carefully constructed gold rationales for supervised training (Camburu et al., 2020; Wiegreffe et al., 2022), which are costly to obtain and taskspecific. Alignment of model rationales with very few human-acceptable gold rationales may raise issues of trust (Jacovi et al., 2021) and the models trained on them may suffer from hallucinations (Maynez et al., 2020).
In this work, we revisit and formalize the task of verbalizing saliency maps, i.e. translating the output of feature attribution methods into natural language (Forrest et al., 2018; Mariotti et al., 2020; Slack et al., 2022). Verbalizations can describe relations between words and phrases and their associated saliency scores. Contrary to conventional heatmap visualizations, we can adjust the comprehensiveness of an explanation more precisely and infuse it with additional semantics such as word meanings, concepts, and context about the task.
We find that verbalization also comes with a few caveats: Similar to human explainers, who communicate only the most relevant explanations to avoid cognitive overload of the recipient (Hilton,
30
2017; Miller, 2019), verbalization methods need to address the problem of deciding “what” to say, i.e. selecting the most informative and useful aspects of the saliency maps and communicating them in a concise manner. We therefore compare different methods for verbalizing saliency maps: Supervised rationales, prompting LLMs, and model/trainingfree templates.
We address the problem of saliency map verbalization (SMV) with the following contributions: • We formalize the underexplored task of SMV and
establish desiderata, i.e. simulatability, explainerfaithfulness, plausibility, and conciseness (§2.1); • We conduct a comparative study on various representations of feature attribution in two text classification setups, measuring the effects of verbalizations methods on both automated (explainerfaithfulness) and human evaluation metrics (simulatability, helpfulness, ease of understanding) (§3, §5). • We propose a novel, model-free, template-based SMV approach, and design instructions for GPT-3.5-generated SMVs (§4) (examples from our two setups are depicted in Fig. 1); • We show that model-free SMVs perform slightly better than heatmaps and extractive rationales on
ease of understanding and are faithful by design, while instruction-based SMVs achieve the highest average simulation accuracy and are preferred in subjective ratings (§6); • We publish a large dataset of model-free and GPT-generated SMVs alongside extractive rationales and results from both evaluations, and opensource code to produce all kinds of SMVs.

heading: 2 Verbalizing saliency maps

heading: 2.1 Formalization
text: The setup of the saliency map verbalization task consists of an underlying (to-be-explained) model m whose prediction ŷ ⊂ Y on source tokens W = w1 . . . wn we want to explain (against the set of possible outcomes Y ). m is equipped with a feature explanation method (or short: explainer) e which produces a saliency map S = s1 . . . sn:
e(W,m) = S (1)
Here, we call token wi salient towards outcome y if its associated saliency score si > 0 and salient against y for si < 0. e can have many sources, e.g. gradient-based methods such as Integrated Gradients (Sundararajan et al., 2017) which
we employ in our experiments (§5), or even human experts assigning relevance scores.
A verbalized saliency map SV is produced by some verbalizer v that receives the output of e:
v(W,S) = SV (2)
v can be any function that discretizes attribution scores and constructs a natural language representation SV. This is connected to the concept of hard selection in DeYoung et al. (2020) and heuristics for discretizing rationales (Jain et al., 2020). In the taxonomy of Wiegreffe and Marasović (2021), verbalized saliency maps can be categorized as freetext rationales with varying degrees of structure imposed through templates. Moreover, verbalized explanations are procedural and deterministic by nature, i.e. they function as instructions that one can directly follow (Tan, 2022) to understand a model’s decision, similar to compositional explanations (Hancock et al., 2018; Yao et al., 2021).

heading: 2.2 Desiderata
text: In the following, we outline the common evaluation paradigms for explanations (faithfulness, simulatability, plausibility) and how we adapt them to saliency map verbalizations.
Faithfulness Saliency maps express that “certain parts of the input are more important to the model reasoning than others” (linearity assumption in Jacovi and Goldberg (2020)). For verbalizations, explainer e and verbalizer v are two separate processes, so the saliency map S can be seen as static. Therefore, the faithfulness of e to the model m is extrinsic to the verbalization. Instead, it is essential to faithfully translate S into natural language, which we coin explainer-faithfulness. The verbalizer breaks faithfulness, e.g. if words are referenced as salient in SV that are made up (do not appear in W ) or if the polarity of any si is falsely interpreted.
Simulatability Another type of faithfulness is the model assumption which requires two models to “make the same predictions [iff] they use the same reasoning process” (Jacovi and Goldberg, 2020). By extension this means a model has to be simulatable (Doshi-Velez and Kim, 2017; Hase and Bansal, 2020), i.e. a human or another model should be able to predict a model’s behaviour on unseen examples while exposed only to the explanation and not the model’s prediction.
Plausibility The plausibility of explanations is commonly measured by correlation with groundtruth explanations (DeYoung et al., 2020; Jacovi and Goldberg, 2020), since gold rationales are influenced by human priors on what a model should do.
Conciseness In addition to these paradigms, verbosity is also an important aspect. A full translation into natural language is nonsensical, however, because all relations between the continuous-valued saliency scores and the associated tokens would normally overload human cognitive abilities. We want SV to be concise, yet still contain the key information, similar to sufficiency and comprehensiveness measures from DeYoung et al. (2020). Thus, we define a coverage measure to indicate how much information is retained going from S to SV, i.e. how much of the total attribution in S = s1 . . . sn is referenced by the tokens mentioned in SV = v1 . . . vm:
Coverage(SV) = ∑ |vi| ||S|| (3)
The goal here is not to achieve a coverage of 1 with all of S, but depending on the use case, SV should mention the most influential tokens, so a trivial solution for k = 5 would be to include the top k tokens with the highest attribution in S.

heading: 3 Study setup

heading: 3.1 Human Evaluation
text: Inspired by previous crowd studies in explainability (Chandrasekaran et al., 2018; Strout et al., 2019; Hase and Bansal, 2020; Sen et al., 2020; González et al., 2021; Arora et al., 2022; Joshi et al., 2023), we propose to measure simulatability as well as ratings for helpfulness and ease of understanding (plausibility). We evaluate the quality of different verbalization methods in a study involving 10 human participants. All participants have a computational linguistics background, with at least a Bachelor’s degree, limited to no prior exposure to explainability methods, and are proficient in English (non-native speakers). After an introduction to the goal of the study and a brief tutorial, annotators are to complete the tasks described below. For each task, we present text instances along with their explanations, using a simple Excel interface.2
2See Appendix C, Figure 7
Task A: Simulation In the first task, participants are asked to simulate the model, i.e. predict the model’s outcome, based only on one type of explanation plus the input text (“What does the model predict?”). They are given the possible class labels and were given an example for each dataset in the tutorial before starting the session. If the explanation does not provide any sensible clues about the predicted label, they still have to select a label, but may indicate this in the following question B1.
Task B: Rating In the second task, participants have to provide a rating on a seven-point Likert scale about (B1) “how helpful they found the explanation for guessing the model prediction” and (B2) “how easy they found the explanation to understand”. A higher rating indicates a higher quality of the explanation.
Task C: Questionnaire Finally, participants are asked to complete a post-annotation questionnaire to obtain overall judgements for each verbalization method. They are prompted for Likert scale ratings about time consumption, coherence, consistency and qualitative aspects of each verbalization method, as listed in Table 1.

heading: 3.2 Automated Evaluation
text: We expect hallucinations (synthesized, factually incorrect text due to learned patterns and statistical cues) from GPT-type models and thus devise the following tests measuring explainer-faithfulness and conciseness: 1. Have the referred words been accurately cited
from the input text? 2. How often do the referred words represent the
top k most important tokens? (Eq. 3) We obtain the results by simple counting and
automated set intersection.

heading: 4 Methods
text: To complement heatmap visualizations and extractive rationales, we propose and analyze two additional verbalization methods: Model-free (§4.1, Fig. 2) and instruction-based (§4.2, Fig. 3) saliency map verbalization.

heading: 4.1 Model-free verbalization
text: For our model-free approach we employ handcrafted templates for surface realization, different binary filter algorithms as search methods (§4.1.1)
and scoring metrics (§4.1.2) to select tokens for filling the templates. This approach does not require architectural changes to the underlying model or modifications to an existing saliency method. The most similar approach to our selection heuristics, to our knowledge, are the discretization strategies in Jain et al. (2020, §5.2).
In the following, we will present two distinct candidate generation methods that can both be combined with one of two scoring metrics. A final candidate selection (§4.1.3) will collect the results from both searches, concatenate them to possibly larger spans and filter the top scoring candidates once more while maximizing coverage (Eq. 3). These salient subsets are then used to complete hand-crafted templates (App. E). We argue that this is more human-interpretable than simple top k single token selection, at the cost of a lower coverage. Our methodology allows to set parameters in accordance to how faithful the verbalization should be to the underlying explainer.

heading: 4.1.1 Explanation search
text: To acquire potentially salient snippets from a given text, we perform a binary selection on a window of attributions from the input of size c and then compare the sum of our selection to one of our scoring methods, performing basic statistical analysis on the window and the input.
Convolution Search Inspired by the convolutions of neural networks, we compare tokens that are located close to each other but are not necessarily direct neighbors. Coherence between pairs of tokens is solely determined by looking at their attributions with the following binary filters. In short, the following method firstly generates templatevectors that we then permute and keep as our binary filters. After computing all valid and sensible permutations, we can start calculating possibly salient or coherent snippets of our input. We choose b ∈ N vectors with a length of c ∈ N. We describe these b vectors vi as follows:
vi = [11,i, 01,c−i], vi ∈ Z1,c. (4)
e.g., for i = 3, c = 5, vi = (1 1 1 0 0)
We only keep those vi where ∑
vi /∈ {0, 1, c} in order to perform sensible permutations. For each vi, we define a filter fi,j , where each distinct entry in fi is a unique permutation of vi. Let A be our attribution input, with A ∈ R1,k, where k is the
length of our input k > c, then we multiply a subset of our input with every binary filter
ri,j,l = fi,j ·Al+cl , l ∈ L,L = {l ∈ Z|1 ≤ l ≤ k − c}.
(5)
From this, we receive result vectors containing possibly coherent attributions and tokens.
Span Search Instead of looking for token pairs in a local neighborhood, we can also look for contiguous spans of tokens by adapting our proposed convolutional search. We generate b vectors of length of c with c being odd. We describe these b vectors as follows: Choose i ∈ N with i being odd, which ensures symmetry of our filters.3
vi = [01,⌊ c−i 2 ⌋, 11,i, 01,⌊ c−i 2
⌋], vi ∈ Z1,c (6)
We calculate attribution vectors ri,l as such:
ri,l = vi ·Al+cl , l ∈ L,L = {l ∈ Z|1 ≤ l ≤ k − c}
(7)

heading: 4.1.2 Candidate scoring metrics
text: We score and filter the snippets r so that we can present the most salient samples. As a threshold, we calculate the average of the n% most salient
3In contrast to our proposed Convolution Search, we don’t need permutations of vi to generate filters f , so we directly use vi. Thus, the result vector r has only two indices.
tokens of the given input sample A. This simple method does not filter for saliency, but it reduces the likelihood of presenting non-salient sample snippets. We call this our baseline β.
Weighted average The weighted average sums up the attribution values of r and divides the resulting scalar by the length of r, calculating the "saliency per word" of r. Then the result gets compared to β. Is the result larger than β, r is considered salient and will be a candidate for the verbalization.
Quantile The quantile method relies on the standard deviation within our current sample A. Given a quantile n, n ∈ R+0 , we calculate the corresponding standard deviation value σ and compare it to the average of the values of our snippet. If the score is greater than σ and β, it will be marked for verbalization.

heading: 4.1.3 Summarized explanation
text: On top of the two search methods in §4.1.1, we construct a summarized explanation to be used in our human evaluation (§3.1) by considering the k single tokens with the highest attribution scores. After generating k candidates from each search method, we concatenate neighboring token indices to (possibly) longer sequences and recalculate their coverage. We compute the q-th quantile of the remaining candidates according to their coverage to
select the final input(s) to our templates. If no candidate is within the q-th quantile, the top-scoring span will be chosen.

heading: 4.2 Instruction-based Verbalizations
text: In light of very recent advances in instructing large language models to perform increasingly complex tasks (Wei et al., 2022), we additionally construct “rationale-augmented” verbalizations (Fig. 3) next to template-based and search-based ones. The instruction contains an overview of the saliency map verbalization task and the associated caveats, e.g. “The classifier cannot base its prediction on the scores, only on the input text itself.”. Our most consistently accurate result was achieved by then representing S as bracketed scores rounded to two digits put behind each word, e.g. “definitely (0.75) a (0.14) girl (-0.31) movie (0.15)”.
In practice, we manually engineered taskagnostic instruction templates to work with GPT-3.5 (March ’23) aka ChatGPT.4 To our knowledge, there are no datasets with gold verbalizations available and we do not want to enforce any specific format of the explanation, so we use the API in a zero-shot setting. We post-process all outputs by removing all occurrences of the predicted label and semantically very similar words (App. G).
4We describe the task-specific instructions in App. F and document the edits to mitigate label leakage in App. G.

heading: 5 Data
text: We choose datasets that cover a selection of English-language text classification tasks. In particular, we select IMDb (Maas et al., 2011) for sentiment analysis, and AG News (Zhang et al., 2015) for topic classification.
We retrieve predictions from BERT models on the test partitions of IMDb and AG News made available through TextAttack (Morris et al., 2020) and their Integrated Gradients (Sundararajan et al., 2017) explanations with 25 samples exactly as they appear in Thermostat (Feldhus et al., 2021).
We then take subsets (IMDb: n = 80, AG News: n = 120) of each dataset according to multiple heuristics (App. D) that make the tasks more manageable for annotators. Each annotator was shown 340 explanations consisting of equal amounts of each type of representation or rationale. We randomize the order in which they are presented to the annotators. Every instance was evaluated by seven different annotators.

heading: 6 Results
text: Human evaluation Tab. 2 shows that both kinds of SMVs are generally easier to understand (B2) than heatmaps or extractive rationales. In a postannotation questionnaire, we asked 8 out of 10 participants 14 questions about both types of SMVs. Tab. 1 lists the results. While template-based explanations are preferred in being less time-consuming, we can see that GPT-generated verbalizations outperform them in all other aspects. Unsurprisingly, associations and summarizations are the preferred characteristics of verbalizations.
Downstream tasks According to Jacovi et al. (2023a), a feature attribution explanation aggre-
gates counterfactual contexts. This becomes apparent in our overall results on the AG News dataset where more than one potential alternative (multiclass classification with |C| = 4) outcome exists. Annotators’ simulation accuracy drops from as high as 94 % (IMDb) to 78 %. SMVGPT beats all other representations across all three measures in IMDb, but surprisingly underperforms in AG News.
Coverage of the verbalization Fig. 4 and App. A show that SMVGPT focuses less on the actual most important tokens that might not be intuitive for recipients, such as function words. The subset of instances with higher-than-average coverage according to SMVTempl (Cov(SVT)↗) is substantially easier to simulate (IMDb) and elicits the highest ratings and accuracies from annotators. We utilize this as a proxy for (low) complexity of S, because usually only a single or few tokens that are very salient make these explanations easy to decipher in most representations.
Therefore, we conducted an automated simulatability evaluation on all SMV types, documented in Appendix B, confirming the suspicions about the faithfulness of GPT verbalizations.
Model predictions Lastly, we investigate the subsets of wrong model predictions: The drop in simulation accuracy and ratings when we filter the instances where the model predicts something different from the true label (y ̸= ŷ) is more severe for IMDb throughout all types of explanations. In AG News, the simulatability and the ease of understanding turn out to be higher for SMVs. Our
consistently worse results in this subset reveal the belief bias (González et al., 2021), i.e. explanations have a hard time convincing humans about a model behavior when they already have prior assumptions about the true label of an instance. For instances where the human simulation mismatched with the predicted label (ŷ ̸= ysim), the drop in scores is even harsher: Only SMVGPT still achieves ratings that are slightly above average.

heading: 6.1 Evaluating instruction-based verbalizations
text: While there are no invented words in the human evaluation subset, our automated mapping between explanation and input text still detected cases where words are auto-corrected and not accurately copied, especially fixing capitalization and small typos. We also found examples in which words or spans are replaced with a synonym, e.g. “not reliable” → “unreliable”, but most strikingly, in an IMDb example, “good premise” was replaced with “bad premise” which entirely changed the meaning and the polarity of the sentiment.
In Tab. 3, we manually count what type of taskrelated information and semantics SMVGPT provides on top of the translation of the importance scores. We can see that the “negative sentiment” in IMDb is often a confounder for the correct interpretation of the negative saliency scores. Without explicit instructions, GPT still questioned some of the wrong prediction the underlying BERT has made, particularly for IMDb. In terms of linguistic aspects of the verbalizations, associations are frequently included, while summarizations of the
input or the decision are rare.

heading: 6.2 Discussion
text: By choosing parameters that prefer longer spans to be selected, we show that SMVTempl can be more plausible to humans than single token selection. We acknowledge that SMVTempl are repetitive and, while the results show that they can guarantee a minimum degree of understandability (Ehsan et al., 2019), sufficiency and conciseness, they will not be satisfying enough for lay recipients on their own.
For SMVGPT, the choice of instruction can greatly impact the faithfulness to the explainer. Plausible explanations driven by world knowledge and semantics allow laypeople to contextualize the prediction w.r.t. the input text, but reliable and generalizable methods for auditing these rationales for faithfulness have yet to be discovered.

heading: 7 Related Work
text: To our knowledge, the only previous saliency map verbalization approach is by Forrest et al. (2018) who used LIME explanations and a template-based NLG pipeline on a credit dataset. While they mostly included numerical values in explanations, we focus on most important features and free-text rationales, because humans are more interested in reasoning than in numerical values (Reiter, 2019). Ampomah et al. (2022) created a dataset of tables summarizing the performance metrics of a text classifier and trained a neural module to automatically generate accompanying texts. The HCI community highlighted the advantages of verbalization as a complementary medium to visual explanations (Sevastjanova et al., 2018; Hohman et al., 2019; Szymanski et al., 2021; Chromik, 2021). Zhang and Lim (2022) advocated for adding concepts and associations to make explanations more understandable, particularly in contrastive setups.
Hsu and Tan (2021) introduced the task of decision-focused summarization. While there are overlaps in the selection of important subsets of the input, the textual nature of the output and the employment of saliency methods, our work is concerned with summarizing the token-level information provided by a saliency map from an arbitrary source for a single instance. Okeson et al. (2021) found in their study that global feature attributions obtained by ranking features by different summary statistics helped users to communicate what the model had learned and to identify next steps for debugging it. Rönnqvist et al. (2022) aggregated attribution scores from multiple documents to find top-ranked keywords for classes.
In early explainability literature, van Lent et al. (2004) already used template filling. Templates in NLE frameworks were engineered by Camburu et al. (2020) to find inconsistencies in generated explanations. While their templates were designed to mimic commonsense logic patterns present in the eSNLI dataset (Camburu et al., 2018), our templates are a means to verbalize arbitrary saliency maps. Paranjape et al. (2021) crafted templates and used a mask-infilling approach to produce contrastive explanations from pre-trained language models. Donadello and Dragoni (2021) utilized a template system to render explanation graph structures as text. Recently, Tursun et al. (2023) used templates together with ChatGPT prompts to generate captions containing verbalized saliency map explanations in the computer vision domain. However, they did not conduct an automated or human evaluation.

heading: 8 Conclusion
text: We conducted a comparative study on explanation representations. We formalized the task of translating feature attributions into natural language and proposed two kinds of saliency map verbalization methods. Instruction-based verbalizations outperformed all other saliency map representations on human ratings, indicating their summarization and contextualization capabilities are a necessary component in making saliency maps more accessible to humans, but they are still unreliable in terms of ensuring faithfulness and are dependant on a closedsource black-box model. We find that templatebased saliency map verbalizations reduce the cognitive load for humans and are a viable option to improve on the ease of understanding of heatmaps without the need for additional resources.
Limitations
Our experimental setup excludes free-text rationales explaining the decisions of a model (Wiegreffe et al., 2022; Camburu et al., 2018), because their output is not based on attribution scores or highlighted spans of the input text, so we argue that they are not trivially comparable. However, there are end-to-end rationalization frameworks that can accommodate arbitrary saliency methods (Jain et al., 2020; Chrysostomou and Aletras, 2021; Ismail et al., 2021; Atanasova et al., 2022; Majumder et al., 2022), but require large language models that are expensive to train and perform inference with, so this is out of scope for this study. However, we also see that high-quality free-text rationales can be more easily generated with LLMs (Wang et al., 2023; Ho et al., 2023), and a comparison between them and our attribution-based explanations is an interesting avenue for future work.
Inferring high-quality explanations from large language models necessitates excessive amounts of compute and storage. Although GPT verbalizations are most promising, we urge the research community to look into more efficient ways to achieve similar results. In the future, we will explore if training a smaller model on top of the collected rationale-augmented verbalizations is feasible.
Emphasizing the concerns of Rogers (2023), we do not recommend the black-box model GPT-3.5 as a baseline for interpretability, because the model’s training data or internal parameters can not be accessed and the dangers of deprecation as well as the lack of reproducibility are serious con-
cerns. However, we do think it has revealed great potential as a surface realization and contextualization tool for the task of saliency map verbalization.
The causality problem explained in Jacovi et al. (2023a) is not solved by our verbalizations, as it is an inherent problem with feature attribution and rationalization. Future work includes verbalizations alongside counterfactuals, e.g. in interactive setups (Feldhus et al., 2022; Shen et al., 2023).
Although multiple models and explanationgenerating methods are available, we specifically focus on one pair for both datasets (BERT and Integrated Gradients), because the focus of our investigation is on the quality of the representation rather than the model.
Finally, explicitly modelling expected highlights to mitigate misalignments as reported on in Schuff et al. (2022), Jacovi et al. (2023b) and Prasad et al. (2021) is still unexplored.

heading: Acknowledgments
text: We would like to thank Aleksandra Gabryszak for her help setting up and conducting the human evaluation. We are indebted to Elif Kara, Ajay Madhavan Ravichandran, Akhyar Ahmed, Tatjana Zeen, Shushen Manakhimova, Raia Abu Ahmad, Ekaterina Borisova, João Lucas Mendes de Lemos Lins, Qianli Wang, and Sahil Chopra for their valuable work as annotators and evaluators. We thank Gabriele Sarti, Arne Binder, Yuxuan Chen, and David Harbecke as well as the reviewers of the NLRSE Workshop at ACL 2023 for their feedback on the paper. This work has been supported by the German Federal Ministry of Education and Research as part of the project XAINES (01IW20005).
Contributions
• NF: Writing, implementation of baselines and summarized template-based verbalization, prompt design and data generation, evaluation of user study, illustrations. • LH: Writing and supervision. • MDN: Conception, implementation and illustra-
tion of search and scoring methods for templatebased verbalization. • CE: Data curation, implementation of instructionbased verbalization pipeline, validation and empirical results on search and scoring methods. • RS: Initial idea and outline. • SM: Supervision and funding.

heading: A Token ranks
text: Figures 5 and 6 show the coverage of the verbalizations, which makes up one aspect of explainerfaithfulness.

heading: B Automated simulatability evaluation
text: We follow Wiegreffe et al. (2021) and Hase et al. (2020) and train a second language model to simulate the behavior of the explained BERT model. Table 4 shows the simulation accuracy of a T5-large receiving various types of verbalizations (plus the original input). We can see for both datasets that SMVGPT induces the most noise and thus results in the lowest accuracy, while the raw output of the search methods (Conv/Span) are most faithful in combination with the original input.
C Efficiency
First, we measure a runtime of less than two minutes on a CPU (i5-12600k) to generate templatebased verbalizations for all 25k instances of IMDb. Given pre-computed saliency maps from any explainer, this is considerably faster than using an end-to-end model for extractive rationales, e.g. Treviso and Martins (2020), which takes several hours for training and then more than 10 minutes for inference on an RTX 3080 GPU. GPT-3.5 with at least 175B parameters, which obliterates the other two setups. This means that there is a considerable carbon footprint associated with using it. Future work has to look into training considerably smaller models on the generated verbalizations.

heading: D Subset selection heuristics
text: • We restrict our experiments to explaining a single outcome – the predicted label ŷ – and thus modify our metric (Eq. 3): Cov+ only considers the positive attributions si > 0. • We select instances achieving at least a Cov+ score of 15% (indicating the attribution mass is not too evenly distributed, making interpretations of saliency maps challenging).
• We find values for q (§4.1.3) of 0.5 ≤ q ≤ 0.75 to produce the right amount of candidates in the end, s.t. there almost always is at least one candidate in the q-th quantile and the resulting verbalization is not longer than most text inputs. • We only consider instances with a maximum token length of 80, s.t. the human evaluation is more manageable for annotators. • We select equal amounts of instances for every true label y (IMDb: 40 positive + 40 negative sentiment; AG News: 30 World + 30 Sports + 30 Business + 30 Sci/Tech) in each dataset. • We select 25% of IMDb and 46.67 % of AG News to be false predictions by the BERT model (y ̸= ŷ). We apply the weighted average for IMDb-BERTIG (β = 0.4) and the quantile scoring metric for AG News-BERT-IG (n = 3). We chose the number of candidates to be k = 5 in all cases and the threshold q to be .75 for IMDb and AG News as the average length of the input is lower for the latter which results in too few candidates with higher qs.

heading: E Templates for Verbalizing Explanations
text: We design our templates as atomic expressions with constraints and blanks that can be filled with words from W . In the most basic cases, we refer to spans, phrases, words and characters as salient or important for some prediction. We design the templates to express saliency information concisely and enable users to reproduce the model’s decision process (simulatability). The set of templates is depicted in Table 8.
Our template-based methodology is task- and model-invariant by design, because no task-specific model or NLG component is involved. Achieving sufficiency (measured by coverage) is harder, because a full translation of any saliency map is too verbose and thus not helpful.

heading: F List of LLM prompts
text: At first, we treated this as table-to-text task – which has recently been tackled with prompt-based large language models (Chen, 2023; Xiang et al., 2022) –
where we provided a list of attribution scores and, separate from that, a list of tokens. However, we registered less hallucinations (the model incorrectly mapping between words and their scores) when we provided the input as a joint representation as shown in Fig. 3.
For the two datasets, we then used the token+score representation as {sample} and a {label_str} being the predicted label (IMDb: positive or negative; AG News: Worlds, Sports, Business, or Sci/Tech) and wrote the instructions in Fig. 9.

heading: G Post-processing of GPT outputs
text: AG News In order to prevent label leakage, we employed the string replacements listed in Tab. 5. In our human evaluation, they were replaced with "{placeholder}", so annotators could perform the simulatability task without cheating.

heading: Future Work
text: Future work has to look into training considerably smaller models on the generated verbalizations