Learning to Scaffold: Optimizing Model Explanations for Teaching
Authors: Patrick Fernandes
Marcos Vinicius Treviso
Danish Pruthi
Andre Martins
Graham Neubig
Venue: NeurIPS 2022 Accept (2022)
Keywords: explainable ai
interpretability
meta-learning
attention

Abstract: Modern machine learning models are opaque, and as a result there is a burgeoning academic subfield on methods that explain these models' behavior.  However, what is the precise goal of providing such explanations, and how can we demonstrate that explanations achieve this goal? Some research argues that explanations should help teach a student (either human or machine) to simulate the model being explained, and that the quality of explanations can be measured by the simulation accuracy of students on unexplained examples. In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model. We train models on three natural language processing and computer vision tasks, and find that students trained with explanations extracted with our framework are able to simulate the teacher significantly more effectively than ones produced with previous methods. Through human annotations and a user study, we further find that these learned explanations more closely align with how humans would explain the required decisions in these tasks. Our code is available at https://github.com/coderpat/learning-scaffold.

Learning to Scaffold: Optimizing Model Explanations for Teaching
Patrick Fernandes V O,,
Marcos Treviso R2.R
Danish Pruthit
André F. T. Martins R.R Graham Neubig "
"Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA
o "nstituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 2 Instituto de Telecomunicações, Lishon, Portugal
Amazon Web Services
Unbabel, Lisbon, Portugal
Abstract
Modern machine learning models are opaque, and as a result there is a burgeoning academic subfield on methods that explain these models' behavior. However, what is the precise goal of providing such explanations, and how can we demonstrate that explanations achieve this goal? Some research argues that explanations should help teach a student (either human or machine) to simulate the model being explained, and that the quality of explanations can be measured by the simulation accuracy of students on unexplained examples. In this work, leveraging meta- learning techniques, themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model. We train models on three natural language processing and computer vision tasks, and find that students trained with explanations extracted with our framework are able to simulate the teacher significantly more effectively than ones produced with previous methods. Through human annotations and user study, further find that these learned explanations more closely align with how humans would explain the required decisions in these tasks. Our code is available
extend this idea to improve the quality of the explanations
Introduction
While deep learning's performance has led it become the dominant paradigm in machine learning, its relative opaqueness has brought great interest in methods to improve model interpretability. Many recent works propose methods for extracting explanations from neural networks ($ 7), which vary from the highlighting relevant input features [Simonyan et al., 2014, Arras et al., 2017, Ding et al., 2019] more complex representations of the reasoning of the network [Mu and Andreas, 2020, Wu et al., 2021]. However, are these methods actually achieving their goal of making models more interpretable? Some concerning findings have cast doubt this proposition; different explanations methods have been found to disagree on the same model/input [Neely et al., 2021, Bastings et al., 2021] and explana- tions do not necessarily help predict model's output and/or its failures [Chandrasekaran al., 2018].
In fact, the research community is still in the process of understanding what explanations are supposed achieve, and how assess success of explanation method [Doshi-Velas Velez and Kim. 2017, Miller, 2019]. Many early works on model interpretability designed their methods around a set of desiderata [Sundararajan et al., 2017, Lertvittayakumjon and Toni, 2019] and relied on qualitative assessment of a handful of samples with respect to these desiderata; a process that is highly subjective and is hard to reproduce. In contrast, recent works have focused on more quantitative criteria: correlation between
Equal contribution. Corresp. to pfernandcce cmu. cmu.edu or marcos.. trevisoftecnico ulisboa.pp pt t Work done while at Camegie Mellon University, prior to joining Amazon.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

student learning time
student evaluation time
simulability loss
classifier
vr
U5
classifer
yr
Ms
Cy Cyg
student
student
simulability loss
explainer
explainer regularizer
X.
explainer
Illustration of our SMaTT framework. First, a student model is trained to recover the classifier's predictions and to match the explanations given by the explainer. Then, the explainer is updated based on how well the trained student simulates the classifier (without access to explanations). In practice, we repeat these two consecutive processes for several steps. Green arrows and boxes represent learnable components.
Figure 1:
explainability methods for measuring consistency [Jain and Wallace, 2019. Serrano and Smith, 2019], sufficiency and comprehensiveness Young et al., 2020], and simulability: whether a human or machine consumer of explanations understands the model behavior well enough to predict its output on unseen examples [Lipton, 2016, Doshi-Velez and Kim, 2017]. Simulability, in particular, has number of desirable properties, such as being intuitively aligned with the goal of communicating the underlying model behavior to humans and being measurable in manual and automated experiments (Treviso and Martins, 2020, Hase and Bansal, 2020, Pruthi et al., 2020J.
For instance, Pruthi et al. [2020] proposed a framework for automatic evaluation of simulability that, given teacher model and explanations of this model's predictions, trains student model to match the teacher's predictions. The explanations are then evaluated with respect to how well they help student learn simulate the teacher (8 2). This analogous to the concept in pedagogy of instructional scaffolding [Van de Pol et al., 2010]. a process through which a teacher adds support for students to aid learning. More effective scaffolding- to lead to better student learning. However, while this previous work provides an attractive way to evaluate existing explanation methods, stops short of proposing a method to actually improve them. In this work, we propose to learn to explain by directly learning explanations that provide better term Ceaffold-Maximize
our case, better explanations
assumed
scaffolding of the student's learning, framework Figure illustrates the framework: the explainer is used to scaffold the student training, and is updated based on how well the student does fest time simulating the teacher model. We take insights from research on meta-learning [Finn et al., 2017, Raghu et al., 2021], I. formalizing our setting bi-level optimization problem and optimizing it based on higher-order differentiation (S 3). Importantly, our high-level framework makes few assumptions about the model we are trying to explain, the structure of the explanations or the modalities considered. To test our framework, we then introduce a parameterized attention- based explainer optimizable with SMaT that works for any model with attention mechanisms (5 4).
Training (SMaT).
We experiment with SMaT in text classification, image classification, and (multilingual) text- based regression tasks using pretrained transformer models ($ 5). We find that framework is able to effectively optimize explainers across all the considered tasks, where students trained with learned attention explanations achieve better simulability than baselines trained with static attention gradient-based explanations. We further evaluate the plausability of our explanations (i.e., whether produced explanations align with how people would justify similar choice) using human-labeled explanations (text classification and text regression) and through a human study (image classification) and find that explanations learned with SMaT more plausible than the static explainers considered. Overall, the results reinforce the utility of scaffolding as a criterion for evaluating and improving model explanations.
2 Background
Consider a model T : X + trained on some dataset Drram V. For example, this could be a text or image classifier that was trained on particular downstream task (with Dunin being the training data for that task). Post-hoc interpretability methods typically introduce explainer for the
module Fr 7 x X - E that takes a model and an input, and produces an explanation output of the model given that input, where& & denotes the space of possible explanations. For instance,

---

interpretability methods using saliency maps define & as the space of normalized distributions of importance over L input elements 7 (where 4 r--1 is the (L 1) p)-ooabailiity simplex).
Pruthi et al. [2020] proposed an automatic framework for evaluating explainers that trains a student model Se X + )' with parameters 0 to simulate the teacher (i.e., the original classifier) in constrained setting. For example, the student can be constrained to have less capacity than the teacher by using a simpler model or trained with a subset of the dataset used for the teacher (Dirain & Drain) In this framework, arg min, E (a.y))-Buui (a (So())) and its simulability SIM(Sg-, is measured on an unseen test set. The actual form of Saim and SIM(Sg- 7) is task-specific. For example, in a classification task, we use cross-entropy as the simulation loss Caim over the teacher's predictions, while the simulability of a model Sg. can be defined as the simulation accuracy, i.c, what percentage of the student and teacher predictions match over held-out test Diest:
baseline student So is trained according to 6"
(IM T) (r. [1 (z) T(x)
Next, the training of the student is augmented with explanations produced by the explainer E. We introduce student explainer Es S X' &. (the S-explainer) to extract explanations from the student, and regularizing these explanations on the explanations of teacher (the T' Teexplainer), using loss Leepl that takes explanations for both models:
0: = arg min B (x. Ceim (So(z), T(x)) +3 Lexpl
(2)
simulability loss
explainer regularizer
For example, Pruthi al. [2020] considered as teacher explainer Ey various methods such LIME [Ribeiro et al., 2016), Integrated Gradients [Sundararajan et al., 2017], and attention mechanisms, and explored both attention regularization (using Kullback-Leibler divergence) and multi-task learning to regularize the student.
The key assumption surrounding this evaluation framework is that a student trained with good explanations should learn to simulate the teacher better than a student trained with bad or no explanations, that is, SIM (Sug. 7) > SIM (Spr,T) T)) For clarity, we will refer to the simulability of model Sez trained using explanations as scaffolded simulability.
Optimizing Explainers for Teaching
As a first contribution of this work, we extend the previously described framework to make it possible to directly optimize the teacher explainer so that can most effectively teach the student the original model's 's behavior. To this end, consider parameterized T T-explainer Egr with parameters Φr, and equivalently a parameterized S-explainer Egg with parameters Φs. We can write the loss function for the student and S-explainer as:
Candenn Esg T, Esr, x) = Laim (Sa(z), (()) + 3.xppt (Esg (Sa, x), Eor (T,x))
(3)
While this framework is flexible enough rigorously and automatically evaluate many types of explanations, calculating scaffolded simulability requires an optimization procedure to learn the student and S-explainerr parameters 0, Øs. This makes it non-trivial achieve our goal of directly finding the teacher explainer parameters Φr that optimize scaffolded simulability, To overcome this challenge, we draw inspiration from the extensive literature on meta-learning [Schmidhuber, 1987, Finn et al., 2017], and frame the optimization as the following bi-level optimization problem (see Grefenstette et al. [2019] for a primer):
8" ((TT),
arg min E (z.g))/mm [Laudem (So, EoSs T, Eupry a)] 0.6s
(4)
6T = arg min (zzg)) [Csim (Sp- (6) (x),
(5)
Here, the inner optimization updates the student and the S-explainer parameters (Equation 4), and in the outer optimization we update the T T-xxplainer parameters (Equation 5). Importantly, our framework does not modify the teacher, as our goal is to explain a model without changing its original behavior. Notice that we also simplify the problem by considering the more tractable simulation loss Ceim instead of the simulability metric (SM(Sg- T') as part of the objective for the outer optimization.
3

---

Now, if we assume the explainers Edy and Egg are differentiable, we can use gradient- gradient-based optimization [Finn al., 2017] optimize both the student (with its explainer) and the T T'explainer. In particular, we use explicit differentiation to solve this optimization problem. To compute gradients for Φr. we have to differentiate through a gradient operation, which requires dessian-vector products, an operation supported by most modern deep learning frameworks [Bradbury et al., 2018, Grefenstette et al., 2019]. However, explicitly computing gradients for ΦT through a large number of inner optimization steps is computationally intractable. To circumvent this problem, typically the inner optimization is run for only a couple of steps or a truncated gradient is computed [Shaban et al., 2019]. In this work, take the approach of taking single inner optimization step and learning the student and S-explainer jointly with the T Teexplainer without resetting the student [Dery et al., 2021]. At each step, we update the student and S-explainer parameters follows:
g 0¹ - 1JINN Vo E. (a.a)-Dain Drain
fstudem (Sor, Fegs. T, Eagy 2))
(6)
- = =os-- TINN Vos E (x.g)) [Cmndenn (Ser, Eog T, Esg, xz)
(7)
After updating the student, we take an extra gradient step with the new parameters but only use these updates to calculate the outer-gradient for ΦT,, without actually updating O. This approach is similar the pilot update proposed by Zhou et al. [2021], and we verified that it led to more stable optimization in practice:
0( ((+l)) 0r+11 TINN V 1 Drm
[Cstuden (Sp: Logi,, Eot,
(8)
Φt11 dT oour V or B(aaai Caim (Soco))()),
Parameterized Attention Explainer
As a second contribution of this work, we introduce a novel parameterized attention-based explainer that can be learned with our framework. Transformer models [Vaswani et al., 2017] are currently the most successful deep-learning architecture across variety of tasks [Shoeybi et al., 2019, Wortsman al., 2022]. Underpinning their success is the multi-head attention mechanism, which computes a normalized distribution over the 1 S i I L input elements in parallel for each head h:
(10) where Q* = [gi........ .9h1 and Kh | ... k are the query and key linear projections over the input element representations for head h. Attention mechanisms have been used extensively for producing saliency maps [Wiegreffe and Pinter, 2019, Vashishth et al., 2019] and while some concerns have been raised regarding their faithfulness [Jain and Wallace, 2019], overall attention- based explainers have been found to lead to relatively good explanations in terms plausibility and simulability [Treviso and Martins, 2020, Kobayashi et al., 2020, Pruthi et al., 2020].
A = SOFTMAX (Q ((")
However, to extract explanations from multi-head attention, we have two important design choices:
1. Single distribution selection: Since self-attention produces attention matrix A E Dt 1. need to pool these attention distributions to produce a single saliency map E 1-1 Typically, the distribution from single token (such as [CLS]) or the average of the attention distributions from all tokens < i < L are used.
2. Head selection: We also need pool the distributions produced by each head. Typical ad-hoc strategies include using the mean over all heads for a certain layer [Fomicheva et al., 2021b) or selecting single head based plausibility since transformers can have hundreds or even thousands of heads, these choices rely on human intuition require large amounts of plausibility labels.
validation set ['Trevioo al., 2021]. However,
In this work, approach the latter design choice in more principled manner. Concretely, associate each head with a weight and then perform a weighted sum over all heads. These weights learned such that the resulting explanation maximizes simulability, as described in $ 3. More formally, given a model Tor and its query and key projections for an input x for each layer and head h H, we define a parameterized, differentiable attention explainer ( (Tor, x))
H
s
I L
(
K*
ESr (T,z) SOFTMAX
Σ N4.h
(11)
where the teacher's head coefficients \T AH-1 1 are \T NORMALIZE( with ØT E R.H.

---

In this formulation, s E R represents the average unnormalized attention log- its over all input elements, which are then combined according to Är and normal- ized with SOFTMAX to produce a distri- bution in 4 1- We apply a normaliza- tion function NORMALIZE to head coeffi- cients involved to create a convex combi- nation over all heads in all layers. In this work we consider the sparse projection SPARSEMAX
heod
R5
11
10
R4
'95
122
92
92
L0o meii
xus
softmax
function NORMALIZE [Martins and Astudillo, 2016], as:
Figure 2: Our parameterized attention-based explainer.
Dashed red boxes represent learned parameters Xς SPARSEMAX(OT) E A weighting average attention logits of each head 1 h < H. softmax the weighted sum generates the attention probabilities.
SPARSEMAX(2)
arg min lpp 2 2112- PEA#-1
We choose SPARSEMAX due to its ben- efits in terms of interpretability, since it leads many heads having zero weight. We also found it outperformed every other projection we tried (see & 6 for a more detailed discussion). Figure illustrates each step of our parameterized attention explainer.
5
Experiments
To evaluate our framework, we attempt to learn explainers for transformer models trained three different tasks: text classification ($ 5.1), image classification ($ 5.2), and machine translation quality estimation (a text-based regression task, detailed in $ 5.3). We use JAX [Bradbury et al., 2018] to implement the higher-order differentiation, and use pretrained transformer models from the Huggingface Transformers library [Wolf et al., 2020], together with Flax [Heek et al., 2020]. For each task, we train a teacher model with Adam [Loshchilov and Hutter, 2019] but, as explained in $ 3, SGD for the student model (inner loop). We also scalar mixing [Peters al., 2018] to pool representations from different layers automatically. We train students with a teacher explainer in three settings:
No Explainer: No explanations are provided, and no explanation regularization is used for training the student (i.e. B = 0 in Equation 3). We refer to studentsin this setting as baseline students.
Static Explainer: Explanations for the teacher model are extracted with five commonly saliency-based explainers: (1) L2 norm of gradients; (2) gradient x input explainer [Denil et al., 2014]; (3) an integrated gradients explainer [Sundararajan et al., 2017]; and attention explainers that uses the mean pooling over attention from (4) all heads in the model and (5) from the heads of the last layer [Fomicheva et al., 2021b, Vafa al., 2021]. More details can be found in Appendix A.
Learned Explainer (SMaT): Explanations are extracted with the explainer described in $ 4, with coefficients for each head that are trained with SMaT jointly with the student. We initialize the coefficients such that the model is initialized to be the same as the static attention explainer (i.c., performing the mean over all heads).
Independently of the Texxlaainer, we always use a learned attention based explainer as the S- explainer, considering all heads except when the T-explainer is static attention explainer that only considers the last layers' heads, where we do the same for the S-explainer. We use the Kullback- Leibler divergence Leept, and gradient-based explainers (since we found smaller values to be better). We set Lsim as the cross- entropy loss for classification tasks, and the mean squared error loss for text regression. For each setting, we train five students with different seeds. Since there is some variance in students' performance (we hypothesize due the small training sets) we report the median and interquantile range (IQR) around it (relative to the 25-75 percentile).
5 for attention-based explainers and
0.2 for
'hiiee scalar mixing reduced variance of student performance, SMaT also worked with other common pooling methods.
5

---

Table 1: Results for the IMDB dataset with respect to student simulability in terms of accuracy (%)
Underlined values indicate higher simulability than baseline with non-overlapping IQR.
500
1,000
2,000
No Explainer
81.72 (81.24:81.75)
83.44 [83.36:83.66 (33
84.84 [84.80:84.88]
Gradient L2
81.66 [81. 320
82.98
[82.72:83,00 .08]
84.78
184,96:85,08) 08]
Gradient x Input
84.83 (84.79:84.88]
81.15
[80.95:81.96 .36]
83.84 [83.59:84.99
Integrated Gradients
82,99 (82.59:82.99 999
81.79
[81.72:81.5) .87]
84,20 [84.03:85.03
Attention (all layers)
83,00 [82. (22.60:83000 00]
85.72 [85.72:86.22 23]
90.08 [89.72:90] 11]
Attention (last layer)
80,91 (79.99:81.07
83.15
[82.91:83.51]
91.41 [91.39911.56]
Attention (SMaT)
91.48 (91.40:91.56]
92.56 (92.28:92.83]
92.84 (92.84:93008
5.1 Text Classification
For text classification, we consider the IMDB dataset [Maas et al., 2011]. a binary sentiment classification task over highly polarized English movie reviews. As the base pretrained model, we use the small ELECTRA model [Clark et al., 2020], with 12 layers and 4 heads in each (total 48 heads). Like the setting in Pruthi et al. [2020], we use the original training set with 25,000 samples to train the teacher, and further split the test set into a training set for the student and a dev and test set. We vary the number of samples the student is trained on between 500, 1,000, and 2,000. We evaluate simulability using accuracy (i.c., what percentage of student predictions match with teacher predictions). The teacher model obtains 91% accuracy on the student test set.
Table shows the results in terms of simulability (Equation 1) for the three settings. We can see that, overall, the attention explainer trained with SMaT leads to students that simulate the teacher model much more accurately than students trained without any explanations, and more accurately than students trained with any static explainer across all student training set sizes. Interestingly, the gradient--ased explainers only improve over the baseline students when the amount of training data is very low, and actually degrade simulability for larger amounts of data (see discussion in A). Using only heads from the last layer seems to have the opposite effect, leading to higher simulability than all other static explainers only for larger training sets.
keiggried anailatt: ao idiaas acvee Vetil one 188) sad Iied
Intonnand gratondi vocil civil decame patcel aubity compace joi the valee bu. the ston inveert the danca ase aradin de mask and the patfernace pentuding
Table 2: Plausibility on MovieReviews in terms of AUC. * represents methods that use hu- man labels.
drsppll en lnd ana Desd was
would give tie Mooe feal H the mat
wheden byynn: offene woo who var the and leed it bu (east a Rnspcll cn and on ani (D02 vai
pense waking coepaes
vide to thin poorone (Eaood) seating trak shng the werll petfermnce
alle stople and
novel
glv
moting
aryene who
AUC
Augged in then cas
skke bat Dr stow jiccens dasce ane keadid de made irquel stig.. and the secrl paffermnce
Grad. L2
0.65
rel pad pat ote bas skmple and uning give dhs
tou
woyting saggest
the
Rover
Grad. Input
Integrated Grad.
0.51
0.53
Figure 3: Explanations given by integrated gradients, attention (last layer), and our learned attention explainer (SMaT) for two movie reviews of the IMDB dataset (negative and positive examples). Green and orange represent positive and negative contributions, respectively.
Arm. (all layer))
0.68
Attn. laver)
Atin. (SMaT) 0.73
Atmn (best laveri* 0.75
Atn. (best Read)* 0.75
Plausibility analysis. We select the median model trained with 1,000 samples and extract explana- tions for test samples from the MovieReviews dataset (DeYoung et al., 2020], which contains binary sentiment movie reviews from Rotten Tomatoes alongside human-rationale annotation. Since the labels are binary (indicating whether a token is part of the explanation not) and the predicted scores are real values, we follow Fomicheva et al. [2021a] and report our results in terms of the Area Under the Curve (AUC), which automatically considers multiple binarization thresholds. The results are shown in Table along with two randomly selected examples of extracted explanations in Figure 3. We found that gradient-baser explanations are less plausible than those using attention (with the exception of Grad. L2, which is similar to static attention) and that ones produced with SMaT achieve the highest plausibility, indicating that our learned explainer can produce human-like explanations while maximiz- ing simulability. Moreover, SMaT achieves a similar AUC score to the best performing attention layer and head, 4 while not requiring any human annotations. This is evidence that scaffolded simulability, while not explicitly designed for it, is a good proxy for plausibility and "human- like"" explanations.
AUC scores obtained by independently trying all attention heads and layers of the model.

---

Table Simulability results, terms of accuracy (%), on the CIFAR100 dataset. Underlined values represent better performance than baseline with non-overlapping IQR
2,250
4,500
9,000
No Explainer
81.16 (8..9)
84.02 [83.98884.24]
85.20 (85.17)
Gradient L2
80.97 [80.91:81.1]
83,98
[83881884.33]
85.13 [84,97:85.55 5500]
Gradient x Input
80.93 (80.82:81.04]
83,99 (83.9%884133]
85.33
(84.85:85.35
Integrated gradients
80.22 180.17:80.35
83,44
[83.25:83.44]
84.99
884.7:855223
Attention (all layers)
82.53 (82.53:82.62]
84.81
[84.74:84.92]
85.92
(85.78:85 94]
Attention (last layer)
82.34 (82.30:82.60]
84.65
[84.56:8488]]
85.31
(4484485,311
Attention (SMaT)
83.09 [82.77:83.28]
85,42
[85.39:85.85]
85.96
(85.74:66335
5.2 Image Classification
To validate our framework across multiple modalities, we consider image classification on the CIFAR-10K dataset [Krizhevsky, 2009]. We use the base model the Vision Transformer (VTT) (Dosovitskiy et al., 2020], in particular the base version with 16 x 16 patches that was only pretrained on ImageNet-21K [Ridnik al., 2021]. We up-sample images to a 224 224 resolution.
Since the self-attention mechanism in the VIT model only works with patch representations, the explanations produced by attention- based explainers will be at patch-level rather than pixel-level. We split the original CIFAR-100 training set into a new training set with 45,000 and a validation set with 5,000. Unlike the previous task, reuse the training set for both the teacher and student, varying the number of samples the student is trained with between 2,250 (5%), 4,500 (10%) and 9,000 (20%). We use accuracy as the simulability metric and the teacher obtains 89% test set.
Table shows the results for the three settings. Similarly to the results in the text modality, the attention explainer trained with SMaT achieves the best scaffolding performance, although the gaps to static attention-base -baeed explainers are smaller (especially when students are trained with more samples). Here, the gradient-basee explainers always degrade simulability across the tested training set sizes and and it seems important that the explanations include attention information from layers other than the last one.
Plausibility analysis. Since there are no available human annotations for plausibility in the CIFAR- 100 dataset, we design user study to measure the plausability of the considered methods. The original image and explanations extracted with Gradient x Input, Integrated Gradients, Attention (all layers), and Attention (SMaT) are shown to the user, and the user has to rank the different explanations answer the question "Which explanation aligns the most with how you would explain a similar decision?". Explanations were annotated by three volunteers. After collecting results, we compute the rank and the TrulSkill rating [Herbrich et al., 2007] for each explainer (roughly, the "skill" level if the explainers where players in game). Further description can be found in Appendix B. The results are shown in Table 4. As in the previous task, attention trained with SMaT outperforms all other explainers in terms of plausibility, and its predicted rating is much higher than all other explainers. We also show examples of explanations for set of randomly selected images in Figure 4.
Input image Integ. Grad. Attn, (all Ix) Attn. (SMaT) Input image Integ. Grad. Attn, (all Ix.) Attn, (SMaT)
Table 4: Plausibility results of the human study on visual expla- nations.
Rank TracSkill
Grad. Input 3-4
-2.7+.67
Figure 4: Explanations given by integrated gradients, attention (all layers), and learned attention explainer for a set input images of CIFAR- 100. Gold labels are: "television", "butterfly", "cockroach", and "sunflower".
Integ. Grad. 3-4
-2.14.67
Attn. (oll [x.
0.7t.67
Atm. (SMaT)
4.3+770
5.3 Machine Translation Quality Estimation
Quality Estimation (QE) is the task of predicting a quality score given a sentence in source language and translation in target language from a machine translation system, which requires models that consider interactions between the two inputs, source and target. Scores tend to be continuous values (making this a regression task) that were collected from expert annotators.
7

---

Table 5:
Simulability results, in terms of Pearson correlation, on the ML-QE dataset. Underlined values represent better performance than baseline with non non-overlapping IQR.
2,100
4,200
8,400
No Explainer
.7457 [,7366.7528]
.7719 (7766077802]
.7891 [.7860:.7944
Gradient 1.2
.8065 (8038:88688
.8535 [.7117:8544]
.8638 [.8411:8657]
Gradient x Input
.6846 [.6781:6894]
.6922
66685
.7141 [.71136.714]]
Integrated gradients
.6686 [66677.6694]
.7086
[6994;7701]
.7036 [,6976.7033]
Attention (all layers)
8120 8.7955:8125]
,8193
82800
.8467 [.8464.8521]]
Attention (last layer)
.7486 [.748477334]
.7720
[7672.77726
.7798 [,7717.7714]
Attention (SMaT)
.8156 [8809:88883]
.8630 (88412.8724)
.8561 [.8512:8889]
Table 6: Plausibility results for source and target inputs for each language pair of the MLQE-PE dataset in terms of AUC. # represents supervised methods that use human labels in some form.
EN-DE
EN-ZH
ET-EN
NE-EN
RO-EN
RU-EN
OVERALL
src.
(gt.
src.
tgt.
SEC.
tgt.
srC.
tgt.
tgt.
SEC.
tgt.
STC.
[gt.
Gradient T.2
0.64 0.65
0.65
0.49
0.67
0.61
0.68
0.55
0.72
0.68
0.65
0.54
0.67
0.59
Gradient x Input
0.58 0.60
0.61
0.51
0.60
0.54
0.61
0.49
0.64
0.59
0.58
0.51
0.61
0.54
Integrated Gradients
0.59 0.60
0.63
0.49
0.60
0.52
0.64
0.48
0.64
0.59
0.60
0.51
0.62
0.53
Attention (all layers)
0.60 0.63
0.68
0.52
0.60
0.61
0.58
0.55
0.66
0.70
0.62
0.55
0.62
0.59
Attention (last layer)
0,51 0.49
0.61
0.49
0.51
0,50
0.55
0.48
0.52
0.57
0.56
0.50
0.54
0,50
Atlention (SMaT)
0.64 0.65
0.68
0.52
0.66
0.64
0.66
0.54
0.71
0.70
0.61
0.54
0,66
0.60
Attention (best layer)
0.64 0.65
0.69
0.64
0.64
0.68
0.68
0.68
0.71
0.76
0.64
0.59
0.65
0.65
Atlention (best heady=
0.67 0.67
0.70
0.65
0.70
0.70
0.70
0.69
0.73
0.75
0.67
0.60
0.67
0.66
Interpreting quality scores of machine translated outputs is problem that has received recent interest [Fomicheva et al., 2021a] since it allows identifying which words were responsible for a bad translation. We use the MI.QE-PE dataset [Fomicheva al., 2020], which contains 7, 7,000 training samples for each of seven language pairs alongside word-level human annotation. We use as the base model a pretrained XLM-R-bass. [Conneau al., 2019], a multilingual model with 12 layers and 12 heads in each (total of 144 heads).
We exclude one of the language pairs in the dataset (si-en) since the XLM-R model did not support it, leading to a training set with 42,000 samples. Similar to the CIFAR- 100 case, we reuse the same training set for both the teacher and student, sampling a subset for the latter. We vary the number of samples the student is trained with between 2,100 (5%), 4,200 (10%) and 8,400 (20%). Since this is a regression task, we evaluate simulability using the Pearson correlation coefficient between student and teacher's predictions. 5 The teacher achieves 0.63 correlation on the test set.
Table 5 shows the results for the three settings. Similar to other tasks, the attention explainer trained with SMaT leads to students with higher simulability than baseline students and similar or higher than static explainer across all training set sizes. Curiously, the Grad. L2 explainer achieves very high simulability for this task. It even has a higher median simulability score than SMaT for 8,400 samples. However, we attribute this to variance in the student training set sampling (that could lead to an imbalance language pair proportions) which could explain why SMaT performance degrades with more samples. For this task, the gradient-basee explainers always degrade simulability across the tested training set size. It also seems that using only the last layer's attention is also ineffective at teaching students, achieving the same performance as the baseline.
Plausibility analysis. We select the median model trained with 4,200 samples and follow the ap- proach devised in the Explainable QE shared task to evaluate plausibility [Fomicheva et al., 2021a], which consists of evaluating the human- likeness of explanations in terms of AUC only the subset of translations that contain errors. The results are shown in Table 6. We note that for all language pairs, SMaT performs on par or better than static explainers, and only being surpassed by Grad. 1.2 in the source-side over all languages. Comparing with the best attention layer/head, an approach used by Fomicheva et al. [2021b]. Treviso al. [2021]. SMaT achieves similar AUC scores for source expla- nations, but lags behind the best attention layer/head for target explanations on *-EN language pairs. However, as stressed previously for text and image classification, SMaT sidesteps human annotation and avoids the cumbersome approach of independently computing plausibility scores for all heads.
$ Pearson correlation is the standard metric used evaluate sentence-level QE models.
8

---

Importance of the Head Projection
A major component of our framework is the normalization of the head coefficients, defined in $ 4. Although many functions can be used to map scores to probabilities, we found empirically that SPARSEMAX performs the best, while other transformations such SOFTMAX and 1.5-ENTMAX [Peters et al., 2019], a sparse transformation more dense than sparsemax, usually lead to poorly performing students (see Table 7).
Table 7:
Simulability results, in terms of accuracy (%), the MLQE dataset with 4200 training examples, with different normalization functions.
SPARSEMAX
SOFTMAX
I.S-ENTMAX
No Normalization
No Explainer
.7719 [,7600-7022]
.7719 (.766078002
.7719 (7660:7802]
.7719 [.7660:7802]
Atiention (all layers)
8193 8280
.7345+ (3703
[1
.7781 [17722:7791]
Attention (last layer)
.7720 à (.7722.776)
.76974 (.7699.7715)
.7807 (7652.78211
.7768 (.7764:7807]
Atlention (SMaT)
8630 [.8412:8724]
.7439 [.7430:7484]
.7163 (
8002 [.7919:8100]
Furthermore, another benefit of SPARSEMAX is that it produces a small subset of active heads. The heatmaps of attention coefficients (AT) learned after training, shown in Figure 5, exemplify this. We can see that the dependency between head position (layer it belongs to) and its coefficient is task/dataset/model specific, with CIFAR-100 and MLQE having opposite observations. We also found empirically that active heads (\/- 0) usually lead to higher plausibility scores than inactive heads, further reinforcing the good plausibility findings of SMaT.
confficients
Head confficinnts
confficients
0.08
0.03
-0.20
0.08
9.05
0.15
00.4
0.01
0.10
0.02
0.02
10-
18
0.02
18-
0.38
0.02
16
30
Miad
Head
Head
Figure 5: Head coefficients for text classification (left), image classification (middle), and quality estimation (right), illustrating that only a small subset of attention heads are deemed relevant by SMaT due to SPARSEMAX.
7 Related Work
Explainability for text & vision. Several works propose explainability methods to interpret decisions made by NIP and CV models. Besides gradient and iitention-based approaches already mentioned, some extract explanations by running the models with perturbed inputs [Ribeiro et al., 2016, Feng al., 2018, Kim et al.. 2020]. Others even define custom backward passes to assign relevance for each feature [Bach et al., 2015]. These methods are commonly employed together with post-pree post-processing heuristics, such as selecting only the top-k tokens/pixels with higher scores for visualization. Another line of work seeks to build a classifier with inherently interpretable components, such as methods based attention mechanisms and rationalizers [Lei et al., 2016, Bastings et al., 2019].
Evaluation of explainability methods. As mentioned in the introduction, carly works evaluated explanations based on properties such as consistency, sufficiency and comprehensivenness Jacovi and Goldberg [2020] recommended the use of a graded notion of faithfulness, which the ERASER benchmark quantifies using the idea of sufficient and comprehensive rationales, alongside compiling datasets with auman-annotale rationales for calculating plausibility metrics [De Young et al., 2020]. Given the disagreement between explainability methods, Neely et al. [2021] showed that without faithful ground-truth explanation is impossible to determine which method is better. Diagnostic tests such as the ones proposed by Adebayo et al. [2018], Wiegreffe and Pinter [2019] and Atanasova al. [2020] are more informative yet they not capture the main goal of an explanation: the ability to communicate explanation to practitioner.

---

Simulability. A new dimension for evaluating explainability methods relies on the forward pre- diction/simulation proposed by Lipton [2016] and Doshi-Velee and Kim [2017], which states that humans should be able to correctly simulate the model's output given the input and the explanation. Chandrasekaran al. [2018], Hase and Bansal [2020], Arora et al. [2022] analyze simulability via human studies across text classification datasets. Treviso and Martins [2020] designed an automatic framework where students (machine or human) have to predict the model's s output given an explana- tion as input. Similarly, Pruthi et al. [2020] proposed the simulability framework that was extended in our work, where explanations are used to regularize the student rather than passed as input.
Learning to explain. The concept of simulability also opens a path to learning explainers. In particular Treviso and Martins [2020] learn an attention explainer that maximizes simulability, However, directly optimizing for simulability sometimes led to explainers that learned trivial protocols (such as selecting only punctuation symbols or stopwords to leak the label). Our approach of optimizing a teacher-student framework similar to approaches that optimize for model distillation [Zhoo et al., 2021]. However, these approaches modify the original model rather than introduce a new explainer module. Raghu et al. [2021] propose a framework similar to ours for learning commentaries for inputs that speed up and improve the training of model. However commentaries are model-indepen indeeedende and are optimised to improve performance on the real task. Rationalizers [Chen et al., 2018, Jacovi and Goldberg, 2021, Guerreiro and Martins, 2021] also directly learn to extract explanations, but can also suffer from trivial protocols.
Conclusion & Future Work
We proposed SMaT, framework for directly optimizing explanations of the model's predictions to improve the training of a student simulating the said model. We found that, across tasks and domains, explanations learned with SMaT both lead to students that simulate the original model more accurately and are more aligned with how people explain similar decisions when compared to previously proposed methods. On top of that, our parameterized attention explainer provides principled way for discovering relevant attention heads in transformers.
Our work shows that scaffolding a suitable criterion for both evaluating and optimizing explainabil- ity methods, and we hope that SMaT paves way for new research to develop expressive interpretable components for neural networks that can be directly trained without human-labeled explanations. However, it should be noted that "iiterprettaility"" is a loosely defined concept, and therefore cau- tion should be exercised when making statements about the quality of explanations based only on simulability, especially if these statements might have societal impacts.
We only explored learning attention-based explainers, but our method can also be used to optimize other types of explainability methods, including gradient-based ones, by introducing learnable parameters in their formulations. Another promising future research direction is to explore using SMaT learn explanations other than saliency maps.
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by P2020 project MAIA (LISBOA-01-0247 FEDER045909), and Fundação para a Ciência e Tecnologia through project PTDCICCCIINF44733/201 (PRELUNA) and contract UIDB/50008/2200. We grateful to Nuno Sabino, Thales Bertaglia, Henrico Brum, and Antonio Farinhas for the participation in human evaluation experiments.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Siddhant Arora, Danish Pruthi, Norman Sadeh, William Cohen, Zachary Lipton, and Graham Neubig. Explain, edit, and understand: Rethinking user study design for evaluating model explanations. In Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI), Vancouver, Canada, February 2022. URL https: // arxiv. org/ abs /ttp//222..0900 09669.
10

---

Leila Arras, Grégoire Montavon, Klaus Robert Müller, and Wojciech Samek. Explaining recur- rent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 159-168, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. 10i:10. 1e.118663/11// 1/W11/21/
org/W17-5220
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 3256-3274, Online, November 2020. Association for Computational Linguistics. 1aii:1..165//1//21 https://acl hattholoyy org/ ww00./20002ca emnlp-main 263.
emnlp mappp-mmi....... URL
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10, 2015.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages
Florence, Italy, July 2019. Association for Computational URL htLpstt::// http://aaaaaanttoooo////oo
Linguistics. doi:
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. "will you find these shortcuts?" A protocol for evaluating the faithfulness of input salience methods for text classification. CoRR, abs/2111.07366 2021. URL https: //wxxv/axivvv//////////////aaaaa org/ abs/2111 221..21111.7....
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miln and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL httt:://githut com/ google/jax.
Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make VQA models more predictable to human? In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pages 1036- 1046, Brussels, Belgium, Dotober-Nowember 2018. Association for Computational Linguistics. URL BRKL:tttts:/aallantbllgggg org/Dog 1128.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information- theoretic perspective on model interpretation. In International Conference on Machine Learning, pages 883-892. PMLR, 2018
Kevin Clark, Minh- Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training text encoders as discriminators rather than generators. In ICLR, 2020. URL https: //openreview.o wwwpaarwwwieeeeeeeddeaaaeeeeeeee netpppdffiddrrr11f1BBB
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: 1et::9911202211 1.02/ 16, 2019.
Misha Denil, Alban Demiraj. and Nando de Freitas. Extraction of salient sentences from labelled documents. ArXiv,, abs//4 142.6815 2014.
Lucio M. Dery, Paul Michel, Ameet S. Talwalkar, and Graham Neubig. Should be praitnni? an argument for end-task aware training as an alternative. ArXiv, abs/2109.07437. 2021.
Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceed- ings the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online, July 2020. Association for Computational Linguistics. doi: 1a1.16663.11/22002ac-mmmmm. URL https : htwpa//aaannntoyo..oeooo org/2020. acl-main. mmmii.40.... 408.
Shuoyang Ding, Hainan Xu, and Philipp Koehn. Saliency-drivar word alignment interpretation for neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Vol- ume 1: Research Papers), pages 1-12, Florence, Italy, August 2019. Association for Computational Linguisties. doi: 1aa:10.116630 11///3////1112000/ URL https:
Ptg/W19-5220
11

---

Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: wetp:::02.000000 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologies neural models make interpretations difficult. Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 1719-3728, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:10.188 1/865//1/D111144000 org/ fegg/D8--o 1407.
URL attps://accanttaaae
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017. URL pestps://proceediee mll parrceeiingg.mr..pres////7 press/v70/ finn17a. html.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 539-555, 2020.
Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. The Eval4NLP shared task on explainable quality estimation: Overview and results. In Proceed- ings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165-178, Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 1oi:10..1653/// 17.
URL http:p/:///aaaaahhll/ooos /atpc//lattholoooor///000....11 1.
Marina Fomicheva, Lucia Specia, and Nikolaos Aletras. Translation error detection as rationale extraction. arXiv preprini arXiv:2108. 12197, 2021b.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. arXiv preprint arXiv:1910.01727 2019.
Nuno M. Guerreiro and André F. T. Martins. SPECTRA: Sparse structured text rationalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6534-6550, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. 1ai:10.18633/1/220 http:/221...mmmmmmiii..5.... URL https: // //aclanthology org/ 2021 emnlp-main 525.
Peter Hase and Mohit Bansal. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540-5552, Online, July 2020. Association for Computational Linguistics.
1e..1866//111222.adlll acl-main.494
URL /tttutt///aalaatttoooaaa /aaaaaaaaaanoooy.....o org/2020.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http://githut www./oogmll.oooooo http//ooglllffla/e
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill(tm): A bayesian skill rating sys- tem. In Advances in Neural Information Processing Systems 20, pages 569-576. MIT Press, January 2007. URL https: wwwp:/wwwwwwww.t.ow./w/t/tw watessotttccomeeee http://uwnu/eeeeeercccaalccceeee trueskilltm - wwwllllaattiggsseeee.ea
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4208 Online, July 2020. Association for Computational
12

---

Linguistics. 1 URL https://aclanthology/ org/2020. ael-main..880
Alon Jacovi and Yoav Goldberg. Aligning Faithful Interpretations with their Social Attribution. Transactions of the Association for Computational Linguistics. 9:294-310, 03 2021. ISSN 2307- 387X. doi: 10. 162/tacl a 00367. URL huttps:///ot wwg/oo/////oo 10. http:/acl_aa____00
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, Volume I (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.186633/1///11113555 URL https: /www. aclweb. org/ anthology/N19t N19-1357.
Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. Interpretation of NL.P models through input marginalization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 315--3667, Online, November 2020. Association for Computational 255. URL https: https////aaaaalllga/aeee org/2020.
Linguistics. doi: 10.1 11111//1122200emmmm//a// momlp-main... 255.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653//112020.emmlpp www.mmmmiin.570501 URL https: //aclanthology - org/ 2200/2020..2 emnlp-main.577 574.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In roceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, 2016.
Piyawat Lertvitaayakumjon and Francesca Toni. Human- grounded evaluations of explanation methods for text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1995-220,, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:10. 10i:100.1653340 1tW.1/86///1111111555 URL D19- 919-.
org/
Zachary C. Lipton.
The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, jun 2016. ISSN 1542-7730. doi:
URL https://doll org/ 10. 1145/3236386. 3241340.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http:::///ppeeerie..eeetttrrtuse Bkg6RicqY7.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Compulational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011 1. Association for Computational Linguistics. URI. http: aclweb. org/ www./rrrggggggggoooooee hatp//atoooo///11--11/ PP11
Andre Martins and Ramon Astudillo. From softmax to sparsemax: sparse model of attention and multi-label classification. In Maria Florina Balcan and Kilian Q Weinherger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https:/ htaps:://proeeee http:///ooeeeenn..//// mlr, wwwpp/eeeseeeee /v8///v8//// ht///martinn6.... html.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial
intelligence, 267:1-38, 2019.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. ArXiv, abs/2006. 14032, 2020.
13

---

Michael Neely, Stefan F Schouten, Maurits JR Bleeker, and Ana Lucic. Order in the court: Explainable ai methods prone to disagreement. In ICML Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI, 2021. URL http::://axxvvv.rrr absa 2105.. .02877.
Ben Peters, Vlad Niculae, and André F. 'T. Martins. Sparse sequeence-f--seequeeee models. In Proceed- ings of the 57th Anmual Meeting of the Association for Computational Linguistiiss pages 1519, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10. 1et.110.1665550 11153//1//111////1 URL https: / /ac//aaaannaa lonhllogy org/ 1146.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: URL fttps://aacaanthllogg org/ N0t//N88333 1202.
Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. Evaluating explanations: How much do explanations from the teacher aid students? CoRR, abs/2012.00893, 2020. URL, attps://arxiii weriii.orrg/aae arg/2bs/2022 00893.
Aniruddh Raghu, Maithra Raghu, Simon Komblith, David Duvenaud, and Geoffrey Hinton. Teaching with commentaries. In International Conference on Learning Representations, 2021. URL htttps://openreviert nett//oouuridddAbaaaaaaa
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
T. Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses. ArXiv, abss/114..ra 10972, 2021.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. on learning now to learn: The meta-metta-mett..-hookk Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL hattp://wwat idsia. ch/ jul//euerrennnaeee hwwpggee/ddplommeee oma. html.
Soñia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-29511 Florence, Italy, July 2019. Association for Computational Linguistics. 19b10.11655//11/199
URL https:
//aclanthology. . org/ 1atg//19-12211
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1723-1733 1732. PMLR, 16-18 Apr 2019. URL attps::/prooceedinn wwagggssm.l.............. mlr. www.ppressvvie hatp///////aaaaee///// 19a. html.
Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2019. URL https:://// / //tpx//ax/vvv///////// org/ wwtg/b///////ooo/e//o////ee atws/1999.00 08053.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1319-3328. PMLR, 06-11 Aug 2017. URL https:://proceee
mlr wwt.:emww..p.../eeeeeeeemme wwwps/vo0/ htto//nnnnaaaaaaaaaa/... html.
Marcos Treviso and André F. T. Martins. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 107-118, Online, November 2020. Association
14

---

for Computational Linguistics. aclweb. org/ htt//annoooly//22000 blackboxmlp-1.110
URL https://www
Marcos Treviso, Nuno M. Guerreiro, Ricardo Rei, and André T. Martins. IST-unbabel 2021 submission for the explainable quality estimation shared task. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 133-145, Pumta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: www.11/6511/000/ http://1122eLeall4lgggg Leval4nlp- 1.14. org/20222 1er222.eval41m eval4nlp- 1.14.
URL https: httpee///aaaaaahooocooo
Keyon Vafa, Yuntian Deng, David Blei, and Alexander Rush. Rationales for sequential predic- tions. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 10314-10332, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 36653//1/222..emmlppmoo attps://acle http//cccnnhhooogyyyooooo org/ org/2021. 20211. emmlp-main... 807.
URL
Janneke Van de Pol, Monique Volman, and Jos Beishuizen. Scaffolding in teacher- interaction: decade of research. Educational psychology review, 22(3):271 2010.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. Attention inter- pretability across nlp tasks. ArXiv, abs/1909. 11218, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / ppwoce/edng./............ neurips. cttpp//r//pppppe////////// 2217
Paa.:05a.c..aaabaaaaaae.... -df..
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.110.111 118818661/1DD/ 1002. URL https://aclanthont www.gy.......oo........... oogoorgg//. Dtt//19-1100 1002.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Compu- tational Linguistics. URL https://////// h/tp::://wwww aclweb. orgganntologyy//////goooo/ /200. emnlp-demos
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi. Yair Carmon, Simon Korblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https:/ www.i.orgg/aa....e.....oaaoo www.//gg////////e/e////eeeeeee/e ab2/ //////2223.22.....2
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th
Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing [Volume 1: Long Papers), pages 6707-6723, Online, August 2021. Association for Computational Linguisties. doi:10.18653/112000 acl--ong..533. URL https: h/tp////aaaaalag////a//// org/2021 2t///2221.a. acl- acl-long.......... 523.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Meta learning for knowledge distillation. arXiv preprint 1rX:5:2006.04570, 2021.
15


Reviews:
[{"review_id": "ppcEd3F0ETd", "reviewer": "Reviewer_b9Jw", "summary": "The authors introduce SMaT (Scaffold-Maximizing Training),  a framework to directly optimize the explanations produced on a teacher model to improve the distillation of a student model simulating the teacher model (and creating an explainer for the student model too). The authors limit the architecture of the models to attention based transformers, and conduct experiments on three tasks (nlp, translation, image classification). Using SMaT resulted in students which more accurately simulate the teacher and produces more plausible explanations.", "questions": "- Would SMaT work in the case of a born-again network? In all the examples the student is limited to a smaller set of data.", "limitations": "The authors do not address this matter.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents an interesting approach to learning explainers in such a way as to more accurately distill a teacher to a student model. The authors demonstrate their results over three very different tasks. The authors point show when other explainability approaches (Gradient L2 for example), results in improved simulability (often when more data is made available). The paper is well written and the experiment is well designed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z5RgdwGWkg3", "reviewer": "Reviewer_Z8bs", "summary": "The paper proposes a bi-level optimization procedure to directly optimize the simulability objective of explanability. In order for the optimization to work well, the paper proposes a new parametrized attention layer that is a weighted sum of the attention heads from Transformer models. This new layer (SMaT) works very well under this objective, showing high simulability (expected) and plausibility rating over other saliency-based methods.", "questions": "- Can you comment on what is the student model? Is it only constrained in a way that it has less parameters than the teacher, but otherwise receive the same input and have the same architecture as the teacher? Can your work be viewed as a type of a model-compression/knowledge distillation? \n- There is a type of work that optimizes through reinforce to hard-select words that shows to have high precision at selecting words that are also selected from human rationale [3]. This also referred to as \"hard attention\" -- have you guys considered showing a baseline of this method for AUC on MovieReviews? Is it better than SMaT or worse?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall comment:\n\nI'm generally impressed by this paper. Explainability work has seen a promising set of development in the last 3 years focusing specifically on the \"why\" and \"so what\" question of explainability -- why do we want explainability and what should explainability do (functionally). I'm excited to see that this paper takes a step in that direction focusing on pushing the boundary on these new properties of explanations. However, though generally well-written, I'm a bit concerned with a few choices the authors took. First of all, the explainers being differentiable. This significantly limits the type of explainers we can use, especially the game-theoretic ones. Attention is not explanation [1] because it does not faithfully represent the output of the model (and does not conform to the axioms of explainability [2]). The authors did use Integrated Gradients -- which mitigates this part of my criticism. Second concern, since the learned attention is the \"winning\" explainer model, I wonder if this is because **attention is the superior explanation method** or because **attention explainer is easier to optimize under the bi-level optimization objective the paper proposed** (aka., any method that can maximize this objective will be a good explanation method, regardless on whether they satisfy other criteria of explainability such as completeness). This is probably a very hard question to address or to design experiments to falsify,  but would like to hear authors' thoughts on this.\n\nStrengths:\n- The paper is clear and well-written\n- The experiment is very thorough (including all three major ML application fields)\n- Plausibility is investigated through either human annotations or user studies\n- The intention to directly optimize \n\nWeaknesses:\n- Some philosophical questions asked above.\n- Could compare to some other interpretability baselines like shapley values (but this is not my main concern, and nor should we require every single paper to run experiment on every method that the reviewer happens to know)\n\n[1] Jain, S., & Wallace, B. C. (2019). Attention is not explanation. arXiv preprint arXiv:1902.10186.\n[2] Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319-3328). PMLR.\n[3] Lei, T., Barzilay, R., & Jaakkola, T. (2016). Rationalizing neural predictions. arXiv preprint arXiv:1606.04155.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SX0xZEibsqV", "reviewer": "Reviewer_HtBu", "summary": "This work introduces Scaffold-Maximizing Training, a process for learning better “explanations” for learned models by optimizing the effect the produced explanations have on downstream “simulatability.” There’s a lot of terminology and nuance to the arguments in this paper, but in reading through it, my understanding is as follows:\n\n- Given a trained model T (off the shelf, black-box, e.g., an image/text classifier), we want to be able to generate good explanations of why it predicted a certain output.\n- How do we know what a “good explanation” is? The field hasn’t yet converged to a single metric, but one plausible measure that seems to be well-adopted and well-motivated is the *simulability* of the trained model T (or teacher model); in other words, how well can we use the explanations to train a student model (lower capacity than the teacher T) that “simulates” how the teacher model T performs on held-out data.\n- The idea is that a student that perfectly simulates the teacher T both in terms of prediction, *and in terms of explanations w/ its own explanation model* is a sign that the original explanations for T are good!\n\nIn general, the paper builds up this narrative cleanly, ultimately arriving at a bi-level optimization problem for learning the student model parameters (as well as the student model’s explanation model), followed by the teacher model’s explanation model as the outer loop. Experiments on image classification, text classification, and multilingual regression (translation quality estimation) show that explainers trained via SMaT lead to higher simulability scores and “plausibility” (human likeness of explanations) than methods that just statically condition on some external explanation process, or that don’t use explanations at all.", "questions": "- Following from the last paragraph of weaknesses — what happens if you remove the explanation regularization from the objective for all of the static and learned models (basically ablate the student explainer model altogether). At this point, I don’t know if the better simulability is actually a result of the bi-level optimization procedure (learning better explanations) or just learning to regress explanations with a student model...\n", "limitations": "This paper is fairly clear about its limitations, acknowledging that the “explanation” methods used in this work are mostly saliency driven (attention maps). Like the paper suggests, I’d be interested in seeing applications of SMaT to other explainability approaches!\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper are in it’s clarity and originality; I strongly believe that each part of the paper is well-motivated, and seeks to build to an answer of the question — what makes better explanations? The experiments are comprehensive and the results while not overwhelmingly convincing, do make enough of a point that the SMaT protocol is effective.\n\nHowever, the biggest weakness of the paper is the assumption driving the main optimization objective of the framework — the inner-level optimization not only optimizes the simulability score of the learned student model relative to the teacher, but the “explanation regularization loss” between the explanations produced under the student and the explanations produced under the teacher. It seems to me that this is an incredibly strong assumption, that doesn’t immediately fall out of the logic from the paper — if the goal is to produce better explanations via simulability, and optimize the explanations for better simulability, then this sort of “explanation matching” loss is not necessary?\n\nEDIT: Rebuttal cleared this up!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept