Gnothi Seauton: Empowering Faithful Self-Interpretability in Black-Box Transformers
Authors: Shaobo Wang
Hongxuan Tang
Mingyang Wang
Hongrui Zhang
Xuyang Liu
Weiya Li
Xuming Hu
Linfeng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Transformers
Interpretability
Efficient-AI
Shapley Value
Primary area: interpretability and explainable AI
TL;DR: We introduce a method that makes a black-box transformer model self-interpretable. By using a side network to generate Shapley values, it reduces memory, training, and inference costs while maintaining prediction accuracy.

Abstract: The debate between self-interpretable models and post-hoc explanations for black-box models is central to Explainable AI (XAI). Self-interpretable models, such as concept-based networks, offer insights by connecting decisions to human-understandable concepts but often struggle with performance and scalability. Conversely, post-hoc methods like Shapley values, while theoretically robust, are computationally expensive and resource-intensive. To bridge the gap between these two lines of research, we propose a novel method that combines their strengths, providing theoretically guaranteed self-interpretability for black-box models without compromising prediction accuracy. Specifically, we introduce a parameter-efficient pipeline, AutoGnothi, which integrates a small side network into the black-box model, allowing it to generate Shapley value explanations without changing the original network parameters. This side-tuning approach significantly reduces memory, training, and inference costs, outperforming traditional parameter-efficient methods, where full fine-tuning serves as the optimal baseline. AutoGnothi enables the black-box model to predict and explain its predictions with minimal overhead. Extensive experiments show that AutoGnothi offers accurate explanations for both vision and language tasks, delivering superior computational efficiency with comparable interpretability.

Published as a conference paper at ICLR 2025
GNOTHI SEAUTON: EMPOWERING FAITHFUL SELF- INTERPRETABILITY IN BLACK-BOX TRANSFORMERS
Shaobo Wang 1.2 Hongxuan Tang² Mingyang Wang" Hongrui Zhang?
Xuyang Liu Weiya Li Xuming Hu
Linfeng Zhang
"ccoool of Artificial Intelligence, Shanghai Jiao Tong University
Efficient and Precision Intelligent Computing Lab, Shanghai Jiao Tong University
3 Sichuan University "Big Data and AI Lab, ICBC
Hong Kong University of Science and Technology, Guangzhou {a [shaobowangl 009, shanglinfent @ jtu..du....
ABSTRACT
The debate between self-interpretable models and post-hoc explanations for black- box models is central to Explainable AI (XAI). Self-interpretable models, such as concept-based networks, offer insights by connecting decisions to human- understandable concepts but often struggle with performance and scalability. Con- versely, post-hoe methods like Shapley values, while theoretically robust, are computationally expensive and resource-intensive. To bridge the gap between these two lines of research, we propose a novel method that combines their strengths, providing theoretically guaranteed self-interpretabiiity for black-box models without compromising prediction accuracy. Specifically, we introduce a -eficieet pipeline, AutoGnothi, which integrates a small side network into the black-box model, allowing it to generate Shapley value explanations with- out changing the original network parameters. This side-tuning approach signif- icantly reduces memory, training, and inference costs, outperforming traditional parameter-efficient methods, where full fine-tuning serves as the optimal baseline. AutoGnothi enables the black-box model to predict and explain its predictions with minimal overhead. Extensive experiments show that AutoGnothi offers ac- curate explanations for both vision and language tasks, delivering superior com- putational efficiency with comparable interpretability.
(a)
(c)
Can we fine-tune the black box model to make it hate-etttrrrlllle?
Predictions
Hard to achieve
Dataset
understand
Black-box Model
White-box Model
0.2% #Params
66.1% Time
Efficient
Predictions
2..9%% 99 Memory
Dataset
Black box Model
120%
#P #Prrams
Not efficient
Predictions
113% Time
121% Memory
Explanations
Dataset
Self-interpretable
Explanations
Dataset
Explainer
Model
Figure Different paradigms towards XAL (a) The ideal paradigm for XAI envisions using white-box models for prediction, which are inherently self-interpretable by design but hard to achieve. (b) The previous paradigm involves post-hoc explanations of black-box models by training separate, weight explainer. (c) We propose novel efffiettrrefffccent paradigm, AutoG- nothi, which fine-tunes the black-bor box model to make it self-interpretable.
"Coreeponning Author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Explainable AI (XAI) has gained increasing significance as Al systems are widely deployed in both vision (Dosovitskiy 2020; Radford et al. [202] Kirillov al. 2023) and language domains (Devlin) et al. 2019 Brown 2020 Achiam et al. 2023). Ensuring interpretability in these systems is vital for
fostering trust, ensuring fairesss, and adhering to legal standards, particularly for complex models such as transformers. As illustrated in Figure Ia), the ideal paradigm for XAI involves designing inherently transparent models that deliver superior performance. However, given the challenges in achieving this, current XAI methodologies can be broadly classified into two main categories: developing self- interreeaabee models and providing post-hoc explanations for black-box models.
Designing Self-lnttrpretabll Models: Several notable efforts have focused on designing self- interpretable models that are grounded in solid mathematical foundations or learned concepts. Among these, concept-based networks have emerged a representative approach linking model de- cisions to predefined, human- undaassndndstt concepts (Kim et al., [2188] Koh et al.) [2020] [Avarez-] Melis & Jaakkola 2018). However, incorporating hand-crafted concepts often introduces trade-off between interpretability and performance, as these models typically compromise the performance of the primary task. Moreover, such methods are often closely tied to specific architectures, which limits their scalability and transferability to other tasks. Furthermore, the explanations generated by concept-based models often lack a rigorous theoretical foundation, raising concerns about their reliability and overall trustworthiness.
Explaining Black-Box Models: Given the chal- lenges of designing selfrintereretable models for practical applications, post-hoc explanations for
0.99
Covert et al.
Explainition Faithfulness
0.98
black-box models have become widely adopted al- ternative. Among these, Shapley value-based meth-
Moda
Methed
SLOPs
Minsary
VIT Shpplty
6.7N
2,635
303.0GMB
0.97
AutiGrans
2J5N
E/000
24-23B
ods (Shapley 1953) are particularly valued for their theoretical rioor and adherencc to four principled axioms Young 1985). However, calculating act Shapley values involves evaluating all possi- ble feature combinations, which scales exponentially with the number of features, making direct compu- tation impractical for models with -ineesiimnsi inputs. To alleviate this, methods like Fast- SHAP (Jethani al. 202T) and ViT-Shapley (Covert et al. 2022) employ proxy explainers that estimate Shapley values during inference, significantly reduc- ing the number of evaluations needed. While these approaches reduce some computational costs, train- ing separate explainer remains resource- intturcee For example, training a Vision Transformer (ViT) explainer requires more than twice the train- ing GPU memory compared to the VIT classifier itself. Moreover, solely depending post-hoc explanations for black-box models is not ideal in high-stakes decision-making scenarios, where im- mediate and reliable interpretability is required (Rudin) 2019).
VIT Stapls
26413
10.360
42410M2
ViT.5
0.96
AskaGaods
0.63M
6.230
91.1248
108.22M
41.70
1000.2194B
L
0.95
AskaGeada
2,034
6.8G
365.510
Ours
Mr7-Stapdey
132,600
5164.2558
3,306
F305 175M
0.94
50
150
200
250
300
350
#Trainable Parameter (M)
Figure 2: Explanation quality on the Ima- geNette dataset using different VITs. Our AutoGnothi significantly reduces the number of trainable parameters, computational costs (FLOPs), and training GPU memory storage without compromising explanation quality.
To bridge the gap between existing methods and address the aforementioned challenges, the core ob- jective of our research is achieve theoretically guaranteed self-interpretabiliiy in advanced neu- ral networks without sacrificing prediction performance, while minimizing training, memory, and inference costs. To this end, we propose novel paradigm, AutoGnothi, which leverages parameter- efficient transfer learning (PETL) to substantially reduce the high training, memory. and inference costs associated with obtaining explainers. As depicted in Figure Fa.) traditional model- specific methods require two training stages: (i) fine-tuning a pre-trained model into a surrogate model, and (ii) training an explainer using the surrogate model. During inference, the original model is used for prediction, while separate explainer network generates explanations, leading to two inference passes and double the storage overhead. In contrast, AutoGnothi utilizes side-tuning to reduce both training and memory costs, shown in Figure b)) By incorporating additive side branch paral- lel to the pre- pre-traine trained model, we efficiently obtain a surrogate side network through side-tuning. We then apply the same strategy to develop the explainer side network, enabling simultaneous predic-
2

---

Published as a conference paper at ICLR 2025
Target: The pre- trained Mack- box model be explained:
Legozad
Self Sul-attattonllock
Foward
Prodictin octainal topus
Torning pertrgpeemttts
Pallycorreecdd leyer
Propegdion
Covert al. inference
(b) Auto nothi (ours) 1x inference / -554% FLOPs/ / -44% Time
Stage Fully P line-tuning blac's box modil obbin surmggato model
Stage Side-tuning black-box model obtain suroggate model
Prodiction for the markedd repet
impuc
Midioed Irput
Preduction for the maskedd japut
Stage Fully Training exglainer modkl with the sumogotic model
Stage 2: Side-tuninng black-box to obrain a self- self-lnterrretable model
Prodictio foe the ocidinal lnput
Iequal
Prsduction reiiinal laput
Ingat
Sansi lspad.
Explorsbon
Explandiio
Figure 3: Overview of AutoGnothi compared to prior work. (a) ViT-Shapley (Covert et al. 2022 fully fine-tunes the black-box model to create surrogate model, then trains a separate explainer based on the surrogate, which is resource-intensive. (b) We employ side-tuning to efficiently obtain both the black-box model and explainer, significantly reducing training costs. AuloGnothi uses single model to simultancously generate predictions and explanations, lowering inference costs by leveraging shared features. In contrast, ViT-lly needs to load two models for prediction and explanation, respectively, and infers two times. AutoGnothi enables seff-interpretability for an arbitrary black-box model. We ignore the positional encoding associated with the pipeline.
tion and explanation in single inference step. An illustrative example comparing the efficiency of AutoGnothi with previous methods presented Figure
More importantly. AutoGnothi achieves self- interrettrtrty without compromising prediction accu- racy. Unlike a simple application of PETL, where full fine-tuning considered the optimal baseline, approach goes further. Experimental results show that relying full fine-tuning to achieve self- interpretability often leads degraded performance contrast, AutoGnothi maintains prediction accuracy while achieving self-interpretability by lever- aging the intrinsic correlation between prediction and explanation. Beyond classical PETL, which primarily focuses on training efficiency, AutoGnothi also enhances inference efficiency through self- interpretation while keeping its faithfulness (see Section 4.2 for further discussion). Our key contri- butions follows:
either prediction or explanation tasks. In
1. Efficient Explanation: We introduce a novel PETL pipeline, AutoGnothi, which enables any black-box models, e.g., transformers, become self-interpretable without affecting the original task parameters. By integrating and fine-tuning an additive side network, suprisingly surpasses previous methods training. inference, and memory efficiency.
Self-interpretabilit:: We achieve theoretically guaranteed self-interpretability for the black-box model with the Shapley value, without any influence the original model's prediction accuracy.
3. Broad Applications the most widely used models, including ViT (Dosovitskiy) (020)) for image classification and BERT (Devlin al. 2019) for sentimental analysis, showing that our methods outperform well on explanation quality. Specifically, for the ViT-base model pre-trained on ImageNette, our surro- gates achieve a 97% reduction in trainable parameters and 72% reduction training memory with comparable accuracy. For explainers, we achieve 98% reduction in trainable parameters, 77% reduction in training memory. For generating explanation, AutoGnothi achieves 54% reduction in inference computation, 44% reduction in inference time, and total parameter reduction of 54%.
both Vision and Language Models: We conducted experiments
RELATED WORK
Among post-hoc explanation methods, the Shapley value (Shapley 1953 is widely recognized as faithful and theoretically sound metric for feature attribution, uniquely satisfying four key axioms: efficiency, symmetry, linearity, and dummy. However, computing Shapley values is computationally expensive, requiring O(2") operations to calculate single Shapley value for one feature in a set size n. To alleviate this computational burden, various approaches have been proposed expedite Shapley value computation, which can
Explaining Black-Box Models with Shapley Values:
3

---

Published as a conference paper at ICLR 2025
be broadly divided into model-agnostic and model-specific methods (Chen al. 2023a). Model- agnostic techniques, such as KemelSHAP (Lundberg) 2017) and its enhancements (Covert Leep 2020), approximate Shapley values by sampling subsets of feature combinations. Nevertheless, when the feature set is large, the sampling cost remains prohibitive, and reducing this cost compro- mises the accuracy of the explanations, importance. Conversely, model-specific methods, such as FastSHAP (Jethani et al., (222]) and ViT- Shapley (Covert al. 2022), employ a trained proxy explainer to accelerate the estimation during inference, though these methods still involve significant training costs to develop the explainer.
fewer samples lead to less reliable estimates of feature
Parameter Efficient Transfer Learning (PETL):: PETL aims to achieve the performance of full fine-tuning while significantly reducing training costs by updating only small subset of parameters. In this context, Adapters (Houlsby et al. 2019 Chen et al. 2023b) introduce trainable bottleneck modules into transformer layers, enabling models to deliver competitive results with minimal pa- rameter adjustments. Another widely adopted method, LoRA (Hu et al., (02T), applies low-rank decomposition to the attention layer weights. Our work aligns more closely with side-tuning meth- ods, where Side-Tuning (Zhang et al. (020)) integrates an auxiliary network that merges its repre- sentations with the backbone the final layer, demonstrating effectiveness across diverse tasks in models like ResNet and BERT. LST (Sung et al. (2022) further improves this approach by reducing memory consumption through ladder side network design. However, none of these methods plore the transfer of interpretability from the main model to the side network, leaving this a largely unexplored area in side-tuning and PETL research.
BACKGROUND
3.1
SHAPLEY VALUES
The Shapley value, originally introduced in game theory (Shapley) 1953), provides a method to fairly distribute rewards among players in coalitional games. In this framework, a set function assigns value to any subset of players, corresponding to the reward earned by that subset. In machine learning scenarios, input variables are typically regarded as players, and a deep neural network (DNN) serves as the value function, assigning importance (saliency) to each input variable.
Let 8 E (0, 10 be an indicator vector representing a specific variable subset for a sample x = [21, T2, F.d T E Réd Specifically, 2g denotes the variables indicated by while those not in are replaced by a masked value (e.g., a baseline value). Let e; E R2 denote the vector with a one in the i-th position and zeros elsewhere. For game involving d players-or equivalently, DNN v : {0, 1]d + R with d input variables- the Shapley values are denoted by (x1))),..... (xa)- Each oe E R represents the value attributed to the i-th input variable 2; in the sample The Shapley value (x;) is computed as follows:
(x.))) = Σ
1
(T&s) (U(E)-e)) - v(x.))
(1)
=:
Intuitively, Eq. (I) captures the average marginal contribution of the i-th player to the overall reward by considering all possible subsets in which player could be included. Shapley values satisfy four key axioms: linearity, dummy player, symmetry, and efficiency (Young) 1985). These axioms ensure a fair and consistent distribution of the total reward among all players.
3.2 MODEL-BASER ESTIMATION OF SHAPLEY VALUES
Calculating Shapley values to explain individual predictions presents substantial computational chal- lenges (Chen al. 2023a). To mitigate this burden, these values are typically approximated using sampling-basee based estimators, such as those in (Lundberg, 2017, Covert & Lee, (020), though the sampling cost remains considerable. Recently, a more efficient model-based approach, introduced in (Jethani al. 2021), accelerates the approximation by training a proxy explainer to compute Shapley values through a single model inference. However, this method has not been validated on advanced neural architectures such transformers.
Building on this, ViT Shapley (Covert et al. 2022) was introduced to train a ViT explainer that interprets a pre-trained ViT model f. As shown in Figure a). the learning process of the explainer
4

---

Published as a conference paper at ICLR 2025
Prediction Quality
Explanation Quality
1.0
R.SE
Leave-cne-out
0.8
0.8
0.8
GredCM
Probstility
Pubbaliity
0.6
0.6
LRP
Alteriinn rellout
Insertion
Deletion
Allenion lest
0.4
Varilla
SmoothGrad
Covertet al:
Vartrad
AutoGnothy (ouns
Covert all
100
150
200
150
200
Masked Patches
# Inserted Patches
Deleted Patches
(a)
(b)
(c)
Figure 4: Training performance of surrogate and explainer models. (a) Prediction accuracy of masked inputs for the original classifier, the surrogate model trained with ViT Shapley (Covert et al. 2022), and our AutoGnothi. AutoGnothi shows greater robustness as the number of masked patches increases. For each mask size, randomly sampled 100 images and generated 10 random masks. The curve represents the average prediction probability. (b) Explanation quality, measured by insertion and deletion metrics, for various explanation methods. We randomly sampled 1,000 images and averaged the prediction probabilities to assess insertion and deletion performance. All experiments were conducted on the ImageNette dataset using the ViT-base model.
consists of two stages. In stage 1, a surrogate model ga with parameters 3 is generated by fine- tuning the pre-trained VIT classifier f handle partial information, which used for calculating the masked variables in the Shapley value. This involves aligning the output distributions of the surrogate model ga with the classifier f. The surrogate is optimized with the following objective:
E Laure (B) =-p(2).--10001))
[KLL (f())199(x.)))
where U(00, d) is a uniform distribution for sampling s. Then, in stage 2, an explainer de with parameters 0 is trained to generate explanations of the predictions of the black-box VIT utilizing the surrogate model gg. This optimization method was first proposed by (Charnes et al. 1988) and later applied in (Lundberg) 2017: Jethani al. 2021; Covert al. 2022). Let p(x) and p(æ, y) denote the distributions of input and input-label pairs, respectively. Specifically, the loss for training the explainer is:
E Lesp(0) (2. ((.)))=(2.)))00q0(()
(gn(<x ga(zolv) - (T)((1,)))
(3)
s.t. 1 1 ch 9) ( (zolv))) v(r
(Efficiency)
where the constraint in the loss function is enforced to satisfy the efficiency axiom of the Shapley value, and gss) is defined with the Shapley kernel CCharnes et al. 1988) as follows:
d-1
((s) d (d) 1
Vs:0< 1s:0<1ss<>
(Shapley Kernel)
In addition, the VIT explainer de has the same number of multi-head self-attention (MSA) layers as the feature backbone, and includes three additional MSA layers and a fully-connected (FC) layer as the explanation head. By learning the explainer the to estimate Shapley values, the computational cost is reduced to constant complexity of ((1).
4 METHOD
4.1 EFFICIENTLY TRAINING THE SHAPLEY VALUE EXPLAINER FOR BLACK-BOX MODELS
As discussed in Section 3.2 existing methods for approximating Shapley values require two stages. First, the black-box model f is fully fine-tuned to obtain surrogate model g with the same trainable parameters and memory cost as f. Then, an explainer ò is fully trained using g. These stages at least double the training, memory, and inference costs compared to using the black-box model f alone, making these methods impractical for large models.
To improve training, memory, and inference efficiency, propose side-tuning pipeline called AutoGnothi, as shown in Figure 3(b). Building on ideas from PETL, we adapt Ladder Side- Tuning (LST) (Sung et al. 2022) by incorporating additive side network. This side network
5

---

Published as a conference paper at ICLR 2025
Comparison training and memory efficiency across different models. We evaluated memory consumption and trainable parameters for previous methods (Covert et al. 2022) and Au- toGnothi across a range of models and tasks. For surrogate models, we measured classification accuracy, while for explainers, we assessed explanation quality using insertion and deletion met- rics. It is important to note that prediction accuracy for the classifier is evaluated on normal inputs, whereas for the surrogate model, accuracy is measured masked inputs.
Table 1:
Datavet
ImageNotte
MURA
Yelp
Model
VII-T
VII-5
VITB
VIT-L
VIT-B
VI-B
BERT-B
Classilier explained
Memory (MB)
AParams (M)
$4.90
5.53
331.80
21.67
1311.11
85.81
4631.25
303.31
1311,50
85.80
1311.93
85.83
1676.59
109.48
Accuracy (1)
0.9791
0.944
0.964
0.8186
0.9469
09010
(Covert et
Samngote
Memory (MB)
#Params (M)
Accuracy (1)
0.9822
84,90
0.9931
331,80
21.67
131161
0,9939
4631,25
0.9975
1311,50
0.8233
1311,93
0,9169
1676.59
0,9490
109.45
Surmugate (AumGGonld)
Memory (MB)
AParams (M)
23.83 (-72%)
0.34 (.97%)
92.38 (-72%)
(:97%)
363.64 (-72%)
2.23
(-978)
(-729))
7.91 (-97%)
438. (-67%)
7.11 (-92%)
363.76 (-72%)
2.23 (-4778)
$32.71 (-68%)
7.15 (-939)
Accuracy (1)
0.9791
019939
0,9959
0.8139
0,9422
0.9260
Memory (MB)
103.05
404.10
160012
$144,05
1599.79
1601.47
1955 87
Explainer
AParamu
6.72
26.41
104.72
336.92
127.79
(Covert et al))
Insertion (1)
0.9344
0.9888
0,9899
0.9843
0.9319
0,9422
0,9600
Deletion (L)
0.5243
0,6865
0.8121
0,7646
0,4199
0.4958
0L1725
Memory (MB)
24.02 (-7795)
(-775)
366.51
(-7798)
1285.87 (-75%6)
449 38 (-72%%)
366 (:77%)
32 (-655)
Explainer (AuoGnook))
*Params (M)
0.15 (-98%)
0.61 (-98%)
2.42 (-98%)
8.24 (-98%)
7.85 (.03%)
2.43 (L988)
17.15 (.87%)
Insertion (1)
0.9802
0.9791
0.9874
0.9588
Deletion (1)
0.5097
0.5667
0.7954
0.6570
0:4116
0.4888
0.1001
separates the trainable parameters from the backbone model f and adapts the model to a different task. It is lightweight version of with weights and hidden state dimensions scaled by factor of 1/r. where r is reduction factor (e.g., r = 4 or 8). For instance, if the backbone / has a 768- dimensional hidden state, then with r 8, the side network has hidden state dimension of 96. By computing gradients solely for the side network, this design avoids backward pass through the main backbone, improving training and memory efficiency. The formulation combines the frozen pre-trained backbone and the side-tuner with learnable parameters |3 as:
main
ga
(Ia),
exp
6
(x),
frozen
trainable
trainable
where 9s is trained by minimizing the loss in Eq.22. and do is trained by minimizing the loss in 33. respectively.
4.1.1 OBTAINING THE SURROGATE
To obtain the surrogate model, AutoGnothi applies LST directly to the black-box model utilizing the additive side branch g with parameters 3 to predict the masked inputs x, of sample Let f() and ( denote the i-th MSA block of the main model f and the surrogate branch g, respectively. Assume there are N MSA blocks in total. Let 2T denote the output for masked input x, of the first MSA layer (2) of f, i.e., 2 (xr)) The forward process of the frozen main model f
with the side-tuning branch y is:
.main
(+) main
surT = (0 (FC )))
(5)
After N MSA blocks, an FC head is applied to generate the prediction for the partial information, ie., yuu FCheaa( surr)) The convergence of the surrogate model is analyzed as follows:
Theorem (Proof in Appendix B). Let the surrogate model be trained using gradiens descent with step size for t iterations. The expected KL. divergence between the original model's predictions f() and the surrogate model's predictions go(x.) ) is upper-bounded by:
1 _ ua:) (Eunr Cauur)
Epwpla) s) [DeL (f(r))l9(x.)) S
(6)
24.
where Bo is the initial parameter value, Dam is the optimal value during optimization, and is the minimal eigenvalue of the Hessian of Luir.
Theorem I establishes a theoretical guarantee that a side-tuned surrogate can achieve performance comparable that of fully trained surrogate. The detailed proof is provided in Appendix B
Figure 3 b) shows the pipeline of obtaining the surrogate in stage An intuitive performance comparison of prediction models is presented in Figure a)). AutoGmothi's surrogate surpasses the

---

Published as a conference paper at ICLR 2025
Table 2:
Inference efficiency comparison various models. We assessed the computational cost (FLOPs), total parameters, and inference time for different methods. For the baseline method (Covert et al. 2022), we calculated these values separately for the classifier and explainer, and then combined them. In contrast, for AutoGnothi, computed these values directly from our -nterretereeee models, who can generate both predictions and explanations simultaneously.
Dataset
ImageNette
MURA
Pet
Yelp
VIT-T
VIT-B
Classifter + Explainer
HLOPS (Ci)
4.78
18.86
74,90
251 96
74.88
74,93
213.51
Time (ms)
19.7
39.0
94,9
310,3
100.3
100.2
166.90
(Covert al.)
(M)
190.53
640.23
self-Interpretable
FLOPs (G)
2.22 (-54%)
8.73 (-54%)
8.73
34,67 (-54%)
122.60 (-51%)
36.81 (:51%)
34.67 (-54%)
116.56 (-55%)
Model
Time (ms)
15.4 (-225))
23.0 (-4155)
52,9
(-445))
179.1 (-4256)
56.7 (-435)
52.0 (-435)
118.55 (-29%)
(AstuGnoth))
(M)
5.68 (-549))
22.28 (-54%)
88.22 (-548)
311.55 (:51%)
(-51%)
88 (-54%)
(-47%)
original classifier in terms of prediction accuracy and matches the performance of (Covert et al., 2022) when handling partial information, but with only 3% trainable parameters. Additionally, AutoGnothi's surrogate exhibits more robust predictions as the number of masked inputs increases. A detailed comparison of the training and memory costs on different models is provided in Table
4.1.2 OBTAINING THE EXPLAINER
For the explainer model, AutoGnothi uses similar LST feature backbone as in the surrogate model g. consisting of N MSA blocks for feature extraction. Let git) represent the i-th MSA block of the explainer branch 6. In addition to the lightweight backbone blocks in the side branch, add M extra FC layers as the explanation head. Together, the side network ş generates explanations based on the backbone features from the main branch. Let zmain denote the output for input æ from the first (z). The forward process of the explainer is: MSA layer f(0) of the main branch f. i.e.e, 21 .mainn
main
p(0 ( mamm)
exp
2600 (FC (=mmain) Vi (1....,,N) N), (... (FC
24
(x)
head
head
where the main branch f remains uncontaminated. We provide theoretical guarantee for the vergence of the trained side branch & as follows:
Theorem (Proof in Appendix Bj. Let (z(x)) denote the exact Shapley value for input-output pair
(r, y) in game v. The expected regression loss (0) upper bounds the Shapley value estimation
error as follows,
bbbl S / (Cop(0) Liese)
E Ep(e [1 (=
(8)
where Ceep represents the optimal loss achieved by the exact Shapley values, and Ha-1 is the (d-1)-
th harmonic number.
Theorem 2 provides a theoretical guarantee that side-tuned explainer can achieve performance on par with fully trained explainer. The complete proof is presented in Appendix
Figure 3b) shows the pipeline of obtaining the explainer in stage 2. We evaluated the explanation quality of AutoGnothi against various baselines, as shown in Figure 4b). AutoGnothi achieved the highest insertion and lowest deletion scores among 12 explanation methods, demonstrating superior explanation quality. Compared to (Covert et al.) 2022). we reduced the trainable parameters for explainers by 98% while maintaining comparable even superior interpretability. Table provides detailed comparisons of training and memory efficiency for different models.
4.1.3 GENERATING EXPLANATION FOR BLACK-BAG -BO MODELS
After obtaining the explainer, we now detail the explanation procedure. In most post-hoc explana- tion methods (Selvaraju et al., 2020 Chattopadhay et al. 2018 Binder et al.) 2016) Covert et al.) 2022), predictions and explanations must be computed separately for a single input. For instance, as illustrated in Figure a). two separate inferences are required to explain a single prediction. In contrast, as shown in Figure B).. AutoGnothi generates both predictions and explanations simul- taneously, needing only inference. To evaluate this efficiency, we measured inference time, computational cost (FLOPs), and total parameters required to generate predictions and explanations for different methods. The comparison of inference efficiency is highlighted in Table 2
7

---

Published as a conference paper at ICLR 2025
4.2 DIFFERENCE BETWEEN AutoGnothi AND PREVIOUS PETL METHODS
In this section, we provide an empirical analysis of why AutoGnothi outperforms previous PETL approaches. We highlight that AutoGnothi is not simple extension of classical PETL, where full fine-tuning is typically the optimal baseline. Additionally, AutoGnothi exploits the intrinsic corre- lation between the prediction task and its explanation, enabling black-box models become self- interpretable without sacrificing prediction accuracy. We elaborate on these two points below.
Full fine-tuning poses challenges for achieving cell-interpretability and is not the optimal base- line for AutoGnothi. For classical PETL. the goal is often to match the performance of fully fine- tuned models on standard tasks (Houlsby et al. 2019)] Chen al.) 2023b; (Hu et al.] 202T; |Mereaa et al., 2024). However, in XAI scenarios, the challenge shifts: it becomes difficult, if not impossible, to train a model that balances both prediction accuracy and explanation quality (Arrieta et al. 22000] Gunning et al., 2019 Dosilović et al. 2018). In fact, fine-tuning models to adapt interpretability without forgetting pre-trained knowledge can be difficult (Li & Hoiem| 2017). Additionally, full fine-tuning the original model also puts us in the Theseus's 's Paradox: won't I be sure if we ex- plaining the very same model anymore. Even if full fine-tuning were practical, it would contradict the goal of interpreting the pre- trained model. In contrast, AutoGnothi pipeline addresses this issue by freezing the primary model and training only a side network to generate explanations, offering an efficient solution that enables self-interpretabiiiit without degrading prediction performance.
(x) Pruyo
Task Similarity
ViT-hase
BERT-base
Prodicion
1:00
10-
Autodratii (ocn
Excoda
Explestion
0.90
Orhoggcc
Dan
(b) Due
0.25
Trenlense
Deplazation
Methol
Ancerary
Inverten
Deleiken
62-
75.75
stol
50.34
USEEI (:3300)
111221 (480L7)
-1.00
Froee
09.44
03800 (33001
0.3383 (44.44)
25
130
Epoch
Esoch
Epoch
00
Figure 5: Other pipelines to
Figure 6: Explanation of why the Duo pipeline underperforms compared to AutoGnothi. (a) The full fine-tuning strategy em- ployed by Duo is not the optimal baseline for AutoGnothi due to significant gradient confficts between the prediction and ex- planation tasks. This conffict results in degraded performance, raising concerns about whether the explanation truly pertains to the same model. Note that gradient similarity can only be mea- sured for the Duo pipeline, as other methods freeze the predic- tion backbones. (b) In contrast, AutoGnothi exhibits stronger correlation between features of the prediction and explanation tasks. We measured the feature similarity with CKA.
achieve the self-interpretability through the full fine-tuning. (a) Freeze the transformer encoder and prediction head, learning only the explanation head. (b) Simulta- neously learn the transformer en- coder, and both task heads. (c) Comparison of classification and explanation performance between different pipelines for ViT-base.
Building on prior work that highlights the challenges of achieving self-interpretabiility through full fine-tuning, we conducted experiments to further explore this issue. As depicted in Figure 5 (b), we introduce two additional pipelines, Froyo and Duo. Froyo adds an explanation head while keep- ing the transformer encoder and prediction head frozen to preserve prediction accuracy. In contrast, Duo jointly learns both the prediction task and its explanation. Both pipelines use the same encoder, prediction head, and explanation head architectures as described in (Covert et al., 2022).
We performed experiments using ViT-base model trained on the ImageNette dataset. Our findings show that the Froyo pipeline underperforms due to the limited trainable parameters in the explanation head, resulting degraded explanation quality. As illustrated in Figure 5)c), c), this led to a reduction in insertion by 0.038 and an increase in deletion by 0.141, despite no impact on prediction accuracy.
For the Duo pipeline, we observed a 4.0% decline in prediction accuracy on the ViT-base model, accompanied by reduction in insertion by 0.041 and an increase in deletion by 0.142. Further empirical evidence, as depicted in Figure G(a), highlights conflicting gradients between the pre- diction and explanation tasks during training. Additionally, as previously discussed, the Theseus's Paradox arises when changes in predictions result in evolving explanations, thereby challenging the consistency and identity of the original model.
8

---

Published as a conference paper at ICLR 2025
AutoGnothi uncovers the intrinsic correlation between predictions and explanations. While the AutoGnothi pipeline enables self-imterpretation with superior efficiency, the underlying mechanisms connecting prediction and explanation remain underexplored. We propose that AutoGnothi lever- ages the intrinsic relationship between the backhone features used for both tasks. This correlation is illustrated in Figure (b). where we evaluated the Central Kernel Alignment (CKA) (Kornblith iet al., 2019) between the backbone features of the original pre-trained model and those of various explainers. Our results show that AutoGnothi exhibits higher feature similarity between prediction and explanation tasks on ViT- and BERT-bass models, supporting our hypothesis.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets and Black-box Models. For image classification, we used the ImageNette (Howard & Gugger 2020), Oxford IIT--eets (Parkhi al. 2012), and MURA (Rajpurkar et al. (017) datasets, following (Covert al.] (022). For sentiment analysis, we utilized the Yelp Review Polarity dataset (Zhang et al. 2015). In terms of black-box models, we employed the widely used ViT models (Dosovitskiy 2020) for vision tasks, including ViT-tiny, ViT-small, ViT-base, and ViT-large. For language tasks, we used the BERT-base model (Devlin et al. 2019).
Implementation Details. For surrogates and ex- plainers, AutoGnothi incorporates the same num- ber of MSA blocks as the black-box model being explained in the side network and utilizes a re- duction factor of branch on both the ImageNette and Oxford IIIT- Pets datasets, and r = 4 for MURA and Yelp Review Polarity. Surrogates use the same task head as the black-box classifiers with one addi- tional FC layer as classification head for handling partial information. Explainers utilize three addi- tional FC layers as the explanation head after the side network backbone. For attention masking, employed causal attention masking strategy, setting attention values to a large negative number before applying the softmax operation (Brown) 2020). More detailed training settings are pro- vided in Appendix [A]
Table 3: Quality metrics (insertion and deletion) for target class explanations of ViT-base across baseline methods and AutoGnothi on the Ima- geNette dataset. More results for other datasets and models are provided in Appendix C
8 for the lightweight side
Method
Insertion (1)
Deletion (L)
Random
0.9578
0.9584
Attention last
0.9633=20.659
0.8524 1748
Attention rollout
0.9408
0.9168-uu:277
GradCAM (Att))
0,9447 #000036
0.9562
GradCAM (LN)
0,9343
0.942610.330
Vanilla (Pixel)
0.9487=000s8
0.8945
Vanilla (Embed)
0.9563.20.043
0.8618
IntGrad (Pixcl)
0.9408 0.9670+00057
IntGrad (Embed)
0.9670
0.9408
SmoothGrad (Pixel)
0,9591 #0,0760
0. =4.59
SmoothGrad (Embed)
0.952910.000
0, 0,9561 10006
VarGrad (Pixel)
0.8600 0.9616200002
VarGrad (Embed)
0.0.52:00000
0.9568=20796
LRP
0.9677
a0.166
Evaluation Metries for Explanations. We used the widely adopted insertion and deletion met- rics [Petsiuk] 2018) to evaluate explanation qual- ity. These metrics are computed by progressively inserting or deleting features based on their importance and observing the impact on the model's predictions. The corresponding surrogate model trained to handle partial inputs used for this pro- cess to generate prediction for masked inputs. We calculated the area under the curve (AUC) for the predictions and average results for randomly selected 1, 000 samples on all datasets.
Leave-one-out RISE Covert et al.
0.9696.10.08
0.933410.169
0.9772.0:025
0.89591..100
0.9839
0.8121 1:: 1:0
AutoGnothi (Ours)
0.9874
0.7954=0.224
Baseline Methods. We considered 12 representative explanation methods for comparison. For attention-based methods, we utilized attention rollout and attention last (Abnar & Zuidemal 2020). For gradient-basee methods, used Vanilla Gradients (Simonyan] 2013). IntGrad (Sundararajan et al. 2017). SmoothGrad (Smilkov et al. 2017). VarGrad (Hooker et al. 2019). LRP (Binder et a 2016), and GradCAM (Selvaraju et al. 2020). For removal-based methods, we employed leave- one-out (Zeiler & Fergus| 2014) and RISE (Petsiuk 2018). For Shapley value-based methods, we utilized KemnelSHAP (Lundberg 2017) and VIT Shapley (Covert et al. 2022) baselines.
5.2 EVALUATING TRAINING, MEMORY AND INFERENCE EFFICIENCY
To evaluate the training and memory efficiency of AutoGnothi, we compared it with various base- lines in terms of trainable parameters and memory usage. I summary of the memory

---

Published as a conference paper at ICLR 2025
AuteGnotii
Ventime
RISE
Grak
Attention milovr
Inford
Snotthirdd
Lewwe-ree-stt
MURIA
Figure 7: Visualization of ViT explanations the ImageNette, Oxford-IIIT Pets, and MURA datasets, AutoGnothi qualitatively outperforms other baseline approaches.
costs and trainable parameters for both surrogate and explainer models across different methods. During training. AutoGnothi achieves significant reduction of over 87% in trainable parameters and more than 65% in GPU memory usage for both surrogates and explainers on both vision and language datasets, while maintaining competitive performance in both prediction accuracy and ex- planation quality. The memory cost is evaluated with training batch size 1.
Next, evaluate the inference efficiency of our self- infeintterreee models by comparing the com- putational cost (FLOPs), inference time, and the total number of parameters required for generating both predictions and explanations. As shown in Table 2 AutoGmothi significantly reduces inference time and FLOPs compared to baseline models that require separate inferences for predictions and explanations. Specifically. when both the predictions and the explanations are required, Auto is capable of reducing at least 45% in FLOPs, 22% in inference time (up to 44%), and 47% in total parameters end-to-end across all datasets and tasks.
5.3 EVALUATING EXPLANATION QUALITY
Quantitative Results. To evaluate the quality of explanations generated by AutoGnothi, we com- pared it against 12 state-of-thh-aa baseline methods using the insertion and deletion metrics. Table presents results from ViT-base model trained the ImageNette dataset, where AutoGnothi sistently achieves the best insertion and deletion scores for target class explanations across various datasets. Further detailed results for other models and tasks are provided in Appendix C For vision task, please refer to Tables 4 5 and 6 for the results of ViT-tiny, ViT-small, and ViT-large on Ima- geNette, respectively. Results for ViT-base on Oxford-IIIT Pets are provided in Table and results for ViT-base on Mura are shown in Table 8 For language task, we also provide results of BERT-base on Yelp Reivew Polarity in Table
Qualitative Results. We also provide visualization results for ViTs on different datasets, as shown in Figure 7 It may be observed that RISE happened fail to provide intrrprattreeeeableeee or intuitive results, which amongst all others AutoGnothi offers more accurate explanations with clearer focus on the subject and diluted colours for irrelevant classes. Additional visualization results for ViT- on more datasets are provided in Appendix E. shown in Figures X 9
and 13
CONCLUSION
This paper introduces AutoGnothi to bridge the gap between self- -effiinterpretable models and post-hoc explanation methods in Explainable AI. Inspired by parameter-efffiient transfer-learnngg, AutoG- nothi incorporates lightweight side network that allows black-box models generate faithful Shapley value explanations without affecting the original predictions. Notably, AutoGnothi outper- forms directly fine-tuning the all the parameters in the model for explanation by clear margin. This approach empowers black-box models with self-interpretability. which is superior to standard post-hoc explanations that require generating predictions and explanations in two separate, heavy ferences. Experiments on ViT and BERT demonstrate that AutoGnothi achieves superior efficiency on computation, storage and memory in both training and inference periods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 41900419999 2020.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXii:2303.0877 2023.
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks, 2018
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges to- ward responsible ai. Information fusion, 58:82-114 1152-115, 2020.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robed Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local remormalization layers. In Artificial Neural Networks and Machine Learning-ICANN 2016: 25th International Confer- ence on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part 25, pp. 63-71. Springer, 2016.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXi::2005.1416. 2020.
A. Charnes, B. Golany, M. Keane, and J. Rousseau. Extremal Principle Solutions of Games in Characteristic Function Form: Core, Chebychev and Shapley Value Generalizations, pp. 123- 133. Springer Netherlands, Dordrecht, 1988. ISBN 978 978-975-00
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 839-847. IEEE, 2018.
Hugh Chen, Ian C Covert, Scott M Lundberg, and Su-In Lee. Algorithms to estimate shapley value feature attributions. Nature Machine Intelligence, 5(6):590-60) 2023a.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision Transformer Adapter for Dense Predictions (ViT-Adapter), February 2023b. arXiv:2205.08534 [cs] Status: In Progress Sead__Statuu_Dae Date: 2024-07-11708:2::222
Ian Covert and Su-In Lee. Improving kemelshaa: Practical shapley value estimation via linear regression. arXiv preprint arXiv:2012.01536. 2020.
Ian Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate shapley values with vision trans- formers. arXiv preprint arXiv:2206.052880 2022.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics, 2019.
Filip Karlo Došilović, Mario Brčić, and Nikica Hlupić. Explainable artificial intelligence: A survey. In 2018 41st International convention on information and communication technology, electronics and microelectronies (MIPRO), pp. 0210-0215. IEEE, 2018.
Alexey Dosovitskiy. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
David Gunning, Mark Stefik, Jacsik Choi, Timothy Miller, Simone Stumpf, and Llang-ZZhon Yang. Xai explanaiin artificial intelligence. Science robotics, 4(37)) 120, 2019.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units, 2017.
11

---

Published as a conference paper at ICLR 2025
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. Advances neural information processing systems, 32, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference machine learning, pp.
PMLR, 2019.
Jeremy Howard and Sylvain Gugger. Fastai: 108, 2020.
layered api for deep learning, Information, 11(2):
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. arXiv:2106.090 0666066606561.0
[es] Rtad_Status: Read Sead_Stttus_DDae Date: 0024-07-11707:26.22
Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. In International conference on learning representations, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-266622222 2677. PMLR, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 4026, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp- 5338-5348. PMLR, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-290) -2999955-29772--- 2017.
Scott Lundberg. A unified approach to interpreting model predictions.
arXiv preprint
arXiv:1005.07874. 2017.
Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab. Time-, Memory- and Parfceter-Efficientt Visual Adaptation (LoSA), February 2024. arXiv:2402.02887 [cs].
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and V. Jawahar, Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
V Petsiuk. Rise: Randomized imput sampling for explanation of black-box models. arXiv preprint arXiv:1806.0744 07421, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp- PMLR, 2021.
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang. Kaylie Zhu, Dillon Laird, Robyn L Ball, et al. Mura: Large dataset for abnormality detec- tion in musculoskeletal radiographs. arXiv preprint
2017.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-25) 2-206-215,20 2019.
Luis M. Ruiz, Federico Valenciano, and Jose M. Zarzuelo. The family of least square values for transferable utility games. Games and Economic Behavior, 24(1): 1998. ISSN 0899- 8256. doi: https://do..or//100 wttp./o//////.//./aomoommm 1006/game..e.e 1997. 00622.
12

---

Published as a conference paper at ICLR 2025
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient- based local- ization. International journal of computer vision, 128:336-359, 2020.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2, 1953.
Grah Simon and Thouvenot Vincent. projected stochastic gradient algorithm for estimating Shap- ley value applied in attribute importance. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pp. 97-115. Springer, 2020.
Karen Simonyan. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: arXiv: 1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:/706.03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: Ladder Side-Tuning for Parameter and Mem-
ory Efficient Transfer Learning, October 2022. LxXiv:2206.06522 [cs] Read_Status. Read
Read_Status Date: 2024-07-11708:24:04222
H Peyton Young. Monotonic solutions of cooperative games. International Journal of Game Theory, 14(2):65-72, 1985.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 313-833. Springer, 2014.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks, July 2020. 11Xiv:1912.13503
[cs] Read Status: Read Read Status Date: 2024-07-11707::::11..000
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification, M.Xiv:1509.0122 [cs], September 2015.
13

---

Published as a conference paper at ICLR 2025
A FURTHER EXPERIMENTAL DETAILS
A.1 ENVIRONMENT
Our experiments were conducted on one 128-core AMD EPYC 9754 CPU with one NVIDIA GeForce RTX 4090 GPU with 24 GB VRAM. No multi-card training or inference was involved. We implemented the training and inference pipelines for image classification tasks under the Py- Torch Lightning framework, and for the sentiment analysis task with just Py PyTorch. For evaluations on baseline methods we leverage the SHAP library (Lundberg 2017), with minor modifications applied to bridge data format differences between Numpy and PyTorch.
A.2 FLOPS AND MEMORY
We used the profile lile function from the thop library to evaluate the FLOPs for each model during the inference stage. Memory consumption was manually calculated during the training stage based on the model parameters, activations, and intermediate results. Our memory estimations were made under the assumption that the memory was evaluated under a batch size of and uses 32-bit floating point precision floch.float32).
A.3 CLASSIFIER
The training of all tasks is split into 3 stages. In the first stage parameters of the classifier inherited from the original base model verbatim, with the exception AutoGnothi adding additional parameters for the new side branch, initialized with Kaiming initialization.
We fine-tune this classifier model on the exact same dataset with the Adam W optimizer, using a learning rate of 10 for 25 epochs, and retain the best checkpoint rated by minimal validation loss. Classification loss is minimum square error with respective to the predicted classes and the ground- truth labels. For AuloGmothi the classes come from the side branch, while all remaining models the classes from the main branch. We train and evaluate these methods on Nvidia RTX 4090, and use batch size of 32 samples provided that fits inside the available GPU memory.
We freeze parameters in all stages likewise. As is described in 4.2 AutoGnothi only trains the side branch and all parameters from the original model are frozen. In ViT-shapley (Covert al.) and Duo pipelines, all parameters are trained, whilst the Froyo pipeline only trains the classification head.
A.4 SURROGATE
Surrogate models have the same model architecture as the classifiers, with a different recipe. We load all parameters from the classifier without any changes additions, further fine-tune the model for 50 epochs, and retain the best checkpoint with minimal validation loss.
Unlike the classifier, the surrogate model focuses on mimicking the classifier model's behavior under a masked context. Consider the logits p(y|#)) from the classifier model, where x is the input and is the corresponding class label. The surrogate model aims at closing in its masked logits distribution, p(y(.,) x.), with the original distribution:
Lsur (B)
E (00(
DEL (f(#))l9(zz)))
= The mask is selected on an equi-categorical basis. We first pick an integer s 1 at uniform distribution, denoting the number of tokens that shall be masked. nt mutually exclusive indices are then randomly chosen from the input uniform distribution. To avoid inhibiting model capabilities, special tokens like the implicit class token VIT or the [CLS] token applied by the BERT tokenizer never masked.
In order to selectively hide or mask inputs from the model, we apply causal attention masks for both the image models and text models in experiments. However, it's worth noting that while they may confuse the transformer's attention mechanisms, certain other methods are also capable of concealing these input tokens, primarily zeroing assigning random values to the said pixels
14

---

Published as a conference paper at ICLR 2025
in image models, assigning [PAD] and [MASK] to the selected tokens. We follow prior work (Covrrt et al. 2022) and adhere to causal attention masks.
Recall that the transformer self-attention mechanism used in ViT (Dosovitskiy 2020 BERT (Devlin al. 2019) Given attention input & R and self-attention parameters Wak Rhx3h whereas d is the number of tokens and is the attention's 's hidden size, and h be the size of each attention head, the output of the self-attention SA(I) for single head is computed follows:
[Q, K, V = u Wakv
(2)
Q KT VI
A softmax
(3)
SA(t) A
(4)
Transformers in practice use a multiple attention heads, holding that k h = h. An attention projection matrix Pasaa R i-th self-attention head's 's output as (t), the final output of the attention layer MSA(t) is thus
h is used to combine the outputs of all attention heads. Denoting the
computed:
MSA(() (A) SA.(())) PasSa
We notice that the attention mechanism is entirely unrelated the number of tokens, and can operate in the absence of certain input tokens. Let an indicator vector E 1) correspond to subset of the input tokens (applying equally to images and text tokens), we calculate the masked self-attention over and as follows:
QQ, K, VJ u. WQ.K.V
(6)
Q-KT (1 5) 000
= softmax
(7)
Vh
SA(t, s) = A.V
(8)
We apply masking to the multi-head attention layers likewise, such that:
MSA(t, s) [ s),(t s). (t, 8)] Passa
This mechanism is widely implemented for attention models in commonplace libraries such as Hug- gingFace's Transformers, named by the argument attent conttiinakkkk in the input tensor.
A.5 EXPLAINER
We load explainer model parameters from surrogate model checkpoints, such that all are copied from the surrogate model to the explainer model, except for the last classification head, which is replaced with an explainer head. The explainer head contains 3 MLP layers and a final linear layer, with GeLU (Hendrycks & Gimpel 2017) activations from in between. For AutoGnothi only the classification head on the side branch is replaced. We train the explainer model for 100 epochs with Adam optimizer, using a learning rate of 10 10-3, and keep the best checkpoint.
In our implementation, we took 2 input images in each mini-batch and generated 16 random masks for each image, resulting parallelism of instances per batch. slight change applied to the masking algorithm in the explainer model from the surrogate model, in order to reduce variance during gradient descent. Specifically, in addition to generating masks uniform, we follow prior work (overt al. 2022) and use the paired sampling trick (Covert & Lee 2020), pairing each subset s. This algorithm equally applies to both image and text classification
s with its complement models.
The explainer model is trained to approximate the Shapley value Φ0(x, y). Let gx(x,ly) be the surrogate values respective to the input and the class y, masked by the indicator vector s, and
15

---

Published as a conference paper at ICLR 2025
(a(ooly) be the surrogate values without masking. Following Covert al. 2022 we minimize the following loss function:
Leap(0) =
E
(00(2s)))) ( (Tosz,
(10)
s.t. 1 10 g3(21 (y) ga(zole) V(a, y).
(Efficiency)
Notice that the explainer model de (z, y) is being trained under a mean squared error loss, and that 16 random masks are generated for each image in the maini The explainer is hence optimized against a distribution of the Shapley values, so an accurate calculation of ground-truth Shapley values for each training sample is not remotely necessary.
Also, the aforementioned efficiency constraint is necessary for the explainer model to output faithful and exact Shapley values. We leverage additive efficient normalization from (Ruiz al.) 1998)
and use the same approach as prior work (Jethani et al. 2021 Covert et al. (022) to enforce this constraint. The model is trained make unconstrained predictions which we then modify using the following transformation to have it constrained:
described in equation TO
Φ0(2, v) do(x.v) + ga(ralv) (o(oll)) - 1 00(x, v)
(11)
d
After training the explainer model, we merge the original classifier model and all relevant interme- diate stages' models into one single, independent model contain both the classification task and the explanation task to have them run concurrently. For the baseline method, ViT-shapley and its modified counterpart for NLP, no parameters overlap between the two tasks, thus inference must be done on both tasks resulting in huge implied performance overhead. AutoGmothi, however, only requires validation pass to ensure that the original classifier head preserved verbatim in the final model. This is done by comparing the output of the original classifier and the final model on any arbitrary input.
A.6 DATASETS
In this section we explain in more detail which datasets are selected and how they are processed for our experiments. Three datasets are used for the image classification task. The ImageNette dataset includes 9, 469 training samples and 3, 925 validation samples for 10 classes. MURA (mus- culoskeletal radiographs) has 36, 808 training samples and 3, 197 validation samples for 2 classes. The Oxford-IIIT Pets dataset contains 5, 879 training samples, 735 validation samples and 735 test samples in 37 classes. For the text classification (sequence classification) task, we use the Yelp Polarity dataset, which originally contains 560, 000 training samples and 38, 000 test samples.
For each epoch, image classifiers (ViT) iterate through all available images in either of the train or test dataset. Specifically to during training, images are normalized by the mean value and standard deviation of each corresponding training dataset, before being down-sampled to 224 x 224 pixels. For text classifiers, each of the epoch trained on exactly 2048 training samples randomly chosen from the dataset, and validated on 256 equally random test samples. This is primarily done to serve our needs frequent checkpoints for more thorough data analysis such as on CKA or parameter gradients.
Due to the sheer cost from some metrics on certain tasks, we reduced the size of our test set during evaluation. We selected 1000 test samples for datasets ImageNette, MURA and Yelp Review Polar- ity, and randomly selected 300 samples for the Oxford-IIIT Pets dataset. For each dataset this subset stays the same between different explanation methods and different model sizes. We emphasize that these samples are deliberately independent from the training set to avoid potential bias from the results.
16

---

Published as a conference paper at ICLR 2025
B PROOFS OF THEOREMS
Here we provide detailed proof for Theorem and Theorem which provide theoretical guarantee for the performance surrogates and explainers. Our proof follows (Simon & Vincent 2020 and (Covert et al. 2022), which exhibit similar results for single data point.
Lemma 1, For a single input x, the expected loss under Eq. 22 is p-strongly convex, where p is the minimal eigenvalue of the Hessian of Luir
Proof. The expected loss for single input under the new objective function is given by:
sur [3) Es-1((0.)) [DKL (f()(ll)))
(12)
This loss function is convex in 3 because the KL divergence DRL (f (fx))lls(xx) is convex in (z,), and (x.) is smooth function of B. The Hessian of (3) with respect to 3 is:
Ca (3) (x-2((0.4)
(13)
The convexity of Caur((e (3) is determined by the smallest eigenvalue of this Hessian, μ.. Since the KL divergence is strictly convex, the minimum eigenvalue is positive, which implies μ-strong convexity.
Li
Let the surrogale model be trained using gradient descent with step size for I itera- tions. The expected KL divergence between the original model's predictions fx) and the surrogate model's predictions 9x(xx) is upper-bounded by:
Theorem 1.
s [DxL (f(=)llg(xx)) S (1 μai' (Bu)
Liurr)
(14)
2p
where Bo is the initial parameter value, and C is the optimal value during optimization.
Proof. Let the surrogate model be trained using gradient descent with step size for t iterations. The optimization process for minimizing the expected KI. divergence between f(xx) and g3(x)) can be written as:
B++1 3 a 2 (B,).
(15)
Because (B) is j--strngly convex, we can apply the standard result for gradient descent conver- gence on strongly convex functions, which gives the following bound:
Saar (Be) L (1 pax)* (L (Bo) S
(16)
where Lin is the optimal value, and Bg is the initial parameter value.
Since the KL divergence is bounded by the expected loss, we have:
Resppa))
S Lun (Bt)-
(17)
Substituting the bound on ( (B:), we obtain:
Ex-p(e), (faa(((())l 24
(1 pax)² (Luure (B0) Laan)
(18)
Lemma 2. For a single input-output pair (x, y). the expected loss under Eq. for the prediction
(0 y) is u-strongly convex with μ = = where Ha-1 is the - 11) harmonic number.
Proof. For an input-output pair (a, y). the expected loss for the prediction = Φe y) is defined as
Lo(a,y) 9) = E)pp(s) (go(x)ll)) - go(2olv) - o)
Eampts) iss T0 2E 2E.P)()) (((0)(x)))) (x) - gs (
(19)
+ Eenp((s (ga(Ecll) g2(2ol1) }
17

---

Published as a conference paper at ICLR 2025
This is quadratic function of @ with its Hessian given by
V Lo(,.y) 2 E BB-p(()
(20)
The eigenvalues of the Hessian determine the convexity of [x, y), and the entries of the Hessian can be derived from the subset distribution p(s). The distribution assigns equal probability to sub- sets with the same cardinality, thus we define the shorthand = p(s) for such that 1 s = k. Specifically, we have:
d
d d-1
Pk Q
and
Q
Σ
(21)
(i) ki) - k)
k(d - k))
We can then write A = Bsmp(s) [ss " and derive its entries as follows:
1
An = Pr(s;= = 1)= Σ
pk
k=1
(22)
= QQ1 1 d-1 d-1 1
Σ dd
d(d - k)
Σ k(d
k=
d
dd2 2
8j = 1)= Ag = Pr(si 8j
Σ
k -
k=
(23)
d-1 k 1
Σk= 2(d
=Q Σ
d(d - k)
d-1
k(d-k)
Based on this, we observe that A has the structure A = (b- c) I4 cll where b = A. Ai and
c= Aij- Following (Simon & Vincent 2020 Covert et al. 2022), the minimum eigenvalue given
by Amin(A) (A) = b A more detailed derivation reveals that it depends on the (d - 1)-th harmonic number, Ha-1:
(A) = b-c Aii Agj
1k=1 k(d-k)
d-1
d-1 d(d-k)
d-1
d-1 k= k(d-)
k= k(-k)
2k= d)
d-1 d-k
+ d-2
(24)
d-1 Σ k(d-k)
d-1 Σ k= k(d-k)
d-1
1
E
d-1 k(d-k
d-1 2
2
>
2H
The minimum eigenvalue is therefore strictly positive, implying that Lo (x. y) is p-strongly convex, where p is given by
== 2 Amin (A) = -
(25)
Note that the strong convexity constant is independent of (æ, y) and is determined solely by the number of input variables d.
Theorem 2.
Let (x(xx) denote the exact Shapley value for input-output pair (x, y) in game The expected regression loss (f) upper bounds the Shapley value estimation error as follows,
((ux)~p(a))) [1 -o)o))))(((rllll (b(crw)))ll]]]
5 / ( (Lesp(0) Lixp)
(26)
where Liep represents the optimal loss achieved by the exact Shapley values.
18

---

Published as a conference paper at ICLR 2025
Proof. We begin by considering single input-output pair (I. y), where the prediction is given by = de(2 y; f). To account for the linear constraint (the Shapley value's 's efficiency constraint) in objective, write the Lagrangian Lz y.
Lo( v, 7) = (z(z, v) + y (ga(x1lv) (s(roly) 1))),
(27)
where y E IR is the Lagrange multiplier. The Lagrangian Lo(x, y. y, is p-strongly convex, sharing the same Hessian as Lo(r, y):
VoLo(I,81)) = (zLL(E, v).
(28)
bound the distance between e and the global minimizer using the Lagrangian's value. Let (#", ) the optimizer of the Lagrangian, such that
By strong convexity,
1o (z, v) (n(
(29)
where (xly) is the exact Shapley value.
From the first-order condition of strong convexity, we obtain the inequality:
L0 v, 2 Lo- (2r, Us v,)")) (Ø- 6o (x, v)) V Lo- (2r v,+") " 11 Φ0- v)..
(30)
By the KKT conditions, VoLo (x, r*) = 0, so the incquality simplifies to:
Lo( u. v,,) Lg (I, + 11 dg (z, 1)
(31)
Rearranging this, we get:
2
| - óz- (I, 111. (Lo(a,y,7)) - Lo (x,,,,,))))
(32)
Since & is feasible solution (i.e., it satisfies the linear constraint), this further simplifies to:
2 v) Los (x,1)).
Φ - Φo (z. v) < f
(33)
Next, we take the expectation over (x,, v) p(a, v). Denote the expected regression loss as Leap (0), which is:
Ceep (0) = E(z.g)<p(a.s)
(95(x)ll)) g y) s Φo (z ())
(34)
Let CTp denote the loss achieved by the exact Shapley values. Taking the bound from the previous inequality in expectation, we have:
2 (Eexp()) Lexp)
B(a.))~p(x.)) [z v: (x (x(
(35)
1
Finally, applying Jensen's inequality the left-hand side, obtain:
2 B(aa))((a.)) [1l0o(z, y; 0) d (xlv))l2) s
v 2 (Eexp(0) Lixp))
(36)
Substituting the value of J = 1/ H/ from Lemma 2 conclude that:
((w.)))((())) (10 yr: 0) (xlx))la s /2Ha-1 (Lesp(0) Ceep)
(37)
C ADDITIONAL RESULTS FOR AutoGnothi
In addition to the results ImageNette included in the main paper, provide more detailed results on other datasets ranging from image classification tasks to text classification tasks, against a number of baseline explanation methods, with respect other model sizes in this section.
19

---

Published as a conference paper at ICLR 2025
Table 4: Performance metrics for ViT-tiny on ImageNette.
Method
Insertion (1)
Deletion (L)
Random
0.9231
0.9229
Attention last
0,9281 =00 1033
0,731 2402
Attention rollout
0.9138:10.11 1102
0.7306:::249
GradCAM (Attn)
0.91555=0. 1242
0.8292++0:000
GradCAM (LN)
0.9280
2.84361022016
Vanilla (Pixel)
0.9006 1173
0.8161 2014
Vanilla (Embed)
0,913 1:0.
0.7708 +
IntGrad (Pixel)
0.9383-20.080
0.8734:10.181
Intrrad (Embed)
0,9317 0.9317=1008000 0800
0.8092:+0.8817
SmoothGrad (Pixel)
0,9153::0. 1199
0.7724:10:225
SmoothGrad (Embed)
0..2668 =0026800.om 1092
0.78522+0::77
VarGrad (Pixel)
0.9219 1147
0.7872:022177
VarGrad (Embed)
0.9317
1.8092::02262
LRP
0.9439==0.052
Leave-one-out
0.9632.#0.0401
0.767 0.7671110222 2:0222
RISE
0.9743-200033
0.65 14:1:030000
Covert et al.
0.9824=0.0230
0.5243++02579
AutoGnothi (Ours)
0.9802-#0.066
0.5097+0.2688
Table 5: Performance metrics for ViT-small on IeageNette.
Method
Insertion (1)
Deletion (L)
Random
0
0.9461 1:00833
Attention last
0.9599
0.7617=00206
Attention rollout
0.9293-20.9940
0.8566220:793
GradCAM (Attn)
0.9217 =00 1206
0.9301 +0.1
GradCAM (LN)
0.9239=200899
0.9184
Vanilla (Pixel)
0.9535 1006
0.81 1811990.100
Vanilla (Embed)
0.9564-100099
1.8240
IntGrad (Pixel)
0.9581
0.9219
IntGrad (Embed)
0.9581 :1007738
190219
SmoothGrad (Pixel)
0.9542-20.0770
0. 0.7998+00200
SmoothGrad (Embed)
0..550=00.07s$
0,8065 2000
VarGrad (Pixel)
0.9535
0.817940.195
VarGrad (Embed)
0.9564-200776
191400101112
LRP
0.9636=20.6377
0.7549
Leave-one-owr
0.9684-20.319
5:0206
RISE
0.9973-=0.000
0.7960++02999
Covert et al.
0.9828.20004
0.6865+0.255
AutoGnothi (Ours)
0.9791 10.0305
0.6667:10263
20

---

Published as a conference paper at ICLR 2025
Table 6: Performance metrics for ViT-large on ImageNette.
Method
Insertion (1)
Deletion (L)
Random
0.9645
0.0747
Attention
0,9251 =0.0794
0.8997+0.130
+
Attention rollout
0.9336:20.083
0.9429:0.099
GradCAM (Att)
0.9398==0.0992
0.9328
GradCAM (LN)
0.9590.10.061
0.9366110.330
Vanilla (Pixel)
0.9040 1046
0.9372:t0
Vanilla (Embed)
0.915 +.00999
0.9258
IntGrad (Pixel)
0.9716
0.94
Intrrad (Embed)
0.9716
0.9596
SmoothGrad (Pixel)
0.9499
1..153
SmoothGrad (Embed)
0.9636=20.088
0.8664+10.143
VarGrad (Pixel)
0.9444=10.0277
0.906020.1456
VarGrad (Embed)
0.9558 000068
0.8827:1.156
LRP
0.9506=200666
0.8814:10
Leave-one-out
0.9743
0,9534:10.088
RISE
0,9801 -2000373
059
Covert et al.
0.9843
0.7646::02002
AutoGnothi (Ours)
0.9837=20.2225
0.6570=a.217
Table 7: Performance metrics for ViT-base on Oxford-IIIT Pet.
Method
Insertion (1)
Deletion (L)
Random
0.8642:s0.185
0.8625.0.1s
Attention last
0,9066 1302
0.5534
Attention rollout
0.861622..1 1384
0.7387
GradCAM (Attn)
0.8726 1585
0.7582
GradCAM (LN)
0.8828 1137
0.7648
Vanilla (Pixel)
0.8855
0.6551 2395
Vanilla (Embed)
0.8996.0.113
0.5783
IntGrad (Pixel)
0.9219
0.845
IntGrad (Embed)
0.9219
0. 0.845 1990
SmoothGrad (Pixel)
0.9140=20 1379
0.55088:0234
SmoothGrad (Embed)
0.8731 +00 1462
0.87 16140.1514
VarGrad (Pixel)
0,9145 1268
0,5801 +:0.3666
VarGrad (Embed)
0.881 0.8818::0.11
0.8778
LRP
0.9192
0.5362+40228
Leave-one-owr
0.9468
0.7341 210051
RISE
0,9581 -100033
0.61 86: 3096
Covert et al.
0.9422 1035
0.4958
AutoGnothi (Ours)
0.9384 1088
0.4888
21

---

Published as a conference paper at ICLR 2025
Table Performance metrics for ViT- on MURA. MURA is a binary classification dataset, we calculated metrics for each of its two categories.
Abnormal
Normal
Method
Insertion
Deletion
Insertion
Deletion
Random
0.8195
0.820610.8599
0.1548
0.1164
Attention last
0.8416+0.18 104.086
0.6303-t0.112
0.1546:0.134
0. 1.
Attention rollout
0.8047:t0 1887
0.7236:20.107
0.1758 1393
0.1636620.18
GradCAM (Attn)
0.8077:10.188
0,817240 19.
0,161 130.
0.1655
GradCAM (LN)
0.8509 1787
0,745 1 +002171
0.177 15071537
041487=0011
Vanilla (Pixel)
0.8384
0.597 4:0 2056
0.1710 1401
00.
Vanilla (Embed)
0.8412:t:.111
0.5709 1993
0.16662:0138
0.1649
IntGrad (Pixel)
0.8677
0.7690
0.201 +:0
0.1326=
IntGrad (Embed)
0.8677:10.661
0.7690 2261
0,201 1:0.8812
0.1326 20.123
SmoothGrad (Pixel)
0.8351 1893
0.6469 +0.69000000 2000
0.1610 1428
0.18422 1885
SmoothGrad (Embed)
0.8293 1:0.1633
1.8006-10.197
0,1605 1:0.
0.1552:a0.1dm
VarGrad (Pixel)
0.8397 ::0 1944
0.6667 +n 2012
0.1592 2:0 1404
0.1813 1-813 1453
VarGrad (Embed)
0.8328 1841
0.8022 1983
0.1575 1461
0.1556::0.441
LRP
0.8524
0,6009 1932
0.1693 +:0 1459
0.1745
Leave-one-ord
0.8996::0.338 1336
0.6887
0. 0.2952-1:220
0.0977
RISE
0.9247+t0.00 1037
0.62558
0.3477++02431
0.0844440076
Covert et al.
1..319
0.0539::00475 0.4516
AutoGnothi (Ours)
0.929210.0597
0.411610.2116
0.4563
0,058 1:0.455
Table 9: Performance metrics for Bert-base on Yelp Review Polarity.
Method
Insertion (f)
Deletion (L)
KernelShap Covert et al.
0.4624 1...94
0.9620::00422
0.1725
AutoGnothi (Ours)
0.9588 1:0.0206
0.1004:0.037
22

---

Published as a conference paper at ICLR 2025
D ABLATION STUDY ON THE SIZE OF SIDE-NETWORKS
In this section, we present an ablation study on the architecture and size of side-networks in Auto- Gnothi. Specifically, we investigated the effect of varying the reduction factor (r 8, 16, 32) on the side-networks of ViT-base model. To evaluate the impact, we compared the performance of AutoGnothi with Covert et al. (Covert et 2022) on the ImageNette dataset. The evaluation was performed using the insertion and deletion metrics, and the results are summarized in Table [O]
This study aimed assess the balance between computational efficiency and explanation quality under different configuration of the side-network. The findings are follows:
Too Large r: When the reduction factor r was too large (resulting in a very small side- network), the network lacked sufficient capacity to learn the explanations effectively. As a result, the explanation quality degraded, with the side-network failing to capture enough information from the main branch.
Too Small Conversely, when was too small (resulting in a larger side-network), the net- work consumed excessive computational resources and interfered with the shared features of the predictor. This interference reduced the feature similarity between the prediction and explanation tasks, negatively impacting the faithfulness of the explainer.
Moderate r: A moderate reduction factor (e.g., r = 8) provided the optimal trade-off between computational efficiency and explanation quality. This configuration allowed the side-network to effectively utilize the features from the predictor while maintaining re- source efficiency.
These results highlight the importance of carefully selecting the reduction factor to optimize the per- formance of AutoGnothi across different transformer architectures. The identified trade-offs ensure that the side-network remains both effective and efficient, making it suitable for various applications.
Table 10: Explanation Quality Metrics of the ViT-base Explainer ImageNette dataset.
Method
Insertion (1)
Deletion (L)
Covert et al.
0.9839
0.8121
AutoGnothi (r=2)
1989411.003
0.8687 1..6871001806
AutoGnothi (y=4)
0.9857 10.0251
0.7846.10.195
AutoGnothi (r=8)
0.9874=0.0266
0.7954+0.229
AutoGnothi (r=16)
0.9720+0.0288
0.6202+0.2023
AutoGnothi (r=32)
0.9520=0.943
0.5547=0.1941
23

---

Published as a conference paper at ICLR 2025
E ADDITIONAL VISUALIZATIONS FOR AutoGnothi
In this section we present a number of image samples on the ViT-base model, regarding explana- tion outputs from 12 representative baseline explanation methods. We ensure that these samples correspond to correct model predictions made by the base model to ensure better clarity.
AutoCmathi
Covert et
Vartrad
RISE
GraICAM
IncCral
SmoothOnd
Lene-nt-oott
Vasilla
IPP
Ancmioon
GrdCAM
Aneenion rollout
French born
Vanilla
Attaniion millor
Cassente player
Incrrd
SmoothGidd
Leave out
Yinilla
LRP
Auemim Inci
AutoCrenti
Conen st
Vartin
RINH
GndlCM
Aticntion solbul
plaee
InCiral
SmoothCCodl
Lawwe-ome-eot
LRP
Ancotion lse
Attention
Gurbogz tuck
Innore
SmasikGna
Levve oze out
Vanilla
LRP
Amtettinn last
Figure 8: Visualization of ViT-base explanation on ImageNette dataset (1/2).
24

---

Published as a conference paper at ICLR 2025
AutoCmochi
Coven
VarGrad
RISE
GuaCCAM
Gold ball
XOXOX
10000
X0XXX
XXXXX
XXXX
XOOX
InrGrad
SmoothGrrad
Lane-mne-cul
Vanilla
LRP
Attintion
10000
GOOO
X0X0X
X0X00
X0X00
XXXX
AutoGmathi
Covest
VarGma
GraCAM
Attencion millout
SmoodsGradd
Attention
AutoCnnchi
Coven
VarGrad
RISE
GraCAMM
Attention nollout
Gar truck
SmoodhGGrad
IntGral
SmanelsGrrdd
Lensenseui
Vanilla
1RP
AutoGinoth
Covant
Vartraa
GradAMM
Attention nollout
French
InrCrad
Figure 9: Visualization of ViT-base explanation on ImageNette dataset (2/2).
25

---

Published as a conference paper at ICLR 2025
AutoCioothi
VarCrall
GradCAM
Attencion coloutt
American
Scottish Terrier
InGGmd
SmoothOmd
Leave one out
Vanille
AutoGinotii
VarOrad
GuaCAM
Atiension collout
inlish Cocker Spaniel
Lonvestno-ou
Vanilla
Attenrion
oglish Cocker Spaniel
Infiinal
SmoothGmmd
Leve-mmeut
Vinilla
L.R.
AutoGrothi
Covest
VarCiaa
RISE
GndCAM
Attomion roliout
English
Intünd
Smootkind
Figure 10: Visualization of ViT-base explanation on Oxford-IIIT Pet dataset (1/2).
26

---

Published as a conference paper at ICLR 2025
AutoCinehi
VarOrnd
GadCAM
Altenties rollout
IntCrd
AntoCnethi
mlal.
VarGmd
Wherecn Terrier
InGiead
SaandiGGnad
Leweeen--uul
Vanilla
AunoGnothi
GndCAM
Attention nollout
IntOred
SmoothGrad
Leave-one our
Vanilla
Accetton lest
AunoGnothi
Covert all.
VarGrnd
RISB
GraCCAM
Attenrion rollout
Leonberger
InrGrea
SmunthGmdd
Letwesenns-nt
Vanilla
IRP
AutoGinuth
GadCAM
Atcetioo nollout
Scottssh
SmoothGrad
Figure 11: Visualization of ViT-base explanation Oxford-IIIT Pet dataset (2/2).
27

---

Published as a conference paper at ICLR 2025
AutoCmnllii
Civant sl.
Vartinal
RISE
GidICMM
Abrunnd
IntGre
SmewihGrnd
Levosene-nd
Vanilla
LRP
Amention last
AutoGrethi
Coven etall
VarGmd
RISE
GudCCAM
Atlenton rollout
Abnornal
Intird
SmeothGrad
Leove-one ost
Vanilla
LRP
Atianiio milloui
Abrunnal
Gas.CA
Attention last
AntoGnothi
Crver
Vartinal
RISE
GndCAM
IntGina
SmoothGradd
Leare one out
Vanilla
1RP
Attentin last
Figure 12: Visualization ViT-base explanation on MURA dataset (1/2).
28

---

Published as a conference paper at ICLR 2025
Covert
VarGind
RISB
Inflndd
SmoothGead
Lemve-ons ous
Vanills
LRP
Attentin het
AstoCinothi
VarGmdd
Attention milleut
Almunal
IntGina
SmaalhGiaa
Lomeemn-mu
Vanilla
LRP
AutoSinati
VanOrdd
GuaCAM
Alteation sollnut
Almunnal
Yauilla
AutoChrathi
VarCmul
GndCAM
Alianion
Attention
AutoGnothi
Covarie et.al
VirGunll
RISE
GraCCMM
Absomal
IntGra
SmaniiGaal
Lonsenzent
Vanilla
LRP
Attention last
Figure 13: Visualization ViT-base explanation on MURA dataset (2/2).
29


Reviews:
[{"review_id": "KZtiuB4nLB", "reviewer": "Reviewer_5F4V", "summary": "This paper aims to address the tradeoff between prediction accuracy and interpretability of machine learning models. That is develop an inherently explainable model without compromising its prediction performance. To achieve this, the authors propose to integrate small side networks into black-box models and use them to generate Shapley value explanations. The paper provides comparisons of computational costs and memory usage, and empirical results on both prediction accuracy and explanation quality.", "questions": "See Weaknesses for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea is intuitive and novel.\n2. The paper is well-written and easy to follow.\n3. The proposed method significantly cuts down the trainable parameters, memory usage, and FLOPs without compromising prediction accuracy and explanation quality.", "weaknesses": "1. Since the proposed method relies on the frozen pretrained black-box model’s parameters, will the biases or spurious correlations in the pretrained model also affect the generated explanations from the side network? Can the side network mitigate these issues?\n2. Following Weakness 1, robustness evaluations are missing from the current experiments. How the side network will behave under the out-of-distribution (OoD) or even adversarial data w.r.t. the pretrained model?\n3. The side network is designed to be parameter-efficient, but is there an optimal balance between its complexity and explanation fidelity? It would be better to see ablation studies on different side network sizes or architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HNdaKGkK9X", "reviewer": "Reviewer_Qio4", "summary": "The paper proposes a method called AutoGnothi for generating Shapley-based explanations in black-box models, such as Vision Transformers (ViTs) and BERT, using side-tuning. Instead of modifying the main model, AutoGnothi trains a lightweight, parallel \"surrogate\" network that learns to approximate the importance of different input features without requiring iterative masking during inference, as is typically needed for Shapley values.\nThe main advantage of AutoGnothi lies in its efficiency. By using side-tuning, AutoGnothi is more memory- and time-efficient compared to previous approaches like ViT-Shapley, which require a fully fine-tuned surrogate model.\n\nThe authors measure the quality of explanations using insertion and deletion metrics, which test whether removing or adding key features impacts the model’s prediction as expected. In these tests, AutoGnothi’s explanations are shown to be comparable to ViT-Shapley (I read through the Appendix as well to make this conclusion). Besides, although qualitative examples suggest that AutoGnothi may better highlight main objects in images, the paper doesn’t fully convince me that it strictly produces better explanations than ViT-Shapley. Putting the explanations into a human-in-the-loop evaluation would help answer my concern.\n\nThe general idea is not new (e.g. compared vs. ViT-Shapley) but I appreciate the efficient solution, clear/detailed writing style, and thorough evaluation. I especially enjoyed reading Section 4.2.\n\nIn summary, I like the paper and recommend accepting it. My ratings can be adjusted during the rebuttal.", "questions": "Q1. The general idea in AutoGnothi is that the main model makes predictions, and the surrogate model explains those predictions. However, how can the authors ensure that the surrogate model **truly** reflects how the main model makes its decisions? Although the paper shows efforts to align the two main and surrogate models, how well would AutoGnothi work if we scale it to more complex models (where aligning networks structure is prohibited) with complicated output distributions, where using KL divergence might not be enough to keep them aligned? Are there better alternatives than KL divergence?\n\nQ2. I’m curious to see how AutoGnothi performs on more complex tasks. Currently, the tasks feel easy (especially the text classification). I’m concerned that side-tuning may not scale well with more complex tasks, like question answering or fine-grained image classification (e.g., CUB-200). How would the authors think the current side-tuning can satisfy interpretability and accuracy for complex tasks?\n\nTo clarify more, Q1 is about alignment and Q2 is about the feasibility of side-tuning.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: While AutoGnothi's goal of providing Shapley-based explanations overlaps with previous work like ViT-Shapley, it introduces a new side-tuning approach to reduce memory and computational demands.\n\nQuality: The paper provides a thorough evaluation across multiple datasets (both vision and language), using various metrics to prove the efficiency and explanation faithfulness.\n\nClarity: The writing is clear and  well-organized.", "weaknesses": "The evaluation of explanation: \nAlthough qualitative examples (Fig.7) suggest that AutoGnothi may better highlight main objects in images, the paper doesn’t fully convince me that it strictly produces better explanations than ViT-Shapley. Putting the explanations into a human-in-the-loop evaluation [1,2] would strengthen claims about its explanation quality.\n\n[1] The Effectiveness of Feature Attribution Methods and Its Correlation with Automatic Evaluation Scores\n\n[2] What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods\n\nMinor: I think the authors could also be upfront about the limitations of the works (e.g. approximation, explanation evaluation). The current writing does not discuss any of its **possible** limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9ScKmpH6n", "reviewer": "Reviewer_2vJv", "summary": "This paper is concerned with the setting of training an explainer for an explainee. \nImprovements to the previous work, which trains a surrogate model that can handle masked inputs and another explainer, that can predict the saliency, are primarily in terms of required ressources, both for training and inference with respect to time and memory. \nThis is achieved by utilizing Ladder-Side-Tuning to only train a reduced number of parameters on top of the explainee.\nWhen saliency and prediction are both desired, only the explainee and the few additional parameters are needed, incurring significantly reduced costs, while still matching the previous SOTA performance.", "questions": "How do the saliency maps transfer across surrogate models?\nAs the saliency maps explain the explainee, they should work across a range of surrogate models, especially those the explainers were not trained on,  but which still try to explain the same explainee.\n\nWhich surrogate model is used to evaluate the baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is presented very clearly, well written and easy to understand.\n\nThe proposed method combines ideas of PETL and Covert et al. in an original way. \n\nThe method at least matches the performance of the significantly more resource intensive method of Covert et al., clearly improving upon it overall.\n\nThe evaluation covers two modalities and multiple datasets with consistent results.\n\nA convincing ablation demonstrates the superiority of the proposed approach to two naive baselines, called Froyo and Duo, showing that explaining and predicting cause orthogonal gradients.", "weaknesses": "The paper discusses a  simple, resource efficient alternative to a niche paradigm of explaining a model, that is only applicable to Transformers.\n\nPatch Dropout has recently emerged as training augmentation for Vision Transformers, that enables transformers to handle masked inputs. (https://openaccess.thecvf.com/content/WACV2023/papers/Liu_PatchDropout_Economizing_Vision_Transformers_Using_Patch_Dropout_WACV_2023_paper.pdf)\nSince the surrogate models are more accurate on multiple datasets, it is not clear why one would want to stick to the explainee.\nThe paper misses a discussion of that.\n\nCombining the two points above, I am unsure how significant this work is, as the field might move towards Transformers trained with Patch Dropout. \n\nWhile the results in Table 3 suggest a gap, they are cherry-picked and no \"clear margin\" is visible to Covert et al. across the different architectures, typcially more evenly matched (As in Tab.  4 -8).\n\nThe evaluation using the surrogate model is frickle, as it is not actually the model to be explained. \n\nIt is unclear why ImageNette is used as main dataset, as it seems too easy, with accuracies above 99%.\n\nFig. 2 is too full. The table is repeated in the paper anyway. (Table 1 and 2)\n\ntypos in lines : 264, 535", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)